Searchlight

Web site owners might be amazed to learn that one of the biggest sources for duplicate content isn't externally, but rather internally.

Certainly, popular sites and blogs that syndicate a lot of content have to deal with external duplication, but as I already touched on external duplicate content, we know that there are steps to minimize those challenges and to establish your site as the canonical source.

Internal, or on-site, content duplication tends to come in a few key ways, the first of which is within the key page elements. The second is from the content itself; similar to e-commerce sites using stock product copy, you may be using your own copy over and over again on your site. Third, it simply may come from too little differentiated copy.

When it comes to Internet retailers, getting found in search results is often just as important as the right location is to brick-and-mortar retailers. When a big part of online success comes down to words, why settle for selling what everyone else is?

All retailers, no matter what their channel of choice, often sell the same products as at least some of their competitors. If you are a big enough fish, you can command enough power to at least obfuscate that fact . . . different product names, model numbers, etc. -- of course the underlying product is often still the same, anyway. Ever wonder how some retailers offer those huge pricing guarantees if you find the same product elsewhere at a lower price -- much easier to do when you have your own guarantee with the manufacturer that no one else can carry that same model.

But online retail is a bit more challenging, because aside from brand loyalty or being at a convenient location, the difference is often about search results . . . obtaining those highly coveted top rankings for the right searches. I began our duplicate content discussion by focusing on the duplicate content filter or penalty topic and the challenges of external content duplication. What better way to bridge the gap from external to internal, or on-site content duplication, than by talking about sales copy.

Looking to get started with a blog? More importantly, do you hope to monetize that blog? There are a lot of articles and books on blogging these days, many with limited or inaccurate information, but How To Make Money With Your Blog is one of the most complete and thorough publications on the subject that I've encountered. Authors Duane Forrester (a search engine marketer) and Gavin Powell (a technical writer) have covered all the important bases from identifying the best blogging platforms to covering the ins and outs of blogosphere culture. Oh yeah, and in between, they explain quite clearly the options for monetizing a blog.

The book contains five main sections, including Quick Starts: Top 10 lists; Blogs and Search Optimization (SEO); Generating Revenue With Your Blog; Managing Your Blog; and Extras and Inspiration. While the quick start lists were interesting and worth a read, they weren't particularly useful. The meat of the book begins with the section on SEO. Like any website, getting found in the search engines is extremely important if you intend to make any money. All of their SEO tips are excellent and would apply to any site, blog or otherwise. The blog management section is also quite good, much of it being basic business management, but some of it also dealing with syndication and promotion on the web. The inspirational section is less focused. It contains many points on blogging culture and blog etiquette, and while those with some blogging experience might find it commonplace, it will interest people completely new to blogging.

Central to the name of the book, and probably the most interesting to those attracted by the pecuniary title, is the section on generating revenue with a blog. Options discussed include affiliate programs, contextual ads, and ad networks. The authors go into detail on each one, and even recommend specific partner services in each of those industries. They also warn that, while it is quite possible to earn income off of a blog, it takes a highly trafficked site for significant money to be made. Still, the book is encouraging and makes the reader feel confident about pursuing a monetized blog.

While this book - and most technical books about the Internet - will become obsolete quickly simply because of how fast the web changes, it is a worthy investment for anyone wanting to create a blog right now. It's an easy read, flows logically, offers well-researched suggestions, and is full of quality information.

In my presentation yesterday here at Web 2.0 Expo, I showed Yahoo Suggest as an alternative to Google Suggest for quick-n-dirty keyword research. Both tools are free. Google Suggest is also built into the Google Toolbar's search box, so it's super convenient: just start typing keystrokes and it makes suggestions (think: auto-complete), the suggestions listed in order of popularity. In fact, it was Google Suggest that my daughter Chloe used to identify her top search term, "neopets cheats" and thus named her site "The Ultimate Neopets Cheats Site" since Google Suggest showed that "neopets cheats" was the second most popular search term after "neopets".

Google Suggest works great if you're comparing keywords that start with the same keystrokes, but you can't compare, say, "new cars" with "used cars". Yahoo Suggest has a leg up on Google Suggest, in that regard, since it returns search terms where your keystrokes may be in the middle or end of the search term. Just go to Yahoo and start typing in their search box.

Sounds great, doesn't it? However, when I was showing this in my presentation, it didn't work as expected. In an instant, and in front of a live audience of 400 people, I lost my faith in this Yahoo tool and its data. I used "cars" as an example, and you'll see from the screenshot below that the third most popular suggestion for "cars" is "toyota car malaysia new car cars".

Yahoo Suggest in action

Hmmmm... Not exactly a credible search term suggestion! I have a "Google, I Suggest..." blog, maybe I'd better start one a "Yahoo, I Suggest..." blog too! This could be my first post there. ;-)

Are you being outranked by you? Is "your" content showing up in searches, but on sites that aren't yours? Do you have multiple websites that compete against each other? Well this discussion on duplicate content from external sources should be right up your alley.

Earlier in the week, I started our discussion on duplicate content by trying to lay to rest the idea of a duplicate content penalty. Now we pick up that discussion with one aspect of duplicate content . . . content duplication from other sites.

While I'd love to start out our discussion with the idea that external duplicate content is the hardest to deal with, that may not always be the case as you'll see when we talk about duplication on our own websites. For now though, we are just going to focus on content duplication from other sites.

At this point, you are probably in one of two camps--the "Yes, help me with this please," camp or the "What in the world are you talking about?" camp. So let's start by getting everyone in the same camp at least. External content duplication can come about, generally, in three ways.

Content Theft

In every aspect of life, there are those who want to get ahead through the hard work of others, even illegally or unethically. The Web is certainly no exception to this, especially given the fact that, of all the ways to take advantage of the hard efforts of others, copy-paste must certainly be the laziest--I mean easiest.

Don't feel that this is an issue that only affects big name brands and sites, because anyone who publishes online is susceptible to this kind of attack. Keep in mind that what we are talking about here is essentially copyright infringement, not phishing sites and things like that, which is a whole other level of criminal activity.

Realistically, this is probably the hardest to combat, but in many cases, probably doesn't cause as much damage as you might think. In many ways, we might thank the search engines for this. They're out to deliver the best results they can to searchers and are certainly aware of these issues. Because of this, I truly believe they work really hard to identify authoritative and original sources of content. They can compare content they find based on when they found it, as well as links leading back to that content, and while purely speculation, I would have to imagine that it would be pretty easy for the engines to assign a score to any site based on the proportion of content on the site that appears elsewhere and determine natural and unnatural patterns.

So what can you do about content theft? While you can file reports with the search engines based on the Digital Millennium Copyright Act (just search on "Google copyright infringement" or the respective search engine for specific details), the ISP that hosts the infringing domain, or seek even greater legal action, it may be better to first weigh the impact you feel it really has as well as the resources it may take to fight it and determine whether it is worth your attention to begin with. And sometimes, just an email or letter to the infringer might be enough

Content Syndication

Ironically, you are probably the most responsible for your own duplicate content on other sites. Writing content and syndicating through article directories or other content syndication services, RSS feeds of blog posts, and press release syndication will probably make up far more of your duplication woes than pirated content.

Each of these instances can be addressed though. Article writing and similar content is best kept unique and different from any content you have on your own site. When it comes to this kind of content, it is often best to develop content for the sites where it is going to be placed anyway, rather than a mass distribution. Of course, you'll also want to include a byline with a link back to your site.

Blog syndication can be handled a little differently. You may decide to include only a summary of your post, or the full post. The pros and cons here must be weighed, since a partial feed may discourage some sites from even syndicating your blog. In many cases, there may be enough differentiation between your blog and the sites where your post is syndicated anyway. However the best solution is to also include an absolute link back to the blog post on your own site. This helps signal to the search engines that your post is the source.

Press releases can be handled the same way as these other content pieces. Whether you are distributing through wire services or using RSS to syndicate from your site, including links back to your site helps signal the source. Press releases also tend to be more temporary on external sites, though you should certainly keep an archive on your own site.

Micro-Sites

The final source of external content also falls under your control. Micro-site strategy consists of creating additional websites, often around niche topical areas. This strategy evolved out of the idea that if one website was good, then many websites must be better, and would increase the chances of ranking in search engines and the number of listings for a particular search. Some view micro-sites as a good thing, while others view them as bad, however neither view is particularly accurate. Rather, it is the implementation that makes them good or bad.

Micro-site strategy is a much bigger topic, but bad implementation is directly related to our discussion of duplicate content. Most micro-site implementations result in identical or nearly identical duplication of the main website's pages on the various micro-sites. This isn't surprising since creating unique content for one site, especially for an ecommerce site, is often challenging enough without having to create unique content for multiple sites. But rather than improving or increasing rankings, the micro-sites tend to directly compete with the main site and greater resources are needed to maintain multiple sites. Needless to say, this is why most micro-site implementations are bad.

Like many things, there are a few tools that can be used in the fight against duplicate content. One tool to help you keep on top of potential content theft issues is Copyscape, that allows you to enter in your page and it comes back with a list of potential duplication.

Flickr announced today that they now support video sharing to go along with their popular photo sharing services. The option is only available to "Pro" accounts, however, so those using the service on the free level will not have the option. Adding video support not only encourages the upgrade to the pro account, but it also takes an obvious swipe at YouTube.

Says the announcement on the Flickr Blog, "If you're a pro member, you can now share videos up to 90 glorious seconds in your photostream...90 seconds? While this might seem like an arbitrary limit...you know that Flickr is all about sharing photos that you yourself have taken. Video will be no different and so what quickly bubbled up was the idea of 'long photos,' of capturing slices of life to share."

The philosophy is decidedly different than that of YouTube, and most likely the intention is not only to control size, but also subject matter; it is positioned to be member video and not the more wide-reaching (and copyright-challenged) posts of YouTube.

Videos can be uploaded and organized in much the same way as photos. They can be organized alongside photos in Sets and Collections or separately. An initial test of posting video shows that, like photos, links embedded in descriptions are nofollowed, but links in Set and Collection descriptions continue to be free of them and pass on PageRank.

A footnote: slipped in at the end of the video announcement is a second announcement that Flickr is doubling the size of photos that can be uploaded to 20MB per photo for pro accounts and 10MB per photo for free accounts.

Several weeks ago at SMX West I had the pleasure of meeting and having lunch with Brian White from Google. White works on Matt Cutts' Web spam team, tirelessly working to make Google's search results the best they can be, ensuring the best user experience. Quite a hefty task indeed.

You'd think that someone who spends his days fighting the never-ending battle that is Web spam might be a bit negative or jaded. If that is the case, he does an amazing job hiding it. Instead, he was upbeat and you could feel the excitement in his voice as he spoke. Here's a guy who loves what he's doing and truly wants to not only improve the searchers' experience on Google, but wants to make the Web a better place. You can't help but like a guy who's fighting the good fight.

Earlier this week I talked about how a recession may be the best thing for SEO. Let's revisit that bold statement and also how to make the most out of a downturn in the economy using SEO.

There will be firms and people within the industry that will feel the same pains of a recession that everyone else will. I'm certainly not claiming that you can sit back and coast in to success. In fact, the statement is less about SEO firms and practitioners, and more about SEO as a tool.

If you are in-house and have been struggling to get the resources or attention you need to make SEO a priority, then this may help to increase the urgency of SEO. Or if your firm provides SEO services, then you may be able to use the concerns and challenges that will come with a recession to get the attention of the decision makers to illustrate how SEO may be a more cost-effective solution.

A recession or economic downturn will lead consumers and businesses to reduce their spending as their confidence in the economy, their business, jobs, investments and/or retirement weakens. While there may be a subset of the market that "quits buying," what we are really talking about is a reduction in spending. There will still be necessities and essentials that must be purchased. Beyond that, we may expect to see purchase adjustments or a scaling back.

For instance:

So here we are, one quarter down and recession to go. Recession has become an unfortunate but popular topic in 2008. Some people follow strict definitions of what qualifies technically as a recession while others speak purely from opinion--or maybe they are speaking from the pinch they are feeling in their wallets. For most, they could care less what you call it; labels don't make the impact they are feeling any better or any less painful.

In most industries, when things slow, something has to go. Will that be cutting back on seemingly frivolous expenses, going to fewer industry shows, reducing ad spending, or worse, cutting jobs? Like many industries, search marketing firms are considering these choices as well, and rather hoping that this slowing, downturn, recession, or whatever you want to call it, actually presents more opportunities than hard choices.

Only time will tell, but I like to think the feeling that search marketing firms may be able to find opportunities during these trying economic times are spot on. But don't think this will just be a slam dunk--every opportunity also presents challenges.

Google's Webmaster Central has become a very important resource for anyone who has a Web site, works on a Web site, or, like SEO practitioners, helps others with their Web sites.

Google continues to roll out more features and better functionality to existing features, and now they just did a little bit of both with the addition of their Generate robots.txt function.

Google had previously added a robots.txt analyzer, which at this point is still the more useful of the two tools. For those who aren't aware, the robots exclusion protocol helps with instructing search engines how to interact with a Web site. There are a number of directives available, but the main purpose of the robots.txt file is to instruct the search engines about content that a site owner doesn't want the robots to crawl.

Why in the world would you not want search engines to crawl any of your content? You may have content that, for whatever reason, you don't want others to find through search results. Note, however, that this is not the same as secure information that requires authentication through a log-in.

Your site may have its own search function that creates "search results" for your site. Search engines generally do not want to include search results within search results, so this content may not be returned for searches on the engines anyway, so you might want to focus the crawlers elsewhere for greater crawler efficiency.

Or you may have duplicate content issues that you could use robots.txt to filter out. This is especially common with a content management system (CMS) that creates a separate printer-friendly page.

Regardless of your specific needs, having a robots.txt file can be important to a site. Rarely is there a site that can't benefit from disallowing at least some content. Even if you have nothing to disallow, you may want to take advantage of the auto-discovery feature for your XML sitemap. Finally, depending on your server log system or analytics package, not having a robots.txt file can be problematic if it inflates your "404 File Not Found" error reporting, which can happen because search engine spiders will request the robots.txt file automatically when they come to your site.

Right now, the robots.txt generator is rather basic and I hope that Google will add more features to it going forward. Currently, site owners have to paste in URLs and URL patterns to build the file. It would be great if it would provide a list of URLs or patterns extracted from a site to help automate the procedure for anyone not familiar with the protocol.

There is more information about the protocol, though a bit more on the technical side, at the robotstxt.org site and you can find more engine specific information on crawling and robots.txt from Google, Yahoo, MSN, and Ask.com.

One important tip is that the following directive tells all spiders they are allowed to go anywhere:

User-agent: *
Disallow:

And, more importantly, the following directive, which I sometimes see when I think people really wanted the above:

User-agent: *
Disallow: /

The latter tells the spiders to stay out of the entire site--clearly two very different results, so be sure you understand which does what.