Google's Webmaster Central has become a very important resource for anyone who has a Web site, works on a Web site, or, like SEO practitioners, helps others with their Web sites.
Google continues to roll out more features and better functionality to existing features, and now they just did a little bit of both with the addition of their Generate robots.txt function.
Google had previously added a robots.txt analyzer, which at this point is still the more useful of the two tools. For those who aren't aware, the robots exclusion protocol helps with instructing search engines how to interact with a Web site. There are a number of directives available, but the main purpose of the robots.txt file is to instruct the search engines about content that a site owner doesn't want the robots to crawl.
Why in the world would you not want search engines to crawl any of your content? You may have content that, for whatever reason, you don't want others to find through search results. Note, however, that this is not the same as secure information that requires authentication through a log-in.
Your site may have its own search function that creates "search results" for your site. Search engines generally do not want to include search results within search results, so this content may not be returned for searches on the engines anyway, so you might want to focus the crawlers elsewhere for greater crawler efficiency.
Or you may have duplicate content issues that you could use robots.txt to filter out. This is especially common with a content management system (CMS) that creates a separate printer-friendly page.
Regardless of your specific needs, having a robots.txt file can be important to a site. Rarely is there a site that can't benefit from disallowing at least some content. Even if you have nothing to disallow, you may want to take advantage of the auto-discovery feature for your XML sitemap. Finally, depending on your server log system or analytics package, not having a robots.txt file can be problematic if it inflates your "404 File Not Found" error reporting, which can happen because search engine spiders will request the robots.txt file automatically when they come to your site.
Right now, the robots.txt generator is rather basic and I hope that Google will add more features to it going forward. Currently, site owners have to paste in URLs and URL patterns to build the file. It would be great if it would provide a list of URLs or patterns extracted from a site to help automate the procedure for anyone not familiar with the protocol.
There is more information about the protocol, though a bit more on the technical side, at the robotstxt.org site and you can find more engine specific information on crawling and robots.txt from Google, Yahoo, MSN, and Ask.com.
One important tip is that the following directive tells all spiders they are allowed to go anywhere:
User-agent: *
Disallow:
And, more importantly, the following directive, which I sometimes see when I think people really wanted the above:
User-agent: *
Disallow: /
The latter tells the spiders to stay out of the entire site--clearly two very different results, so be sure you understand which does what.
Search is a constantly evolving and changing entity, and this year has certainly seen more than its fair share of change. Possibly highest on the list has been the move to blended or universal results. Much of the focus and discussion has revolved around Google Universal, but they aren't the only game in town and all the majors have now entered into blended results to some extent or another.
This presents interesting opportunities and challenges. Those who focus on developing a fuller and broader Web presence, adding video, news, blogs, images, local content and social media to the mix, greatly increase their opportunities to gain rank positions across all the engines. The challenge however will be that the results across the engines may become even more varied, as each engine puts its own spin on the blending.
It's still very early in the game as the engines continue to ramp up and tweak their algos as they evolve and introduce searchers to blended results. It will be interesting to watch the evolution of blended search across the different engines. Today, results vary within the engines. Some searches show little to no signs of blended results, and others, show considerably more.
For an interesting example, let's take a look at a single search across the engines and see how they compare. Since we are talking about how blended search is evolving, what better search term than "cavemen" to use as a test. It is an interesting term for our test because it has such a wide frame of reference and actually poses a rather interesting challenge for the engines. Depending on the searcher, it may carry very different connotations:
- Prehistoric humans
- The definition of the word itself
- The ad campaign for the Geico commercials
- The newly launched sitcom on ABC based on the Geico commercials
Here are screenshots of that search in Google, Yahoo, MSN (Live Search) and Ask. It is interesting to see the variety of the impact of blended results across the engines, from very little to considerable impact. As you experiment with different searches though, you may see the engines' position on this scale shift up or down.
Along with calling out some of the specific blended results, I've also noted some of the refinement options, such as related searches, narrowed or expanded searches, and Yahoo's new "Search Assist."
Ask.com SERPs.
Google SERPs.
Yahoo SERPs with Search Assist.
MSN Live SERPs.
While the survival of the new Cavemen series may be in question, the continued evolution and survival of blended search is without doubt.
Whether you have multiple locations or just one, local is fast becoming an important market online. Traditionally, the local marketing battlefield consisted of phone directory listings, billboards, newspapers, TV, radio, and anywhere else that made sense to get your name seen. The web and search engines have opened up a new local battlefield, and smart businesses are moving quickly to gain a foothold.
Surprisingly though, many businesses have failed to tap into one of the more powerful tools available for reaching the local market. If that's surprising, then what makes this amazing is that this tool is free and will become even more important as local search information continues to be blended into regular search results.
The major search engines saw the growth and importance in local search years ago and started putting tools in place to not only help searchers, but also businesses. What the search engines may have discovered was that many searchers weren't aware of these local search tools, like local.google.com and local.yahoo.com. Instead, people just searched for local things where they searched for everything else, which further helps to explain things like universal search.
What was really exciting though was that the search engines also provided businesses with a way to get listed. Yet many businesses have yet to tap into this opportunity. If this sounds like you, then there is no time to waste, and better yet, getting listed takes no time at all. The process varies by engine, but is often as simple as having a postcard sent to your business address, receiving an automated phone call at your business location, or filling out a form or email.
Go to local.google.com and click on the "Add or Edit your business: Learn more" link along the left side. If you don't have a Google Account yet, you'll need to create one. Once you've done that, you'll get access to the Local Business Center, where you can add or edit your business listing.
Yahoo
At listings.local.yahoo.com, you'll see that Yahoo provides different listing levels, including a basic free listing. The Basic listing is done in 4 easy steps, while the paid versions have an extra step or two.
MSN Live Search
Not as direct, you'll need to click on the Help link in the upper right corner at local.live.com, and then select the first FAQ item, "How do I add my business listing to Live Search Maps?" From there you'll get a link to an infoUSA.com form for adding or updating your information as need be.
Ask
Even Ask wants to list your local business. Even less intuitive than the others, you'll need to follow the Help link in the upper right corner at city.ask.com, then toward the bottom of the FAQs is one labeled, "I'm not finding my business in AskCity. How can I get it added?" With Ask, you'll need to send off an email with some information to Ask's Customer Service.
- prev
- 1
- next
