robots.txt

Recovery.gov blocked search engine tracking

Update: As of 8 a.m. PST, within three hours of this story first going live, it appears that President Obama's Web team has (silently) pulled the robots.txt file from the Recovery.gov Web site. The site is now open to Web crawlers of all kinds.

The Obama administration has apparently opted to forbid Google and other search engines from indexing any content on the newly launched Recovery.gov.

Is this even more evidence that the administration's much-publicized commitment to transparency is simply hype?

Recovery.gov, which went live Tuesday, is set to act as a central … Read more

White House expands use of search-blocking code

The White House has silently tripled the number of Web pages that it forbids Google and other search engines from accessing. Is this a bad omen or much ado about nothing?

Within hours of Barack Obama being sworn in as president, bloggers and tech journalists began to closely examine the new White House Web site for hidden indicators as to how he would shape future tech policy.

While I focused my efforts on the White House privacy policy, others looked to the new administration's robots.txt file, which lays out boundaries that search engines like Google should follow when … Read more

Google jokesters ward off zombies

Google, whose servers constantly crawl the Web, doesn't have anything against spiders. But zombies, well, that's another matter.

Showing some timely techie humor, the search giant updated its robots.txt file for Halloween. For the uninitiated, search engines trying to index Web sites look for robots.txt files for instructions about whether they're permitted access to particular pages.

The Google robots.txt file on Friday begins with the following exclusion:

User-agent: zombies Disallow: /brains

Most of the time when people do this sort of thing it's called an Easter egg. Is there such a thing as … Read more

Analyze, create robots.txt files in Google

Google's Webmaster Central has become a very important resource for anyone who has a Web site, works on a Web site, or, like SEO practitioners, helps others with their Web sites.

Google continues to roll out more features and better functionality to existing features, and now they just did a little bit of both with the addition of their Generate robots.txt function.

Google had previously added a robots.txt analyzer, which at this point is still the more useful of the two tools. For those who aren't aware, the robots exclusion protocol helps with instructing search engines … Read more