Google dips toes into 'deep Web' search

Google's ever-active search bots, which scour the Web constantly for new pages, have begun a new, more active phase of their indexing jobs.

In a blog post Friday, Jayant Madhavan and Alon Halevy of Google's crawling and indexing team said the company has begun an experiment in which its indexing software experimentally enters text in Web site forms to see what previously undiscovered pages may appear.

"In the past few months, we have been exploring some HTML forms to try to discover new Web pages and URLs that we otherwise couldn't find and index for users who search on Google," they wrote. "This experiment is part of Google's broader effort to increase its coverage of the Web. In fact, HTML forms have long been thought to be the gateway to large volumes of data beyond the normal scope of search engines."

The new Google indexing practice involves only "high quality" Web sites and doesn't run on sites with "robots.txt" files or other standard mechanisms of warding off indexing software.

To decide what words to "type" into the forms, the indexing software samples from among words on the Web page with the form, Google said.

The technology looks related to a company called Transformic that Google acquired, according to a blog post by Anand Rajaraman, who was involved with the technology earlier in his career, while working for Halevy.

[SteveCherry]

Another tool for recruiters and headhunters?

Deep web is nothing new, but it would be a new capability to an already dominant force in the research toolbox. Google's interface has been the tool of choice for recruiters and headhunters for years now, but an added deep web capability could really change the game.

Yahoo has never even come close to the older capabilities of Google, so this would ensure a healthy lead for some time.

Steve Cherry
www.sandiegoemployer.com

[zumpa1]

I wonder if it would post on forums and stuff. If so I think this site may be a test run of google talking to itself: http://www.f2bb.com

[t8]

I noticed that

Google had some URLs for my website that were search strings.

I am happy about that too.

[Shankland]

Interesting--were pages ranked high?

I'm guessing you noticed the string-based results via Google search. Did your pages show up high in the results?

[pbarnhart01]

Economic Stimulus Package For SEOs

We looked at this on the 13th in "172 and one-half ways Google Form Spidering Is Kewl" - this will be a boon for SEO/SEM consultants seeking new ways to game it all: http://pbarnhart.wordpress.com/2008/04/13/172-and-one-half-ways-google-form-spidering-is-kewl/

[47project]

Sketchy...

Just think about how many HTML forms do not check their referrer or protect their data effectively for injection by spammers/bots. It's kinda scary to think about how Google can potentially get into data that may have not been intended for the public eye but was accessed by them because of it's insecure implementation.

Make sure you have web developers that know what they're doing! Eeeek.