Crawl the Web with Wales' open-source search tool

Friday morning, at the O'Reilly Open Source conference in Portland, Ore., Jimmy Wales, founder of Wikipedia and Wikia, announced the release of an open-source Web crawling site called Grub.

Grub was acquired from LookSmart under the open-source project Wikia. The platform, now available for downloading and testing, is built on users donating their personal computer power. It's meant to operate through open protocol and community collaborative added functions combined with the wiki.

Last year, Wales claimed that Internet search as we know it is broken. Grub is one of his attempts to gather open-source technologies to organize free content on the Web.

[ben::zen]

Grub???

What about the GRand Unified Bootloader
(GRUB)??? Why two grubs? This makes no sense. At
least it isn't weevil, or we'd all be making
Master and Commander jokes right about now...

[ir8piir8]

That's neat, but your sites broke...

I believe that this is the best way to crawl the web, but fix your website! There is no contact link and I got MySQL errors when registering my team. So I'll leave my comment publicly. Feel free to reply.

[flowgar]

faroo.com goes even a step further

www.faroo.com goes even a step further: distributed crawling and distributed search.
A distributed crawler makes sense only, when the index is distributed too.