Yahoo to distribute its version of Hadoop

Yahoo announced plans Wednesday to release an open-source version of its take on Hadoop, a grid-computing framework used to run many parts of its business.

Yahoo is a major force in the development of Hadoop, which is principally overseen by the Apache Software Foundation. Hadoop is essentially an open-source version of the software Google uses to run its Web indexing servers, and Yahoo uses it for much the same purpose internally.

Hadoop runs on tens of thousands of servers inside Yahoo, said Nigel Daley, quality and release engineering manager for Yahoo Grid Technologies, in a blog post Wednesday. That's a much larger implementation than other companies and organizations might wish to deploy, but at the same time they would like to benefit from the reliability tweaks that Yahoo has made to Hadoop in order to support its enormous Web properties.

"This distribution is largely a response to the numerous requests that we have received to share Yahoo!'s internally tested and scale-proven releases," Daley wrote. The code is available for download immediately here on Yahoo's site.

Hi Tom, Yahoo should not distribute its own version of Hadoop. It will create a fork in the current Apache Hadoop distribution. And it undermines the work of Apache. Better way is for them to include whatever enhancements they have made back to Apache Hadoop distro.

[ddesy]

I think you're right about this. Many OSS projects end up hurt by forking, and Hadoop might be joining those ranks now. Contributing changes to the main project might take a little longer, but in the end it would probably be more beneficial to everyone involved.

[Seaspray0]

So let me get this right... Yahoo is giving out the software that allows people to run their own mini search engine? This could be useful for say... an intranet site where the amount of data to be indexed is small. To do the internet, like what yahoo has done, would take one huge database beyond the capability of everyone unless they have the computer resources like yahoo.

Could this also be a sign that yahoo doesn't want to make search the backbone of its services?

[lonestarState]

Yahoo has released Hadoop enhancements to the Apache foundation. Life is a fork so why not release their own version. Case in point: LINUX distros