Comments on: Internal cloud's big test: Amazon vs. Cloudera

The debate about the market for private cloud infrastructure finally has an objective test case: the choice between commercial Hadoop implementations from Cloudera and Amazon.com.

[BlitzBoy1120]

Is it me, or is Amazon starting to do everything?

[marvin25]

I have one question is how can we do cloud computing when we don't have enough capacity on the Internet right now. We are short of capacity and this why the economy is going downhill. This added requirements can't be handled at all with the current capacity on the Internet. So we are talking something in the future when we have capacity in the Internet. The data streams that is required can't be handled with the current capacity at all or is it your goal to take down the Internet completely. All the major nodes are working at maximum capacity and all it is absorbed as fast as additional bandwidth by one ISP. Come back in a year for update on the area of capacity. This is really an idea that can't take place under current conditions.

[jhammerb]

Hey James,

Thanks for the analysis. One addition: you don't need to own any hardware to run Cloudera's Distribution for Hadoop. We have an AMI for our distribution and a set of supporting scripts to get moving with our distribution on EC2, documented at http://www.cloudera.com/hadoop-ec2.

In addition, one of Cloudera's engineers, Tom White, wrote the code that enabled Hadoop clusters to be deployed on EC2 and read their data from S3 (http://issues.apache.org/jira/browse/HADOOP-930). This code underlies Amazon's current implementation of Elastic MapReduce.

We're pretty excited to see Amazon lowering the barrier for adoption of Hadoop even further!

Later,
Jeff

[jamesurquhart]

Fair enough. I should have added that Cloudera will do business on EC2 as well as in internal clouds. However, that would in many ways be an extension of the public cloud approach, not an alternative to Amazon's implementation. That being said, its good to see Cloudera leverage as many opportunities as possible, and hedging their bets.

Congrats to Tom White for his possibly historically important contribution.

[cloudment]

Just want to add that Amazon slashed S3 transfer prices to 3 cents per GB/months. This makes it even more attractive to the vast audience and I hope it will also help to grow CloudBerry Explorer for S3 freeware user base. http://cloudberrylab.com/

[asmann25]

This is a great discussion. I would like to add that Aster Data Systems provides an enterprise-class implementation of MapReduce. It runs inside Aster's massively parallel relational database for large-scale analytics. Read here for more details:

http://www.asterdata.com/blog/index.php/2009/04/02/enterprise-class-mapreduce/