Scaling Amazon EC2 to 512 active nodes

Among all the cloud-computing hype, one thing that hasn't been evident is just how far you can scale across a provider. I haven't seen any other vendor come near Amazon.com's ability to reach the massive scale that the cloud itself connotes.

Max Gorbunov from Grid Dynamics ran a 512-node Monte Carlo simulation to find out how well Amazon EC2--short for Elastic Compute Cloud--would perform. He used GridGain, a Java-based open-source grid computing infrastructure for the test.

All in all, this test clearly shows that you can utilize Amazon's massive infrastructure for high-end processing with an acceptable performance hit. And while I am sure I am oversimplifying the difficulty associated with getting this all set up, based on the development notes it seems like it was fairly easy (at least for Max.)

GridGain deployment on EC2

(Credit: Grid Dynamics)

The test consisted of a custom setup based on open-source components including GridGain and Open MQ running on the default EC2 Fedora Core 8 distribution and using a custom test harness developed for this project.

The performance degradation of 3 seconds (about 20 percent) should be considered minor given roughly 250-fold increase in scale. The curve rises two times: in the ranges 2-8 and 256-512, while 8-256 remains almost flat.

While I don't have the math available to me about what this cost to run (I would guess well less than $5,000), the way you might have done something like this in the past would have involved expensive software like DataSynapse or very technical open-source tools like the Globus Toolkit.

It's a new world out there.

Thanks to Eugene at TSS for the pointer.

[alexisbellido]

Interesting news. It seems cloud computing will be taking the world by storm in the next years.

I'm just waiting for persistent storage to be available in EC2 to move a few sites for production in the cloud.

[benjaminstraight]

Good article.