Report: Cloud services can't handle the pressure

According to a new report by researchers in Australia, stress tests have revealed that the "infrastructure-on-demand services offered by Amazon, Google and Microsoft suffer from regular performance and availability issues."

The seven-month study of Amazon's EC2, Google's App Engine, and Microsoft's Azure cloud computing services simulated 2,000 concurrent users connecting to services from each of the three providers, with researchers measuring response times and other performance indicators.

The results were at best mixed, and at worst, severely dysfunctional. For example, I'd never heard that when using Google App Engine, none of your data-processing tasks can last longer than 30 seconds, lest the service throw an exception back at you.

Researchers found that the three platforms "delivered wildly variable performance results as Amazon, Google and Microsoft trialled, added and dropped new features."

Response times on the service also varied by a factor of 20 depending on the time of day the services were accessed. Anna Liu, associate professor in services engineering at the University of New South Wales School of Computer Science responsible for the study, also noted the immature monitoring tools and the inability to accurately estimate cost:

"None of the platforms have the kind of monitoring required to have a reasonable conversation about performance," she said, in the report from Australia's ITnews. "They provide some level of monitoring, but what little there is caters for developers, not business users. And while Amazon provides a dashboard of how much it is costing you so far, for example, there is nothing in terms of forecasts about what it will cost you in the future.

It's way too early to suggest that cloud services can't meet the customer needs, but it's important to know what you are getting into if you want to use these services now. As with any nascent technology, early adopters will benefit in some ways and suffer in others. Cloud services still offer one of the most intriguing ways to consume IT and software applications.

Follow me on Twitter @daveofdoom.

[BTRoundtable]

Dave, perhaps this study further validates why AT&T in the U.S. and BT in the UK have entered the managed cloud services market -- to fullfill the apparent need for services designed with enterprise applications in mind. Purpose-built unified computing platforms will enable a quality of services that the most demanding IT teams require.

Also, the recent launch of unified service delivery offerings has fueled the early-adopter interest.

David Deans
Business Technology Roundtable

[cvaldes1831]

That's why I keep saying that mainstream journalism covering cloud computing should be put off for maybe five to ten years. Cloud computing simply isn't ready for primetime.

As you've noted, these slobs muck around with production systems and they often don't use the two-person rule, sending down services for hours because of some boneheaded and easily-preventable maneuver. If you want to be taken seriously, run your web service seriously, not like a bunch of 22-year-old CS grads hopped up on Rockstar or Red Bull.

[billsmith1961]

I recommend that anyone who reads the above article also read the article he references. Dave Rosenberg's article is a lot more pessimistic than the Australian article, which also said "The research found some merit to vendors' claims of "perceived infinite scalability."

[Vegaman_Dan]

No Google app can take more than 30 seconds of processing time or it throws an exception?

Huh, I've never heard of that limitation before. I wonder if that is purely time on Google's side, or includes the total time from when your system submits the request to when it is delivered.

Online / cloud services aren't quite ripe yet for use. There's still a need for client side office apps through iWorks, OpenOffice, and MS Office. Enterprise level customers simply cannot afford this level of performance at this time. It will get better in time though, I expect.

[affinity13]

i don't even develop for Google's app engine and that was one of the first things i found out when looking into it...You have to do your homework and not just assume...

[GothAlice]

The company I work for has migrated from externally managed and self-managed dedicated hosting (at the peak, three servers) to fully self-managed 'dedicated hosting' using Amazon EC2 (currently two instances). Even using "Small" instances, ping times (latency) and throughput has been excellent with 17 clients hosted on the first instance. Uptime has been a continuous 128 days with literally no service interruption despite one network upgrade. (Throughput was reduced for a period of abo. an hour.)

And yes, we do have a system in place to move spawn instances under high load and almost seamlessly migrate clients from instance to instance. This also covers the eventuality of an instance being nuked due to Xen host OS failure.

For replacing traditional hosting infrastructure, with two exceptions (multiple static IPs and IPv6) I have been thoroughly impressed with cloud offerings, especially Amazon's.

I'm pretty sure the Model T had issues as well :)

[enovikoff]

Fascinating article. Being a cloud provider myself, I always notice resistance in any group of people talking about Cloud to acknowledging performance problems, so this is a welcome change. Clearly, as a new technology, there will be issues, which don't necessarily render it a bad choice. In the future, there will be a wider array of choices at various price/performance tradeoff points, some of which will not exhibit any of these problems. That doesn't make low-cost offerings with performance problems "bad", just as an economy car with high gas mileage isn't "bad." Since we've released a high-end cloud offering that was designed to address performance problems, it has done well with enterprise customers but we've seen considerable resistance among traditional cloud customers (startup developers) to paying for the extra performance and reliability: essentially their comments are, "it's great, now can I have it for the same price as EC2?" As a result, I think there will always be a place for price-focused cloud services alongside value-focused ones.

-Eric Novikoff
ENKI PrimaCloud

[BrettWinterford]

The Australian researchers at UNSW have released some more data on this.
http://www.itnews.com.au/News/153819,more-data-released-on-cloud-stress-tests.aspx

[VanniGreene]

I certainly know that if I ran a commercial web service, I would want more comprehensive control of my infrastructure than what any of the popular cloud providers offer. A note about App Engine, though, from someone who has worked with it pretty extensively (shameless plug: I put together nicksmap.org and a couple similar toys with GAE) -- there is a perfectly sufficient workaround for the 30-second process cutoff you mention in the form of the Remote Api (http://code.google.com/appengine/articles/remote_api.html). Really, though, if your product is web-based and more than just a toy, I think that you'd serve yourself to not put yourself in a position where you must worry about quotas, arbitrary restrictions, data portability concerns, etc. and run your infrastructure yourself.