Lightning zaps Amazon cloud

Amazon.com is blaming the latest outage to hit its Elastic Compute Cloud service on a lightning strike at one of its data centers.

In a statement on the Amazon Web Services "health dashboard," the online retailer and cloud-computing provider addressed concerns from some U.S. customers whose EC2 service had been disrupted around 6:20 p.m. Pacific Daylight Time on Wednesday.

"A lightning storm caused damage to a single Power Distribution Unit (PDU) in a single Availability Zone. While most instances were unaffected, a set of racks does not currently have power, so the instances on those racks are down," the company said initially on the health dashboard.

The disruption lasted about seven hours, during which time Amazon asked any affected customers to use alternative parts of the network. "Users with affected instances can launch replacement instances in any of the U.S. Region Availability Zones or wait until their instance(s) are restored," Amazon said.

The company later attributed the outage to a problem on one "availability zone" and that the outage was localized. "We would like to reconfirm that this issue was limited to the single Availability Zone where this power issue occurred, and that a very small percentage of instances in that AZ were affected; this was not a generalized service issue," Amazon said.

Despite acknowledging that Amazon had dealt with the issue fairly efficiently, one user was concerned that a single lightning strike was able to bring down the service, if only in a limited way.

"I was under the impression that your architecture had more resiliency built into it. Yes we can use multiple availability zones to help with a single point of failure, but I thought that even within a single availability zone there was not a single point of failure for hardware/power," the user posted to an Amazon forum on the issue.

The EC2 service provides customers with virtual access to Amazon's computing infrastructure, using virtual machines that can be created using the Xen virtualization platform. First launched in a limited beta in August 2006, the EC2 service went fully live in October 2008.

Not including the latest issue, the service has suffered two major disruptions during that time in February 2008 and October 2007. In June 2008, Amazon's main retail site suffered an outage that the company blamed on the complexity of its own systems.

A series of outages that have hit other online or cloud computing services including Google's Gmail and other applications over recent months have led some critics to question whether the cloud approach to computing is really capable of providing the resilience required by enterprise users.

In mid-May, Google services were hit by an outage which apparently affected one in 10 of its users. In January, software-as-a-service pioneer Salesforce.com experienced an outage that disrupted all its customers for about an hour.

Here is the string of messages from Amazon about this week's outage.

(Credit: Amazon.com)

Andrew Donoghue of ZDNet UK reported from London.

[Zer0Wolf]

Cloud... Lightning.... Amusing headline! Will lightning be the biggest downtime cause, if ever, for cloud computing?

[MadLyb]

This is a fundamental problem with many of the cloud implementations today.

If I am consuming capacity from your cloud, I shouldn?t even care about underlying infra details, much less be required to pick up the pieces behind a failure in your system. That is the whole point of a cloud, for the consumer to focus on his core strengths and the provider to make the service transparently available at all times.

Otherwise, you are just hosting in a server farm.

[jayhawk73]

I'm going to throw the BS flag here. If they're in a data center they should have lightning arresting system on the top of the building, a ground ring inside the building that is less than .01 Ohms to ground, and each rack should be tied to that ground ring. They should also have A & B power feeds (redundant PDUs) as well as generator and battery backup. There is no way a data center should have one section go down due to lightning.

I have seen contractors drop wrenches across power supply buses and take down an entire data center line-up, but lightning is just too unlikely.

[Tony McCune]

The title to this article is very misleading. The DigitalChalk system and many other customers on AWS never saw any type of interruption. The idea that any infrastructure is 100% immune to acts of God is kind of silly I think.

[hutchike]

Which availability zone? I'd like to know.

[Vegaman_Dan]

Why was lightning allowed to even affect the system in the first place? The internal power grid should be isolated entirely from external affects. A lightning strike shouldn't have affected anything internally- that's a failure right there.

Now this may not be a design failure- data centers are designed to be protected against outside conditions with their own UPS's, generators, etc. This could have been a building construction issue insteaed. Something unplanned / unforseen such as a metal conduit from the roof that happens to go through the data center in some completely bizarre way- it's happened before where a stairway railing becomes the vector all the way through the system. It's a tricky thing to totally isolate a room and network, but that's why they cost so much too. All of which can be negated by some single oversight.

Still, even if a rack of systems goes down, there should be a redundant parallel array that takes up the slack, and that IS a design failure.

[luke_marsh]

If Industry started using decent grounding then the radiological world would have to spend far to much on Signal processing R&D to keep up with their agendas and what the odd lightening strike anyway to a company. Just a new UPS device or a couple of server fixes or insurance claims.I do however agree with what should have been done but if human development did half of what it should have done in the first place would we even be in the pickle we're in.
When Economy is disposable people are more dependent on the system at large which makes other more powerful. If people were doing better and thought less impulsively and hence took all the right precautions then people would be more inclined to tell policy makers wanting power where they could put their fab new idea for control of them.

[Scott Trotter]

This is a completely pointless story. This kind of stuff happens all the time. The real issue here is web application architecture. If ANY web application became unavailable as the result of this lightning strike, then that application's architect SHOULD BE FIRED for failing to provide for adequate fault-tolerance and redundancy.

[luke_marsh]

Ahh I knew the met office groups were having a crack I'm little surprised they succeeded. Although I noted they managed on the second attempt to get the temperature flux required in ground temperature ionic s it did not seem to me as if they had connected enough charge off the earth field to induce then a a lightening storm but I suppose this was then compensated by the conductivity of the silver trails in the upper atmosphere enough to allow them to get the effect out in some other rigorously Silvered up Atmosphere zones. All I can say is well done to them but I doubt they'll manage much more than a crackle and the odd wap.

[luke_marsh]

I must admit they seem to be quite good with ionics and the basic's of the planets field but they aren't that good with more advanced features of the field and unlike all these bunk UFO conspirators claims they have a **** poor understanding of gravity. Yes With little competition in moments they can indeed hover a magnetic superfluid based centralised engine around a bit but that sort of thing belongs in a circus and I suspect that big new high speed US jet isn't that much cop either considering the poor coding dynamics an overly complex hardware it's probably using. No wonder the US Military is at war with a bunch of sheep farmers in a Locally endorsed war campaign as opposed to anything else because if they flew those drones anywhere else why one reusable emp and a couple of Micro needle tranquillizer pea shots later and there would be no top brass or good engineers left would there.

[Draq Wraith]

All your clouds are owned by lightning.

Ha just imagine if there were an energy crunch and power could not be delivered.

D~W

[luke_marsh]

You might laugh about that now but If indeed the recessional periods to come aren't to deadly to consumer growth then Indeed in five or so years countries like the USA might actually be suffering a lot more with power demand issues.
Just times 300 million people by a Killo watt and tell me where that kind of supply would come from. I can almost see the Doc off back to the futures face at the thought of it. This means that if growth in Electrical usage does expand fast then Energy companies won't only stop stifling good personal Energy generation additions to the AC power grid they will be heavily encouraging It as to stop the grid collapsing. Already today they promote more Energy solutions and Energy saving concepts than they would dare even consider before and this although used to paint a nice picture of the good caring guy of themselves is for energy groups and economic groups one of necessity more than simply out of the nature loving kindness of their hearts.

[luke_marsh]

Do you know what the worst bit is if you related by 2030 the wishes of then roughly 10 to 12 billion people to the axioms of human know how as to deduce how you would manage 11 billion people times 20Kilo watts. The only way off the cureent R&D paths by that time to obtain such a degree of energy for that degree of demand would be to tap it off the Suns Ionic flux and considering the Degrees of inefficiencies relative to so called free energy which is nothing of the sort you would then be Managing Ionic sterility effects that would be very difficult to regulate with that degree of Zetta scaling. I suppose by 2050 you might work out it far more effective to alter the Temperate states of the other solar system and tap the energy straight off heaven itself where even if you were looping round a Terra watt each you would still be the equivalent to a microscopic nat.
If of course you did want to compete with say local advanced project in say M32 and to some degree M31 you might want to develop tritiral plane reseach technologies that tap the the neck zone of the cosmos for time and space travel ect purposes. Then you would have to access more exotic points of energy as Hevan probably wouldn't accommodate you well in that pursuit unless you were content with the wider Islands cosmic regulatory evolutionary rules especially being of such a primordial species. Of course if you were to tap More subordinate to the three local Island zones with Irregularly bypass the the rules tapping in to wider oceanic resources. What would you do with such technology seeming as you can barely comprehend linear concept of time let alone advanced aspect of Cosmic tritational extensibility. So lightly you will Suck a bit of the sun with poor being technologies charge off some rocks and dot around a fraction of the local Island but later species of this planet within the neck zone of the cosmos might go a bit further than humanity.
At the moment though your stuck with 300 million plus US citizens alone wanting probably 1 Kilo watt each from the energy resource pool, you can use with using current magna charging techniques very limited spare Ionic energy to pull from. Not to mention a fast growing electric using generation on top of those demands also wanting a slice of the action.