Lightning zaps Amazon cloud
Amazon.com is blaming the latest outage to hit its Elastic Compute Cloud service on a lightning strike at one of its data centers.
In a statement on the Amazon Web Services "health dashboard," the online retailer and cloud-computing provider addressed concerns from some U.S. customers whose EC2 service had been disrupted around 6:20 p.m. Pacific Daylight Time on Wednesday.
"A lightning storm caused damage to a single Power Distribution Unit (PDU) in a single Availability Zone. While most instances were unaffected, a set of racks does not currently have power, so the instances on those racks are down," the company said initially on the health dashboard.
The disruption lasted about seven hours, during which time Amazon asked any affected customers to use alternative parts of the network. "Users with affected instances can launch replacement instances in any of the U.S. Region Availability Zones or wait until their instance(s) are restored," Amazon said.
The company later attributed the outage to a problem on one "availability zone" and that the outage was localized. "We would like to reconfirm that this issue was limited to the single Availability Zone where this power issue occurred, and that a very small percentage of instances in that AZ were affected; this was not a generalized service issue," Amazon said.
Despite acknowledging that Amazon had dealt with the issue fairly efficiently, one user was concerned that a single lightning strike was able to bring down the service, if only in a limited way.
"I was under the impression that your architecture had more resiliency built into it. Yes we can use multiple availability zones to help with a single point of failure, but I thought that even within a single availability zone there was not a single point of failure for hardware/power," the user posted to an Amazon forum on the issue.
The EC2 service provides customers with virtual access to Amazon's computing infrastructure, using virtual machines that can be created using the Xen virtualization platform. First launched in a limited beta in August 2006, the EC2 service went fully live in October 2008.
Not including the latest issue, the service has suffered two major disruptions during that time in February 2008 and October 2007. In June 2008, Amazon's main retail site suffered an outage that the company blamed on the complexity of its own systems.
A series of outages that have hit other online or cloud computing services including Google's Gmail and other applications over recent months have led some critics to question whether the cloud approach to computing is really capable of providing the resilience required by enterprise users.
In mid-May, Google services were hit by an outage which apparently affected one in 10 of its users. In January, software-as-a-service pioneer Salesforce.com experienced an outage that disrupted all its customers for about an hour.
Here is the string of messages from Amazon about this week's outage.
(Credit: Amazon.com)Andrew Donoghue of ZDNet UK reported from London.
If I am consuming capacity from your cloud, I shouldn?t even care about underlying infra details, much less be required to pick up the pieces behind a failure in your system. That is the whole point of a cloud, for the consumer to focus on his core strengths and the provider to make the service transparently available at all times.
Otherwise, you are just hosting in a server farm.
I have seen contractors drop wrenches across power supply buses and take down an entire data center line-up, but lightning is just too unlikely.
Now this may not be a design failure- data centers are designed to be protected against outside conditions with their own UPS's, generators, etc. This could have been a building construction issue insteaed. Something unplanned / unforseen such as a metal conduit from the roof that happens to go through the data center in some completely bizarre way- it's happened before where a stairway railing becomes the vector all the way through the system. It's a tricky thing to totally isolate a room and network, but that's why they cost so much too. All of which can be negated by some single oversight.
Still, even if a rack of systems goes down, there should be a redundant parallel array that takes up the slack, and that IS a design failure.
When Economy is disposable people are more dependent on the system at large which makes other more powerful. If people were doing better and thought less impulsively and hence took all the right precautions then people would be more inclined to tell policy makers wanting power where they could put their fab new idea for control of them.
- by Draq Wraith June 12, 2009 10:55 AM PDT
- All your clouds are owned by lightning.
- Like this Reply to this comment
-
-
- by luke_marsh June 12, 2009 11:41 AM PDT
- You might laugh about that now but If indeed the recessional periods to come aren't to deadly to consumer growth then Indeed in five or so years countries like the USA might actually be suffering a lot more with power demand issues.
- Like this
-
- by luke_marsh June 12, 2009 1:09 PM PDT
- Do you know what the worst bit is if you related by 2030 the wishes of then roughly 10 to 12 billion people to the axioms of human know how as to deduce how you would manage 11 billion people times 20Kilo watts. The only way off the cureent R&D paths by that time to obtain such a degree of energy for that degree of demand would be to tap it off the Suns Ionic flux and considering the Degrees of inefficiencies relative to so called free energy which is nothing of the sort you would then be Managing Ionic sterility effects that would be very difficult to regulate with that degree of Zetta scaling. I suppose by 2050 you might work out it far more effective to alter the Temperate states of the other solar system and tap the energy straight off heaven itself where even if you were looping round a Terra watt each you would still be the equivalent to a microscopic nat.
- Like this
-
(13 Comments)Ha just imagine if there were an energy crunch and power could not be delivered.
D~W
Just times 300 million people by a Killo watt and tell me where that kind of supply would come from. I can almost see the Doc off back to the futures face at the thought of it. This means that if growth in Electrical usage does expand fast then Energy companies won't only stop stifling good personal Energy generation additions to the AC power grid they will be heavily encouraging It as to stop the grid collapsing. Already today they promote more Energy solutions and Energy saving concepts than they would dare even consider before and this although used to paint a nice picture of the good caring guy of themselves is for energy groups and economic groups one of necessity more than simply out of the nature loving kindness of their hearts.
If of course you did want to compete with say local advanced project in say M32 and to some degree M31 you might want to develop tritiral plane reseach technologies that tap the the neck zone of the cosmos for time and space travel ect purposes. Then you would have to access more exotic points of energy as Hevan probably wouldn't accommodate you well in that pursuit unless you were content with the wider Islands cosmic regulatory evolutionary rules especially being of such a primordial species. Of course if you were to tap More subordinate to the three local Island zones with Irregularly bypass the the rules tapping in to wider oceanic resources. What would you do with such technology seeming as you can barely comprehend linear concept of time let alone advanced aspect of Cosmic tritational extensibility. So lightly you will Suck a bit of the sun with poor being technologies charge off some rocks and dot around a fraction of the local Island but later species of this planet within the neck zone of the cosmos might go a bit further than humanity.
At the moment though your stuck with 300 million plus US citizens alone wanting probably 1 Kilo watt each from the energy resource pool, you can use with using current magna charging techniques very limited spare Ionic energy to pull from. Not to mention a fast growing electric using generation on top of those demands also wanting a slice of the action.