GameSpot and Craigslist, where did you go?
SAN FRANCISCO--A power outage hit downtown San Francisco Tuesday afternoon, leaving thousands of residents without power and knocking popular Web sites such as Craigslist, GameSpot, Yelp, Technorati, TypePad and Netflix offline for a few hours.
The power failure apparently hit 365 Main, a 227,000-square-foot data center in downtown San Francisco, particularly hard. The data colocation center's client list includes Craigslist and CNET Networks' GameSpot, a sister site of News.com.
It wasn't immediately clear if the other affected Web sites were customers of 365 Main or of other Web hosting companies, or whether the sites were blacked out for all visitors.
(Credit:
CNET Networks)
At 4 p.m. PDT in an e-mailed statement, Miles Kelly, 365 Main's vice president of marketing, had this to say: "At 1:45 pm today, there was a major power event in San Francisco that impacted business operations for many San Francisco based companies, including 365 Main's San Francisco data center. PG&E has not yet determined the cause of the failure. Some customers within the 365 Main facility were temporarily effected by the utility failure. The building is currently 100 percent operational and running on back-up power (generators) until the company can confirm that utility power is stable."
Most sites seemed to be working again by 4:45 p.m. What this means for 365 Main's service agreements with its customers, which promises 24-hour-a-day, 7-days-a-week, 365-days-a-year power, is still unclear. We're waiting to hear more from 365 Main. In an ironic note, a press release from 365 Main dated Tuesday noted the company had provided Red Envelope "two years of continuous uptime."
Update 5:00 p.m.: Asked why the data center company, which bills itself as "the world's finest data centers," failed to meet its standard of ensuring service to customers even in the event of a power failure, Kelly said "I don't know."
According to the San Francisco Chronicle, Pacific Gas & Electric said that more than 30,000 of its customers lost power after an explosion under a manhole cover on Mission Street.
A man exits the main entrance of 365 Main on Tuesday.
(Credit: Elinor Mills/CNET News.com)Contrary to prior reports, there was no mob of angry customers outside the 365 Main building, and no drunk employee had gone on a rampage, according to Kelly. The "mob" was actually a line of customers who were forced to enter through the front door and have badges checked manually to get into the building because the parking garage gate was affected by the power outage, according to Chris Hutchens, a network engineer at SF Data, which is a customer of 365 Main.
Update July 25, 9 a.m.: Kelly released some more details of went wrong at the San Francisco data center. While they still don't know all the answers, here's what they're saying now:
--Power was restored at 2:34 p.m. Tuesday afternoon, after a 45-minute outage at 365 Main. At least three of the eight colocation rooms were affected, and possibly more.
--So far it is estimating that the power failure impacted 20 to 40 percent of its San Francisco customers.
--The company is still investigating why parts of the back-up generator system failed during Tuesday's power surge.
Erica Ogg is a CNET News reporter who covers Apple, HP, Dell, and other PC makers, as well as the consumer electronics industry. She's also one of the hosts of CNET News' Daily Podcast. In her non-work life, she's a history geek, a loyal Dodgers fan, and a mac-and-cheese connoisseur. E-mail Erica. 
211: FAILBOAT - Some read only time this evening
Updated Tue, Jul 24 - 01:01 PDT
We need to do some unscheduled db maintenance this evening. Gonna fire it off as soon as i figure it all out. I'm a chikin, LOL.
Read only from 11pm pst - 11:45 pm pst.
back for more: read only to about 1:30am pst We are back. Let's see how this runs.
I'm eagerly awaiting what our operations staff finds out about this.
:(
Given the fact that most big customers have no idea which/what systems are critical, don't forget that everything in the path would need protection (transport gear, routers, switches, firewalls, load balancers, etc) I find it very unlikely anyone would go for it.
Fact is, most datacenters rely on either active or battery-backed UPS to provide power for a short period of time (10s-60s) until the generators kick in. Looks like in this case, the gens failed to come up for a portion of the floor.
It's going to be a long quarter for management...
Of course it could be like the article said - "...backup generators failed". Considering even the largest UPSs won't work for longer than a half-hour (and 365's were probably spec'd out to work for considerably less time, as they had backup generators in place) they might have actually had UPS in the room, and the customers may even have had UPS on their individual servers as well.
But we'll just stick with the "they are idiots who don't know the obvious" stance. It makes those of us who think of everything feel much better.
ouch
http://brain.com
- Eeks!
- by iggysnitx July 25, 2007 5:45 PM PDT
- The only line longer than the one in front of 365 Main will be the one at the courthouse when folks start suing them for violating their TOS and the service agreements due to the power failure.
- Reply to this comment
-
(12 Comments)Curiouser and curiouser.