Google networking error caused outage
Updated at 12:25 p.m. PDT with word that Google has confirmed an error on its end caused the outage, and at 3:30 p.m. PDT with Google's comment on McAfee's description of the events.
Widespread outages involving several Google services--including search, Google Docs, and Gmail--were caused by an upgrade gone awry inside of Google, according to engineers.
Dmitri Alperovitch, vice president of threat research for McAfee, said that Google this morning attempted to make changes to key Internet routing numbers--known as autonomous system numbers--as part of its ongoing transition from an older networking standard to a newer one called IPv6. An unknown "bug" inside Google's network involving some sort of hardware failure or glitch prevented Internet service providers from finding Google's new ASNs on the Internet--effectively sealing it off from many customers, he said.
Not all Internet users were affected, but some that use larger providers--such as AT&T or Verizon--appeared to be disproportionately hurt because large ISPs "peer" with Google, or interconnect their networks with Google's networks in order to improve speed and reduce bandwith costs, Alperovitch said. Not all customers at those providers were affected, and smaller ISPs that didn't interconnect their networks were able to route around the problem. But just like when a bad car accident shuts down a key highway, the ripple effects were felt elsewhere.
The outage began at 8:13 a.m. PDT, according to McAfee's data, and was fixed by 9:14 a.m. PDT. The issue was discussed inside forums dedicated for ISPs and their engineers, such as the North American Network Operators Group. McAfee's customers reported the issue to the security company, which monitors network traffic for some customers.
Google is a major fan of IPv6 and makes many of its services available through the new network technology. However, IPv6 has been slow to arrive overall, in part because it's a very difficult transition from the current IPv4 network.
Google spokesman Eitan Bencuya wouldn't confirm what caused the problem but said the company plans to detail what happened in a company blog to be published "shortly."
Update at 12:25 p.m. PDT: Google has confirmed that "an error in one of our systems caused us to direct some of our Web traffic through Asia, which created a traffic jam." The company did not elaborate on what caused the error in a blog post, but claimed just 14 percent of users were affected.
"We've been working hard to make our services ultrafast and 'always on,' so it's especially embarrassing when a glitch like this one happens. We're very sorry that it happened, and you can be sure that we'll be working even harder to make sure that a similar problem won't happen again," Google wrote.
Updated 3:30 p.m. PDT: Google has denied that work on the transition to IPv6 is to blame for this morning's outage, but will not specify what was to blame. "This issue is unrelated to any work we are doing in transitioning toward supporting IPv6," a spokesman said. McAfee said it obtained its information from Google on a private mailing list for networking professionals of which Google is a member, but declined to provide a copy of the thread in question.
Tom Krazit writes about the ever-expanding world of Internet search, including Google, Yahoo, online advertising, and portals, as well as the evolution of mobile computing. He has written about traditional PC companies, chip manufacturers, and mobile computers, spending the last three years covering Apple. E-mail Tom. 
@Aaron: Custom built stuff always has pros and cons, but to be fair I think their average is almost 99.9 for the last 30 days, including this (extra-ordinary) outage. The world is a single point of failure, and you can't engineer every possible problem 100% out of the equation (but Yes, 5 9's would be better ... there is always a cost-benefit aspect :). ).
I am not saying differences aren't present, of course they are(!), but there is a large amount of shared ancestry inside ... and from what I have heard the outage was primarily due to an (BGP/ASN?) misconfiguration - nothing due directly to IPv6 as a protocol.
www.ipv4depletion.com
I guess my point is that Google has the ability to recover *very* quickly from such events. They are very prepared - lesson to be learned.
Tried proxies and i can access google.
We all going to next grade level all those hidden botnet look lost .
I didn't had any Problems! >>Google is just great<<
- by the liquid man May 15, 2009 7:01 AM PDT
- why all the crying? It's not like you get charged for using Gmail/Google/etc
- Like this Reply to this comment
-
(27 Comments)