Webware

Read all 'outage' posts in Webware
December 23, 2009 5:23 PM PST

Web staggers under pre-Christmas DDoS attack

by Tom Krazit
  • 11 comments

Editor's note: This post was continuously updated as this story developed. For a more complete account of what happened, see our followup story here.

Update: A customer support representative for NeuStar, the company that provides the UltraDNS service, confirms the outage was the result of a DDoS (distributed denial of service) attack. More details below.

Procrastinators, beware: Amazon.com and a host of Internet shopping sites are having trouble Wednesday evening.

Two days before Christmas is likely not the best time for Amazon to go down, but at some point around 5 p.m. PST Wednesday evening, Amazon was loading extremely slowly. One CNET reader wrote in to say his pending transaction failed just as he attempted to complete it, and the site was extremely sluggish as the sun set on the West Coast Wednesday.

Twitter users of Amazon's S3 service for Web hosting reported outages as well, and customers of Salesforce.com and Walmart.com were also reporting problems on Twitter.

We'll update as we learn more. If you can't (or can) access Amazon from your location, please let us know in the comments.

Updated 5:35 p.m. PST: It appears the problem may be larger than Amazon. Reports began to immediately circulate that UltraDNS, the DNS provider for several West Coast Internet companies, was having serious problems. In addition to Amazon, Salesforce.com, and Walmart.com, problems were also reported with Expedia.com.

Rusty Hodge, general manager and program director at Internet radio station SomaFM, tweeted that UltraDNS simply went down, taking down customers of Amazon's S3 and EC2 services as well as Amazon.com itself. Jeff Barr, lead Web services evangelist at Amazon.com, retweeted Hodge's statement without comment.

In perhaps a related problem, the Internet Health Report shows severe latency and packet loss on a connection between Qwest and Savvis.

Updated 5:45 p.m. PST: UltraDNS representatives could not be reached for comment. Amazon.com, Walmart.com, and Salesforce.com all seemed to come back to life around 5:40 p.m. PST, but some problems were still being reported.

Amazon representatives did not immediately return a call seeking comment.

Updated 5:53 p.m. PST: Amazon's AWS Service Health Dashboard reports "DNS resolution errors" affecting Amazon Simple Storage Service customers for Northern California and U.S. Standard.

Updated 6:10 p.m. PST: A customer support representative for Neustar, the company that provides the UltraDNS service to several e-commerce sites, confirmed that its network was hit by a DDOS attack targeting their California network in Palo Alto and San Jose.

As of 6:15 p.m. PST, things seemed back to normal. The Internet Health Report also showed an improvement on the Qwest-Savvis line noted earlier, and Amazon's Web Services dashboard confirmed that while there were problems resolving DNS requests, "the service is successfully responding to requests."

Originally posted at Relevant Results
November 23, 2009 1:03 PM PST

'Technical issue' downs eBay search over weekend

by Caroline McCarthy
  • 8 comments

eBay on Sunday confirmed that a "technical issue" had caused search queries on the auction site to be messed up over the weekend, resulting in limited or no search results. The company says that it's being cautious, though, and is holding back on some advanced search features until the issue is fully solved.

"We are happy to report that critical search functionality was restored overnight on Saturday and we are seeing normal activity levels today," a post on the company's eBay Ink blog read Sunday. "As part of our effort to restore critical search functionality as quickly as possible for sellers and for buyers, we have kept some secondary search features temporarily offline. This includes refining search by certain item specifics, such as color or clothing size, and having Store Inventory Format results included in the main search results."

In a statement, eBay also said the technical issue was caused by "a surge in live listings as sellers ramp up for the holiday season. eBay currently has more than 200 million live listings, 33 percent more than at this time a year ago."

Some eBay members still weren't satisfied with the explanation. "I had a one day auction ending today, (and) no one was obviously able to bid on it because they couldn't search for it," one commenter said on the eBay Ink blog. "Will I get a credit for this?"

"eBay should credit all sellers with active listings during this time," another said. "These issues have cost sellers many bids and sales. Once again eBay is screwing sellers."

Much like Twitter's today, outages at eBay were rather prominent in the company's early days. They're not too frequent anymore. But this one came at a time when there are some sentiments of malaise among eBay sellers, some of whom use the auction site to make a living, and when it also faces increased competition in the e-commerce sector.

An analyst release from JP Morgan Chase said that it did not anticipate the outage would have an effect on eBay's fourth-quarter earnings. But, it contained a warning: "Although we recognize it is virtually impossible for a site of this complexity to not encounter occasional issues," the report from analyst Imran Khan read, "we continue to believe that eBay needs to make greater investments in the robustness and functionality of its site in order to remain competitive within the e-commerce space."

Originally posted at The Social
October 20, 2009 10:10 AM PDT

Flickr hit with Tuesday morning outage

by Josh Lowensohn
and
Tom Krazit
  • 7 comments

Yahoo-owned social photo site Flickr went dark Tuesday at around 8:50 a.m. PDT. The outage, which remains ongoing at time of this initial post, is keeping users from accessing all parts of the site, however photos that had been embedded on third-party sites are still able to be viewed.

An update on Flickr's official blog, timestamped at 9:51 a.m. PDT, says "all hands are on deck," and the problem will soon be resolved. That was followed shortly thereafter by a post at 10:05 a.m. PDT saying that that outage "shouldn't be too much longer!"

Flickr's last major outage, which took place back in February 2007, resulted in the company revealing some details about the immensity of the photo sharing site, which at that time was serving close to a billion photos a day.

More details as they come...

Updated 10:55 a.m. PDT: A Yahoo representative had no details on the nature of the outage, but it appears to be a problem with the Web servers rather than a data issue. Yahoo updated the Flickr blog to inform users that photos embedded into a Web site should still be appeared on those sites.

Updated 11:35 a.m. PDT: Flickr is back up and running.

Updated 12:01 p.m. PDT: Flickr released a statement on the outage.

"Flickr regularly makes routine updates to the site - and once in a blue moon we hit a snag in the road. Flickr is now back to normal and no data was lost during this morning's outage. Members who might have been uploading at the time should have received an error message, but should be able to share photos and videos now. We continued to serve photos to 3rd party sites throughout the service interruption. Thanks for bearing with us and feel free to let the team know if you continue to experience any issues."

Correction 11:35 a.m. PDT: This story initially misstated that embedded images could not be viewed during the outage.

Originally posted at Web Crawler
October 15, 2009 8:33 PM PDT

Facebook's COO: Response to disabled accounts was 'too slow'

by Caroline McCarthy
  • 15 comments

PALO ALTO, Calif.-- Facebook Chief Operating Officer Sheryl Sandberg admitted in a talk here on Thursday evening that the company's response to a database outage that knocked out approximately 150,000 user accounts was "too slow."

"It's a very small percentage of our users, but it's a lot of people," Sandberg said of the affected users. "We want them to be able to (access Facebook) every day. We resolved it in about a week and a half. I think that was too slow."

Numerous Facebook users began complaining early this month that they could not access the social network, instead receiving a notice that their accounts were "down for maintenance." Many of them claimed that repeated requests for information from Facebook went unanswered, and clamored for better customer service and communication.

The whole affair was "a little frustrating, but it ended," Sandberg said, and chalked it up to the social network's extremely rapid growth. It now has more than 300 million active users around the world.

"We are, I promise, doing our best to scale," she continued, reiterating that all data (except for some recent updates, a statement from Facebook said last week), "and our growth means we're sometimes a little bit behind."

Originally posted at The Social
October 12, 2009 10:32 PM PDT

Sidekick's lesson: We learn by failing

by Rafe Needleman
  • 46 comments

Technically, we can't blame the loss of Sidekick users' data on a failure of either the concept or the technology of "cloud computing." But Microsoft's clear bungling of basic information management practices (apparently, there were backups--but they didn't work) does cast a pall over not just Microsoft but the cloud concept entirely.

T-Mobile is trying to keep customers' data alive.

(Credit: Screenshot by Rafe Needleman/CNET)

Microsoft, as one of the giant infrastructure technology companies that's saying through its product offerings that data is safe in the "cloud," has a responsibility not just to its customers but to the growth of cloud computing overall to keep the data it's holding safe.

The company's failure to keep the data safe shows the world how fragile cloud computing is. Even though, really, it isn't. The world knows how to build systems that safeguard data from hardware and software and network failures, and even from hacking and other forms of sabotage. The fact that Microsoft failed to keep the Sidekick data backed up indicates, rather, how management can fail.

But do consumers, or corporate IT managers considering cloud-based services, care where the failure was? All we know is that it failed.

Travel by commercial airliner is neither unsafe nor inherently safe because of the technology itself. It is as safe or as dangerous as the procedures followed to certify and maintain the equipment that people put their life's trust in.

Microsoft's Sidekick outage shows that sadly, in fact, it's true: you cannot trust the cloud because you can't trust the people who run it. It indicates another scary truth: We haven't had enough cloud failure yet. We're going to have more. We need more. We learn from each failure. And we're all thinking the same thing: I hope I'm the beneficiary, and not the victim, of the hard lessons still to come.

Data in the cloud can be safe. And it will become more safe thanks to this outage. Failures of trust, like this one, have costs, but there are benefits as well.

More: Full coverage of the Sidekick data loss

Originally posted at Rafe's Radar
October 12, 2009 3:13 PM PDT

Facebook database outage cut off about 150,000

by Caroline McCarthy
  • 37 comments

Thousands of Facebook users who have been unable to access their accounts for nearly a week and a half now are now seeing their profiles restored--but some data related to recent profile updates may have been lost.

What happened? According to Facebook, the replacement of profiles and login screens with a "down for maintenance" notice--which appears to have started on October 3--stemmed from "a technical issue with a single database." The company has stressed that there is no chance that it was due to hackers or other malicious activity.

Profiles should be restored over the course of the next day, the company estimates.

"Our engineering team has worked around the clock, and as of today, all of these users should begin to regain access to their Facebook accounts," Facebook spokeswoman Brandee Barker said reading from a statement. "We apologize for the inconvenience this may have caused and we are taking additional measures to uphold the reliability users come to expect from Facebook."

Less than 0.05 percent of Facebook's users have been affected by the outage, the company estimated. The social network's last head count, about a month ago, was 300 million active users, so that comes out to be a total of about 150,000 affected users. Not very many but enough to put some of them in a panic over not being able to access a primary mode of communication and (in some cases) business.

Profiles have not been lost or deleted, Facebook has continually said--even though the company has been otherwise tight-lipped about the maintenance issue until this point. When affected users' access is restored, however, some things may be different and very recent updates may be missing. According to a notice that Facebook is displaying to members who may have been affected:

You may not have been able to access your account over the last several days. We're sorry for this inconvenience; an extended technical issue affected a small number of Facebook accounts, including yours. We have done our best to restore your account to its most recent state, but some data and settings may not be current. In order to be cautious, we defaulted some of your privacy settings to their most restrictive settings. You may wish to review your privacy settings and reset them.

Facebook added that "some of (affected members') content may not be up to date: in other words, some minor data loss regarding recent updates to profiles. This, according to Facebook, may include photos that were recently added or deleted, recent updates to friends lists (additions and deletions), and "other content you've added, sent, received, or posted."

As for the company's relative silence about the matter until now, Barker explained in a phone call that the company wanted to nail down the specifics of the outage and figure out the situation, rather than provide details to users that could turn out to be inaccurate.

Many of the complaints pertaining to the outage alleged poor customer service on Facebook's part, and as a sort of olive branch, the company is encouraging feedback pertaining to the specific outage. The alert displayed to affected members whose accounts have been newly restored directs them to a form to report any further details or additional problems.

Whether Facebook will step it up a notch for future unexpected technical problems remains to be seen.

Originally posted at The Social
October 10, 2009 8:20 AM PDT

Downed Facebook accounts still haven't returned

by Caroline McCarthy
  • 76 comments

Something is really odd here.

As a reporter covering Facebook, I do get the occasional cranky complaints from members who, for one reason or another, are experiencing errors when they try to access their accounts. But it's never been anything like the past week, with a steady stream of e-mails continuing to come in from Facebook members who say they remain shut out of their accounts--despite assurance from Facebook that profiles have not been deleted and that the company is working on the problem.

"This is now seven days and counting," an e-mail sent on Saturday morning read. "It's beyond ridiculous and extremely frustrating."

"The experience completely reversed the Facebook opinion and experience for me," one reader complained. "I see many people bitch and complain, many more beg and a few threaten. To me, the route to take is fairly obvious. Mark Zuckerberg on his own page invites democratic input from Facebook users in one of his most recent videos. Given that statement especially, I find the way their user base is being treated with respect to their disabled account policy hypocritical at best."

"My account has now been held hostage for a week," another reader wrote. "Some of my friends think that I have deleted (my profile) or even blocked them...None of my friends or family can see my profile or even find it in search. It's as if I simply deleted my account or blocked all of them from seeing it without even a word."

Some users have started threads on Get Satisfaction and Yahoo Answers. A few others have pointed me to blogs and YouTube channels devoted to the subject.

The inaccessible accounts appear to be limited to a very small subset of Facebook's over 300 million active users, which means that it's not a large-scale issue for the health of the site. And Facebook is supported by neither subscription money or taxpayer dollars (though it wouldn't have advertising revenue without its users) so there's an argument to be made that users shouldn't be complaining about something they don't pay for. But that's an argument that many of the people who have come to rely on Facebook as a channel of communication simply don't buy.

Whether the string of complaints is warranted or not, Facebook hasn't disclosed exactly what's caused the "extended maintenance issue," and that's what I find puzzling.

Originally posted at The Social
October 8, 2009 9:42 AM PDT

Something is clogging the Twitter stream!

by Caroline McCarthy
  • 14 comments

Something has dammed the Twitter river. I bet it was this guy.

(Credit: Creative Commons licensed: flickr.com/photos/sherseydc)

Holy cow. Is nothing on the Internet working these days? Facebook's acknowledged that a number of members have had account maintenance issues, and now Twitter has confirmed that "many" users are experiencing timeline delay problems.

Basically, the lowdown is that you can post tweets, and they'll publish, but that your timeline--the stream of updates from the Twitter accounts you follow--isn't bringing up any new tweets. For me, it looks like this started at around 8:00 a.m. PT.

Twitter, which has been prone to many an outage in its three-year history, says it is investigating the problem and will provide an update shortly.

(Photo by Flickr user sherseydc, licensed under Creative Commons).

Originally posted at The Social
October 8, 2009 8:58 AM PDT

Facebook's mounting customer service crisis

by Caroline McCarthy
  • 85 comments

There are some things that are nice to wake up to. The smell of bacon, for example. On Thursday morning, however, I woke up to something a little less pleasant: an in-box full of e-mails from Facebook members whose accounts are still inaccessible. Some were more or less on the verge of, well, panic.

Earlier this week, we wrote about Facebook's acknowledgment that some members could not access their accounts for several days, instead receiving a "down for maintenance" error. At the time, a Facebook representative explained that it was a "technical issue with one of our databases" and estimated that it would be resolved within 24 hours. It's unclear how many accounts have been affected.

But a resolution of the problem doesn't seem to have occurred, judging by the e-mails that were still showing up in my in-box well into Thursday morning. I sent another request to Facebook to find out more.

"We are continuing to work on the extended maintenance issue that is restricting some users from accessing their accounts," a statement e-mailed by a Facebook representative explained. "No accounts have been compromised during this process, and access will be restored as soon as possible. We apologize for any inconvenience."

Reader e-mails indicated quite a bit of frustration.

"I lost my job back in March and have been using this site as a networking tool," one reader's e-mail said. "It's frustrating that it's been down for so long."

Some were paranoid that their accounts had been deleted and all their contacts lost. And many of the e-mails cited unresponsiveness on Facebook's part despite multiple customer service complaints. Third-party customer service forum Get Satisfaction was filled with chatter about Facebook login and access problems, including at least one threat of a class-action lawsuit.

"Accounts are still down as of this Thursday morning," another e-mail read. "Facebook has been completely non responsive to its users. My account has been down with site maintenance issues since Saturday. I have sent over 20 requests to FB and joined help user groups looking for answers."

From yet another e-mail: "So far Facebook has been largely unresponsive to my emails, saying that this issue can't be reported as a security issue. It seems absurd to me that Facebook customers have no way to directly contact Facebook regarding problems."

With over 300 million active users around the world, we shouldn't expect Facebook to be able to respond to every inquiry it receives. And Facebook is a free product, so it arguably doesn't have a customer service obligation on par with your cable company or the Web site where you bought your last pair of shoes. But this is still a real problem for the social network, which has become so ingrained in culture and communication that for some people it's replaced the address book, the e-mail client, and the personal Web site. Many of the e-mails I received came from people who say that Facebook is their primary method of communication with far-flung family and friends. Others said it's crucial to how they do business.

Here's something else: Facebook doesn't offer a way for members to export their contact information into an address-book format, something that took center stage when blogger Robert Scoble had his Facebook account temporarily banned after testing a script that would export his contacts' information to Plaxo. Even now that Facebook has launched its Facebook Connect login product, there still is no easy way to access your contacts offline. The current account-access snafu indicates that this is a big void.

At the very least, Facebook could make some kind of mass message available explaining what exactly the problem is and reassuring people that inaccessible accounts have not been permanently deleted (assuming that's the case)--something easier to find and more detailed than the brief statement now posted to its company "fan page." On a more long-term level, this seems like a big red flag that Facebook needs to streamline its customer service operations somehow so that this sort of hysteria can be prevented.

A way to export basic contact information for offline access--phone numbers, e-mail addresses, instant-message screen names--wouldn't be bad either.

This post was updated at 10:41 a.m. PT with comment from Facebook.

Originally posted at The Social
September 24, 2009 8:17 AM PDT

Gmail outage hits 'small subset of users'

by Stephen Shankland
  • 35 comments

Gmail was unavailable Thursday morning for what Google said was a "small subset of users," the latest outage from a company that prides itself in running advanced computing systems.

On the Google Apps status dashboard, the company said at 7:29 a.m. PDT that it was aware of the problem. However, using IMAP (Internet Message Access Protocol) to access e-mail through software such as Outlook or Thunderbird still works, the company said.

Even a small subset can be a lot of people, though, as carping on Twitter indicates.

Gmail outages hit Google itself sometimes, providing extra incentive to improve reliability. One company spokesman, Adam Kovacevich, said on Twitter, "Gmail down (for Googlers too)."

Google had Gmail outages in February, April, and very widely on September 1.

Gmail was working for me Thursday morning, but slowly and without access to my contacts at 8 a.m. PDT. By 8:13 a.m., it was behaving properly.

Updated 9:15 a.m. PDT - Many users are reporting that their e-mail is back to normal, but there are still problems with Gmail contacts. Google posted the following advisory at 8:29 a.m. PDT.

"The Gmail issue should now be resolved for most of our users. There still might be issues with your contacts. For Gmail users: Use www.google.com/contacts to access your contacts For Google Apps Customers: www.google.com/contacts/a/yourdomain-name.com."

Updated 9:54 a.m. PDT - Google asked for more time before it feels ready to declare an all-clear. "We are continuing to investigate this issue. We will provide an update by September 24, 2009 10:30:00 AM UTC-7 detailing when we expect to resolve the problem."

Updated 10:10 a.m. PDT - It's now safe to return to your computer, according to Google. "The problem with Google Mail should be resolved. We apologize for the inconvenience and thank you for your patience and continued support."

Tom Krazit contributed to this report.

The Google Apps status dashboard flagged the Gmail problem Thursday morning.

The Google Apps status dashboard flagged the Gmail problem Thursday morning.

(Credit: Screenshot by Stephen Shankland/CNET)
Originally posted at Deep Tech
advertisement

About Webware

Say No to boxed software! The future of applications is online delivery and access. Software is passé. Webware is the new way to get things done.

Add this feed to your online news reader

Webware topics

15 sites that went kaput in 2009

Web sites launch all the time, but they also shut their doors. We highlight 15 that bit the dust this year.

Top 10 news stories of the decade

Let the debate begin: Was the iPhone more important than iTunes? Was anything bigger than Google finding a great business model? CNET offers its list of the 10 most important stories of the '00s.

Most Discussed

Inside CNET News

Scroll Left Scroll Right