The Social

PALO ALTO, Calif.-- Facebook Chief Operating Officer Sheryl Sandberg admitted in a talk here on Thursday evening that the company's response to a database outage that knocked out approximately 150,000 user accounts was "too slow."

"It's a very small percentage of our users, but it's a lot of people," Sandberg said of the affected users. "We want them to be able to (access Facebook) every day. We resolved it in about a week and a half. I think that was too slow."

Numerous Facebook users began complaining early this month that they could not access the social network, instead receiving a notice that their accounts were "down for maintenance." Many of them claimed that repeated requests for information from Facebook went unanswered, and clamored for better customer service and communication.

The whole affair was "a little frustrating, but it ended," Sandberg said, and chalked it up to the social network's extremely rapid growth. It now has more than 300 million active users around the world.

"We are, I promise, doing our best to scale," she continued, reiterating that all data (except for some recent updates, a statement from Facebook said last week), "and our growth means we're sometimes a little bit behind."

Thousands of Facebook users who have been unable to access their accounts for nearly a week and a half now are now seeing their profiles restored--but some data related to recent profile updates may have been lost.

What happened? According to Facebook, the replacement of profiles and login screens with a "down for maintenance" notice--which appears to have started on October 3--stemmed from "a technical issue with a single database." The company has stressed that there is no chance that it was due to hackers or other malicious activity.

Profiles should be restored over the course of the next day, the company estimates.

"Our engineering team has worked around the clock, and as of today, all of these users should begin to regain access to their Facebook accounts," Facebook spokeswoman Brandee Barker said reading from a statement. "We apologize for the inconvenience this may have caused and we are taking additional measures to uphold the reliability users come to expect from Facebook."

Less than 0.05 percent of Facebook's users have been affected by the outage, the company estimated. The social network's last head count, about a month ago, was 300 million active users, so that comes out to be a total of about 150,000 affected users. Not very many but enough to put some of them in a panic over not being able to access a primary mode of communication and (in some cases) business.

Profiles have not been lost or deleted, Facebook has continually said--even though the company has been otherwise tight-lipped about the maintenance issue until this point. When affected users' access is restored, however, some things may be different and very recent updates may be missing. According to a notice that Facebook is displaying to members who may have been affected:

You may not have been able to access your account over the last several days. We're sorry for this inconvenience; an extended technical issue affected a small number of Facebook accounts, including yours. We have done our best to restore your account to its most recent state, but some data and settings may not be current. In order to be cautious, we defaulted some of your privacy settings to their most restrictive settings. You may wish to review your privacy settings and reset them.

Facebook added that "some of (affected members') content may not be up to date: in other words, some minor data loss regarding recent updates to profiles. This, according to Facebook, may include photos that were recently added or deleted, recent updates to friends lists (additions and deletions), and "other content you've added, sent, received, or posted."

As for the company's relative silence about the matter until now, Barker explained in a phone call that the company wanted to nail down the specifics of the outage and figure out the situation, rather than provide details to users that could turn out to be inaccurate.

Many of the complaints pertaining to the outage alleged poor customer service on Facebook's part, and as a sort of olive branch, the company is encouraging feedback pertaining to the specific outage. The alert displayed to affected members whose accounts have been newly restored directs them to a form to report any further details or additional problems.

Whether Facebook will step it up a notch for future unexpected technical problems remains to be seen.

Something is really odd here.

As a reporter covering Facebook, I do get the occasional cranky complaints from members who, for one reason or another, are experiencing errors when they try to access their accounts. But it's never been anything like the past week, with a steady stream of e-mails continuing to come in from Facebook members who say they remain shut out of their accounts--despite assurance from Facebook that profiles have not been deleted and that the company is working on the problem.

"This is now seven days and counting," an e-mail sent on Saturday morning read. "It's beyond ridiculous and extremely frustrating."

"The experience completely reversed the Facebook opinion and experience for me," one reader complained. "I see many people bitch and complain, many more beg and a few threaten. To me, the route to take is fairly obvious. Mark Zuckerberg on his own page invites democratic input from Facebook users in one of his most recent videos. Given that statement especially, I find the way their user base is being treated with respect to their disabled account policy hypocritical at best."

"My account has now been held hostage for a week," another reader wrote. "Some of my friends think that I have deleted (my profile) or even blocked them...None of my friends or family can see my profile or even find it in search. It's as if I simply deleted my account or blocked all of them from seeing it without even a word."

Some users have started threads on Get Satisfaction and Yahoo Answers. A few others have pointed me to blogs and YouTube channels devoted to the subject.

The inaccessible accounts appear to be limited to a very small subset of Facebook's over 300 million active users, which means that it's not a large-scale issue for the health of the site. And Facebook is supported by neither subscription money or taxpayer dollars (though it wouldn't have advertising revenue without its users) so there's an argument to be made that users shouldn't be complaining about something they don't pay for. But that's an argument that many of the people who have come to rely on Facebook as a channel of communication simply don't buy.

Whether the string of complaints is warranted or not, Facebook hasn't disclosed exactly what's caused the "extended maintenance issue," and that's what I find puzzling.

Something has dammed the Twitter river. I bet it was this guy.

(Credit: Creative Commons licensed: flickr.com/photos/sherseydc)

Holy cow. Is nothing on the Internet working these days? Facebook's acknowledged that a number of members have had account maintenance issues, and now Twitter has confirmed that "many" users are experiencing timeline delay problems.

Basically, the lowdown is that you can post tweets, and they'll publish, but that your timeline--the stream of updates from the Twitter accounts you follow--isn't bringing up any new tweets. For me, it looks like this started at around 8:00 a.m. PT.

Twitter, which has been prone to many an outage in its three-year history, says it is investigating the problem and will provide an update shortly.

(Photo by Flickr user sherseydc, licensed under Creative Commons).

There are some things that are nice to wake up to. The smell of bacon, for example. On Thursday morning, however, I woke up to something a little less pleasant: an in-box full of e-mails from Facebook members whose accounts are still inaccessible. Some were more or less on the verge of, well, panic.

Earlier this week, we wrote about Facebook's acknowledgment that some members could not access their accounts for several days, instead receiving a "down for maintenance" error. At the time, a Facebook representative explained that it was a "technical issue with one of our databases" and estimated that it would be resolved within 24 hours. It's unclear how many accounts have been affected.

But a resolution of the problem doesn't seem to have occurred, judging by the e-mails that were still showing up in my in-box well into Thursday morning. I sent another request to Facebook to find out more.

"We are continuing to work on the extended maintenance issue that is restricting some users from accessing their accounts," a statement e-mailed by a Facebook representative explained. "No accounts have been compromised during this process, and access will be restored as soon as possible. We apologize for any inconvenience."

Reader e-mails indicated quite a bit of frustration.

"I lost my job back in March and have been using this site as a networking tool," one reader's e-mail said. "It's frustrating that it's been down for so long."

Some were paranoid that their accounts had been deleted and all their contacts lost. And many of the e-mails cited unresponsiveness on Facebook's part despite multiple customer service complaints. Third-party customer service forum Get Satisfaction was filled with chatter about Facebook login and access problems, including at least one threat of a class-action lawsuit.

"Accounts are still down as of this Thursday morning," another e-mail read. "Facebook has been completely non responsive to its users. My account has been down with site maintenance issues since Saturday. I have sent over 20 requests to FB and joined help user groups looking for answers."

From yet another e-mail: "So far Facebook has been largely unresponsive to my emails, saying that this issue can't be reported as a security issue. It seems absurd to me that Facebook customers have no way to directly contact Facebook regarding problems."

With over 300 million active users around the world, we shouldn't expect Facebook to be able to respond to every inquiry it receives. And Facebook is a free product, so it arguably doesn't have a customer service obligation on par with your cable company or the Web site where you bought your last pair of shoes. But this is still a real problem for the social network, which has become so ingrained in culture and communication that for some people it's replaced the address book, the e-mail client, and the personal Web site. Many of the e-mails I received came from people who say that Facebook is their primary method of communication with far-flung family and friends. Others said it's crucial to how they do business.

Here's something else: Facebook doesn't offer a way for members to export their contact information into an address-book format, something that took center stage when blogger Robert Scoble had his Facebook account temporarily banned after testing a script that would export his contacts' information to Plaxo. Even now that Facebook has launched its Facebook Connect login product, there still is no easy way to access your contacts offline. The current account-access snafu indicates that this is a big void.

At the very least, Facebook could make some kind of mass message available explaining what exactly the problem is and reassuring people that inaccessible accounts have not been permanently deleted (assuming that's the case)--something easier to find and more detailed than the brief statement now posted to its company "fan page." On a more long-term level, this seems like a big red flag that Facebook needs to streamline its customer service operations somehow so that this sort of hysteria can be prevented.

A way to export basic contact information for offline access--phone numbers, e-mail addresses, instant-message screen names--wouldn't be bad either.

This post was updated at 10:41 a.m. PT with comment from Facebook.

No, it's not just you.

Facebook confirmed on Monday afternoon that there have been sitewide problems that saw log-in credentials turned down, status messages eaten up, and other various unpleasant occurrences over the course of the past few days. But the social network, which recently surpassed 300 million active users worldwide, hasn't yet disclosed the source of the problem.

"Some users are experiencing errors across a number of site features," a statement e-mailed to CNET News read. "This includes content occasionally disappearing, difficulty logging in or viewing profiles, and error messages when posting content. We are working to resolve these issues as soon as possible."

Outages at major social media sites have drawn particular attention since a massive distributed denial-of-service attack last month threw Facebook into flux and took down Twitter altogether.

Twitter recently changed its default user avatar--you know, the one that pops up if you haven't uploaded a picture of your own--to a cute little bird icon. Unfortunately, then the service got a little bit overzealous and started chomping up existing users' photos, replacing them with the defaults. Oops!

"Many people in my timeline suddenly have default user icons," Twitter engineer Alex Payne posted to his Twitter account on Thursday. "This is probably not intentional. I have inquired with colleagues."

By the time the end of the day rolled around on Friday, there were still plenty of missing user avatars.

"We're having an issue with disappearing user and background images," Twitter wrote on its status blog on Friday morning. "Those affected will have their custom images replaced by defaults. This is due to a caching error on our side which means that your images are not lost and have not been deleted. They are not displaying correctly and we are working to get them to load properly."

This is the second time that a disappearing-user-avatar incident has occurred on Twitter this year. Back in April, infrastructure problems resulted in many Twitter user icons temporarily disappearing--including CEO Evan Williams'. It's unclear whether the same or a similar error downed avatars this time around.

Twitter's servers were on the fritz again on Tuesday, with members receiving server timeouts and third-party applications unable to access the microblogging service. This appears to have begun around 11:45 a.m. PDT.

Twitter posted an update to its status blog when the servers had been in flux for about 10 minutes: "Responding to site downtime. We're working to recover from a site outage and will update as we learn more."

The service was back up about a half hour later. At 12:17 p.m. PT, Twitter confirmed that it was an attack. "We're back up and analyzing the traffic data to determine the nature of this attack," the company said.

Outages used to be commonplace at Twitter when the small start-up's servers were unable to keep up with the massive amount of data flowing through them. They gradually became less and less frequent. But this one's particularly notable because it happens as Twitter is still reeling from a denial-of-service attack last week that targeted a Georgian activist blogger but ended up knocking Twitter's servers offline for several hours. Other services, like Facebook and LiveJournal, were also affected by the attack.

More updates when we hear them...last updated at 3:57 p.m. PT.

Oh, snap! I'm not even getting a fail whale!

Twitter was inaccessible for several hours on Thursday morning, followed by a period of slowness and sporadic time-outs (and more outright downtime). The company is blaming an "ongoing" denial-of-service attack but has not said anything further. Facebook has also confirmed that it was targeted by a DoS attack that rendered some of its features slow or non-functional.

Judging by the timeline of my TweetDeck client, it looks like the problems started right around 6 a.m. PDT.

"We are determining the cause and will provide an update shortly," Twitter's staff posted at 6:43 a.m. PDT on the service's status blog.

Then, around 7:49 a.m. PT, the company posted, "We are defending against a denial-of-service attack and will update status again shortly."

Around 8:15 a.m., the status blog post was updated with "The site is back up, but we are continuing to defend against and recover from this attack." (I still was unable to access Twitter.)

Perfomance monitoring firm AlertSite says that Twitter's home page went down at 6:05 a.m. PT and was showing 40 percent availability at 8:04 a.m. PT, but that timeouts were continuing from most of its monitoring locations at 8:30 a.m.

Way back when, Twitter outages were so commonplace that it was worth reporting when it didn't crash--as when it stayed afloat during the entire South by Southwest Interactive Festival in 2008. Now, a few million dollars of venture capital later, the service is far more stable.

Twitter wants to establish itself as a communications standard rather than just a social-media brand. It's been a crucial platform for information exchange in the face of global events where more traditional means of broadcasting have been inaccessible or blocked.

Problems at Facebook, too
Some features of Facebook were also experiencing uptime issues on Thursday--one reader speculated that log-in servers may have been down--which raises the issue of whether a hosting company problem is to blame. Alternately, a denial-of-service attack could have been targeting both high-profile companies.

Facebook responded later in the morning on Thursday with a statement. "Earlier this morning, we encountered issues within our network that resulted in a short period of degraded site experience for some visitors," the statement read. "No user data was at risk and the matter is now resolved for the majority of users. We're monitoring the situation to ensure that users continue to have the fast and reliable experience they've come to expect from Facebook."

About an hour later, the company revised the statement to confirm that a denial of service attack was involved. "Earlier this morning, Facebook encountered network issues related to an apparent distributed denial of service attack, that resulted in degraded service for some users," the updated statement read. "No user data was at risk and we have restored full access to the site for most users. We're continuing to monitor the situation to ensure that users have the fast and reliable experience they've come to expect from Facebook."

But the Facebook outages were not on the same scale as Twitter's by any means, said Ben Rushlo, a senior consulting manager at performance firm Keynote. "There's been a few slow data points but you couldn't even put them in the same sort of stratosphere of comparison," Rushlo told CNET News.

Publishing site LiveJournal also appears to have been affected by attacks on Thursday.

Botnets, bot herders, and DDoS attacks
DDoS (distributed denial-of-service) attacks typically come from a collection of compromised computers called a botnet, said Graham Cluley, a senior technology consultant at Internet security firm Sophos. The botnet computers can inundate a Web site's servers with communication requests, legitimate or malformed to cause extra trouble.

Botnet-based DDoS attacks are difficult to deal with because it can be hard to distinguish legitimate communications from those that are part of the attack. And just blocking access from the IP addresses of offending computers poses complications: "You don't want to block legitimate users. The computers probably sending (the DDoS) traffic to Twitter belong to legitimate people," Cluley said.

DDoS attacks can be motivated by people seeking ransom money or seeking to make a political statement, but Cluley suspected that's not the case in this particular attack. "My guess is this is most likely some kid in a back bedroom who has access to a large botnet and is showing off to his friends what he can do," Cluley said.

Twitter is unusual in that much of its use comes not through its Web site but through an application programming interface (API) that lets software such as TweetDeck interact with the service. API access also suffered during the outage.

"Often there is collateral damage" during a denial-of-service attack, Cluley said. "Other servers can begin to fall over."

There have been a notable number of DoS attacks recently in the social-media space: On Wednesday, URL shortener Trim claims that one such attack rendered its truncated URLs inaccessible for some time; earlier in the week, blog network Gawker Media was downed by an attack that targeted The Consumerist, a property that it recently sold but still hosts on its servers.

Denial-of-service attacks are actually waning these days as bot herders rent their botnets to those who want to use them to send spam or host malicious software that can be used to compromise other computers, said John Harrison, group product manager of security response at security software company Symantec.

"Organized crime and other groups have gone off to other things. It's more lucrative for them to use the Internet, not to take the Internet away," Harrison said. Using a botnet in a denial-of-service attack can reveal computers to be part of a botnet, for example when an administrator notices high network traffic from a compromised machine, so keeping a low profile can save the botnet for use another day.

To keep a PC from becoming part of a botnet, Harrison recommended keeping the operating system, browser, browser plug-ins such as Adobe Systems Flash and Reader, and other software up to date, and naturally to install antivirus software. "All it takes is one vulnerability to potentially have malware installed," he said.

A massive series of DoS attacks hit the Web a decade ago, long before either Facebook or Twitter was remotely close to existence. They hit the likes of CNN.com, Amazon, E*Trade, eBay, and Buy.com, and were such a serious problem that the FBI held a series of press conferences to address concerns.

There has been no indication that a single party, or groups of hackers in tandem, was responsible for the Facebook and Twitter attacks, or whether there was any connection to the other DoS attacks on smaller sites earlier this week. But it's probably not a coincidence that they all happen to coincide with the annual Defcon hacker convention.

One security expert thinks he may have found a connection. "Today's outage is happening at the same time a new version of the Koobface malware has been found in the wild that is using both Twitter and Facebook messages to send invitations that are designed to lure potential victims to fake AV web pages," an e-mailed statement from Paul Henry, a security analyst at the firm Lumension, explained. "The speculation is that the onslaught of bogus messages that are directing users to malicious pages may in fact be overwhelming Twitter."

More to come when we hear it. Last updated at 12:10 p.m. PT.

CNET News' Stephen Shankland contributed to this report.