Beyond Binary

T-Mobile said on Wednesday that a software glitch was to blame for a massive outage on Tuesday that left many customers unable to send or receive calls or text messages.

"After investigating the cause, we have determined that a back-end system software error had generated abnormal congestion on the network," T-Mobile said in a statement. "T-Mobile has since implemented additional measures to help prevent this from happening in the future."

The wireless service provider did not say which software caused the issue.

"We again apologize to those customers who were affected and may have been inconvenienced," T-Mobile said. I've also asked the carrier what, if any, compensation it plans to give those who were without service.

The service disruption began on Tuesday afternoon and lasted, for some, until late into the evening Pacific Time.

T-Mobile has stated that the outage affected about 5 percent of its users.

T-Mobile customers are still seething after a major outage on Tuesday that left many people across the country unable to text and call their friends and business associates.

The outage, which started Tuesday afternoon and lasted through the evening before being resolved, affected a wide swath of users, though T-Mobile said that only 5 percent of customers were affected. T-Mobile has yet to say what caused the problems.

"Our sole focus during the service disruption on Tuesday was to quickly restore normal service to affected customers," T-Mobile said in a statement on Wednesday. "We are now working to determine the root cause and facts surrounding the interruption."

After CNET News reported on the outage and asked readers to share their experience, dozens of e-mails poured in.

From a husband unable to connect with his pregnant wife, to small-business owners unable to reach clients, to people getting grief for seemingly ignoring text messages from their significant others, people wrote in with their grievances.

Electrician Casey French, of Flower Mound, Texas, said that the outage is a major issue for his business, given that he can't afford to sit in an office with a landline phone.

"This is a catastrophic blow to businesses like mine, losing a day or more of production means losing not only money, but potential new customers, which in this economy are extremely hard to come by already," French said.

T-Mobile, which confirmed the outage Tuesday afternoon, released an updated statement around 5 p.m. PST, saying that "some T-Mobile customers may be experiencing intermittent service disruptions impacting voice and some data services."

However, plenty of folks e-mailed me to say they were having more than intermittent problems.

Around 6:15 p.m., the company said it was making "good progress restoring voice and messaging service to affected customers." The company added that, "at this time, approximately 5 percent of T-Mobile customers are experiencing service disruptions."

From that point, though, another 50 people e-mailed me to say they were still having problems with their service--many saying that they had multiple phones that weren't working as well as friends who were also having problems of one variety or another.

Some questioned T-Mobile's 5 percent estimate, saying that nearly everyone they knew with T-Mobile was experiencing some sort of outage. (As of the second quarter, T-Mobile had 33.5 million subscribers, meaning that even if 5 percent of users were affected, that would still be more than 1.5 million people.)

John Bystrom, of Elk Grove Village, Ill., said he also doubted the 5 percent figure, given the number of people who packed a local store he stopped in to inquire about the outage. Bystrom said he had just switched from AT&T to T-Mobile to get the BlackBerry 8900, but now hopes to switch back to AT&T.

"Hopefully I can get out without being charged the fee since T-Mobile in my opinion has broken the contract first by not delivering a stable system," Bystrom said.

At 10:30 p.m., T-Mobile e-mailed another statement, to say that things had been resolved.

"T-Mobile confirms it has fully restored voice and text/picture messaging services for customers affected by intermittent service disruptions on Tuesday," the company said. "About five percent of our customers across various geographies were affected for much of Tuesday evening, and by late Tuesday PST their service was restored... We sincerely apologize for the inconvenience that this has caused our customers."

Some took issue with the way T-Mobile handled the outage.

"My frustration with T-Mobile is not that they had an outage, but the way they buried their head in the sand," said Carlos Ovalle, an architect in Long Beach, Calif. "They have just about everyone's email and could have notified us of the issue. Had that been the case I could have immediately notified customers that rely on being able to reach me at a moment's notice."

Clarence Barnes, a TV and radio host in Los Angeles, said he also objected to how T-Mobile managed the issue. "The problem for me was that if you called my phone, it would say 'The number you dialed is no longer in service'," said Barnes who is looking for full-time work after the radio station where he worked switched formats. "If you get that message it generally means that the person no longer has that number or simply didn't pay the bill--either answer doesn't make me look that responsible."

Tuesday's outage is the latest blow for T-Mobile, which is still working through a month-long ordeal for its Sidekick service, in which some customers have lost their address books and many more are still waiting to get back other data, such as calendars, to-do lists, and photos.

Of course, T-Mobile customers are not the only ones with cell phone issues. AT&T customers regularly complain about service problems with their iPhones. An outage last year interrupted service for BlackBerry customers on various networks across North America. Earlier this year a cut fiber line left many AT&T customers in Silicon Valley without service.

On Wednesday, T-Mobile customers by and large had their service back, but many were still looking for answers. Bystrom said that several hours after calling customer care he got a call back offering a $5 credit. "When I protested that it was unacceptable (I) was pretty much told take it or leave it," Bystrom said.

T-Mobile Sidekick LX

(Credit: CNET)

Microsoft has begun the process of restoring data to Sidekick owners who have been without it since a massive outage began at the beginning of the month, the software giant said Tuesday.

In a statement, Microsoft said it has posted a tool to T-Mobile's Web site that allows Sidekick owners to restore their address book.

Although it initially feared that much data might be lost, Microsoft said last week that it expected to be able to recover most, if not all, of the information. However, the company also said that the process of bringing back the data will go beyond this week.

Here is the full statement:

The Danger/Microsoft team continues to work around the clock and has completed its latest round of rigorous tests. We are now ready to make the first phase of the content restoration process available to you, starting with personal contacts.

This data restoration effort is only necessary for the minority of customers who lost data from their Sidekick devices.

Beginning today, log into the My T-Mobile website, where there will be a recovery tool to restore contacts you may have lost during the recent service outage. This tool will enable you to view the contacts you had on your device as of October 1. With a few clicks and a confirmation, you will be able to restore these contacts to your Sidekick. If you have recreated some of the same contacts on your Sidekick since October 1, you can choose to keep both sets of contacts, merge them, or just keep the set of contacts now on your device. You may also edit any partial or complete duplicates on your Sidekick after restoration.

We continue to work 24 hours a day, 7 days a week to restore your data. We're making solid progress on the next phase in this restoration process, including your photographs, notes, to-do lists, marketplace data and high scores.

We appreciate your ongoing patience.

Microsoft on Monday apologized for the length of time it is taking to restore missing data to T-Mobile Sidekicks. The company said it expects to begin restoring data this week, but added that bringing back all data will take longer than that.

T-Mobile Sidekick LX

(Credit: CNET)

In a note on its Web site, Microsoft said that the reason for the delay is that the company wants to make sure that it doesn't risk messing up data as it restores information to users' phones.

"The Danger/Microsoft team is continuing to work around the clock on the data restoration proces," Microsoft said. "We apologize that this is taking so long, but we want to make sure we are doing everything possible to maintain the integrity of your data."

A significant number of Sidekick owners have been without their data since the beginning of the month, when Sidekick data service became interrupted amid a massive outage. At one point, Microsoft feared much of the data was lost, but the company said early last week that things were looking better and later added that it expected to be able to bring back most, if not all, of the data.

"We continue to make steady progress, and we hope to be able to begin restoring personal contacts for affected users this week, with the remainder of the content (photographs, notes, to-do-lists, marketplace data, and high scores) shortly thereafter," Microsoft said.

Although Microsoft has said it expects to recover most if not all lost data for Sidekick users, T-Mobile said on Thursday that new sales of the product remain on hold.

The carrier declined to say when--or if--Sidekick sales would resume.

(Credit: CNET)

"We're pleased that Microsoft/Danger is continuing to make progress," T-Mobile vice president David Beigie said in a statement to CNET News. "T-Mobile's sole focus remains on helping Sidekick customers recover from this disruption."

Service for the device has been impaired for going on two weeks. Last Saturday, Microsoft, whose Danger subsidiary powers the Sidekick service, said it believed that any data that was not on users' devices might be permanently lost.

However, by Monday, Microsoft said it was more optimistic and early on Thursday Microsoft issued a letter to customers saying it expected to recover most, if not all, user data.

All models of the Sidekick have remained listed as "temporarily out of stock" on T-Mobile's Web site.

The T-Mobile Sidekick LX.

(Credit: CNET)

When the T-Mobile Sidekick started having data problems two weeks ago, it was a big pain for nearly all of the company's roughly 800,000 subscribers. But it became particularly rough for the many deaf people who have been among the Sidekick's most loyal fan base.

"It was severely limiting and made me very dependent on others," Carla McCraw, a Web designer from San Antonio, Texas, said in an e-mail interview. "This loss of independence was very stressful on me."

McCraw said she relies on her Sidekick to manage everyday tasks, from keeping up with closures at her son's day care to whether soccer practice will be rained out.

"It was extremely frustrating, not being able to know, and my mother had to call the soccer coach and day care, making me feel extremely inadequate," she said. "I had to rely on the Internet through (alternative services such as) Yahoo and Facebook to let my friends know my Sidekick was down."

The good news, for both deaf and hearing users, is that Microsoft and T-Mobile appear to be making headway at restoring much of the data that, at one time, appeared gone permanently.

But even an outage can be a big deal for those that use the Sidekick as something of a communications lifeline.

Lisa Gault, a deaf Sidekick owner in Katy, Texas.

(Credit: Lisa Gault)

"As the outage went on, I became concerned about how my deaf teenage son would be able to communicate in an emergency," Jamie Berke, an About.com guide based in the Washington metro area, said in an e-mail.

"I know he is not the only one," said Berke, who is also deaf. "The outage probably meant that thousands of deaf children who depend on their Sidekicks to communicate with parents were unable to communicate in the event of an emergency. Plus, I myself, would have been unable to communicate in an emergency."

The Sidekick became an early favorite in deaf circles because of its good keyboard, then-state-of-the-art instant-messaging abilities, and ability to connect to relay services. T-Mobile improved things further by adding a data-only option so that deaf users weren't paying for voice minutes they didn't use.

Because the Sidekick didn't evolve as fast as other smartphones, many users--deaf and hearing alike--moved to the BlackBerry and other devices. And after their recent experiences, many of the remaining Sidekick users in the deaf community have said they are considering jumping ship, once their T-Mobile contracts end.

"I am going to look into Sprint's BlackBerry, which more and more deaf people are taking up, and have complimented, and (said) that it is a lot more reliable," McCraw said.

Lisa Gault, a deaf Sidekick owner in Katy, Texas, said she in an e-mail interview that she relies on the Sidekick as a means to stay in touch with her family.

"It's a way for the school to get a hold of me, if something were to occur with my son who is (not deaf)," Gault said.

Gault said that even short of an emergency, it is a problem not to get her e-mail for an extended period.

"It's annoying, as my friends think I'm ignoring them, when in reality, I didn't get the e-mails yet," Gault said. "It really put the deaf community at more of a disadvantage--more so than for hearing people, since we're so reliant on e-mail (devices) to keep in touch."

Microsoft said Thursday that it believes it has recovered most of the Sidekick data that it initially feared might have been permanently lost.

"We are pleased to report that we have recovered most, if not all, customer data for those Sidekick customers whose data was affected by the recent outage," corporate vice president Roz Ho said in a letter to customers."We plan to begin restoring users' personal data as soon as possible, starting with personal contacts, after we have validated the data and our restoration plan. We will then continue to work around the clock to restore data to all affected users, including calendar, notes, tasks, photographs and high scores, as quickly as possible."

On Wednesday, at least two lawsuits were filed against Microsoft and T-Mobile over the Sidekick outages, which began at the beginning of the month with data service interruptions and have left many users without access to their address books.

Microsoft had said on Saturday that it believed all data that was not on users' phones was probably permanently lost. However, by Monday it was sounding more optimisitic it could recover the data.

Here's the full text of the letter:

Dear T-Mobile Sidekick customers,

On behalf of Microsoft, I want to apologize for the recent problems with the Sidekick service and give you an update on the steps we have taken to resolve these problems.

We are pleased to report that we have recovered most, if not all, customer data for those Sidekick customers whose data was affected by the recent outage. We plan to begin restoring users' personal data as soon as possible, starting with personal contacts, after we have validated the data and our restoration plan. We will then continue to work around the clock to restore data to all affected users, including calendar, notes, tasks, photographs and high scores, as quickly as possible.

We now believe that data loss affected a minority of Sidekick users. If your Sidekick account was among those affected, please continue to log into the T-Mobile Sidekick forum at http://www.t-mobile.com/sidekick for the latest updates about when data restoration will begin, and any steps you may need to take. We will work with T-Mobile to post the next update on data restoration timing no later than Saturday.

We have determined that the outage was caused by a system failure that created data loss in the core database and the back-up. We rebuilt the system component by component, recovering data along the way. This careful process has taken a significant amount of time, but was necessary to preserve the integrity of the data. We will continue working closely with T-Mobile to restore user data as quickly as possible. We are eager to deliver the level of reliable service that our incredibly loyal customers have become accustomed to, and we are taking immediate steps to help ensure this does not happen again. Specifically, we have made changes to improve the overall stability of the Sidekick Service and initiated a more resilient backup process to ensure that the integrity of our database backups is maintained.

Once again, we apologize for this situation and the inconvenience that it has created. Please know that we are working all-out to resolve this situation and restore the reliability of the service.

Sincerely,

Roz Ho

Corporate Vice President

Premium Mobile Experiences, Microsoft Corporation

Update 10:10 a.m.: Amended letter to include Microsoft's final wording of letter, which said it believes it has recovered "most, if not all" data.

It was only a matter of time, but the T-Mobile Sidekick issue has now spilled over to the courthouse.

A number of lawsuits have been filed, including two filed in federal court in Northern California on Wednesday that allege both negligence and false claims on the part of Microsoft and T-Mobile.

T-Mobile Sidekick LX

(Credit: CNET)

The suits come amid a massive outage of the service that powers the Sidekick, which has hampered data service since early this month and left many users without access to their calendars, address books, and other information. At one point, Microsoft and T-Mobile indicated that any data not yet recovered was probably lost permanently, however the companies said Monday that they were more optimistic about being able to bring back users' information.

One suit, filed on behalf of a Bakersfield, Calif., man "and all others similarly situated" charges that, among other things, Microsoft and Danger failed to use reasonable care in handling Sidekick owners data and that the Sidekick was falsely advertised. That suit seeks monetary damages as well as an order requiring the companies to fix the Sidekicks and service or offer a full refund.

"T-Mobile and its service providers ought to have been more careful the use of backup technology and policies to prevent such data loss" said Ira P. Rothken, an attorney working on that case. "We are hopeful that T-Mobile and the rest of the defendants will do the right thing, use this as an opportunity to redesign the system as a new standard for cloud computing storage, and provide full compensation for the data loss."

Another suit, filed on behalf of Maureen Thompson "and all others similarly situated" seeks unspecified damages for Thompson and others who have lost data as a result of the recent Sidekick problems.

According to her lawyer, Thompson owns a Sidekick used primarily by her daughter, an aspiring model, singer, and songwriter who used her Sidekick to store personal and business contacts, appointments, and even irreplaceable song lyrics not stored anywhere else. The lawyer said that Thompson bought the device "primarily because T-Mobile promised that any data would be protected and available no matter what happened to the phone."

"T-Mobile's initial efforts to reimburse Sidekick users are a step in the right direction, but fail to sufficiently compensate Sidekick users for this disastrous loss of data," Thompson attorney Jay Edelson said in a statement. "T-Mobile and Microsoft promised to safeguard the most important data their customers possess and then apparently failed to follow even the most basic data protection principles. What they did is unthinkable in this day and age."

In that lawsuit, Thompson's lawyers argue why the outage of the Sidekick was particularly devastating, noting the device's cloud-based architecture in which the primary copy of the data is stored, not on the devices, but on servers operated by Microsoft's Danger unit.

"Further complicating the data loss is the fact that Sidekicks, unlike iPhones, BlackBerrys and other smartphones, are not designed to sync locally with a user's personal computer without additional software and hardware," the suit states. "This means that most users were not able to backup their data locally, but were encouraged and required to rely on Microsoft/Danger."

Microsoft declined to comment on the lawsuit, but, a representative said on Wednesday that the company is "obviously very sorry for the inconvenience that this situation has caused Sidekick users, and we are working around the clock in an effort to recover and restore the data for any affected users. While it is still too early to say for sure, we announced on Monday that our engineering teams were increasingly optimistic."

For its part, T-Mobile said in a statement that it "does not comment on pending litigation."

"We are focused on helping our Sidekick customers recover from this recent service disruption and are continuing to support Microsoft's ongoing efforts to address and resolve the Danger platform issues," the company said.

T-Mobile has halted sales of the Sidekick amid the ongoing issues.

T-Mobile said late on Monday that it may yet be able to recover Sidekick users' information that it had previously thought was lost as part of a massive server failure by Microsoft's Danger subsidiary.

"Recent efforts indicate the prospects of recovering some lost content may now be possible," it said.

(Credit: CNET)

Those who do suffer permanent data loss will get a $100 "customer appreciation card" good toward T-Mobile service or products, the carrier said in a statement.

"For those who fall into this category, details will be sent out in the next 14 days - there is no action needed on the part of these customers," T-Mobile said. "We however remain hopeful that for the majority of our customers, personal content can be recovered."

That marks a significant change in tone. On Saturday, the carrier and Microsoft had warned that any data not on a customer's phone at that point was likely gone forever.

Monday's business day came and went with little public comment from the companies, but apparently efforts to restore data were more fruitful.

T-Mobile did halt sales of the Sidekick as it investigated the issue.

T-Mobile continued to urge customers not to remove the battery on the device, reset the Sidekick or let it run out of power while the company works to restore its servers.