Wireless

A massive outage left many T-Mobile USA customers across the country unable to make calls or get data on their cellular phones on Tuesday.

In a statement, T-Mobile said it is working to get its service back up and running.

"T-Mobile customers may be experiencing service disruptions impacting voice and data," the company said in a statement. "Our rapid response teams have been mobilized to restore service as quickly as possible. We will provide updates as more information is available."

The move comes as the carrier is still working to restore data to T-Mobile Sidekick users, many of whom have been without some or all of their information for an entire month.

In a statement on Friday, T-Mobile said many of its Sidekick customers have been able to restore at least their contacts, although it acknowledged not everyone got their address books back.

"There was a very small group of customers for whom we were unable to recover any contacts," the carrier said. It is still working to bring back other data, including calendars, to-do lists, notes, pictures and other data. "We will share more details around this restoration process very soon," it said.

Microsoft, whose Danger service powers the Sidekick, initially thought all data might be lost, but later said it expected to be able to restore most, if not all user data.

Update 5:04 p.m. PT: In a slightly updated statement, T-Mobile said "Some T-Mobile customers may be experiencing intermittent service disruptions impacting voice and some data services."

Reports from users seem to vary. In San Francisco, I was unable to make or receive calls or get e-mail. Meanwhile, a few users in Boston and Washington, D.C. told me that their T-Mobile phones appeared to be working.

If other people want to drop me a note of where they are and what's working or not, I'll get a better sense of things.

5:20 p.m. PT: T-Mobile subscribers in eastern Tennessee, Honolulu, Long Island, N.Y. all e-mailed me to say they, too, were experiencing outages.

Gretchen Wuersching, an accountant who lives in Mount Vernon, N.Y. and works in Manhattan said she has been without service in New York City and Westchester County. Wuersching said she first noticed her service out in Manhattan and was still out in both Westchester County and later when she drove to the Bronx.

Meanwhile, a T-Mobile subscriber in Orange County, Calif. said service was intermittent, while another customer in Denver said her BlackBerry could make calls and handle e-mail, as could friends on T-Mobile's network there, but that text messages were not working.

Several people in the Salt Lake City area said they were having problems, while another in Utah said that calls and text messages were not working, but that Web and e-mail access was continuing.

All in all, it sounds like a very mixed bag for T-Mobile subscribers. Thanks to everyone who wrote in and I'll let you know when I hear more from T-Mobile.

Update 6:24 p.m. PT: T-Mobile issued an updated statement saying it now believes only 5 percent of customers are experiencing outages.

"We're making good progress restoring voice and messaging service to affected customers," a T-mobile representative said. "At this time, approximately 5 percent of T-Mobile customers are experiencing service disruptions. Issues began at approximately 5:30 p.m. Eastern time. Our rapid response team is working continuously to fully resolve this disruption. We sincerely apologize for the inconvenience that this has caused our customers."

The e-mails are literally flooding into my in-box from around the country and far more people are telling me they are experiencing outages of one variety or another than say they have full service. But perhaps that's the nature of who e-mails.

Among the dozens of e-mails of problems I have gotten was this message from Karen Jacobson, an analyst and programmer in North Carolina. "My daughter called me from Chicago to say that her text messaging doesn't work and asked me to call T-Mobile," Jacobson said. "I called them and was told that the network was busy. I then tried texting ... and was unable to text from down here. My friend Crystal called from Florida, on her landline, to tell me that her T-Mobile has not been able to make or accept phone calls for at least an hour (and now her boyfriend is angry cause he thinks she ignored him!)."

For my part, I have data access on my Blackberry Curve, but no ability to make calls. E-mail service, which was out for a time, seems to be back.

Update 10:50 p.m. PT: T-Mobile issued an updated statement saying it has restored voice and text/picture messaging services to customers who experienced outages and apologizing for the inconvenience.

Microsoft on Monday apologized for the length of time it is taking to restore missing data to T-Mobile Sidekicks. The company said it expects to begin restoring data this week, but added that bringing back all data will take longer than that.

T-Mobile Sidekick LX

(Credit: CNET)

In a note on its Web site, Microsoft said that the reason for the delay is that the company wants to make sure that it doesn't risk messing up data as it restores information to users' phones.

"The Danger/Microsoft team is continuing to work around the clock on the data restoration proces," Microsoft said. "We apologize that this is taking so long, but we want to make sure we are doing everything possible to maintain the integrity of your data."

A significant number of Sidekick owners have been without their data since the beginning of the month, when Sidekick data service became interrupted amid a massive outage. At one point, Microsoft feared much of the data was lost, but the company said early last week that things were looking better and later added that it expected to be able to bring back most, if not all, of the data.

"We continue to make steady progress, and we hope to be able to begin restoring personal contacts for affected users this week, with the remainder of the content (photographs, notes, to-do-lists, marketplace data, and high scores) shortly thereafter," Microsoft said.

Although Microsoft has said it expects to recover most if not all lost data for Sidekick users, T-Mobile said on Thursday that new sales of the product remain on hold.

The carrier declined to say when--or if--Sidekick sales would resume.

(Credit: CNET)

"We're pleased that Microsoft/Danger is continuing to make progress," T-Mobile vice president David Beigie said in a statement to CNET News. "T-Mobile's sole focus remains on helping Sidekick customers recover from this disruption."

Service for the device has been impaired for going on two weeks. Last Saturday, Microsoft, whose Danger subsidiary powers the Sidekick service, said it believed that any data that was not on users' devices might be permanently lost.

However, by Monday, Microsoft said it was more optimistic and early on Thursday Microsoft issued a letter to customers saying it expected to recover most, if not all, user data.

All models of the Sidekick have remained listed as "temporarily out of stock" on T-Mobile's Web site.

The T-Mobile Sidekick LX.

(Credit: CNET)

When the T-Mobile Sidekick started having data problems two weeks ago, it was a big pain for nearly all of the company's roughly 800,000 subscribers. But it became particularly rough for the many deaf people who have been among the Sidekick's most loyal fan base.

"It was severely limiting and made me very dependent on others," Carla McCraw, a Web designer from San Antonio, Texas, said in an e-mail interview. "This loss of independence was very stressful on me."

McCraw said she relies on her Sidekick to manage everyday tasks, from keeping up with closures at her son's day care to whether soccer practice will be rained out.

"It was extremely frustrating, not being able to know, and my mother had to call the soccer coach and day care, making me feel extremely inadequate," she said. "I had to rely on the Internet through (alternative services such as) Yahoo and Facebook to let my friends know my Sidekick was down."

The good news, for both deaf and hearing users, is that Microsoft and T-Mobile appear to be making headway at restoring much of the data that, at one time, appeared gone permanently.

But even an outage can be a big deal for those that use the Sidekick as something of a communications lifeline.

Lisa Gault, a deaf Sidekick owner in Katy, Texas.

(Credit: Lisa Gault)

"As the outage went on, I became concerned about how my deaf teenage son would be able to communicate in an emergency," Jamie Berke, an About.com guide based in the Washington metro area, said in an e-mail.

"I know he is not the only one," said Berke, who is also deaf. "The outage probably meant that thousands of deaf children who depend on their Sidekicks to communicate with parents were unable to communicate in the event of an emergency. Plus, I myself, would have been unable to communicate in an emergency."

The Sidekick became an early favorite in deaf circles because of its good keyboard, then-state-of-the-art instant-messaging abilities, and ability to connect to relay services. T-Mobile improved things further by adding a data-only option so that deaf users weren't paying for voice minutes they didn't use.

Because the Sidekick didn't evolve as fast as other smartphones, many users--deaf and hearing alike--moved to the BlackBerry and other devices. And after their recent experiences, many of the remaining Sidekick users in the deaf community have said they are considering jumping ship, once their T-Mobile contracts end.

"I am going to look into Sprint's BlackBerry, which more and more deaf people are taking up, and have complimented, and (said) that it is a lot more reliable," McCraw said.

Lisa Gault, a deaf Sidekick owner in Katy, Texas, said she in an e-mail interview that she relies on the Sidekick as a means to stay in touch with her family.

"It's a way for the school to get a hold of me, if something were to occur with my son who is (not deaf)," Gault said.

Gault said that even short of an emergency, it is a problem not to get her e-mail for an extended period.

"It's annoying, as my friends think I'm ignoring them, when in reality, I didn't get the e-mails yet," Gault said. "It really put the deaf community at more of a disadvantage--more so than for hearing people, since we're so reliant on e-mail (devices) to keep in touch."

It was only a matter of time, but the T-Mobile Sidekick issue has now spilled over to the courthouse.

A number of lawsuits have been filed, including two filed in federal court in Northern California on Wednesday that allege both negligence and false claims on the part of Microsoft and T-Mobile.

T-Mobile Sidekick LX

(Credit: CNET)

The suits come amid a massive outage of the service that powers the Sidekick, which has hampered data service since early this month and left many users without access to their calendars, address books, and other information. At one point, Microsoft and T-Mobile indicated that any data not yet recovered was probably lost permanently, however the companies said Monday that they were more optimistic about being able to bring back users' information.

One suit, filed on behalf of a Bakersfield, Calif., man "and all others similarly situated" charges that, among other things, Microsoft and Danger failed to use reasonable care in handling Sidekick owners data and that the Sidekick was falsely advertised. That suit seeks monetary damages as well as an order requiring the companies to fix the Sidekicks and service or offer a full refund.

"T-Mobile and its service providers ought to have been more careful the use of backup technology and policies to prevent such data loss" said Ira P. Rothken, an attorney working on that case. "We are hopeful that T-Mobile and the rest of the defendants will do the right thing, use this as an opportunity to redesign the system as a new standard for cloud computing storage, and provide full compensation for the data loss."

Another suit, filed on behalf of Maureen Thompson "and all others similarly situated" seeks unspecified damages for Thompson and others who have lost data as a result of the recent Sidekick problems.

According to her lawyer, Thompson owns a Sidekick used primarily by her daughter, an aspiring model, singer, and songwriter who used her Sidekick to store personal and business contacts, appointments, and even irreplaceable song lyrics not stored anywhere else. The lawyer said that Thompson bought the device "primarily because T-Mobile promised that any data would be protected and available no matter what happened to the phone."

"T-Mobile's initial efforts to reimburse Sidekick users are a step in the right direction, but fail to sufficiently compensate Sidekick users for this disastrous loss of data," Thompson attorney Jay Edelson said in a statement. "T-Mobile and Microsoft promised to safeguard the most important data their customers possess and then apparently failed to follow even the most basic data protection principles. What they did is unthinkable in this day and age."

In that lawsuit, Thompson's lawyers argue why the outage of the Sidekick was particularly devastating, noting the device's cloud-based architecture in which the primary copy of the data is stored, not on the devices, but on servers operated by Microsoft's Danger unit.

"Further complicating the data loss is the fact that Sidekicks, unlike iPhones, BlackBerrys and other smartphones, are not designed to sync locally with a user's personal computer without additional software and hardware," the suit states. "This means that most users were not able to backup their data locally, but were encouraged and required to rely on Microsoft/Danger."

Microsoft declined to comment on the lawsuit, but, a representative said on Wednesday that the company is "obviously very sorry for the inconvenience that this situation has caused Sidekick users, and we are working around the clock in an effort to recover and restore the data for any affected users. While it is still too early to say for sure, we announced on Monday that our engineering teams were increasingly optimistic."

For its part, T-Mobile said in a statement that it "does not comment on pending litigation."

"We are focused on helping our Sidekick customers recover from this recent service disruption and are continuing to support Microsoft's ongoing efforts to address and resolve the Danger platform issues," the company said.

T-Mobile has halted sales of the Sidekick amid the ongoing issues.

So, just what the heck happened?

That's one of many questions that Sidekick owners and the broader tech community are asking after one of the largest data failures in recent memory.

Two days after warning customers that their contacts, calendar, and other information may be gone, Microsoft and T-Mobile spent most of Monday in silent mode as they continued to work to try to recover the data from thousands of Sidekick owners.

(Credit: CNET)

Microsoft has said that the hardware failure that caused the problem took out both the primary and backup copies of the database that contained Sidekick users' information. But the question remains, why wasn't there a true independent backup of the data?

T-Mobile has said that it is exploring what to do to try to compensate customers who have lost their data, but as of 4:30 p.m. PDT on Monday had not offered a promised update on where things stood.

For those who don't have their data, there was little to do but vent on various forums and hope that the data recovery efforts bear fruit. (There may be some hope on that front, as some users did report some data re-appearing on their devices on Monday).

Still, those who do have some or all of their information on their device, might want to back that up pronto. Enthusiast site Hiptop3.com and T-Mobile itself have offered up a few ways to back up contact information, in particular.

And, because it bears repeating, T-Mobile is warning those who do have information on their device not to reset their Sidekick, take out the battery, or let the device fully run out of power.

(Credit: Screenshot by Ina Fried/CNET)

Wireless carrier T-Mobile USA has, at least temporarily, stopped selling all models of the Sidekick in the wake of a massive hardware failure that resulted in many customers losing their e-mail, contacts, and other data.

As of Sunday, all models of the Sidekick were listed as "temporarily out of stock" on T-Mobile's Web site. T-Mobile retail store workers also said on Monday that they have been instructed to halt new sales of the device as the company continues to investigate the recent problems that have plagued the handheld.

To recap, Sidekick customers started experiencing problems connecting to the data network more than a week ago. Microsoft, whose Danger subsidiary powers the Sidekick service, said it was investigating the problems.

On Saturday, Microsoft and T-Mobile posted an updated notice saying all data that was not currently on customers devices was likely lost permanently.

Microsoft and T-Mobile have not said how many of the roughly 800,000 Sidekick customers have lost data. Microsoft said a server failure impacted the main and back-up databases. One theory is that the problems cropped up as Hitachi was doing work on the storage network that manages the Sidekick data.

T-Mobile has promised an update for customers sometime Monday. For now, the carrier has advised customers not to reset their devices, remove the battery, or let them run out of power, as doing so could result in losing whatever data they do have.

Microsoft acquired Danger last year, saying it hoped to use its service architecture more broadly in its mobile strategy. The software maker has been working on a project code-named Pink that was to be essentially the future of the Sidekick. The company had not planned for any more versions of the current Java OS-based Sidekick.

Update, 12:30 p.m. PT: T-Mobile confirmed that "Sidekick sales are temporarily on hold." A company representative told CNET News in an e-mail that the company doesn't have an exact number of customers who lost data but that "we believe it is a minority of customers."

Although there are reports that customers are being let out of their wireless contracts and being offered discounts on other T-Mobile phones, the company is officially offering only one month of Sidekick data service. "We are also considering additional measures for those who have lost their content to help reinforce how valuable they are as T-Mobile customers," the representative said.

As for why there weren't better backup mechanisms in place, T-Mobile referred that question to Microsoft.

Update, 2 p.m. PT: Even though T-Mobile has said it has temporarily halted sales of the Sidekick, retailers in New York were still selling the device Monday. At three different locations on the Upper West Side of Manhattan, sales representatives said the Sidekicks were still available, but they were warning any potential customers that mobile Web and e-mail services might not work until the server problem is resolved. They said the phone and the accompanying text-messaging service have not been affected, so customers buying a new Sidekick would be able to make calls, as well as send and receive SMS messages.

In an e-mail to retail sales managers dated October 10, T-Mobile instructed managers on how to deal with Sidekick customers. The message informed them that some personal information backed up by the Microsoft/Danger servers had been lost. This information included pictures, contacts, e-mails, text messages, calendar entries, and to-do lists.

"Our teams continue to work around the clock in hopes of discovering a means for a network recovery solution. However, the likelihood of a successful outcome is extremely low," the e-mail stated.

Sales representatives were told to direct customers with questions to T-Mobile Forums on its Web site for details and to get updated information.

As part of its "action steps," sales reps were also directed to tell customers not to reset their devices by removing the battery or letting their battery drain completely.

Sidekick's LX, the newest model

(Credit: CBSInteractive)

After suffering through a weeklong outage, T-Mobile Sidekick users got even worse news on Saturday, when the company advised them that any data not on their phone was likely gone forever due to a hardware failure.

Since then, Sidekick owners have been sharing their stories with me via e-mail. Here are some of those tales. Feel free to add your own in the comments section or e-mail me (ina dot fried at cnet dot com)

Bram Weiser, a sign language interpreter and computer specialist in New York, wrote on Monday:

Since the outage occurred on October 1st, I've not been able to receive (or, I presume, send) email to/from my Sidekick. (All along, though, I've had phone and text message capability.)

On Monday, October 5th, roughly 100 emails seemed to suddenly appear, leading me to think that the outage was FINALLY over, days(!) after it started. However, while I got notification of those new emails (e.g., italicized, boldface sender & subject line information), the actual bodies of those emails was painfully slow in arriving, if they were ever going to do so.

I THOUGHT I saw an advisory at tmobile.com not to try a "hard" reset as an attempt to retrieve data and/or "kickstart" (my word) my Sidekick, so I steered clear of that. So, thinking I saw it written that way (note: I later realized it didn't say "hard," but puzzlingly advised against ANY reset at all), I tried powering my Sidekick on and off, albeit to no avail, and then tried a "soft" reset in good faith by clicking the pinhole on the body of the device. In more "normal" circumstances (read: just about any other time in the years that I've owned a Sidekick and paid for service through T-Mobile), this would usually bring the device back to life with its information intact, because, as we now know, it would download it from the T-Mobile (read: Microsoft/Danger) servers after connecting to the network. Not this time, though...

After doing all of that, and letting my Sidekick power up again, I, as did many thousands of other users, suddenly lost saved emails, device settings in ALL applications, bookmarks for the Web Browser, my entire Address Book (188 or so entries, although three recent entries did reappear later), and all of the nearly 1,000 entries (past, present and future) for my Calendar. To date, all of these remain lost, though I CAN surf the Internet on my own, as well as define settings, add contacts to my Address Book, send/receive instant messages and, I presume, add Calendar entries on my own as well, though I've not tried much of that yet as I await the safe return of my data.

You're absolutely right when you ask how, for instance, a company like Microsoft/Danger (Microsoft!) didn't have sufficient backups of its servers in place, be they daily, weekly or whatever. This is beyond belief in this day and age!

People need to know about this and not give Microsoft a free pass. For us to be without important data that we paid for the privilege of entrusting to Microsoft/Danger for safekeeping only to have this happen 1-1/2 weeks ago, and continuing to this day, to potentially lose it all permanently(!), and (for now at least) to get a credit of only(!) one month's data service (honestly, is that REALLY satisfactory to anyone?!) boggles the mind.

Jeff McGaha, an electrical engineer in Indianapolis wrote that his wife has a Sidekick, was without access to the data network for four to five days, and has now lost all of her contacts:

She's dropping the Sidekick now, something she was on the fence about for a long time. T-Mobile is lucky we're staying with them. They can thank Google and Motorola for that. The Cliq is going to see bigger sales because of this sidekick outage. Long live Android.

The massive data failure at Microsoft's Danger subsidiary threatens to put a dark cloud over the company's broader "software plus services" strategy.

A key tenet of that approach is that businesses and consumers can trust Microsoft to reliably store valuable data on their servers.

T-Mobile Sidekick Slide

(Credit: Corinne Schulze/CNET)

A week ago, though, Microsoft's Danger unit experienced a huge outage that left many T-Mobile Sidekick users without access to their calendar, address book, and other key data. That's because the Sidekick keeps nearly all its data in the cloud as opposed to keeping the primary copy on the devices themselves.

Things got even worse on Saturday, as Microsoft said in a statement that data not recovered thus far may be permanently lost. It's not immediately clear how many people lost their data. The outage earlier in the week affected a broad swath of Sidekick users, though many had data return during the week.

While outages in the cloud computing world are common (one need only look at recent issues with Twitter or Gmail), data losses are another story. And this one stands as one of the more stunning ones in recent memory.

The Danger outage comes just a month before Microsoft is expected to launch its operating system in the cloud--Windows Azure. That announcement is expected at November's Professional Developer Conference. One of the characteristics of Azure is that programs written for it can be run only via Microsoft's data centers and not on a company's own servers.

It should be pointed out that the Azure setup is entirely different from what Danger uses: the Sidekick uses an architecture Microsoft inherited rather than built (Microsoft bought Danger last year). Still, the failure would seem to be enough to give any CIO pause.

Update, 2 p.m. PT, 10/11/2009: I asked Microsoft for comment Saturday when I was writing this, in particular as to how the rest of its cloud might differ from the Danger set up.

Microsoft said Sunday that its the fabric controller that manages the Azure service is built with redundancy in mind.

"We write multiple replicas of user data to multiple devices so that the data is available in a situation where a single or multiple physical nodes may fail," Windows Azure general manager Doug Hauger said in a statement to CNET News.

That doesn't mean Azure is immune from data loss, though I'm told an entire data center would have to be wiped out, as opposed to just a server or collection of servers. I'd be interested to know whether Microsoft will also offer multiple location options so that users that want to can have their data in more than one physical spot as well.

But that's just one of many questions raised by this spectacular failure. Among the other questions still looming large in my head are:

1. What backup procedures did Danger have?

2. Just how many of T-mobile's Sidekick customers lost their data? (Feel free to let me know, Sidekick users.)

3. What impact will this have on the Pink project, which was largely seen as the evolution of the Sidekick, and some say was already in trouble?

4. Will this hurt Microsoft's efforts to build a brand around the notion of Windows Phone even though that uses a different architecture (with its own challenges, to be sure)?