Unanswered questions loom large in Sidekick fiasco

So, just what the heck happened?

That's one of many questions that Sidekick owners and the broader tech community are asking after one of the largest data failures in recent memory.

Two days after warning customers that their contacts, calendar, and other information may be gone, Microsoft and T-Mobile spent most of Monday in silent mode as they continued to work to try to recover the data from thousands of Sidekick owners.

(Credit: CNET)

Microsoft has said that the hardware failure that caused the problem took out both the primary and backup copies of the database that contained Sidekick users' information. But the question remains, why wasn't there a true independent backup of the data?

T-Mobile has said that it is exploring what to do to try to compensate customers who have lost their data, but as of 4:30 p.m. PDT on Monday had not offered a promised update on where things stood.

For those who don't have their data, there was little to do but vent on various forums and hope that the data recovery efforts bear fruit. (There may be some hope on that front, as some users did report some data re-appearing on their devices on Monday).

Still, those who do have some or all of their information on their device, might want to back that up pronto. Enthusiast site Hiptop3.com and T-Mobile itself have offered up a few ways to back up contact information, in particular.

And, because it bears repeating, T-Mobile is warning those who do have information on their device not to reset their Sidekick, take out the battery, or let the device fully run out of power.

For the time being, T-Mobile has halted sales of the Sidekick but the long-term future for the current product line, as well as its planned successor (code-named Pink) remain unclear.

Justine Castro, a student at San Francisco's Academy of Art school, had been having problems with her Sidekick, but hadn't heard about the massive outage or data loss issues. She said she had almost reset her device a couple times--a move that probably would have caused her data to disappear.

(Credit: Ina Fried/CNET)

What is clear is that both T-Mobile and Microsoft are going to have some serious work to do to regain customers' trust.

Despite all the buzz in technology circles, some Sidekick users weren't even aware of the extent of the issue. Justine Castro, a student at San Francisco's Academy of Art, was tapping away on her gadget when I approached her Monday morning.

Castro said that she was having problems for the last week and had thought about resetting the device but thankfully had yet to do so. Castro said her T-Mobile contract is up and she was already thinking about getting a new phone as soon as she gets the money.

"I don't know what I'd do if all my data got erased," said Castro, who is on her second Sidekick after getting her first one about three years ago.

[gggg sssss]

fear the cloud. What part of that do they not understand?

[rcrusoe]

MS/Danger's system, as I understand it, isn't a "cloud" where multiple redundant copies of your data exist on servers scattered around the planet (ala Google Docs). This is just a case of one "big ole server" in one site that someone effed up when they botched a hardware upgrade. Apparently someone skipped SysAdmin 101 where they teach you to have multiple, VERIFIED, backups of your data before you do anything that might put it at risk.

I use Google Apps for all my email, spreadsheet and word processing work, and have full confidence that they will protect my data.

Oh yeah, I also make backups all my Google Apps data - I didn't skip sysadmin 101. :)

[cvaldes1831]

Backing up your data is elementary: SysAdmin 1, not 101. It is not an upper-division concept.

The people who run Danger are a bunch of shoemakers.

[t8]

The Microsoft part.
The Google Cloud is fine.
Better than your own secure network.

[Been_there_Saw_it_before]

You refer to it as SysAdmin 101 or SysAdmin 1, well, I disagree. That implies collage-level numbering of courses. I would hope this is now taught in junior high school. Elementry Computer Operations!

[Michichael]

I give em until Wednesday before they get hit with some kind of class action. Seriously, I would have lost 5 years of e-mails, important files, and other stuff, with the assurance that it's all backed up (and no alternative options of backup) on their servers. Luckily I switched to android. :)

[stever451]

customers dont have to settle for this they are responsible for loosing ppls info and addressbook should be a class action lawsuit in the works, also for ppl that add mins they been nickle and diming ppl for years time to pay up tmobile and i hope the customers and the judeges have no mercy on you serves you and any other cell company a good lessonfor taking advantage of ppl and serving up crap, thank god for tmobile idiocracy they helped the customers out in a victory i hope every sidekick owner or otherwise to find a new company i know i will be and cant wait for the settlement and them begging the govt for help and they will get none, for all the crappy things they did and the crappy outsourced customer service. yaaaay tmobile way to go

[TotallyMadeUpName]

Well stever451, I gave up reading your post after the first two lines. Capital letters and periods have a place. Learn how to use them.

[ittesi259]

Stever....back up your own crap...its not hard to have your contacts written down....or er...wow what a concept synced on your computer.

Crap happens, and there hasn't been a long standing history of problems like this with the sidekick.

My question is why would anyone use a device where things aren't saved on the device itself?

[eekitsericc]

so what caused it to be lost?

[gggg sssss]

stick your email on an Excahneg server. Then teh device is only a copy of the data, not tehdata itself.

[Random_Walk]

Depends on the Exchange setup.

Even though it is based on a crap design idea (a modified JET database? c'mon...) You can peel off copies of that db onto other servers via continuous replication, back it up (lots of third-party apps for that), and for the truly paranoid, peel off .pst files until the cows come home and back those up.

There's plenty of faults with Exchange, but copies and backups aren't among them (though those would go smoother if the frickin' emails stored in mbox format, FFS...)

[groupeone]

It is amazing that Microsoft's Danger division could have such an epic fail. That a company so synonymous with all things IT could make such a fundamental mistake is pretty terrifying. Hardware and software glitches with any system are a given. But the knowledge on how to prepare for a major system update, to have a hot site ready, to have off-site back-ups of data is common knowledge with any IT enterprise, from banks to brokerage houses to insurance carriers. Mission-critical data can never be treated carelessly, and this sad story is a cautionary tale of IT management failure. Not only is this a black eye on one of the world's leading technology companies, it's bound to be the start of a legal nightmare that will make some attorneys quite wealthy.

For some very interesting insight on the mess that is the Windows Mobile/Danger/Pink group within Microsoft, and a deeper look at what went wrong with the Danger epic fail, see:

http://www.appleinsider.com/articles/09/10/12/microsofts_sidekick_pink_problems_blamed_on_dogfooding_and_sabotage.html

[jpmays]

@groupeone

Why is it when something goes awry with Microsoft, or for that matter, any tech company, all of the Apple fanboys had to come out of the woodwork? Quite frankly it's getting rather old and boring... to tell you the truth!

[o2bpitching]

Just because he linked to an AppleInsider article doesn't make him an Apple Fanboy. And the question of "How did this happen?" is a very good question, especially with regards to Microsoft. I mean if the largest tech company in the world has stuff like this happen, it raises a lot of questions about who we can trust our data to. And it also begs the question, if Microsoft has stuff like this happen with its Danger division, how is it ready to launch Azure?

[hce_hce]

The most laughable part of this issue is that the author, who has spent years covering tech issues, is as mystified as everyone else as to the cause of this problem.

It's there in front of your face folks, and it's the virus called Microsoft.

[XiroMisho]

wow... Fail.

The idea of the cloud is that it's set-up on a series of servers and is redundant, should something happen in the cloud, it's at least retrievable.

I do have to say though.... your cloud service's name is "Danger"

Never tempt fate... okay? Fate loves a good pun.

"All my data! I thought it was safe!"
"Well... it was in... DANGER!"

[scdecade]

How can they serve up this excuse with and expect to be taken seriously: "Microsoft has said that the hardware failure that caused the problem took out both the primary and backup copies of the database..." How does a hardware failure take out your primary and backup database? It should basically only be possible to do that on purpose. For them to claim a hardware failure caused unrecoverable loss of both databases must mean they were running on the same piece of hardware. That doesn't really make much sense to me.

[tech_crazy]

Possible that they were running a RAID like thing and the hardware failure was an issue with the controller malfunctioning and not truly mirroring the data and lost the primary too.

In real life, anything is possible. A few years I had 2 hard drives (on 2 different computers) that backed each other up, go bad in 2 days, no power fluctuations/brownouts or anything. Since the drives did the active backups, I had not been backing up in a 3rd place that frequently and I hence lost a lot of data. Serves to prove that even physical isolation of the backup mechanisms may not always be foolproof.

[cp256]

I alternate backups between two different backup hard disks and always have a periodic third archival backup for worst case scenarios. I buy hard disks in threes, even for my notebooks, so there are always at least two spares and it's saved my bacon a few times.

If in fact M$ allowed a single point of failure to take out both primary and secondary data storage systems with no tertiary backups available is absolutely amateur. Everyone who knew about that and didn't raise hell to institute more backup systems should be thrown under the bus.

[QA_Tester]

Also Why weren't backups redundant?

[bluemist9999]

I've heard when buying hard drives for a RAID, it's best NOT to buy them from a single vendor. The reason is that a set of hard drives from a vendor will all probably be from the same batch---meaning they will all fail around the same time.

However, if the hard drives are from different vendors, they will be different batches and hopefully, will fail at vastly different times.

[bicparker]

It's not the "primary and backup copies of the database". It's the primary and failover copies. Just using the information provided in this article, it sounds like there is huge misunderstanding between what a "failover" is compared to a "backup". They aren't the same and if you treat a failover as a backup, you could very well end up with what happened here.

Failover is just a design to maintain high availability. And in an environment of this complexity, both the primary and failover should have some built-in redundancies. It isn't just a hard drive failing here, it is a mirrored complex of systems failing and the failure of the communication systems used to keep the primary and mirrored "Failover" systems synchronized.

In cases like these in a telecommunications environment (at least all of the ones I have seen), the primary and failover are housed in two physically separate data centers and may even be redundant data centers altogether (in the case of very large systems).

The true backup is whatever is used to restore the mirrored systems (typically while one or the other is still running).

[cp256]

Typically mirrors rebuild themselves unless both copies are hosed and in that case the backup is restored to both of them simultaneously. Anyone who thinks they are safe with only a storage mirror has a false sense of security. I regularly clone my mirror systems to other disks.

[dbloyd]

No syncing to a PC? That is strange.

[tgrenier]

I once worked for a supposedly forward thinking CIO/COO. He insisted that tape had no place in a modern data center. Shortly after I left, someone deleted a very important disk group in the one of the SANs while "looking around." Years of data and email was gone forever. The CIO/COO quickly canned the curious admin and blamed him for the mistake and the lack of backups which he refused to fund. That Exec now has a high profile job with a very prominent national bank in an extremely wealthy country. I wonder if they have off-site tape backups now?

[jospark]

Conceivably, another possibility is that data was corrupted, and then it was copied to the failover or backup copies. ie: The backup would be no good, if the data on it is also compromised. Also, it's the same issue with simple home system, with backups of a hard drive being contaminated with a trojan or virus.

The lesson would be that data integrity & data failsafe contingencies are as important as hardware. If the database is well designed, robust, and simple (ie: not convoluted, with one bad byte ruining a critical data system file like a house of cards), then it would be less of an issue.

It's surprising that they couldn't recover with from another dated, versioned backups? What? They don't have multiple, time versioned backups? Did they store all their backup data in some proprietary, no-longer-available hardware restricted data format? Ie: If a RAID 5 card breaks, you'd often need the exact same RAID 5 hardware card to read the data from your "redundant" hard drives. It's hard to believe if it was set up that way too. But that could also explain it.

[QA_Tester]

With increase in amount of data redundancy should be increased as well. Companies must plan for the most catastrophic scenarios possible. If Danger had multiple data centers with multiple backups this wouldn't hav happened to the extent that it did. Most intellegent user would have had only a small percentage of the data loss. Only those that rarely synced to Dangers servers to keep their data safe would have had major problem

[zboot]

Microsoft only recently bought Danger and in likelihood just left the company as-is. That was their mistake, allowing it to continue to operate without some basic improvements like backups of customer data. Someone's head is probably already rolling.

Danger is probably to them an experiment. Look how someone else does something and what can we learn. Well, this experiment has bourne some, arguably pretty common sense for those who are computer savvy, fruit.

[Zand222]

Danger was fine for what, 8 years? One of the first smartphones in the American market.

And then Microsoft buys them and disaster happens in <1 year.

Either Microsoft is totally incompetent or this was deliberate. You have to wonder.