October 12, 2009 10:32 PM PDT

Sidekick's lesson: We learn by failing

by Rafe Needleman
  • Font size
  • Print
  • 46 comments

Technically, we can't blame the loss of Sidekick users' data on a failure of either the concept or the technology of "cloud computing." But Microsoft's clear bungling of basic information management practices (apparently, there were backups--but they didn't work) does cast a pall over not just Microsoft but the cloud concept entirely.

T-Mobile is trying to keep customers' data alive.

(Credit: Screenshot by Rafe Needleman/CNET)

Microsoft, as one of the giant infrastructure technology companies that's saying through its product offerings that data is safe in the "cloud," has a responsibility not just to its customers but to the growth of cloud computing overall to keep the data it's holding safe.

The company's failure to keep the data safe shows the world how fragile cloud computing is. Even though, really, it isn't. The world knows how to build systems that safeguard data from hardware and software and network failures, and even from hacking and other forms of sabotage. The fact that Microsoft failed to keep the Sidekick data backed up indicates, rather, how management can fail.

But do consumers, or corporate IT managers considering cloud-based services, care where the failure was? All we know is that it failed.

Travel by commercial airliner is neither unsafe nor inherently safe because of the technology itself. It is as safe or as dangerous as the procedures followed to certify and maintain the equipment that people put their life's trust in.

Microsoft's Sidekick outage shows that sadly, in fact, it's true: you cannot trust the cloud because you can't trust the people who run it. It indicates another scary truth: We haven't had enough cloud failure yet. We're going to have more. We need more. We learn from each failure. And we're all thinking the same thing: I hope I'm the beneficiary, and not the victim, of the hard lessons still to come.

Data in the cloud can be safe. And it will become more safe thanks to this outage. Failures of trust, like this one, have costs, but there are benefits as well.

More: Full coverage of the Sidekick data loss

Rafe Needleman writes about start-ups, new technologies, and Web 2.0 products, as editor of CNET's Webware. E-mail Rafe.
Recent posts from Rafe's Radar
Reporters' Roundtable Podcast: Biggest tech stories of 2009
Dear newspapers: I will pay for your content, once
Blippy launches the Twitter of personal finance
Reporters' Roundtable Podcast: Google Chrome OS
How to fix Facebook's new privacy settings
Milo.com and Google Products search store shelves
Boxee Beta is cleaner, better, still closed
Hands-on with the JooJoo
Add a Comment (Log in or register) (46 Comments)
  • prev
  • 1
  • next
by cvaldes1831 October 12, 2009 10:58 PM PDT
The only thing I've learned from all these cloud computing failures is that cloud computing isn't ready for primetime. Actually, that's not something I "learned"; these episodes just reiterate my original stance.

Frankly, we should stick a fork in cloud computing. We could revisit it in 5-10 years, but it's simply a waste of column-inches right now. These services are being run by a bunch of shoemakers.
Reply to this comment
by Lerianis3 October 12, 2009 11:05 PM PDT
There are always going to be computer failures. Even in 5-10 years, there are going to be computer failures, you just learn to deal with it. What I want to know is why there was not a backup of all these computers and the data on them, and why they were not running in RAID configurations that are 'safe'..... i.e. two drives with the EXACT SAME INFORMATION ON THEM, if one fails... your information is safe.
by Khurt October 13, 2009 4:51 AM PDT
"all these cloud computing failures". How many is that? How long did they last? How risky is it compared to the large internal data failures at public/private companies that we never hear about?
by Super2online October 13, 2009 5:02 AM PDT
The article says that there were backups but they didn't work. The failure here was that "people" didn't make sure that the backup did in fact save 100% of the data.
by CoolSilver October 13, 2009 6:04 AM PDT
""There are always going to be computer failures. Even in 5-10 years, there are going to be computer failures, you just learn to deal with it. What I want to know is why there was not a backup of all these computers and the data on them, and why they were not running in RAID configurations that are 'safe'..... i.e. two drives with the EXACT SAME INFORMATION ON THEM, if one fails... your information is safe.""

Data will be safe if a single drive fails. If the system is writing corrupted data or the system is breached you will not have safe data. If the system replicating backup database info is being send bad data from the primary database, all your data is corrupt across all mirrored drives unless setup correctly.

Obviously this was the case. The database crashed and corrupted the backup when replicating.
by Random_Walk October 13, 2009 6:51 AM PDT
*sigh*

RAID 1 (mirrored) will keep identical copies, but performance suffers a bit.

RAID 5 (striped/parity) is the standard on most servers. You can lose one (or two, depending on how many disks you have) and be okay, but you have to let the set rebuild after a failure. Requires at least three drives, which most users do not have stashed in their desktops.

Note that RAID will not save your data if that data gets corrupted (through malware, OS failure, etc), because the corruption will simply be written across the set (and any network drives attached to your device). RAID will not save you from human error - if your kids accidentally re-format the disk, they'll re-format all of the disks in a given set. RAID only prevents a single hard disk failure from destroying your data. That's it.

Only offline backups will reliably save you from most types of failure and mishap. Shadow copies and such are helpful, but if the disk gets reformatted, so do the shadow copies. If the OS gets infected, you can't rely on them to be clean. (hint: get a USB drive, plug it in to your computer once in awhile to run a backup, then UNPLUG THE DAMNED THING when you're done w/ the backup).
by cvaldes1831 October 13, 2009 7:18 AM PDT
Random_walk is correct. Lerianis3 has not thought this through.

RAID only protects from hard-drive failure, nothing more. It does not protect you from corrupted data (bad software, bad controller, bad cables). It does nothing against theft or environmental disasters like flood or fire (you need to store a backup copy offsite).
by Mergatroid Mania October 13, 2009 11:17 AM PDT
I'll never store any of our company data off site. I don't trust other companies to keep the data safe. I have been taking care of our data for 20 years. Regular backing up and good security keep our data safe.

In 20 years I have lost nothing.

People can keep using "clouds" if they want, but my track record speaks for itself. If you want something done right then do it yourself. It's not computers you have to trust, but the people operating the computers.

Computers don't kill data, people kill data ;)

I might be old fashioned, but I'll stick with how I do things now since I don't have to rely on anyone but myself.
by TotallyMadeUpName October 13, 2009 12:45 PM PDT
So, Mergatroid Mania, your facility is immune from fire, earthquake, tornado, plane falling from the sky, terrorist attack, ect?
by brian.lee October 14, 2009 1:37 PM PDT
I think Microsoft is giving cloud computing a bad reputation. This failure seems to be because Microsoft cut corners because it isn't planning on keeping the sidekick around so it's putting it on a skeleton crews. This is why people hate Microsoft regardless what their business plans are they should have done one of two things either maintain a good level of service by properly staffing the team the runs the sidekick OR... if they plan to kill off the service than promptly kill it off don't do a half a$$ed job of it. People are paying good money, if you can take their money than you should be able to live up to your side of the deal and provide a quality service.
by Emmy2 October 12, 2009 11:17 PM PDT
As someone who worked on "cloud computing" back when it was just a gleam in the architects' eyes, my biggest fear was not that our company could not safeguard the customers' data. I knew that we could, and that we had a long history of doing so. My biggest concern, and one that cannot be mitigated by technology, is that, even though your precious data may be held securely by a company in which you have absolute trust, your trusted partner can abruptly go out of business or be taken over by another company, for example, in a leveraged takeover. I saw an analogous situation on occasion internally in my own company during Corporate IT shakeups that moved us to better technologies but sometimes left the migration puzzle to the employees. So, my advice to friends and colleagues has always been the same: if you have data that you consider critical, you *must* keep your own backup of the data. It doesn't necessarily need to be in the same format as the "official" data, but, unless you can easily recreate needed content from memory, *you* must have a backup in your own possession.
Reply to this comment
by Sumatra-Bosch October 12, 2009 11:44 PM PDT
We've learned nothing from this fiasco.

But we have been reminded of MSFT's barking incompetence.

http://www.roughlydrafted.com/2009/10/12/microsofts-sidekickpink-problems-blamed-on-dogfooding-and-sabotage/

Cloud computing architectures can be designed for highly available criteria - and have been, among them Danger, the system that secured the Sidekick customers' data. Until MSFT bought the company and lobotomized it.

Word has it that Ballmer ordered a truck load of chairs to throw at the server farm, hurled everyone of them with that girlie arm toss to no avail.

Duncan Hines,

RSB
Reply to this comment
by Splashes October 13, 2009 12:18 AM PDT
+1

Ever heard of a comparable data loss at a large well-known firm? No, you haven't, so this fiasco says nothing about cloud computing, yet confirms what we already knew about Microsoft (managed by morons).
by Lerianis3 October 13, 2009 1:39 AM PDT
Yep, you have heard of a comparable data loss at a large well-known firm, if you are listening and have a 'in' to the big companies as I do.
The only reason that you do not hear of OTHER companies having these problems is that they cover it up, say it isn't a loss of data, and their user agreements have severe penalties if you say anything publicly about them losing data.
by Splashes October 13, 2009 3:12 AM PDT
Golly gee whillikers, really? A data-loss coverup conspiracy? Zounds! What won't they think of next? Goodness, I wish I had an "in" like you and all the cool people!
by Super2online October 13, 2009 5:13 AM PDT
Just as I suspected. I read the article you quoted and the source of the article it quotes and ALL of this supposed "inside" information came from an Apple enthusiast website. None of the articles name any of the people they supposedly talked to who were on the inside of Danger, the pink project or the Zune team. Explain to me how you can trust any of this information please? This is Apple biased hogwash at it's finest, plain and simple.
by Random_Walk October 13, 2009 6:58 AM PDT
@Splashes:

Other companies have had data losses, but rarely do you see them happen in orgs that store customer-owned data. Microsoft/Danger is the first really big one.

@Lerianis3: Bull Excrement. No NDA can save a corp from being sued over a customer's lost data (even if it's small claims). No NDA can save a corp from having to compensate paying customers for lost data (only an SLA could do that, and in the business world, only a very stupid company would agree to eating lost data w/o compensation).

As for the whole Microsoft Pink thing? Meh - we'll see. It sounds plausible, and Windows Mobile is choking on its own blood right now, but only time will tell.
by anchova October 13, 2009 12:12 AM PDT
The idea that a commercial airliner is neither safe nor unsafe because of the technology is a baffling claim. Are you sure about that? So all the safety innovations in planes over the past six decades - those were a joke?

The truth is, cloud computing is fragile. It takes your data and sends it over an internet connection (driving through 8 mile in a convertible with a suitcase full of hundred dollar bills) and fails intermittently. So far we have seen periodic failures with Google Apps (more than enough to scare most any business or serious user away) and situations such as these where data evaporates.

I doubt we will have this down pat in the next five years. Cloud computing needs time. I'd love to wake up and see technology writers just giving it a break already and go back to talking about virtualization or something else that is truly revolutionizing industry.
Reply to this comment
by Lerianis3 October 13, 2009 1:44 AM PDT
So what? In case you didn't know, computers in homes fail intermittently, about on the same schedule as cloud computing apps do.
The fact is that cloud computing is only fragile IF THE COMPANY IN QUESTION CHEAPS OUT, and doesn't have numerous backup servers to take over when the inevitable failures happen.
by Khurt October 13, 2009 4:46 AM PDT
Here we go with the anecdotal "if it happened once" it happens all the time argument. Dude. The cloud fails less often than there are car accidents and no one dies. Yet, I bet you drive to work every day. Based on your reasoning if I have one car accident this year, I should just stop driving?
by myles taylor October 13, 2009 7:21 AM PDT
78% of aviation accidents are caused by pilot error (cite FAA 2004). The safety innovation are mostly to take the pilot out of the equation to minimize the possibility for pilot error. They were also to increase load, fuel efficiency, and other things. Airliners have been quite safe for some time.
by brian.lee October 14, 2009 1:40 PM PDT
Would you fly on a plane if the software the runs the guidance controls were written by Microsoft? it's a yes or no answer would you. Given this companies track record?
by theantibush October 13, 2009 12:20 AM PDT
I don't buy the teething pain explanation.
Untested backups are a fundamental error, committed by the most incompetent of the incompetent IT.
It has nothing specifically to do with cloud computing.

You can expect more basic, fundamental errors that instead of affecting one company takes out dozens simultaneously as cost competition squeezes competence out.

And its not just the cloud but the communications with it. I don't need my internal business hampered by external problems. We had a landslide last year that took out internet for a week.
We'd be busted today if we were cloud-dependent.

Cloud computing is IT's version of greasy fast food, complete with the e.coli.

That said, there will be those that are gonna wolf it down like the morons they are.
Reply to this comment
by cvaldes1831 October 13, 2009 12:37 AM PDT
You're right: I don't see this getting much better in the next few years. One would *think* that offline/offsite backups would be a no-brainer in this day and age, but again we are witness to incompetents running the show.

Sadly, it looks like tech journalists are wolfing down the cloud computing fast food like it is going out of style. The last few days have been a god-awful puke-fest. It's almost like these writers are bulimic or something.
by dshkipin October 13, 2009 12:34 AM PDT
its all about the chips.
Reply to this comment
by Splashes October 13, 2009 12:39 AM PDT
Rafe: "We haven't had enough cloud failure yet. We're going to have more. We need more. We're learn from each failure."

Even ignoring the typo, this is pure, unadulterated bunk. Rafe writes as if cloud computing is in its infancy, and there's no way we can hold Microsoft responsible for this unprecedented data loss. Need I point out that RIM has managed to avoid this sort of problem for the last 10 years? Amazon has managed not to lose my purchase history, wish lists, etc. Same with Gmail, Yahoo, and a dozen other high-profile cloud services I could name. Heck, even the two-bit hosting company I utilize hasn't ever lost anyone's data in the 8 years they've been in business.

The fact is, Microsoft screwed up big-time. The attempt to spread the blame over the entire cloud is nothing more than a defensive reflex kicking in, and grasping at the best excuse that came to mind. Pitiful, Rafe.
Reply to this comment
by Lerianis3 October 13, 2009 1:45 AM PDT
RIM has been very lucky. Plus, the fact that they are smart and having everything in Mirrored Raid setups where it's almost IMPOSSIBLE to lose data.
by Splashes October 13, 2009 3:20 AM PDT
RIM has been "lucky", eh? Tell me, besides Sidekick/Microsoft, which prominent company has been "unlucky" as regards data loss?

(That's a rhetorical questions, by the way. I deliberately used the word "unprecedented".)

Several quotes regarding luck come to mind. I'll go with this one: "Luck has a peculiar habit of favoring those who don't depend on it."
by Khurt October 13, 2009 4:49 AM PDT
I agree this is a failure of the parties involved not a failure of the cloud.
by cvaldes1831 October 13, 2009 7:25 AM PDT
@Khurt:

But the cloud is operated by people. People make the architectural decisions. They implement it. They maintain and repair it. The cloud doesn't magically create itself.

The SideKick lesson is this: the cloud failed for 1 million SideKick users. Epic data-losing fail. Did anyone learn anything from this? If you are running cloud computing services and don't understand the basic concepts of data backup, you shouldn't be a system administrator. You shouldn't need lessons like this to learn.

Tech writers should stop writing about cloud computing. Give it a rest. It might be worth revisiting in 5-10 years, but then again, it might not. These clowns who are running these services today may be replaced by cheap, even less competent replacements.
by swimflyfast October 13, 2009 3:53 AM PDT
I wonder if Bill Gates would be happy if his contacts and emails were lost and he was offered $100 and expected just to smile and stay with the service?
Reply to this comment
by Anti-BS-Advocate October 13, 2009 5:52 AM PDT
And whilst this does not excuse the action of losing the Data, Bill (and me and other supposed "tech-savvy" folks) would already have our personal data backed up via ActiveSync or other technologies. It's a no-brainer, really. Goes back to the old saying: Save Early, Save Often. (By whatever means necessary)...
by myles taylor October 13, 2009 7:23 AM PDT
I'd say the same as I tell customers. Make redundant backups and check them to make sure they are there. This is 100% Microsoft failure. There is no reason cloud computing shouldn't work given proper procedures.
Reply to this comment
by 247mark October 13, 2009 7:59 AM PDT
The risk inherent in cloud computing is too great for critical data but people are seemingly blinded by the "convenience" of cloud computing. Los Angeles County recently signed on to have Google host their data. That should be interesting.

First, some other party is holding your data. Trust all you want but back in the dotcom bust, companies were falling out left and right. In a down economy, if the company going out of business runs your cloud, you're screwed. I haven't seen it yet but it's a definite possibility.

Secondly, privacy is an issue. Privacy policies change all the time, usually for the benefit of the company, seldom for the benefit of the consumer.

Finally, control is an issue. This fiasco says it all as far as control.

I understand that this was associated with a cell phone and probably beyond the users' power to decide "to use the cloud/not to use the cloud" but it illustrates the dangers of cloud nonetheless. Not everything that tech companies dream up is a good idea. Cloud computing is the current version of thin client computing, which is old and better suited to companies than consumers. I'm sure I'll be criticized for this opinion in some way but I definitely won't be crying when a cloud blows away and I still have my data.
Reply to this comment
by mikekrause October 13, 2009 8:09 AM PDT
I think the title of this article should be "We have the opportunity to learn from failure". Clearly Microsoft has had opportunities, but hasn't learned much in this case.
Reply to this comment
by Ralph Doctorow October 13, 2009 8:09 AM PDT
Well, many organizations take the view that with Microsoft products it's best to wait for version 3 (or later) because the first few are pretty flawed.

It seems that cloud computing at least as run by MS is following the same trajectory.
Reply to this comment
by AppleSuxLeo October 13, 2009 11:09 AM PDT
Anyone dumb enough to only have their important info on such a service needs to learn how to do backups
or use a reliable online backup service like Carbonite.
Reply to this comment
by Warp09 October 13, 2009 3:07 PM PDT
AppleSuxLeo - What assures that Carbonite won't lose your data or have hardware failures or find out their backups don't work?

I am continually amazed at how much information people upload to Facebook and backup to some CLOUD with total faith it will always be there. The idea of having your data accessible from anywhere, anytime has it's price.

Image your hard drives at home to an external drive and then go put that external HD in your bank's safe deposit box.
Reply to this comment
by corelogik October 13, 2009 5:37 PM PDT
Cloud Computing = Mainframe 2.0

There is a reason the computing paradigm moved away from the mainframe/client model decades ago. None of my data resides solely in the "cloud". Who came up with that name anyway, it's a stupid name. I will always entrust my data to only one person, me.
Reply to this comment
by alexacker October 14, 2009 10:29 AM PDT
Can you backup your own data from a sidekick? Before the iphone, before the treo, i had the first sidekick and it didn't sync to my mac, like my iphone or treo did.

Sidebar: although the treo did, it always was a painful experience.

If you can backup from a sidekick locally, please advise. And if it's truly easy, then I agree with this comment. Customers should keep their own backups. Although the demographic for a sidekick user is pretty much a teen, I doubt they'd backup even it came with a cable and an 'easy' button to push for backups.
by QA_Tester October 13, 2009 6:13 PM PDT
Some people blame Microsoft exclusively for this. Some people blame the cloud. There is no substitute for users backing up their own data no matter what.
Reply to this comment
by Frank_Nospam October 13, 2009 7:23 PM PDT
Quoting a funny comment I read years ago, about the exact same situation happening at some other company:

Jerry: I don't understand. Do you have my data?
IT: We have your backup, we just can't restore it.
Jerry: But the backup keeps the data here, that's why you have the backup!
IT: I think I know why we have backups.
Jerry: I don't think you do. You see, you know how to MAKE the backup, you just don't know how to RESTORE the backup. And that's really the most important part of the backup: the restoring. Anybody can just make them.
Reply to this comment
by dasumner October 14, 2009 3:42 AM PDT
This was not a cloud problem. This was a problem of relying on an untested backup strategy. If I had a dollar for every time I've listened to some manager explain a backup plan that sounded like it came out of the Wiley Coyote School of IT Management, I'd be rich. I can almost guarantee that this event involves a similar story. Even worse is that there are probably a 1/2 dozen other service providers who are currently thanking God that it wasn't THEM and their wobbly backup plan that fell victim to Murphy's Law this time around.
Reply to this comment
by rinap732 October 14, 2009 11:49 AM PDT
tmobile is the worst phone service ever.
Reply to this comment
by Forked_Tongue October 16, 2009 7:08 PM PDT
I agree, this isn't the first time this has happened with Tmobile or the sidekick (I should know, I've been an employee for over 7 years), most of the outtages are intermittent but this is the third time I know of where the loss affected the customer's data on the sidekick. The problem is how they implement it and their reasoning, they want to hold your data hostage so it works in their favor to resign you to another contract renewal. The item should store your contact information locally as well as on the danger server, just like how android devices do with google, it's not foolproof but all this would have been avoided if this was implemented.
by Mikebanks October 17, 2009 3:36 PM PDT
Damn--when I saw all those "Sidekick" headlines I had a flashback to DOS (Borland Sidekick).
--Mike
Reply to this comment
(46 Comments)
  • prev
  • 1
  • next
advertisement

Behind the scenes: NORAD's Santa tracker

For decades, the defense group has let you follow the Christmas Eve travels of the jolly old elf. These days, technology is playing a bigger role than ever.

Intel redesigns Atom chip for Netbooks

The chipmaker officially announces the next generation of its popular Atom CPUs for Netbooks, the N450, weeks before the CES trade show.

About Rafe's Radar

Rafe Needleman has been reviewing technology products and businesses since 1988. Formerly editor-in-chief of Byte Magazine, and author of the Catch of the Day column for Red Herring, he's interviewed thousands of tech execs. For this blog he talks to entrepreneurs and start-up CEOs to explore the strategies behind new technologies.

Add this feed to your online news reader

Rafe's Radar topics

advertisement
advertisement

Inside CNET News

Scroll Left Scroll Right