Salesforce outage angers customers

A Salesforce.com outage lasting nearly a day cut off access to critical business data for many of the company's customers on Tuesday in what appears to be Salesforce's most severe service disruption to date.

Salesforce stores customer information for thousands of businesses, delivering data "on-demand" via the Web. The lack of that data interfered with some customers' sales and customer service activities on a critical pre-holiday business day.

"This is not just an inconvenience. We're losing sales," said Charlie Crystle, CEO of Mission Research, a software company in Lancaster, Pa. "It's a busy time of the year."

Bruce Francis, a Salesforce spokesman, said he doesn't know how many of Salesforce's 18,700 customers were affected by the outage, which began at about 6:30 a.m PST. The cause was a faulty database, which was repaired by about 2 p.m., he said.

"We apologize to any customer who was inconvenienced by this," Francis said. "We take that very, very seriously."

It's clear the problem was not isolated. Complaints from affected customers have surfaced across the Web, including on several blogs.

Salesforce touts an "uptime" rate of greater than 99 percent. Outages are "a rare occasion," according to Francis. He said Salesforce's systems are as reliable or more reliable than other comparable systems, including the type that companies run on their own servers.

Yet several Salesforce customers that contacted CNET News.com about Tuesday's glitch said outages happen more frequently than they had expected. About once a month, Mission Research experiences Salesforce outages that typically last an hour or so, Crystle said. Another customer, an East Coast consulting firm, has been struck by outages about a half a dozen times over the past year, according to the firm's vice president, who requested anonymity. Frustration levels are rising.

"I'm really, really angry about this because (Salesforce is) out there marketing themselves as something they're just not living up to," Crystle said.

Salesforce has been a forerunner in a movement to make software cheaper and easier to use by delivering it as a "service" over the Web. But questions about reliability have long been a sticking point for skeptics. Tuesday's outage could give them more reason to stay on the sidelines.

"It's like losing your Internet connection for a whole day," the executive who requested anonymity said. "That's pretty severe. We had technical support people that couldn't talk to customers. When (Salesforce) becomes unavailable, it really shuts you down pretty badly."

Salesforce, which has been growing rapidly, has undertaken efforts to bolster its computing infrastructure. For instance, it has configured its database to run on four different computers so if a machine fails, others will pick up the slack, Francis said. But the "failover" feature didn't prevent Tuesday's problems.

Salesforce's database supplier helped to restore service, Francis said. While he declined to identify who that supplier was, he did identify Oracle as Salesforce's biggest database supplier.

[Lucky Lou]

Morons to rely on Salesforce anyway

It's pretty easy to gripe at Salesforce. That way you don't have to
blame your own stupid decision to rely entirely on an online app
provider for the core of your business and not having a
redundant system yourself in case of an outage. If your business
is so important and your customer database is so crucial to that
business, then don't be an idiot and rely on an outside provider
to give you that service unless you can exist for sometime
without it. It's called a backup plan, Einsteins.

[scdecade]

Where's your money?

Do you use a bank to store your money? Why?

Are They Really Using Oracle?

That last television commercial I heard from Oraclew was the one where they stated their grid system "NEVER FAILS". And, maybe that loud mouth head of Salesforce.com is eating crow now about more realiable client-server systems such as Microsoft or UNIX. On-demand is here to stay, but you have to be prepared for such a calamity (i.e. taking down thousands of businesses at once). Saleforce's apparent attempt to hide all of the downturn will only end up stalling the move towards on-demand computing. They should start fessing up about the vulnerability of such systems until the market matures.

[markdoiron]

Told You So

web-based applications are always going to be susceptible to this because no software is bug-proof. at least with software residing within a corporation/small business, the executives/managers of that business can ***see*** that someone is working to resolve the problem, rather than being uncertain if the web-based provider is off for a company golf day.

mark d.

[adhack]

B/C There are no glitches with ACT?

Yes, Salesforce was down, but it's not like I've never had a software glitch with ACT or Goldmine, or God forbid a Microsoft product! Salesforce is the best solution around today.

[Revscan]

Salesforce is the best solution around today.

It would appear by this belief that you do NOT own the business you work at. If you did you would understand the folly of someone else holding all your customer information. Your customer list is all that keeps you in business. Yes, in house systems will give you problems, but at least all your critical info is in house.

Gord

[lazura]

The Hidden Flaw of Web-Based Apps

For all those prophets who tout web-based application hosting and usage as 'the future' of information technology, this story points out the huge flaw in this model; the customers are totally at the mercy of the application provider's reliability (some may say they have you by the you know what)! This model only works for non-essential applications that won't stop your business. As pointed out in another post, any bonehead company who relies on Salesforce (or other web hosted apps) to keep their business up and running deserves to be impacted!

They do offer better than 99% uptime

I have come accross many customers while consulting who do not realize that 99% uptime means you can have 87 hours of unscheduled downtime per year. So, you are looking at at close to 4 days of downtime per year.

[Joe Blow]

As in Casino Gambling ...

where the average yokel doesn't realize that 99% payouts from slot machines mean that you don't break even, on average (i.e., 100% payout would be break-even - you would get back all of the money you put in). In either case of casino gambling or Web-app gambling, you're going to lose, sooner or later (and, with Murphy running the casino/Web, you can bet your sweet bippy it will be sooner!).

This is yet-another example of how putting MBAs in charge of even the simplest things is a guarantee of failure, usually sooner rather than later. It's a wonder that only 90% of businesses fail in their first five years, mostly due to undercapitalization and naively hoping against inevitable lows in economic cycles. Only about eight out of 1,000 companies ever makes any big money (i.e., gets large enough to make it into the Fortune XXXX and stays there for more than a decade). If computer hardware and software development had that kind of track record, we'd still all be chiseling our data into stone (although that _would_ be easier than some of the user interfaces I've had to endure over the years, not to mention the enhanced reliability!).

I guess Salesfarce.com customers are going to have to wait for The Web 3.0 (3.1, for the Microsloth portion, of course!) before downtime drops below four days per year, huh? Actually, if you only count an eight hour business day, and the nominal 250 business days per year (assuming about 10 holidays annually), that's only 2,000 hours of operations, and 99% of that would be 1,980 hours of uptime, or 20 hours of downtime. Look, I've improved reliability by a factor of 88*100/20 = 438%!

Ladies and gentlemen, I would like it to be noted that at no time did my hands leave my wrists, or enter your wallets, either, thanks to open source calculations. I am hereby patenting this "business process" and plan to go public before you even finish reading this. You'll all get your engraved invitations to the bankruptcy proceedings, but only after I've retired to some South Pacific island I'll own by then, renamed in my honor, and with no extradition treaties to anywhere else, of course! Darn, now I've spilled the beans on how, at the first day of MBA school, they teach you how to set up your golden parachute, and on the second day, they teach you how to execute it.

Notice to all of my dear brand-new customers paying through the wazoo: our systems are guaranteed to be down from now through the holidays, and until I feel like showing up again for work, if even then.

All the Best,
Joe Blow

[lscheirer]

Quoted "customer" is a Salesforce competitor

Just wanted to mention that Mission Research, which is the main customer quoted in the article as unhappy, makes a software tool that competes head-to-head with Salesforce in the nonprofit space. Mission Research makes a software package called GiftWorks, which is a donor CRM database for the nonprofit space. Salesforce is doing a major push into the nonprofit space (they're giving out lots of free licenses); undoubtedly GiftWorks is seeing some serious competition from them.

I guess it's possible that GiftWorks is using Salesforce to manage their own sales process (I certainly don't know anything to the contrary), but this outrage seems pretty convient.

[acitrano2]

From Quoted Customer: Outage was Real

The outage was real, system-wide, and it wasn't convenient, Laura, and our complaints have nothing to do with competition. We were royally screwed last Tuesday. We've used SalesForce since last March. We're not happy campers and we're leaving next week to a modified version of GiftWorks. All sales since last March have been processed and tracked through SalesForce. Except Tuesday.

Tuesday was not the first time. We have had outages or slowdowns consistently over the past 5 months, mostly within the last 10 days of every month, always without any warning or notification. With slowdowns, the response time per page was over a minute and lasted for hours, just killing productivity. On Tuesday we lost an entire day of work. Now, I'll agree, shame on us for not figuring out a way to back up, but it's not easy--it takes 7 separate exports to get out of SF and then a re-import into something useful. And then training the sales team to use something new, etc, etc. It's not trivial.

Most of their US customers were nailed. What bothered me so much was the lack of response, the lack of communication, and the lack of contrition until after stories were printed. They consistently said "we apologize for any inconvenience..."; it wasn't inconvenient, it was costly.

Other article quoted other customers:
http://www.infoworld.com/article/05/12/20/HNsalesforceoutage_1.html

Finally, Microsoft is a competitor in a very broad sense too, but it's apples and oranges, just like with SalesForce. We have not yet run into SalesForce as a competitor in the nonprofit sector--it's just a completely different category of software (like they claim, "no software) and target markets; we're web-enabled desktop, and they are web-only and target for-profits. They have the free program for nonprofits, but again, we haven't run into them in the field at all (we mostly serve small and very small nonprofits) and welcome their presence or anyone else's--as long as it comes with reliability.

[AE109]

Yes we are Einsteins, Thank you.

Obviuosly the last comment is from an mid-level IT manager
who is afraid of losing their job because the infrastructure is no
longer managed in-house.

[jhambacher]

At least you're not bitter

If the last comment was from a mid-level IT manager, they're likely afraid of losing their job by being blamed for their CEO's decision to use Salesforce (because the CEO didn't understand that 99% uptime really means significant outages in service). That's the nature of the IT manager's job - taking sub-optimal purchase decisions (made by others typically) and trying to make them work in real life.