December 21, 2005 2:11 PM PST

Salesforce outage angers customers

A Salesforce.com outage lasting nearly a day cut off access to critical business data for many of the company's customers on Tuesday in what appears to be Salesforce's most severe service disruption to date.

Salesforce stores customer information for thousands of businesses, delivering data "on-demand" via the Web. The lack of that data interfered with some customers' sales and customer service activities on a critical pre-holiday business day.

"This is not just an inconvenience. We're losing sales," said Charlie Crystle, CEO of Mission Research, a software company in Lancaster, Pa. "It's a busy time of the year."

Bruce Francis, a Salesforce spokesman, said he doesn't know how many of Salesforce's 18,700 customers were affected by the outage, which began at about 6:30 a.m PST. The cause was a faulty database, which was repaired by about 2 p.m., he said.

"We apologize to any customer who was inconvenienced by this," Francis said. "We take that very, very seriously."

It's clear the problem was not isolated. Complaints from affected customers have surfaced across the Web, including on several blogs.

Salesforce touts an "uptime" rate of greater than 99 percent. Outages are "a rare occasion," according to Francis. He said Salesforce's systems are as reliable or more reliable than other comparable systems, including the type that companies run on their own servers.

Yet several Salesforce customers that contacted CNET News.com about Tuesday's glitch said outages happen more frequently than they had expected. About once a month, Mission Research experiences Salesforce outages that typically last an hour or so, Crystle said. Another customer, an East Coast consulting firm, has been struck by outages about a half a dozen times over the past year, according to the firm's vice president, who requested anonymity. Frustration levels are rising.

"I'm really, really angry about this because (Salesforce is) out there marketing themselves as something they're just not living up to," Crystle said.

Salesforce has been a forerunner in a movement to make software cheaper and easier to use by delivering it as a "service" over the Web. But questions about reliability have long been a sticking point for skeptics. Tuesday's outage could give them more reason to stay on the sidelines.

"It's like losing your Internet connection for a whole day," the executive who requested anonymity said. "That's pretty severe. We had technical support people that couldn't talk to customers. When (Salesforce) becomes unavailable, it really shuts you down pretty badly."

Salesforce, which has been growing rapidly, has undertaken efforts to bolster its computing infrastructure. For instance, it has configured its database to run on four different computers so if a machine fails, others will pick up the slack, Francis said. But the "failover" feature didn't prevent Tuesday's problems.

Salesforce's database supplier helped to restore service, Francis said. While he declined to identify who that supplier was, he did identify Oracle as Salesforce's biggest database supplier.

20 comments

Join the conversation!
Add your comment
Morons to rely on Salesforce anyway
It's pretty easy to gripe at Salesforce. That way you don't have to
blame your own stupid decision to rely entirely on an online app
provider for the core of your business and not having a
redundant system yourself in case of an outage. If your business
is so important and your customer database is so crucial to that
business, then don't be an idiot and rely on an outside provider
to give you that service unless you can exist for sometime
without it. It's called a backup plan, Einsteins.
Posted by Lucky Lou (88 comments )
Reply Link Flag
Where's your money?
Do you use a bank to store your money? Why?
Posted by scdecade (329 comments )
Link Flag
Are They Really Using Oracle?
That last television commercial I heard from Oraclew was the one where they stated their grid system "NEVER FAILS". And, maybe that loud mouth head of Salesforce.com is eating crow now about more realiable client-server systems such as Microsoft or UNIX. On-demand is here to stay, but you have to be prepared for such a calamity (i.e. taking down thousands of businesses at once). Saleforce's apparent attempt to hide all of the downturn will only end up stalling the move towards on-demand computing. They should start fessing up about the vulnerability of such systems until the market matures.
Posted by (50 comments )
Link Flag
Told You So
web-based applications are always going to be susceptible to this because no software is bug-proof. at least with software residing within a corporation/small business, the executives/managers of that business can ***see*** that someone is working to resolve the problem, rather than being uncertain if the web-based provider is off for a company golf day.

mark d.
Posted by markdoiron (1138 comments )
Reply Link Flag
B/C There are no glitches with ACT?
Yes, Salesforce was down, but it's not like I've never had a software glitch with ACT or Goldmine, or God forbid a Microsoft product! Salesforce is the best solution around today.
Posted by adhack (1 comment )
Reply Link Flag
Salesforce is the best solution around today.
It would appear by this belief that you do NOT own the business you work at. If you did you would understand the folly of someone else holding all your customer information. Your customer list is all that keeps you in business. Yes, in house systems will give you problems, but at least all your critical info is in house.

Gord
Posted by Revscan (1 comment )
Link Flag
The Hidden Flaw of Web-Based Apps
For all those prophets who tout web-based application hosting and usage as 'the future' of information technology, this story points out the huge flaw in this model; the customers are totally at the mercy of the application provider's reliability (some may say they have you by the you know what)! This model only works for non-essential applications that won't stop your business. As pointed out in another post, any bonehead company who relies on Salesforce (or other web hosted apps) to keep their business up and running deserves to be impacted!
Posted by lazura (23 comments )
Reply Link Flag
They do offer better than 99% uptime
I have come accross many customers while consulting who do not realize that 99% uptime means you can have 87 hours of unscheduled downtime per year. So, you are looking at at close to 4 days of downtime per year.
Posted by (2 comments )
Reply Link Flag
As in Casino Gambling ...
where the average yokel doesn't realize that 99% payouts from slot machines mean that you don't break even, on average (i.e., 100% payout would be break-even - you would get back all of the money you put in). In either case of casino gambling or Web-app gambling, you're going to lose, sooner or later (and, with Murphy running the casino/Web, you can bet your sweet bippy it will be sooner!).

This is yet-another example of how putting MBAs in charge of even the simplest things is a guarantee of failure, usually sooner rather than later. It's a wonder that only 90% of businesses fail in their first five years, mostly due to undercapitalization and naively hoping against inevitable lows in economic cycles. Only about eight out of 1,000 companies ever makes any big money (i.e., gets large enough to make it into the Fortune XXXX and stays there for more than a decade). If computer hardware and software development had that kind of track record, we'd still all be chiseling our data into stone (although that _would_ be easier than some of the user interfaces I've had to endure over the years, not to mention the enhanced reliability!).

I guess Salesfarce.com customers are going to have to wait for The Web 3.0 (3.1, for the Microsloth portion, of course!) before downtime drops below four days per year, huh? Actually, if you only count an eight hour business day, and the nominal 250 business days per year (assuming about 10 holidays annually), that's only 2,000 hours of operations, and 99% of that would be 1,980 hours of uptime, or 20 hours of downtime. Look, I've improved reliability by a factor of 88*100/20 = 438%!

Ladies and gentlemen, I would like it to be noted that at no time did my hands leave my wrists, or enter your wallets, either, thanks to open source calculations. I am hereby patenting this "business process" and plan to go public before you even finish reading this. You'll all get your engraved invitations to the bankruptcy proceedings, but only after I've retired to some South Pacific island I'll own by then, renamed in my honor, and with no extradition treaties to anywhere else, of course! Darn, now I've spilled the beans on how, at the first day of MBA school, they teach you how to set up your golden parachute, and on the second day, they teach you how to execute it.

Notice to all of my dear brand-new customers paying through the wazoo: our systems are guaranteed to be down from now through the holidays, and until I feel like showing up again for work, if even then.

All the Best,
Joe Blow
Posted by Joe Blow (175 comments )
Link Flag
Quoted "customer" is a Salesforce competitor
Just wanted to mention that Mission Research, which is the main customer quoted in the article as unhappy, makes a software tool that competes head-to-head with Salesforce in the nonprofit space. Mission Research makes a software package called GiftWorks, which is a donor CRM database for the nonprofit space. Salesforce is doing a major push into the nonprofit space (they're giving out lots of free licenses); undoubtedly GiftWorks is seeing some serious competition from them.

I guess it's possible that GiftWorks is using Salesforce to manage their own sales process (I certainly don't know anything to the contrary), but this outrage seems pretty convient.
Posted by lscheirer (1 comment )
Reply Link Flag
From Quoted Customer: Outage was Real
The outage was real, system-wide, and it wasn't convenient, Laura, and our complaints have nothing to do with competition. We were royally screwed last Tuesday. We've used SalesForce since last March. We're not happy campers and we're leaving next week to a modified version of GiftWorks. All sales since last March have been processed and tracked through SalesForce. Except Tuesday.

Tuesday was not the first time. We have had outages or slowdowns consistently over the past 5 months, mostly within the last 10 days of every month, always without any warning or notification. With slowdowns, the response time per page was over a minute and lasted for hours, just killing productivity. On Tuesday we lost an entire day of work. Now, I'll agree, shame on us for not figuring out a way to back up, but it's not easy--it takes 7 separate exports to get out of SF and then a re-import into something useful. And then training the sales team to use something new, etc, etc. It's not trivial.

Most of their US customers were nailed. What bothered me so much was the lack of response, the lack of communication, and the lack of contrition until after stories were printed. They consistently said "we apologize for any inconvenience..."; it wasn't inconvenient, it was costly.

Other article quoted other customers:
<a class="jive-link-external" href="http://www.infoworld.com/article/05/12/20/HNsalesforceoutage_1.html" target="_newWindow">http://www.infoworld.com/article/05/12/20/HNsalesforceoutage_1.html</a>

Finally, Microsoft is a competitor in a very broad sense too, but it's apples and oranges, just like with SalesForce. We have not yet run into SalesForce as a competitor in the nonprofit sector--it's just a completely different category of software (like they claim, "no software) and target markets; we're web-enabled desktop, and they are web-only and target for-profits. They have the free program for nonprofits, but again, we haven't run into them in the field at all (we mostly serve small and very small nonprofits) and welcome their presence or anyone else's--as long as it comes with reliability.
Posted by acitrano2 (3 comments )
Link Flag
Yes we are Einsteins, Thank you.
Obviuosly the last comment is from an mid-level IT manager
who is afraid of losing their job because the infrastructure is no
longer managed in-house.
Posted by AE109 (1 comment )
Reply Link Flag
At least you're not bitter
If the last comment was from a mid-level IT manager, they're likely afraid of losing their job by being blamed for their CEO's decision to use Salesforce (because the CEO didn't understand that 99% uptime really means significant outages in service). That's the nature of the IT manager's job - taking sub-optimal purchase decisions (made by others typically) and trying to make them work in real life.
Posted by jhambacher (1 comment )
Link Flag
I think the response that asked why we use a bank sasy it all. Whilst we dislike banks they do a better job than the majority of us at looking after money, present financial global crash excluded......, muppets the whole lot6 of them.

Most organisations are not Microsoft, Google, IBM, Wal-Mart and so on and SaaS gives way more benefits to the masses than an local solution hosted by the end user or via old style ASP solutions. The proof will be in the how many customers leave SalesForce, very few I would guess. We were effected badly but wont be leaving.... Why? Because it's better than doing it yourself for the majority.

For those in disagreement try to open up your mind in a none biased manner or accept becoming a dinosaur....., extinct. Ps. No axe to grind either way but a little amused at the politics and self-serving views posted. As I said we mostly dislike bank's but they are dominate in money management and day to day finances. Using the argument above, why would ALL OF US put all our money with banks. You all know why, the alternative is not as good. QED....
Posted by Coastey (3 comments )
Reply Link Flag
If your business and customers can't wait, check out a standby solution "Hubcase for Salesforce". When salesforce is down, your 24x7 call center agents can log cases in Hubcase, and have sent to your Salesforce automatically.

http://appexchange.salesforce.com/listingDetail?listingId=a0N30000003IQuWEAW
Posted by rayzhu2006 (5 comments )
Reply Link Flag
Get alert and notification by email. Check status dashboard and history at:

http://salesforcestatus.com
Posted by rayzhu2006 (5 comments )
Reply Link Flag
 

Join the conversation

Add your comment

The posting of advertisements, profanity, or personal attacks is prohibited. Click here to review our Terms of Use.

What's Hot

Discussions

Shared

RSS Feeds

Add headlines from CNET News to your homepage or feedreader.