Security

On August 11, Randi Levin, the chief information officer of the city of Los Angeles, stood before City Council members at a hearing of the information technology committee and made her case for why the nation's second-largest city should adopt Google Apps.

"The ability to get whatever information the city needs, whenever they need it, on whatever device they need it on will fundamentally change the way the city works and enhance productivity greatly," she said. "In a fiscal crisis it is difficult to find technology solutions that will save money without requiring a significant capital outlay to achieve those objectives."

Security concerns have kept many government agencies and large corporations away from Google Apps. That is starting to change. A number of small U.S. cities are using the suite and there are Google Apps pilots in more than a dozen federal agencies. If Los Angeles signs on, it would join the District of Columbia as one of the largest government adoptions.

Randi Levin, chief information officer of the city of Los Angeles and a key player in the city's move to adopt Google Apps.

(Credit: City of Los Angeles)

Security experts and officials in other state and federal government departments tend to be wary about outsourcing the management and storage of highly sensitive data to an outside company.

Even in Los Angeles some persuasion was needed. Tony Cardenas, a councilman and chair of the IT committee, and a police official last month said they were worried that sensitive police investigations could be compromised if data were exposed somehow under Google's control. "Drug cartels would pay any sum of money to be aware of our progress on investigations," Cardenas told the Associated Press.

The concerns came to light after the poor e-mail practices of an employee at Twitter and an easy password reset mechanism at Yahoo gave a hacker access to sensitive Twitter documents stored on Google Apps.

While that breach had nothing to do with level of security of Google Apps, it prompted renewed scrutiny of the cloud computing service. It's unclear exactly what effect the exposure of financial and business plan data will have on Twitter. But the specter of confidential police records being hosted by a search company has some people concerned, particularly with a service that is just 3 years old and that added enterprise-level security enhancements 18 months ago.

Google Apps creates "a value proposition. No doubt about it," Mike Hamilton, chief information security officer of the city of Seattle, said in a phone interview after the LA proposal to adopt Google Apps made the news. Seattle uses the e-mail security outsourcing service Google offers through its Postini Message Security, and Hamilton said he is pleased with that service.

"My personal opinion is there's still some shaking out to do around this," he said. "Some bad things are going to happen before all this gets worked out and we don't want to be an early adopter."

During a chief security officer panel at the Black Hat security conference in Las Vegas last month, a group of CSOs from several corporations and one government agency said Google Apps was on their radar, but they didn't seem too eager to embrace it just yet.

"We are looking at those apps and have a number of pilots underway," said Bob Lentz, CSO for the U.S. Department of Defense. "There are a number of security requirements that have to be addressed."

LA concerns assuaged
The sentiment over the past month among LA officials about Google Apps has turned from righteous skepticism to cautious acceptance of the contract, under which systems integrator Computer Sciences Corp. will help the city deploy the Google technology. Cardenas hardly mentioned security during the August 11 committee meeting.

In what appeared to be mere preaching to the choir, Levin made some strong points in favor of the five-year, $2 million contract in her final comments before the committee.

"We own the data, not Google," she said, kicking off a list of key points. "Their security is better than ours," she added. And "cloud computing is safe."

Also, city employees would be using the version of Gmail that is designed for corporate and government customers, not the free consumer product, and they will benefit from electronic discovery, archiving, and disaster recovery functions that they don't have access to today, she said.

Meanwhile, Microsoft Office will remain on all desktop PCs that have it and future purchases will be evaluated for necessity, she added.

Officials from the police department and city attorney's office told the committee that their concerns were being resolved in negotiations with Google. The committee promptly approved the Google Apps adoption proposal, given certain caveats on Google customization for those departments and a later rollout for the police department. The proposal now goes to the budget and finance committee and then the full City Council for a vote in early September.

Specifically, 17,000 of the 30,000 total LA city workers would be migrating to Google Apps if the plan gets approved. The 13,000 workers in the police department and the city attorney's office would initially only use Gmail (instead of Groupwise) and not the other Google Apps, officials said.

In order to meet security requirements from the California Department of Justice (which is within the Attorney General's office), Google needs to either provide some additional level of background checks for people able to access the data or provide an encrypt option and allow city officials to hold the key, according to city officials.

Google also needs to add other functions to the e-mail service, such as auto-acknowledgment of receipt, Tim Riley, chief information office for the LA Police Department, said in an interview.

"We deliver 400,000 electronic subpoenas to our employees every year and there are requirements for the e-mail," he said. "When an officer opens his e-mail, there has to be acknowledgment back to the server that shows he opened it and was (officially) served."

Google has committed to resolving the issues, Riley said.

As for productivity applications, the police department is in year three of a five-year contract with Microsoft for Office that covers about 8,500 PCs, Riley said.

"We have a number of Microsoft databases that are not currently accessible from Google Apps," he said. "That's not to say that down the line we wouldn't" consider Google Apps.

The LA City Attorney's Office has similar security constraints to the police department, and in addition it is required by the courts to use either Word or WordPerfect to file documents, an official said during the hearing.

"People are comfortable with Microsoft Office and there are compatibility issues," Ted Jordan, a city attorney said in an interview.

Meanwhile, "we are still talking to Google about risk management issues, indemnification, limits of liability, data security and disclosure and loss," Jordan said.

For Levin, the cost-benefit analysis is a no-brainer. Google provides enhanced collaboration and remote access ability and more storage and would replace an "antiquated e-mail system" that needs to be replaced, Levin said in an interview.

Because the Google productivity apps are less robust than the Microsoft apps and there are compatibility problems for workers using the two different programs, power users who need the high-end functions of Microsoft applications will still be able to use them, she said.

LA city officials have said the contract is projected to save the city $13.8 million, according to Matt Glotzbach, product management director for Google Enterprise.

More precisely, if the city were to pay for all the capabilities it will get with Google that it currently doesn't have--including archiving, automated electronic discovery and video conferencing--the cost could run as high as $50 million, Kevin Crawford, the city's assistant general manager, said in an interview.

The Google contract would mean the city won't need to dedicate at least 16 servers to run its current system and it can deploy that same number of staff members who operate and maintain it to other areas, he said.

Rebates are unique
But there is an added financial incentive for the city to move to Google. The contract the city is drawing up with Google is written so that any public agency within the state of California can amend it and adopt it for its own use, Crawford said. If 100,000 users sign up, the city will get $1.2 million in rebates, he said.

Crawford told committee members that his office had heard from more than 20 state agencies that are interested in using Google Apps under the contract terms.

"This is unheard of in technology contracts," Levin said during the hearing.

Meanwhile, all the authentication for the single sign-on system is designed to go through the city's authentication system before it hits Google, Crawford said.

As far as concerns about reliability, the city's current in-house system was down about 300 times more than Google was in the last 12 months, he said.

"Google Apps has much higher reliability than on-premise systems tend to have," said Glotzbach, adding that the company promises paying customers uptime of greater than 99.9 percent. "We make redundant copies, multiple live copies, and other things most government agencies can't and don't do."

Outages with Google Apps in the past have been infrequent and short-lived and most did not affect paying customers, according to Google. But, still, they have happened and they make headlines.

Last month there was a four-hour outage in the Google App Engine application hosting infrastructure service. In February, business customers were affected by a 2.5-hour Gmail outage. A Google Docs outage hit them in July 2008, while a Google Apps outage affected some Gmail users in March. Google Docs, meanwhile, had a privacy glitch in March that Google said led to "inadvertent sharing" for a small fraction of documents.

Security concerns weren't enough to keep executives from large corporations like Genentech, Motorola's handset division, Johnson Diversey, and Fairchild Semiconductor from jumping on board.

Blazing the trail for government agencies is the District of Columbia, where deployment of Google Apps was completed in July 2008. The project was driven by DC Chief Technology Officer Vivek Kundra, who has since left to become chief information officer for the federal government, where next year's budget includes funding for cloud computing initiatives.

The District of Columbia paid $479,560 for the Enterprise Google Apps license and is leveraging Google Apps to create the city's intranet at a cost of $500,000, according to a city spokeswoman. As a result, DC saved $3.5 million by replacing an earlier plan to pay another company $4 million for the portal project, she said.

Google Apps is available to 38,000 DC city employees, 4,000 of whom are actively using it, she wrote in an e-mail response to questions. Gmail could potentially replace Microsoft Exchange there, "but this decision has not been made yet," she wrote.

"Our policy has been to put into Google Apps only information that could be transmitted via e-mail; at this time we are not placing sensitive data into Google Apps," the DC spokeswoman wrote. "The District mostly uses Google Apps collaboration tools (e.g. Google Docs, Sites, Video) which filled a void we had for Intranet collaboration tools."

Google is working with the governments to help them overcome their fears.

"Government, by definition, tends to be a little bit more conservative and generally with good reason," Glotzbach said. "But security experts in the city of LA have dug into it and found that in a number of ways the cloud can be more secure than existing in-house systems."

Security guru Bruce Schneier says that as far as cloud computing is concerned, resistance is futile in the long run.

"This is the future of computing and don't think for a minute that this isn't what people will be doing in a few years," he said in his question-and-answer session at the Defcon security conference late last month in Las Vegas.

"I think it's nutty to give up that much control," he said after mentioning Google Apps specifically. "The problem here is trust. They're going to go out of their way to secure their systems more than any user because their reputation is at stake."

Updated 4:52 p.m. PDT to add that Computer Sciences Corp. would be the systems integrator of the Google Apps technology for Los Angeles and that the proposal goes to the city's budget and finance committee before it goes to the full city council.

Google discovered a privacy glitch that inappropriately shared access to a small fraction of word-processing and presentation documents stored on the company's online Google Docs service.

"We've identified and fixed a bug which may have caused you to share some of your documents without your knowledge. This inadvertent sharing was limited to people with whom you, or a collaborator with sharing rights, had previously shared a document," the company said in a note, quoted at TechCrunch, that the search giant sent to affected people. "The issue only occurred if you, or a collaborator with sharing rights, selected multiple documents and presentations from the documents list and changed the sharing permissions. This issue affected documents and presentations, but not spreadsheets."

Google said in a later statement that the problem affected only 0.05 percent of documents stored at the site and that affected Google Docs users had been notified.

Though the documents were shared only with people whom the Google Docs users had already shared documents, rather than with the world at large, the problem illustrates one downside of cloud computing, in which Internet servers host software previously run on a person's own computer. The flip side of a cloud-computing advantage, that a person can get access to those documents from any Internet-connected computer or smartphone, is that technical problems or hacking attempts also can expose private information.

It should be noted, though, that housing data on a local machine has risks of its own. A lost or stolen laptop can reveal any number of secrets, as Boeing, Hewlett-Packard, the National Institutes of Health, and others have found.

(Via Google Blogoscoped.)

To critics, cloud computing can't be trusted because you aren't in control of the data outside your network.

But if that's the case, then how secure are the data and collocation centers that corporations contract with to host their data?

"It does come down to vetting the practices of the provider and making sure they meet the standards you want for your business," Phil Hochmuth, a senior analyst at Yankee Group, said Monday, the eve of Cloud Computing Innovation Day in Santa Clara, Calif.

Companies like Salesforce.com, Amazon.com, and Google have built businesses around serving up on-demand services to enterprises that would rather pay a service provider than buy hardware and hire staff to manage their databases. However, handing over the data is still a cause for concern among many corporations.

"What are they doing to the data? Is it persistently encrypted? Are there access controls in place? Do you get to monitor who they hire and who cleans the data centers at night?" said Phil Dunkelberger, chief executive of PGP Corp. in relaying the concerns on peoples' minds about cloud computing.

How secure is the data? "It's one of the first questions we get, especially from enterprises," said Adam Selipsky, vice president of product management and developer relations for Amazon Web Services.

Securing the data is key to a cloud service provider's business, Selipsky said. "We can afford to devote resources to it that, quite frankly, most of our customers can't," he added.

"Cloud computing can be as secure, if not more secure, than the traditional environment," said Eran Feigenbaum, director of security for Google Apps. "Most organizations really struggle, whether they want to admit it or not, securing their networks."

Feigenbaum points to data breaches that hit the headlines, such as the one that exposed credit card information held by payment processor Heartland recently.

Then there are the statistics that show that one-third of breaches result from stolen or lost laptops and other devices and from employees accidentally exposing data on the Internet, with nearly 16 percent due to insider theft.

"Cloud computing can fix some of these issues," Feigenbaum said.

Not only can Google apply patches more quickly than most enterprises to plug holes in software, but the Google Apps Premier edition offers the ability to protect data in transit by encrypting it in the pipe between Google and the user's desktop, as well as offer control over who can access the data, he said.

Cloud service providers are held to high standards, must offer evidence of security certifications, and are subject to inspections by auditors, placing them under much higher scrutiny than typical in-house security teams, according to Peter Coffee, director of platform research at Salesforce.com.

Most data theft results from someone authorized to access the data doing so improperly or handling the data carelessly, he said. With cloud-based services, when a user logs out, the browser cache can be set to flush automatically, leaving nothing on the desktop to be lost or stolen, and logs can show who did what to which data, he added.

"This is inherently safer than the typical client-server model of downloading data that remains on the end-user device, and is far more secure than distributing data as e-mail attachments whose subsequent use and transmittal are largely uncontrolled," Coffee wrote in an e-mail reply to questions.

The security concern with cloud computing is a cultural issue, said Rebecca Wettemann, a vice president at Nucleus Research.

"The question is would I rather be at a huge data center where a vendor is contractually required to keep my data secure or would I rather rely on my staff to do it properly?" Wettemann said. "You need to trust that your vendor will manage your data."

So far, there haven't been any significant security breaches with an on-demand services vendor, she said. And people are getting used to the idea of being able to access their data anytime and from anywhere because it is out on the Internet, she added.

There have also been precursors to cloud computing that people are familiar with, such as the evolution of answering machines to voice mail services, said Peter Evans, director of security strategy and technology integration at IBM Security Systems.

"It is as much an emotional thing as anything," Evans said. "When my data is on my server in my building, there is a good gut feeling about that. When it's out in the ether, how do I know it's protected?"

Google on Thursday offered administrators of its Premier version of Google Apps more control over the passwords their users choose to access data in Gmail, Docs, and other hosted applications.

Google Apps administrators can now set a minimum password length and will be able to see how strong each user's password remains over time. They can then suggest that users change them if the passwords become weakened. Password strength degrades as the words and names on which they are based become more common and more subject to dictionary attacks.

"Customers were asking for (this) and looking for better visibility" into their end users' security choices, said Eran Feigenbaum, director of security for Google Apps.

The experience for end users will not change. Users of the premier version, typically corporations and educational institutions, are able to see a visual gauge of the strength of their passwords when they create them, as users of the free consumer Google Apps can.

More information is available on the Google Enterprise Blog. Google password tips are here.

Experts say stronger, more secure passwords are longer, have little resemblance to a common word, and have more upper-case and special characters.