Green Tech

Google for the first time showed off its server design. (Click to enlarge)

(Credit: Stephen Shankland/CNET)

Updated at 4:08 p.m. PDT April 1 with further details about Google's data center efficiency and shipping containers modules and 6:30 a.m. April 2 to correct the time frame of efficiency statistics.

MOUNTAIN VIEW, Calif.--Google is tight-lipped about its computing operations, but the company for the first time on Wednesday revealed the hardware at the core of its Internet might at a conference here about the increasingly prominent issue of data center efficiency.

Most companies buy servers from the likes of Dell, Hewlett-Packard, IBM, or Sun Microsystems. But Google, which has hundreds of thousands of servers and considers running them part of its core expertise, designs and builds its own. Ben Jai, who designed many of Google's servers, unveiled a modern Google server before the hungry eyes of a technically sophisticated audience.

Google server designer Ben Jai

(Credit: Stephen Shankland/CNET)

Google's big surprise: each server has its own 12-volt battery to supply power if there's a problem with the main source of electricity. The company also revealed for the first time that since 2005, its data centers have been composed of standard shipping containers--each with 1,160 servers and a power consumption that can reach 250 kilowatts.

It may sound geeky, but a number of attendees--the kind of folks who run data centers packed with thousands of servers for a living--were surprised not only by Google's built-in battery approach, but by the fact that the company has kept it secret for years. Jai said in an interview that Google has been using the design since 2005 and now is in its sixth or seventh generation of design.

"It was our Manhattan Project," Jai said of the design.

Google has an obsessive focus on energy efficiency and now is sharing more of its experience with the world. With the recession pressuring operations budgets, environmental concerns waxing, and energy prices and constraints increasing, the time is ripe for Google to do more efficiency evangelism, said Urs Hoelzle, Google's vice president of operations.

"There wasn't much benefit in trying to preach if people weren't interested in it," said Hoelzle, but now attitudes have changed.

The company also focuses on data center issues such as power distribution, cooling, and ensuring hot and cool air don't intermingle, said Chris Malone, who's involved in the data center design and efficiency measurement. Google's data centers now have reached efficiency levels that the Environmental Protection Agency hopes will be attainable in 2011 using advanced technology.

"We've achieved this now by application of best practices and some innovations--nothing really inaccessible to the rest of the market," Malone said.

The rear side of Google's server.

(Credit: Stephen Shankland/CNET)

Why built-in batteries?
Why is the battery approach significant? Money.

Typical data centers rely on large, centralized machines called uninterruptible power supplies (UPS)--essentially giant batteries that kick in when the main supply fails and before generators have time to kick in. Building the power supply into the server is cheaper and means costs are matched directly to the number of servers, Jai said.

"This is much cheaper than huge centralized UPS," he said. "Therefore no wasted capacity."

Efficiency is another financial factor. Large UPSs can reach 92 to 95 percent efficiency, meaning that a large amount of power is squandered. The server-mounted batteries do better, Jai said: "We were able to measure our actual usage to greater than 99.9 percent efficiency."

Urs Hoelzle, Google's vice president of operations

(Credit: Stephen Shankland/CNET)

The Google server was 3.5 inches thick--2U, or 2 rack units, in data center parlance. It had two processors, two hard drives, and eight memory slots mounted on a motherboard built by Gigabyte. Google uses x86 processors from both AMD and Intel, Jai said, and Google uses the battery design on its network equipment, too.

Efficiency is important not just because improving it cuts power consumption costs, but also because inefficiencies typically produce waste heat that requires yet more expense in cooling.

Costs add up
Google operates servers at a tremendous scale, and these costs add up quickly.

Jai has borne a lot of the burden himself. He was the only electrical engineer on the server design job from 2003 to 2005, he said. "I worked 14-hour days for two and a half years," he said, before more employees were hired to share the work.

Google has patents on the built-in battery design, "but I think we'd be willing to license them to vendors," Hoelzle said.

Another illustration of Google's obsession with efficiency comes through power supply design. Power supplies convert conventional AC (alternating current--what you get from a wall socket) electricity into the DC (direct current--what you get from a battery) electricity, and typical power supplies provide computers with both 5-volt and 12-volt DC power. Google's designs supply only 12-volt power, with the necessary conversions taking place on the motherboard.

Google's data center efficiency has been improving gradually.

(Credit: Stephen Shankland/CNET)

That adds $1 or $2 to the cost of the motherboard, but it's worth it not just because the power supply is cheaper, but because the power supply can be run closer to its peak capacity, which means it runs much more efficiently. Google even pays attention to the greater efficiency of transmitting power over copper wires at 12 volts compared to 5 volts.

Google also revealed new performance results for data center energy efficiency measured by a standard called power usage effectiveness. PUE, developed by a consortium called the Green Grid, measures how much power goes directly to computing compared to ancillary services such as lighting and cooling. A perfect score of 1 means no power goes to the extra costs; 1.5 means that ancillary services consume half the power devoted to computing.

Google's PUE scores are enviably low, but the company is working to lower them further. In the third quarter of 2008, Google's PUE was 1.21, but it dropped to 1.20 for the fourth quarter and to 1.19 for the first quarter of 2009 through March 15, Malone said.

Older Google facilities generally have higher PUEs, he said; the best has a score of 1.12. When the weather gets warmer, Google notices is that it's harder to keep servers cool.

An excerpt from a video tour Google presented of its data center containers. Like conventional data centers, Google's shipping containers have raised floors.

(Credit: Stephen Shankland/CNET)

Shipping containers
Most people buy computers one at a time, but Google thinks on a very different scale. Jimmy Clidaras revealed that the core of the company's data centers are composed of standard 1AAA shipping containers packed with 1,160 servers each, with many containers in each data center.

Modular data centers are not unique to Google; Sun Microsystems and Rackable Systems both sell them. But Google started using them in 2005.

Google's first experiments had some rough patches, though, Clidaras said--for example when they found the first crane they used wasn't big enough to actually lift one.

Overall, Google's choices have been driven by a broad analysis on cost that encompasses software, hardware, and facilities.

"Early on, there was an emphasis on the dollar per (search) query," Hoelzle said. "We were forced to focus. Revenue per query is very low."

Mainstream servers with x86 processors were the only option, he added. "Ten years ago...it was clear the only way to make (search) work as free product was to run on relatively cheap hardware. You can't run it on a mainframe. The margins just don't work out," he said.

Operating at Google's scale has its challenges, but it also has its silver linings. For example, a given investment on research can be applied to a larger amount of infrastructure, yielding return faster, Hoelzle said.

A diagram of a Google modular data center

(Credit: Stephen Shankland/CNET)

SANTA CLARA, Calif.--Revamping existing data centers can achieve energy efficiency close to those built from scratch to be greener, according to an early report Thursday from Accenture, which analyzed results of case studies backed by the Silicon Valley Leadership Group.

The energy savings explored, if widespread, could prevent the release of carbon dioxide equivalent to taking 8 million cars off the road, researchers said.

Data center energy use could double by 2011, amounting to $7.4 billion in U.S. electricity costs and requiring the equivalent of 10 new power plants, according to the Environmental Protection Agency.

"Just because you're not (running) a new data center doesn't mean you can sit back," said Teresa Tung, senior researcher at Accenture Technology Labs, which pooled results of studies in which Lawrence Berkeley National Laboratory and tech heavyweights including Yahoo, Sun, and Oracle collaborated.

Online activity may exist in the cloud, but for each e-mail sent or movie downloaded, a "little puff of carbon" is emitted somewhere on the back-end, as Sun vice president of engineering Subodh Bapat has described.

Projected growth in IT power consumption outpaces that of any other industry, noted Ray Pfeifer, chair of data center efficiency for the Silicon Valley Leadership Group, which organized a summit at Sun's headquarters, where study results were presented.

Many tech leaders fear the sector will suffer from rising energy costs and anticipated California caps on carbon emissions.

Some consider their data center details to be trade secrets, but more are opening up to each other to try to reduce the industry's carbon footprint. Case in point, some say, are the collaborative demonstration projects, which put into practice recommendations on data center efficiency (PDF) presented to Congress in 2007 by the EPA.

New data centers using the suggested technologies could achieve 79 percent infrastructure efficiency, according to Tung of Accenture. Older data centers applying the tweaks weren't far behind at 74 percent efficiency.

A holistic approach to IT transformation can reduce the electrical use of data centers better than individual site improvements, Tung added.

Useful systems included airflow management with variable fan drives and water side economizers. In addition to promoting consolidation and virtualization, researchers underscored the importance of better managing virtual environments.

Standards are needed to improve security and optimize the virtual environment, said David Thompson, chief information officer at Symantec, which he said saved $40 million through consolidating data centers last year.

Sun Microsystems has reduced its number of data centers from a dozen five years ago to seven today, said chief information officer Bob Worrall, adding that he'd like that to drop to three data centers in the next several years.

"People are saying there are five fuels: coal, nuclear, gas, and oil," said PK Agarwal, chief technology officer for the state of California. "But efficiency is the fifth fuel."

The full report of the data center demonstration projects' results is set to become available July 11.

Other data center hosts for the case studies were Symantec, NetApp, Synopsys, the U.S. Post Office, and Digital Realty Trust. Also participating in the research were SynapSense, APC, Cassatt, IBM, Liebert, Modius, Power Assure, PowerSmiths, Rittal, SprayCool, the California Energy Commisison, and PG&E.

BURLINGAME, Calif.--Chipmakers have been applying lessons learned in mobile computing to servers in an effort to increase efficiency by lowering power consumption. But a noted Google engineer threw some cold water on the approach on Monday, arguing the two styles of computing are too different.

"The data center is a different device than the key targets for mobile electronics, laptops, and mobile devices," said Luis Barroso, a Google engineer who closely studies the company's power consumption, speaking at the O'Reilly Velocity conference here.

And naturally, with at least hundreds of thousands of servers in operation and its data centers placed near power plants to cut electricity costs, Google is trying to get computing equipment makers more excited about efficiency.

"Maybe if you call this a land-held computer, perhaps they'll help us," he quipped, showing an aerial view of a sprawling Google data center.

The basic problem is that mobile devices and servers have different modes of activity.

Mobile devices have been improving through better exploitation of the fact that they spend a lot of time dormant with occasional bursts of activity. That lets processors and other electronics save power by spending most time in low-power sleep modes, then snapping awake for peak-power high-performance modes when necessary.

Google's servers, though, have the opposite type of activity: they spend most of their time doing modest amounts of work, with frenzied moments of peak activity and complete lulls a rarity, Barroso said. The measurements are based on measurements of about 5,000 servers performing four different Google applications, he added.

The company's servers simply can't go to sleep, he said. Each machine is "rarely fully idle," he said. "The fraction of time the servers are actually doing exactly nothing is very small."

Thus, Google is urging electronics designers to create products that more gracefully reduce power demands as activity diminishes. Servers naturally consume peak power at peak activity, but what's bad is that they still consume about half peak power when at zero activity.

Processors have gotten a bad rap for squandering ever more energy--indeed, Barroso himself, once a chip designer for Digital Equipment, has expressed such concerns. But chips actually are better than hard drives, memory, and network adapters at reducing power consumption during periods of moderate activity.

Some sophisticated hard drives, for example, can slow down their rotational speed to save power during periods of lower activity. However, "They need to bump to higher RPM to do something useful," to read and write, he said, unlike processors, which can actually still process data when in low-activity modes.

The "slow food" movement came first, followed by "slow work" and even "slow medicine." Next, will people let the Internet relax a little for the sake of ecological sustainability?

Researchers are finding that data centers can make relatively simple power consumption tweaks that mimic those long available for personal computers, as New Scientist reports.

Energy-saving settings take several clicks to set up on Windows or Mac personal computers. But at data centers, where power consumption counts on a grander scale, equipment is often left on even when dormant.

Data centers' emissions of global warming gases exceed those of Argentina and the Netherlands combined, according to an April study by McKinsey & Co. and the Uptime Institute.

However, research from labs at Intel and the University of California at Berkeley has found that network hardware could consume up to 80 percent less energy if allowed to sleep, or if set up for data to travel in clusters rather than in an even flow. Changes to delay the flow of data by milliseconds, not enough for Web surfers to notice, reportedly cut energy use in half.

And in tests with Windows Live Messenger chatting software, Microsoft cut energy use by one-third by clustering active network connections rather than spreading them evenly across servers, noted New Scientist.

Greenhouse gas emissions from data centers rank higher than the countries of Argentina and the Netherlands and right behind airlines, shipyards, and steel plants.

Those comparisons were compiled by consulting firm McKinsey & Company and the Uptime Institute, which on Wednesday published a report on the worsening picture--environmentally and economically--of energy consumption from data centers.

Data centers represent .5 percent of all greenhouse gas emissions. Click on the image for a photo gallery tour of efforts by Silicon Valley data centers to go green.

(Credit: Elsa Wenzel/CNET Networks)

McKinsey called for the creation of a metric that combines the energy efficiency of a data center facility with the utilization of IT gear that runs in data centers. When combined, the financial impact of data centers' energy bill will become more clear and cause people to act, which as a whole, has not been happening.

"Typically the costs are in different silos," said Kenneth Brill, the founder of research firm the Uptime Institute, who worked on the study. "The shocker is when you do put them together, it's clear that people should do something."

Although there have been different industry groups created, such as the Green Grid, on the whole, energy usage at data centers continues to go up.

That's because servers consume more power, the number of servers is increasing, and the cost of electricity is going up, said William Forrest, the associate principal for IT at McKinsey.

Global data center energy consumption doubled from 2000 to 2006. The projected growth in the U.S. between now and 2010 is the equivalent of 10 new power plants.

As a result, companies that are heavily dependent on IT, such as Google, Microsoft, Facebook, and financial services firms, are getting push-back from their boards on proposals to spend hundreds of millions of dollars on new facilities, Forrest said. Those that don't manage energy well will have a bit taken out of their profits.

"The more you look, the more you find that the entirety of the IT stack, from the CPU up, is very inefficient," Forrest said. "Data centers are becoming a major business issue."

Heavy consumers will also come under growing regulatory scrutiny to lower the carbon footprint of their operations, with greenhouse gas emissions projected to quadruple between now and 2020.

"We think it's going to become a regulatory concern that will drive scrutiny not just in (corporate) boardrooms but with regulators as well," said Forrest.

Back to bite IT
There are a number of factors that contribute to runaway energy use and cost--and several steps that operators can take that have more to do with management than technology. Rather than build new "green" data centers, companies should focus on making existing ones run more efficiently.

Data centers are filled with relatively low-cost servers, which are often dedicated to one specific task or application. That simplifies the set-up of applications but is also highly inefficient: a typical server only uses 6 percent of its processing capacity while it continues to draw power.

Historically, the bill for energy falls outside the responsibility of chief information officers or data center operators.

Yet the total cost of operating a server is four to five times more than the purchase price over the 5-to-10-year lifetime of a server, according to the study.

And the cost of managing facilities is going up at 20 percent, higher than IT budgets' projected rise of 6 percent, according to the Uptime Institute.

What to do
McKinsey and the Uptime Institute concluded that users should aim to double energy efficiency by 2012 through more efficient individual components and a comprehensive data center management plan.

They call for companies to appoint an "Energy Czar," much the way a chief security officer manages security across an IT organization.

They also recommend an energy dashboard to give better information on obvious areas to improve efficiency. Cooling, in particular, can be greatly improved with better sealing of cabling and cheaper water-cooling systems.

On the equipment side, IT organizations should use more virtualization, retire old equipment, upgrade some equipment, and adopt a strategy of better matching equipment's processing power with an application's needs.

At an industry level, McKinsey and the Uptime Institute proposed a new financial metric for measuring data center efficiency that combines efficiency of facility operations, such as space cooling, and measures the utilization of IT equipment.

This standard would be the server equivalent to the CAFE (Corporate Average Fuel Economy)--the levels of fuel efficiency the U.S. government mandates that auto manufacturers meet.

"This metric would deliver immediate financial and transparency benefits to executive management of enterprises large and small and could become a government-recognized measure of efficiency," according to the study's summary.