SAN FRANCISCO--The inner workings of Google just became a little less secret.
The search colossus has shed only occasional light on its data center operations, but on Wednesday, Google fellow Jeff Dean turned a spotlight on some parts of the operation. Speaking to an overflowing crowd at the Google I/O conference here on Wednesday, Dean managed simultaneously to demystify Google a little while also showing just how exotic the company's infrastructure really is.
Google fellow Jeff Dean
(Credit: Stephen Shankland/CNET News.com)On the one hand, Google uses more-or-less ordinary servers. Processors, hard drives, memory--you know the drill.
On the other hand, Dean seemingly thinks clusters of 1,800 servers are pretty routine, if not exactly ho-hum. And the software company runs on top of that hardware, enabling a sub-half-second response to an ordinary Google search query that involves 700 to 1,000 servers, is another matter altogether.
Google doesn't reveal exactly how many servers it has, but I'd estimate it's easily in the hundreds of thousands. It puts 40 servers in each rack, Dean said, and by one reckoning, Google has 36 data centers across the globe. With 150 racks per data center, that would mean Google has more than 200,000 servers, and I'd guess it's far beyond that and growing every day.
Regardless of the true numbers, it's fascinating what Google has accomplished, in part by largely ignoring much of the conventional computing industry. Where even massive data centers such as the New York Stock Exchange or airline reservation systems use a lot of mainstream servers and software, Google largely builds its own technology.
I'm sure a number of server companies are sour about it, but Google clearly believes its technological destiny is best left in its own hands. Co-founder Larry Page encourages a "healthy disrespect for the impossible" at Google, according to Marissa Mayer, vice president of search products and user experience, in a speech Thursday.
To operate on Google's scale requires the company to treat each machine as expendable. Server makers pride themselves on their high-end machines' ability to withstand failures, but Google prefers to invest its money in fault-tolerant software.
"Our view is it's better to have twice as much hardware that's not as reliable than half as much that's more reliable," Dean said. "You have to provide reliability on a software level. If you're running 10,000 machines, something is going to die every day."
Breaking in is hard to do
Bringing a new cluster online shows just how fallible hardware is, Dean said.
In each cluster's first year, it's typical that 1,000 individual machine failures will occur; thousands of hard drive failures will occur; one power distribution unit will fail, bringing down 500 to 1,000 machines for about 6 hours; 20 racks will fail, each time causing 40 to 80 machines to vanish from the network; 5 racks will "go wonky," with half their network packets missing in action; and the cluster will have to be rewired once, affecting 5 percent of the machines at any given moment over a 2-day span, Dean said. And there's about a 50 percent chance that the cluster will overheat, taking down most of the servers in less than 5 minutes and taking 1 to 2 days to recover.
A look at a custom-made Google rack with 40 servers from a modern data center. Infrastructure guru Jeff Dean showed the snapshot at the Google I/O conference.
(Credit: Stephen Shankland-CNET News.com/Jeff Dean-Google)While Google uses ordinary hardware components for its servers, it doesn't use conventional packaging. . And, Dean said, the company currently puts a case around each 40-server rack, an in-house design, rather than using the conventional case around each server.
The company has a small number of server configurations, some with a lot of hard drives and some with few, Dean said. And there are some differences at the larger scale, too: "We have heterogeneity across different data centers but not within data centers," he said.
As to the servers themselves, Google likes multicore chips, those with many processing engines on each slice of silicon. Many software companies, accustomed to better performance from ever-faster chip clock speeds, are struggling to adapt to the multicore approach, but it suits Google just fine. The company already had to adapt its technology to an architecture that spanned thousands of computers, so they already have made the jump to parallelism.
"We really, really like multicore machines," Dean said. "To us, multicore machines look like lots of little machines with really good interconnects. They're relatively easy for us to use."
Although Google requires a fast response for search and other services, its parallelism can produce that even if a single sequence of instructions, called a thread, is relatively slow. That's music to the ears of processor designers focusing on multicore and multithreaded models.
"Single-thread performance doesn't matter to us really at all," Dean said. "We have lots of parallelizable problems."
The secret sauce
So how does Google get around all these earthly hardware concerns? With software--and this is where you might think about dusting off your computer science degree.
A Google data center, circa 2000. Note the fan on the floor to cool servers.
(Credit: Stephen Shankland-CNET News.com/Jeff Dean-Google)Dean described three core elements of Google's software: GFS, the Google File System, BigTable, and the MapReduce algorithm. And although Google helps with a lot of open-source software projects that helped the company get its start, these packages remain proprietary except in general terms.
GFS, at the lowest level of the three, stores data across many servers and runs on almost all machines, Dean said. Some incarnations of GFS are file systems "many petabytes in size"--a petabyte being a million gigabytes. There are more than 200 clusters running GFS, and many of these clusters consist of thousands of machines.
GFS stores each chunk of data, typically 64MB in size, on at least three machines called chunkservers; master servers are responsible for backing up data to a new area if a chunkserver failure occurs. "Machine failures are handled entirely by the GFS system, at least at the storage level," Dean said.
To provide some structure to all that data, Google uses BigTable. Commercial databases from companies such as Oracle and IBM don't cut the mustard here. For one thing, they don't operate the scale Google demands, and if they did, they'd be too expensive, Dean said.
BigTable, which Google began designing in 2004, is used in more than 70 Google projects, including Google Maps, Google Earth, Blogger, Google Print, Orkut, and the core search index. The largest BigTable instance manages about 6 petabytes of data spread across thousands of machines, Dean said.
MapReduce, the first version of which Google wrote in 2003, gives the company a way to actually make something useful of its data. For example, MapReduce can find how many times a particular word appears in Google's search index; a list of the Web pages on which a word appears; and the list of all Web sites that link to a particular Web site.
With MapReduce, Google can build an index that shows which Web pages all have the terms "new," "york," and "restaurants"--relatively quickly. "You need to be able to run across thousands of machines in order for it to complete in a reasonable amount of time," Dean said.
The MapReduce software is increasing use within Google. It ran 29,000 jobs in August 2004 and 2.2 million in September 2007. Over that period, the average time to complete a job has dropped from 634 seconds to 395 seconds, while the output of MapReduce tasks has risen from 193 terabytes to 14,018 terabytes, Dean said.
On any given day, Google runs about 100,000 MapReduce jobs; each occupies about 400 servers and takes about 5 to 10 minutes to finish, Dean said.
That's a basis for some interesting math. Assuming the servers do nothing but MapReduce, that each server works on only one job at a time, and that they work around the clock, that means MapReduce occupies about 139,000 servers if the jobs take 5 minutes each. For 7.5-minute jobs, the number increases to 208,000 servers; if the jobs take 10 minutes, it's 278,000 servers.
My calculations could be off base, but even qualitatively, that's enough computing horsepower to make the mind boggle.
Fault-tolerant software
MapReduce, like GFS, is explicitly designed to sidestep server problems.
"When a machine fails, the master knows what task that machine was assigned and will direct the other machines to take up the map task," Dean said. "You can end up losing 100 map tasks, but can have 100 machines pick up those tasks."
The MapReduce reliability was severely tested once during a maintenance operation on one cluster with 1,800 servers. Workers unplugged groups of 80 machines at a time, during which the other 1,720 machines would pick up the slack. "It ran a little slowly, but it all completed," Dean said.
And in a 2004 presentation, Dean said, one system withstood a failure of 1,600 servers in a 1,800-unit cluster.
Next-generation data center to-do list
So all is going swimmingly at Google, right? Perhaps, but the company isn't satisfied and has a long to-do list.
Most companies are trying to figure out how to move jobs gracefully from one server to another, but Google is a few orders of magnitude above that challenge. It wants to be able to move jobs from one data center to another--automatically, at that.
"We want our next-generation infrastructure to be a system that runs across a large fraction of our machines rather than separate instances," Dean said.
Right now some massive file systems have different names--GFS/Oregon and GFS/Atlanta, for example--but they're meant to be copies of each other. "We want a single namespace," he said.
These are tough challenges indeed considering Google's scale. No doubt many smaller companies look enviously upon them.
The data centers used by tech companies to run their Web sites and corporate networks are notorious energy hogs.
The information and communications technology sector currently accounts for about 6 percent of the nation's power consumption, up from about 2 percent to 3 percent in 2000, according to a report in February from the American Council for an Energy-Efficient Economy.
In a report to Congress last August, the Environmental Protection Agency predicted that the amount of power used by U.S. data centers would more than double over the next five years, at a cost of $7.4 billion each year. The EPA also suggested that the nation could save up to $4 billion in energy costs, if it made its data centers more energy-efficient.
Those figures have led many tech giants, such as Microsoft, Google, IBM, and Dell, to get behind efforts to reduce power consumption in data centers. Now IBM is ramping up its business of selling power-saving technologies with new tools designed to track and cap data center energy consumption, including power for air conditioning to cool server computers, according to a report from Reuters. The products were announced at an IBM conference Wednesday in Los Angeles.
IBM is also expanding to 27 countries a program begun last year as part of its Big Green Innovations that lets companies earn and trade certificates awarded for verified energy savings, Reuters reported.
"Energy efficiency has become a critical business metric, like product reliability and customer satisfaction," William Zeitler, head of IBM's systems and technology group, told Reuters. "This is a critically important problem in the industry."
Certainly, Big Blue is landing a lot of the Big Green by helping other companies go green.
The initiative has generated nearly $200 million of technology services contract signings in the first quarter and about $300 million in the fourth, Reuters quoted Chief Financial Officer Mark Loughridge as saying during recent earnings presentations.
A map view of Google's European data centers.
(Credit: Pingdom)Ever wonder where Google's data centers are? Now you can use Google Maps to get a good overview.
Pingdom put together a Google map via Wayfaring based on data center location information from Data Center Knowledge. The map shows general areas, so don't expect to zoom in on a satellite photo.
The result is an artful scattering of 36 pushpins, one for each known data center. Yes, that's a lot of data centers, and Data Center Knowledge has some details on Google renting further capacity, too.
And there are likely more tucked away here and there. "Since Google tends to be quite secretive about their data centers in general, the information we have presented here most likely isn't 100 percent complete," Pingdom said.
Cisco Systems said Monday that it's added to its arsenal of data center technology with a new switch and the purchase of start-up Nuova Systems.
Cisco, which already owned 80 percent of Nuova, worked with the start-up to build the new Nexus 5000. On Monday, Cisco introduced the new switch, and also announced that it has bought the remaining 20 percent of the start-up.
Cisco announced its $70 million investment in Nuova in 2006. The company didn't disclose details of the current buyout. But in April of last year, it expanded its funding agreement and raised the maximum potential payout of the transaction to $678 million.
Cisco will continue to use the Nuova technology to further develop and expand the Nexus product line, a new line of data center switches the company introduced earlier this year. The first product in the line was the Nexus 7000. The Nexus 5000 will provide a smaller, fixed version of the product.
The new Nexus switches combine Ethernet switching, IP routing, storage, and security into a single device. And while these switches don't necessarily take over the functions of servers or storage area devices in the data center, they will allow companies to use their servers and storage devices more efficiently.
Cisco has spent more than two decades building its brand as a switching and routing powerhouse. Now, it is tackling the data center, where it hopes its Nexus products will dominate. These new switches are expected to become the next high-ticket item Cisco can sell to large companies to help fuel the company's growth. Cisco hopes the data center will be worth about $10 billion over the next five years, which means it is a key opportunity for a company that needs to grow at least between 10 percent and 15 percent a year to satisfy Wall Street.
But Cisco's move into the data center could pit it against some of its largest partners, such as IBM, Hewlett-Packard, and EMC. Cisco claims its products complement products from these companies. IBM and Hewlett-Packard are more server-centric. And EMC is more focused on the issue from a storage angle. Meanwhile, Cisco sees the intelligent network, with its Nexus 7000 sitting in the center, as the best answer for virtualizing the data center.
Still, competing with its partners is starting to become a familiar tune at Cisco, which has found itself also competing head-to-head with software giant Microsoft in some areas of its business.
Levanta, a Linux data center automation company that was reborn from the ashes of Linuxcare, has closed its doors.
Levanta, which laid off the bulk of its 20 or so employees on Monday, is now seeking to sell the assets of the company, said a Levanta investor.
"Levanta won't continue as a standalone company, but we hope to find a home for its technology soon," said the investor. "Levanta has the best technology for managing Linux servers...but we couldn't wait for the market to develop. In the U.S., there's only a 10 percent penetration rate for Linux data centers."
The company's closure comes less than six months after it received an $8 million round of venture funding. That funding round was designed to accelerate the company's move into Linux data center automation. Previously, Levanta was focused on Linux management and data virtualization.
Levanta was created in 2004, rising from the demise of Linux operating system services company Linuxcare. The former Linuxcare used to offer a product called Levanta, which was designed to make it easier to run Linux on IBM mainframe computers.
Over the four years that Levanta operated, it raised more than $10 million in venture funding.
And, so, while Levanta as an independent company now ceases to exist, its technology may get its third home soon. Keep an eye out for the phoenix to rise again.
Systems management company BMC Software on Monday said it intends to buy BladeLogic for $28 per share, or about $800 million net of cash acquired.
BladeLogic, which went public last year under the ticker BLOG, makes tools for automating jobs in data centers, such as configuring servers and provisioning storage appliances.
BMC said the software will be added to its existing product portfolio and bring it a "significant, high-growth revenue stream."
The acquisition comes at the tail end of a wave of consolidation in the data center software field, which started out earlier this decade.
With the growing complexity of data centers for running public Web sites or corporations, IT professionals need productivity tools to manage their operations.
Last July, Hewlett-Packard said it would spend $1.6 billion to buy data center automation firm Opsware, a company founded by Marc Andreesen. Several other smaller companies with niche tools have been bought by other hardware providers, including IBM, Sun Microsystems, and EMC.
Update: In an interview, BMC CEO Bob Beauchamp and BladeLogic CEO Dev Ittycheria said the combination of the two companies will create an integrated suite of products that will allow IT professionals to more quickly install and change applications.
Beauchamp said the vision is to allow people to provision applications in minutes, a task that sometimes can take months.
"The ability to have this closed-loop model for break/fix and change reconciliation is a powerful concept," Ittycheria said.
Remember the panic when the first computer worm hit? We're going to have a crisis like that next year when we get the first data center meltdown, predicted Subodh Bapat, a vice president in the eco-computing team at Sun Microsystems.
"You'll see a massive failure in a year," Bapat said at a dinner with reporters on Monday. "We are going to see a data center failure of that scale."
"That scale" referred to the problems caused by the worm created by Cornell grad student Robert Morris Jr. in 1988. His worm infected about 5 percent of the Unix boxes on the Internet, freaked people out, and helped jump-start the security industry.
Of course, it's just a prediction, so there is no guarantee that it will happen. But it does seem possible. Data centers have mushroomed with the flood of processes and jobs being turned over to the Internet. Companies have built up their data centers, but even with technologies like virtualization it's been tough to keep up. At some point, a data center is going to crash and people are going to go spastic.
On a more cheery note, Bapat and other Sun executives said that the IT industry is also on the verge of a construction boom that, if it happens, will lead to big orders for equipment for makers of servers, storage systems, and other data center equipment.
The typical life span of a data center is only about 10 to 12 years, said the Sun executives. Thus, a lot of those data centers built at the beginning of the dot-com era need to be rebuilt. Other companies like Facebook are expanding rapidly as well. (Sun CTO Greg Papadopoulos mentioned Facebook several times, so it sounds like maybe Sun is working with, or trying to work with, them. Just a thought.)
National labs and universities are also looking at new centers. Next year, one of the national labs has plans to build a data center that will take up 500,000 square feet and consume 50 megawatts. (Big data centers now take up 400,000 square feet and chew up 40 megawatts, Sun executives said.)
Other organizations are looking at 50-megawatt to 70-megawatt data centers.
Though beaten to the punch by the FCC, Dell introduced the world to its forthcoming all-in-one PC at Oracle OpenWorld in San Francisco Wednesday.
Well, it wasn't a formal introduction as much as an all-too-brief glimpse of the XPS One A2010, which resembles a wide-screen TV with separate keyboard. Chief Executive Michael Dell drew the crowd's attention to the all-in-one during his keynote when he asked Chief Technology Officer Kevin Kettler, who joined him on stage, what "that" device was. Playing along, Kettler "guessed" that it looked like a TV, since it had a remote and was playing a Blu-ray disc.
The Dell XPS One leaked on the FCC site last month.
(Credit: Dell)"It looks like an all-in-one machine. But I probably shouldn't say anymore about that," Kettler concluded, tongue firmly in cheek. He did mention that it will be announced "next week."
Though an enterprise software conference might seem an odd place to introduce a firmly consumer-oriented product, Dell certainly wasn't the only one to deviate from standard OOW content. And it didn't stop with the XPS One. While on stage, Dell took the opportunity to demonstrate its forthcoming convertible tablet PC, the Latitude XT (click here for video), which Kettler said is on track to ship in the next few months. And for good measure, Kettler briefly noted how the XPS M1330 laptop he was holding would be a great holiday gift for the whole family.
It wasn't all a consumer hardware commercial, however. Dell also talked up on-demand desktop streaming as an alternative to thin-client computing. On-demand streaming features a desktop client with its own CPU and graphics processor. Saying it had comparable security and costs to thin clients, and the performance of traditional clients, Dell said desktop streaming would allow IT departments to push software updates instantly to all stations and allow for better video playback.
And it wouldn't be a Dell keynote for the company's green initiative. Dell again challenged the industry to join his efforts in the greening of the IT industry. Citing a Gartner report predicting widespread data center brownouts several years from now, Dell called it "absolutely unacceptable."
Without revealing many details, he said that Dell will soon roll out a program called Greenprint, which allows companies to check their power efficiency and then enable them to find ways to make themselves even more "green."
He ended on the note challenging other companies to "consider the impact green technologies can have on (return on investment) and on our planet."
SAN FRANCISCO--For the record, Hewlett-Packard CEO Mark Hurd would not let Oracle grand poobah Larry Ellison beat him at tennis.
These and other important questions were answered in a more conversational approach to the afternoon keynote slot at this year's Oracle OpenWorld here. In lieu of a traditional speech, Hurd, chairman and chief of the world's largest PC company, answered previously videotaped questions--most of which seemed canned--about a variety of topics.
The questions, posed by ostensibly random conference attendees, ranged from the obvious ("Should I buy an HP or a Dell?"), to the unnecessary ("Would you let Ellison beat you at tennis?"), to the mundane ("How much was Jerry Seinfeld paid to do the HP 'hands' commercial?").
Video: Hurd on
infrastructure
And what did we learn? Hurd says to buy an HP (duh.); no, he played tennis at the college level and would never let anyone win on purpose; and nothing--Seinfeld's HP bit coincided with his recent movie from DreamWorks, with whom HP is partners.
But wait, what about software? Hurd says HP remains "heavily focused" on the management end of software--servers, systems, network, storage. "We're now the sixth biggest software company in the world," Hurd said. "We're different from other companies because we're very focused on that one category (management), and we want to lead in that category."
He was also asked to switch into clairvoyant mode and predict which tech companies will still be independent, major players three years from now. Though reticent to name names, his answer was essentially "not many." Noting the day's news of IBM's acquisition of Cognos, he predicted an increased rate of consolidation in the industry, and "more vertical integration, not less." It will all come down to the bottom line, he predicted.
"In the end, math wins. When you look at the math right now, in the tech industry, there are only a handful of players that have in excess of $100 billion worth of cash," he said. The better question, he said, is how fast that consolidation will occur. Though it's still unclear, "if the right combination of companies were to occur, it could accelerate" in the next two or three years.
SAN FRANCISCO--CORRECTION: Right now, if you buy an energy-efficient dishwasher, utilities like PG&E will give you a cash rebate.
They may do the same for energy-efficient PCs.
The Climate Savers Computing Initiative, a consortium of tech companies that is trying to get the industry and consumers to adopt energy-efficient components, has started to explore the idea of direct rebates with utilities, said Erik Teetzel, a program manager at Google during a meeting at the Intel Developer Forum taking place in San Francisco. Google and Intel are the driving forces behind Climate Savers. (correction: we spelled his name wrong. He wrote it on a business card.)
Some component makers already qualify for rebates from utilities. The payments, though, go directly to these manufacturers. Consumers might enjoy lower retail prices as a result, but it's not the same as getting a check in the mail.
Climate savers earlier this year touted a supposedly more efficient power supply that it is promoting to PC and server makers. Its next initiative will be to persuade IT managers and consumers to adopt the power saving settings on their computers.
