Beyond Binary

LOS ANGELES--During Tuesday's keynote speech, Ray Ozzie outlined how Windows Azure works from a software perspective.

Across the Los Angeles Convention Center, though, developers had a chance to see just what Azure is running on. Microsoft uprooted one of its containers from its Washington data center and brought it to the Professional Developers Conference.

The container was one of the more popular attractions on the PDC show floor as attendees had a chance to peek in and even step inside the container.

It is Microsoft's fourth generation of data center design----newer even than the containers used at the recently opened Chicago data center, which CNET toured earlier this year.

It's about half as long as the containers in Chicago and holds hundreds rather than thousands of servers. On the other hand, it has its own cooling system built in and can operate in a much wider range of climates. It can operate with at a temperature of anywhere from 50 to 95 degrees Fahrenheit and anywhere from 20 percent to 80 percent relative humidity. That--combined with its rugged design--means the fourth-generation units can literally be run outdoors.

The units still require power and high-speed networking, of course, as well as water. However, they use only two to three gallons of water per minute as opposed to hundreds of gallons of water for some other designs.

The public display also allowed a chance to talk about some details Microsoft generally prefers not to talk about--such as whose servers are used. The unit on display at PDC, for example, was running Dell boxes.

The goal of the fourth-generation devices is to further reduce the amount of lead time Microsoft needs to add capacity--from a matter of months if it has to build a new data center wing to as little as six weeks to equip and install a new self-contained unit.

A PDC peek at a Microsoft server container (photos)

CHICAGO--On the outside, Microsoft's massive new data center resembles the other buildings in the industrial area.

Even the inside of the building doesn't look like that much. The ground floor looks like a large indoor parking lot filled with a few parked trailers.

It's what's inside those trailers, though, that is the key to Microsoft's cloud-computing efforts. Each of the shipping containers in the Chicago data center houses anywhere from 1,800 to 2,500 servers, each of which can be serving up e-mail, managing instant messages, or running applications for Microsoft's soon-to-be-launched cloud-based operating system--Windows Azure.

Upstairs, Microsoft has four traditional raised floor server rooms, each roughly 12,000 square feet and consuming, on average, 3 megawatts of power. It's all part of a data center that will eventually occupy 700,000 square feet, making it one of the world's largest.

"I think, I'm not 100 percent sure, but I think this could be the largest data center in the world," said Arne Josefsberg, general manager of infrastructure services for Microsoft's data center operations.

Even with only half the site ready for computers, the center has 30 megawatts of capacity--many times that found in a typical facility.

On a hot day, Microsoft would rely on 7.5 miles worth of chilled water piping to keep things cool, but general manager Kevin Timmons smiled as he walked in for the facility's grand opening in late September. It was around 55 degrees outside.

"When I stepped out, I said 'what good data center weather'," he said. "I knew the chillers were off."

Although Microsoft is open about many of the details of its data centers, there are others it likes to keep quiet, including the site's exact location, the names of its employees, and even which brand of servers fill its racks and containers.

The software maker also won't say exactly which services are running in each facility, but the many Bing posters inside the upstairs server rooms in Chicago offer a pretty good indication of what is going on there.

Microsoft originally intended to open the Chicago facility last year, but the company has slowed its data center pace some amid the weaker economy and an array of cutbacks companywide. Instead, the facility had its grand opening in late September.

Of Sidekick--and Azure
Within a month, though, Microsoft's data centers were attracting attention for a wholly different reason. A massive server failure at an older facility--one that Microsoft acquired as part of its Danger acquisition--left thousands of T-Mobile Sidekick owners without access to their data as part of an outage that is now stretching into its second month.

Although Sidekick uses an entirely different architecture, the failure represented a tangible example of the biggest fear of cloud computing--that one will wake up one day to find their data gone.

Photos: Inside a Microsoft data center

Microsoft is quick to highlight the differences between the Sidekick setup and what Microsoft is building in Chicago and elsewhere. "We write multiple replicas of user data to multiple devices so that the data is available in a situation where a single or multiple physical nodes may fail," Windows Azure general manager Doug Hauger said in a statement after the Sidekick failure.

As for Azure, Microsoft is expected to talk about its commercial launch at this month's Professional Developers Conference in Los Angeles, including offering more details on how the system will provide its redundancy. Microsoft has already announced some new Azure details, noting last week that it will begin charging for Azure as of February 1.

Microsoft is still trying to figure out just how much capacity at Chicago and elsewhere it needs to assign for Azure.

"Azure is incredibly hard to forecast," said Josefsberg. "We're probably erring toward having a little more capacity than we need in the short term."

What is clear is that, over time, Microsoft will need even more capacity. That's what has Josefsberg returning to a custom "heat map" that figures out the best place to build data centers based on factors including cheapness, greenness, and availability of power, political climate, weather, networking capacity, and other factors. Choosing the right spot is critical, Microsoft executives say, noting that 70 percent of a data center's economics are determined before a company ever breaks ground.

Josefsberg said he already has the next spot picked out.

"We know exactly where it is going to be but I can't tell you right now," he said.

But Microsoft has indicated how the next generation of data center will improve upon the Chicago design.

Moving to containers allows Microsoft to bring in computing capacity as needed, but still requires the company to build the physical building, power and cooling systems well ahead of time. The company's next generation of data center will allow those things to be built in a modular fashion as well.

"The beauty of that is two-fold," Josefsberg said. "We commit less capital upfront and we can then accommodate the latest in technology along the way."

Microsoft's Chicago data center offers a merge of old and new techniques. The ground floor features sealed containers with tightly packed racks of servers, while the second floor houses more traditional server rooms.

(Credit: Microsoft)

CHICAGO--On most days it takes the right access badge and a biometric scan to make it inside the doors of Microsoft's massive data center. But on Wednesday, the company allowed a group of reporters, customers, and partners to tour the 700,000 square foot facility.

The data center, along with another just-opened facility in Dublin, Ireland and existing centers in San Antonio and Quincy, Wash., serve as the guts behind Microsoft's online ambitions, from Bing to Hotmail to Windows Azure.

But, for all its strategic import, the ground floor of the Chicago plant looks more like a truck parking lot than a traditional data center. In each parking spot, though, Microsoft can drop off a container packed with up to 2,000 servers.

Right now, only about a dozen of the 56 container spots are filled, but Microsoft executives said they expect that to change quickly. The software maker expects to eventually spend up to $500 million filling up the Chicago site with gear.

The site was originally slated to open months earlier, but Microsoft delayed things due to the economy. Eventually, though, it decided to move forward.

"Investing in these uncertain economic times is always a tough choice," said Arne Josefsberg, general manager of infrastructure services Microsoft's data center operations. But, he added, "We take a very long-term approach to the business.

The data center itself is housed in an unmarked warehouse in one of the Chicago area's many industrial districts. (The software maker didn't want the exact location disclosed.)

Microsoft picked the spot because of its convenient spot close to cheap and abundant power as well as the fact it sits atop a major Internet connection point that houses major east-west and north-south fiber routes.

"It's a lot about location, location, location," Josefsberg said.

I'll have a ton more to say in a follow-up post, including a bunch more pictures and some video interviews, but I wanted to share a few initial thoughts before hopping a plane to the Seattle area, where I will be working for the rest of the week.

Debra Chrapaty, who has been leading Microsoft's data center charge, is reportedly heading to a product role at Cisco.

(Credit: Dan Farber)

One of the top executives in charge of Microsoft's data center operations is leaving the company for a product role at Cisco, according to a report on All Things D.

Debra Chrapaty, who as vice president of global foundation services is among a small number of women in Microsoft's top technology ranks, is said to be taking a role at Cisco heading collaboration products, according to All Things D. A Microsoft representative declined comment, while Chrapaty and a Cisco representative were not immediately available for comment.

According to the leadership pages on Microsoft's corporate Web site, just 14 of the company's 131 top executives are women, including Chrapaty.

Before joining Microsoft, Chrapaty worked at ETrade, AllBusiness, and the National Basketball Association, where she served as chief technology officer.

Chrapaty was among the executives who spoke at Microsoft's recent financial analysts meeting.

There has been some changeover within the leadership in the data center unit of late. Microsoft nabbed a Yahoo executive, Dayne Sampson, back in March, while general manager Mike Manos left the company back in April to join Digital Realty Trust.

Update 4:05 p.m. PT: A Microsoft representative confirmed Chrapaty is leaving Microsoft for Cisco, but was not immediately able to provide further details.

The fallout from Thursday's cost-cutting moves by Microsoft continues. On Friday, Microsoft announced that it is delaying its plans for a massive data center to be built in West Des Moines, Iowa.

It is an abrupt reversal for Microsoft, which announced its plans for Iowa just five months ago as part of its massive push into the data center business. By October, though, the software maker was already detailing plans to cut its data center spending by $300 million. Now Microsoft is looking to cut things even further.

"We are still continuing construction of our facilities in Chicago and Dublin, and are planning to open them as customer demand warrants, Microsoft's Mike Manos and Arne Josefsberg said in a blog posting on Friday. "But given the current economic climate we're going to do the right thing for our business and shareholders and revisit our plans on a quarter-by-quarter basis."

Microsoft tried to put its best spin on the move, saying that long-term the cash crunch will help drive customers to turn to Microsoft rather than building out pricey data centers themselves. Josefsberg and Manos also said that Microsoft will find less expensive ways to have the capacity it needs to run Windows Live Hotmail, Windows Live Messenger, and all of Microsoft's other online services.

"We've been preparing for lean times for a while," the pair said. "This recession is the ideal backdrop to implement small changes that target big needs. Frugality drives innovation, and limited resources are just another forcing function to develop creative solutions to infrastructure needs. For our industry, this means more reasons to identify the small tweaks to products or operational approaches that can unlock big opportunities."

An aerial view of the site in West Des Moines, Iowa, where Microsoft is now delaying plans to build a massive data center.

(Credit: Microsoft)

When it comes to picking a spot for a data center, Google and Microsoft often have the same sites in mind. But when it comes to how they build, the two companies take far different approaches.

Google relies largely on its own design expertise, contracting for and building its own server designs. Microsoft, meanwhile, relies on outside companies to build the hardware, though it certainly takes an active role in designing the centers themselves.

A custom Dell server known as Xanadu built for an unnamed data center customer.

(Credit: Dell)

Dell is one of the companies that helps power Microsoft's server farms, including the ones that power Microsoft's operating system in the cloud, Windows Azure.

Data centers have been a bright spot for Dell, which has struggled in recent years. On its own, Dell's data center business would be a top 5 server vendor, said Forrest Norrod, the Dell vice president who heads its data center effort. In its most recent earnings conference call, CEO Michael Dell called out Facebook, Microsoft, Amazon.com, Akamai, and Baidu as key customers in that arena.

The story of how Dell got into the data center business is an interesting one. A couple of years back, the company was noticing that its share of the largest data centers was less than it might expect. Plus, it noticed that whether it won or lost a bid, the terms tended to be such that Dell didn't stand to make money.

"That was curious," Norrod said.

It turned out that customers at the highest end didn't really need some of the hallmarks of Dell's servers. Built-in management code and redundancy might appeal to the average business, but to a customer that expects to burn through their servers, such features are costly and unnecessary.

Meanwhile, other features like extreme power efficiency and density were the things that companies would pay a premium to get.

Dell data center chief Forrest Norrod

(Credit: Dan Farber/CNET News )

At the beginning of 2007, Dell set up a separate unit to explore this area. Its mission was to look at the 50 biggest data center customers worldwide and work to understand what those companies needed.

It turns out there are a lot of things you don't need when building a server that is going to go in one of these data centers. For example, expansion ports are definitely out, as are legacy I/O ports. Memory slots need to be limited to the minimum necessary (and then kept full so they don't change the thermodynamics).

Instead of redundant fans or power supplies, Norrod said, Dell learned what customers really needed was one good one, since a system wasn't likely to be touched until it failed, at which time it would be replaced.

Norrod said that Dell has learned a lot by working with Microsoft, including the need to start a system's design by knowing where the server is going. And that is a rapidly changing environment as servers move from racks, to pre-equipped containers and even to entire prefabricated data centers.

"Whether the room is a room or a container, looking at the environment is one of the key parameters in system design," Norrod said.

And companies like Microsoft are looking to get rid of anything they can.

"They want computers and servers sitting in a field with a power cord and network cord going to them. Period," Norrod said. "Everything else is overhead--the building, the rest of the infrastructure. We're at the end game of how close can you get to that. "

Microsoft announced on Tuesday that its so-called Gen 4 data centers will consist of prefabricated buildings that can be up and running in three to six months.

Microsoft is taking its container approach to data centers a step further, making the building housing the data center itself a module.

In a blog posting on Tuesday, Microsoft detailed what the "generation four" data centers will look like.

"This is a significant step forward, and one that Microsoft believes will reshape how companies build data centers and support cloud computing," a Microsoft representative said in a statement.

The generation four concept "builds on the innovation at Microsoft's Chicago data center, which houses shipping containers packed with up to 2,500 servers each," the representative said. "A container facility helps ensure that we don't overbuild server capacity, while allowing the company to reduce the time to build a data center from 24 to 12 months."

The new approach goes a step further, building the center itself out of prefabricated mechanical, electrical, and security components, as well as the containerized servers. Such facilities can be deployed in just three to six months and expanded when demand grows.

Microsoft says the new approach will cut capital costs by 20 percent to 40 percent.

"In short, we are striving to bring Henry Ford's Model T factory to the data center," Microsoft's Mike Manos said in the blog posting. "We intend to have our components built in factories and then assemble them in one location (the data center site) very quickly. Think about how a computer, car or plane is built today."

For those that are in to data centers, check out the blog. it goes into just a ton of detail. I've also embedded a video that Microsoft did.

Meanwhile, I'm working on some separate posts for this week on some of the vendors that are powering Microsoft's efforts.

Microsoft on Monday officially opened its San Antonio, Texas, data center, the latest in a string of giant facilities aimed at powering Microsoft consumer and business online services.

The company said the data center occupies nearly half a million square feet and cost $550 million to establish.

Microsoft touted some of the environmental features of the facility, including the fact that it is using 8 million gallons of recycled water per month as part of its cooling system.

"Microsoft looked at 31 variables in narrowing its site selection to San Antonio, including the availability of fiber-optic networks, affordable energy rates, and a work-life balance for our employees that the city offered," Microsoft General Manager Mike Manos said in a statement.

The facility joins other Microsoft data centers, including one in Quincy, Wash. Another site, due to open later this year in the Chicago area, will be Microsoft's first to employ containers of servers, in addition to traditional rack-based set-ups.

Microsoft also recently announced plans to build a center in West Des Moines, Iowa.

Correction, 9:31 a.m. PDT: This story cited the wrong state for Microsoft's Quincy data center. That facility is located in Quincy, Wash.

Once upon a time, Microsoft used to fill its data centers one server at a time. Then it bought them by the rack. Now it's preparing to load up servers by the shipping container.

Starting with a Chicago-area facility due to open later this year, Microsoft will use an approach in which servers arrive at the data center in a sealed container, already networked together and ready to go. The container itself is then hooked up to power, networking, and air conditioning.

"The trucks back 'em in, rack 'em, and stack 'em," Chief Software Architect Ray Ozzie told CNET News. And the containers remain sealed, Ozzie said. Once a certain number of servers in the container have failed, it will be pulled out and sent back to the manufacturer and a new container loaded in.

It's just one way that Microsoft is trying to cope in a world where it adds roughly 10,000 servers a month.

"You contain your infrastructure but you also contain the heat that's generated from the servers," Arne Josefsberg, Microsoft's general manager of infrastructure, said in an interview this week. "We are working incredibly hard to improve the energy efficiency of our data centers."

Only a couple of years ago, Microsoft was adding capacity one server at a time, adding individual servers to racks and taking a couple of hours to wire in each new server.

"That's way too expensive, way too slow," said Josefsberg.

An aerial view of the San Antonio, Texas, data center under construction.

(Credit: Microsoft)

Microsoft also used to lease much of its space, until it realized that data centers were going to be a very big part of its future as more and more software moved into the cloud. A couple of years back, though, it found itself running tight on capacity and bought two San Francisco Bay Area data centers in which it had been leasing space.

Over the past 18 months, though, Microsoft has been on a buying--and building--spree. The company has opened a data center in Quincy, Wash., and will open the Chicago facility, as well as another in San Antonio, Texas, later this year. A facility is due to open in Dublin, Ireland next year.

Microsoft is close to announcing yet another data center, Josefsberg said. The software maker also has signed a memorandum of understanding to build a data center in Russia.

No more off-the-shelf hardware
Gone are the days in which Microsoft settled for off-the-shelf hardware to fill its server farms. These days, Microsoft is looking for servers designed to its exact needs. It's not just that Microsoft doesn't want servers that have keyboard or USB ports--it wants motherboards that don't even have the added wiring necessary to support those things that it will never use. Such moves eliminate cost, space, and power consumption.

"We are not physically building our servers, but there is very deep engagement (with the computer makers)," Josefsberg said.

Even a 1 percent or 2 percent reduction in power consumption makes a big difference, Josefsberg said. As it is, Microsoft is trying to cram a whole lot of gear into a small space. While server racks at a Web hosting facility might have power densities of 70 watts to 100 watts per square foot, things are packed far more tightly in the containers, which might be consuming in the thousands of watts of power per square foot.

The container approach is easiest to implement on the ground floor of a facility. In Chicago, for example, it will use containers on the first floor and more traditional racks on the second level. But Josefsberg said that, though it poses some logistical challenges, the company is also considering using multiple levels of containers at other sites, including at a Dublin, Ireland data center due to open next year.

What the servers are serving up
So just what are all these data centers doing? Outsiders got a glimpse into this thanks to a slide included as part of a video that Microsoft put on its Web site touting its environmental efforts. The chart shows search accounting for the vast number of the servers--nearly 80,000 or so--with Hotmail and Messenger distant runners-up in terms of server usage.

Josefsberg said the figures were accurate, but out of date, reflecting where things were at a year or 18 months ago.

"Search was a very large portion of our demand in fiscal year 2008," he said. "Going into this year it is still a very large proportion. It is now not as dominant as it was last year."

A conceptual rendering of the planned data center due open next year in Dublin.

(Credit: Microsoft)

Microsoft is seeing new demands, he said, such things like consumer video and photo services as well as its collection of hosted enterprise services under the Microsoft Online moniker.

Josefsberg said his goal is to keep capacity a certain number of months ahead of where Microsoft's utilization is running. To do that, he said, takes some serious planning. Business unit heads who used to have to just create a forecast for revenue and headcount, now need to be able to predict how much server capacity they will need, or at least give Josefsberg the data he needs to make such calculations.

He points to things like Microsoft's work with the Olympics as indicative of the kinds of demands his data centers will see in years to come.

"One of the big drivers for us that I see is the move to IP-based delivery of rich video," Josefsberg said.

And not all of his problems will be solved just because Microsoft can now get its servers by the containerful. Microsoft has sophisticated "heat maps" that plot the best locations for new data centers based on everything from government policy to water supply to power prices. But in other areas, such as networking technology, Microsoft is counting on the industry to make some quantum leaps.

"When you think about large-scale data centers there are a number of limitations in the technology," he said. "Some of the network protocols were designed years ago...Some are 30 years old."