Inside one of the world's largest data centers
CHICAGO--On the outside, Microsoft's massive new data center resembles the other buildings in the industrial area.
Even the inside of the building doesn't look like that much. The ground floor looks like a large indoor parking lot filled with a few parked trailers.
It's what's inside those trailers, though, that is the key to Microsoft's cloud-computing efforts. Each of the shipping containers in the Chicago data center houses anywhere from 1,800 to 2,500 servers, each of which can be serving up e-mail, managing instant messages, or running applications for Microsoft's soon-to-be-launched cloud-based operating system--Windows Azure.
Upstairs, Microsoft has four traditional raised floor server rooms, each roughly 12,000 square feet and consuming, on average, 3 megawatts of power. It's all part of a data center that will eventually occupy 700,000 square feet, making it one of the world's largest.
"I think, I'm not 100 percent sure, but I think this could be the largest data center in the world," said Arne Josefsberg, general manager of infrastructure services for Microsoft's data center operations.
Even with only half the site ready for computers, the center has 30 megawatts of capacity--many times that found in a typical facility.
On a hot day, Microsoft would rely on 7.5 miles worth of chilled water piping to keep things cool, but general manager Kevin Timmons smiled as he walked in for the facility's grand opening in late September. It was around 55 degrees outside.
"When I stepped out, I said 'what good data center weather'," he said. "I knew the chillers were off."
Although Microsoft is open about many of the details of its data centers, there are others it likes to keep quiet, including the site's exact location, the names of its employees, and even which brand of servers fill its racks and containers.
The software maker also won't say exactly which services are running in each facility, but the many Bing posters inside the upstairs server rooms in Chicago offer a pretty good indication of what is going on there.
Microsoft originally intended to open the Chicago facility last year, but the company has slowed its data center pace some amid the weaker economy and an array of cutbacks companywide. Instead, the facility had its grand opening in late September.
Of Sidekick--and Azure
Within a month, though, Microsoft's data centers were attracting attention for a wholly different reason. A massive server failure at an older facility--one that Microsoft acquired as part of its Danger acquisition--left thousands of T-Mobile Sidekick owners without access to their data as part of an outage that is now stretching into its second month.
Although Sidekick uses an entirely different architecture, the failure represented a tangible example of the biggest fear of cloud computing--that one will wake up one day to find their data gone.
Microsoft is quick to highlight the differences between the Sidekick setup and what Microsoft is building in Chicago and elsewhere. "We write multiple replicas of user data to multiple devices so that the data is available in a situation where a single or multiple physical nodes may fail," Windows Azure general manager Doug Hauger said in a statement after the Sidekick failure.
As for Azure, Microsoft is expected to talk about its commercial launch at this month's Professional Developers Conference in Los Angeles, including offering more details on how the system will provide its redundancy. Microsoft has already announced some new Azure details, noting last week that it will begin charging for Azure as of February 1.
Microsoft is still trying to figure out just how much capacity at Chicago and elsewhere it needs to assign for Azure.
"Azure is incredibly hard to forecast," said Josefsberg. "We're probably erring toward having a little more capacity than we need in the short term."
What is clear is that, over time, Microsoft will need even more capacity. That's what has Josefsberg returning to a custom "heat map" that figures out the best place to build data centers based on factors including cheapness, greenness, and availability of power, political climate, weather, networking capacity, and other factors. Choosing the right spot is critical, Microsoft executives say, noting that 70 percent of a data center's economics are determined before a company ever breaks ground.
Josefsberg said he already has the next spot picked out.
"We know exactly where it is going to be but I can't tell you right now," he said.
But Microsoft has indicated how the next generation of data center will improve upon the Chicago design.
Moving to containers allows Microsoft to bring in computing capacity as needed, but still requires the company to build the physical building, power and cooling systems well ahead of time. The company's next generation of data center will allow those things to be built in a modular fashion as well.
"The beauty of that is two-fold," Josefsberg said. "We commit less capital upfront and we can then accommodate the latest in technology along the way."
During her years at CNET News, Ina Fried has changed beats several times, changed genders once, and covered both of the Pirates of Silicon Valley. These days, most of her attention is focused on Microsoft. E-mail Ina. 
It`s sad that all that equipment is needed just so the old guy on the Verizon advert can Twitter
"I`m Twittering on the porch"
Tell me- how exactly does running Unix on a piece of hardware prevent hardware faults, networking hardware faults, or redundant physical backup faults?
Methinks you have overlooked the purpose of a data center with redundancy. It doesn't matter what OS you run if the hardware fails and for that, you need a lot of it.
I suppose if you run a *nix datacenter 1/3 the size, then you just have a much greater chance of physical failure as a a result. Not exactly wise thinking in data management.
Tell me, kids - why does a typical SharePoint install require four machines (CA, Index, NLB, and DB server) just to run the same workload that a single *nix box running Joomla and a bit of Perl can do?
Why does Microsoft's recommendations on a patching server setup (SCCM) sprout into at least 2-3 servers and a license for each one?
When you get done with that, how about telling us how many of you out there actually (and honestly now!) run a Windows Server variant in CLI-only, because I have yet to see one that I haven't built myself. Long story short, those GUIs eat more power than they're worth, and damn if you're stuck with it anyway in far too many cases...
Oh, and Futuredude - Google is a whole lot bigger traffic-wise than Microsoft could ever hope to be at this time; yet it's Microsoft that's bragging on having the most hardware and the biggest DCs. Not exactly a boost to the TCO, is it?
==
"Tell me- how exactly does running Unix on a piece of hardware prevent hardware faults, networking hardware faults, or redundant physical backup faults? "
Ask VMWare - they managed to pack far more servers into far less physical hardware space overall - and guess what they run (hint: Not Windows ;) ).
"I suppose if you run a *nix datacenter 1/3 the size, then you just have a much greater chance of physical failure as a a result"
...and it's exactly that kind of simplistic and ignorance-laden thinking that will keep you on the help desk.
I have 100 users on a P3/512MB server running Linux and Kerio email server and it is FAST. Exchange couldn't run one person on this hardware.
Dude seriously it's 2009, nobody cares what you can run on a 500mhz p3. Resources are cheap.
Kerio MailServer System Requirements:
Windows / Linux
Minimal HW configuration (1-20 users):
CPU 1GHz, 512 MB RAM, 40GB of free space for message store and backup
Recommended HW configuration (20-100 users):
CPU 2GHz, 2GB RAM, 160GB of free space for message store and backup
Recommended HW configuration (100+ users):
CPU 2.8GHz Dual Core, 4 GB RAM, 200GB+ of free space for message store and backup
Source: http://www.kerio.com/mailserver/requirements
So watch yourself next time you call somebody an idiot and change your nickname to just-bs-no-facts.
Please dont bother to reply!
@random
"Why does Microsoft's recommendations on a patching server setup (SCCM) sprout into at least 2-3 servers and a license for each one? "
Ever heard of redundancy? Thats the reason, if you were a sys admin than you should know bout those things.
Thats kinda thing you learn in first class.
DATA LOSS = NO GOOD FOR BUSINESS
"Long story short, those GUIs eat more power than they're worth, and damn if you're stuck with it anyway in far too many cases..."
Are you serious?!??!? Do you really think that start menu and blue background make any preformance impact on a server with 500CPUs????
@rest *nix ppl
You guys think that MS is running default installation of operating system on that machine? Well... its probably extremely customized installation, and I'm almost certain that they build that hardware to fit the software not the other way around.
It would appear that you've been resoundly dismissed as uninformed here.
You didn't realy think that by changing your username here that people would forget your reputation as Penguinisto, did you? You can't hide from your own past just by running away from it.
The smart move is to locating data center modules where low grade heat is desired, for example apartment buildings.
Google seems to be a long way ahead in this because they already have "Spanner" which can instantly shift loads in and out of any given "droplet" of the cloud.
Even smaller data centers should be warming water.
Water, especially fresh water, is such an in demand resource it seems strange people are still using it for mission critical use. Maybe there is no alternative.
With regard to google is farther ahead because they can share load between sites...... they all do that. Nothing new.
they should have at least 2 cooling towers and few chiller water compressors. If 1 breaks down, other(s) will cut on.
even if water company cut the main water line off, there'll be enough water in those cooling towers to cool down the chillers. Water (condenser) is re-usable.
Chicago threatens increased taxes, Microsoft points to containers.
Con-Ed suggests higher energy costs, Microsoft points to nearby freight rail line.
State of Illinois wants to increase business property tax. Microsoft says, "Remember the Baltimore COLTS"....
*eyeroll*
Putting servers in shipping containers is actually a good idea Google came up with: http://www.youtube.com/watch?v=zRwPSFpLX8I
no wondering you're just a labor. if MS is stupid, they wouldn't have 95% of the market :P
it's stupid to relocate IT equipments around. Energy management is the key.
Ah yes, once again you prove your ignorance knows no bounds. :)
Since you've so completely dismissed and denounced the very concept of shipping containers as being effecient ways of moving equipment around, perhaps you could tell the multi-trillion dollar shipping industry this too? They apparently haven't heard your wisdom yet and yet keep raking in the money. Funny how they haven't noticed their failure.
But seriously, read the article. Using containers is smart. How else will you be able to bring online several thousand servers in just a few hours? With redundancy in place, they can have machines go down without taking out the entire system. They can upgrade on the fly. They can replace equipment or even expand as needed. Flexibility is the name of the game and you just lost.
I know you're strongly in the belief that we should all be hunkered down in cement bomb shelters with racks of equipment taking up every possible space, but really, the world moved on without you. Stay with the times.
http://www.foreignaffairs.com/articles/65499/wesley-k-clark-and-peter-l-levin/securing-the-information-highway
BTW, insecurity isn't always about the OS or software.
Probably. IBM also ! The fundamental question is this just the tip of the iceberg for data center market or the iceberg itself...
Just south of the Northlake water tower. Easy to see from the road.
It would be interesting to contrast the 'computing' efficiency of these large data centers by comparing the technologies used by Google and others over a short series articles.
As 'cloud' services continue to drive these large installations, efficient use of local resources becomes critical, not only for the services provider, but for the public at large. Physical limitations exist in every locality regarding electricity, connectivity, the effects of heat exchange etc. These (and many other factors) have an impact within the communities selected.
For example, Google initiated this summit:
http://www.google.com/corporate/green/datacenters/summit.html
And the EPA has this initiative (which may not scale to the size mentioned in the article):
http://www.energystar.gov/index.cfm?c=prod_development.server_efficiency
Obviously, the more efficient the design of hardware and software, the smaller the 'footprint' of the data center...more computing and with less resources.
I'm sure you've already given this some thought, but I hope your able to inform us readers with additional articles on the topic.
Get a tripod please. It's completely unprofessional to see the video shaking when you're sitting interviewing two people from MSFT.
Oh wait, those servers are no doubt running Windows Server 2008 R2, and if you paid attention, as of Windows Server 2008 R2, Windows Server has dropped all support for the x86 architecture.
Redundancy.
When the planar assembly in that mainframe goes down, then the entire system goes down. It doesn't matter how many CPU's you have on it- it's dead. Even Google understands that using smaller lighter multiple systems is more effecient for reliability and redundancy.
Also when it comes time to upgrade the hardware on that mainframe, you have to down the system- that's downtime that is very expensive. The redundant small PC hardware model means the system can stay up while you do repairs/upgrades/etc.
The container idea also allows you to assemble the setup, test it out in a controlled environment and then truck it to your location. All the testing and bug fixing of integration is done ahead of time, minimizing downtime.
- by vabiro November 3, 2009 10:36 AM PST
- Ina,
- Like this Reply to this comment
-
Showing 1 of 2 pages (48 Comments)The video that we see glimpses of in your piece, where could I see the whole video?
Victor