The Pervasive Data Center

SAN FRANCISCO--General manager of Intel Architecture Group Sean Maloney's announcement of a reference design for a "micro server" during his Tuesday afternoon keynote at the Intel Developer Forum brought me a sense of deja vu.

Intel's Sean Maloney holding microserver.

(Credit: Intel)

He disclosed "a new ultra-low-voltage Intel Xeon 3000 series processor featuring a TDP (Thermal Design Power) of only 30 watts. To complement the broad range of dense and power-optimized platform offerings, Intel also demonstrated publicly for the first time a single-socket 'micro server' reference system which will help enable micro server innovation and future specification." Intel plans to ship the 30-watt dual-core chip in Q1 on 2010; a 45-watt quad-core version is set to ship immediately.

A reference system is primarily intended to demonstrate a concept. It provides a hands-on experience for partners and customers and therefore an opportunity to experiment with and fine-tune the basic approach. The microserver reference design will accommodate 16 server modules in a 5U-high (8.75-inch) chassis. The server boards are approximately 8-inches by 4.5-inches.

Jason Waxman, the general manager of high density computing at Intel, told me that they see the primary target for this class of system as "hosting companies that do a lot of white boxes." White boxes are systems that are often assembled in-house from component parts such as motherboards and cases. Waxman added that such companies nonetheless want many of the features associated with servers--such as memory with error correcting code (ECC).

In Intel's view of the world, microservers very much target service providers and companies that host busy Web sites and otherwise are associated with high-scale network computing. It sees this market as distinct from large high-performance computing (HPC) installations. Vendors such as HP tend to treat network computing and HPC as more of an overlapping customer group.

My deja vu when it comes to microservers relates to the fact that we've seen them before. They used to be called blades.

That's not to say that blade servers don't already exist today, but they've largely evolved into a much different concept from how they were initially conceived. The blades sold today by the likes of Cisco, Dell, HP, and IBM are about virtualization and integration. They pull together computing, networking, and storage and tightly integrate them both physically and through software. They are, in a sense, a form of scale-out consolidation.

Sun has largely eschewed this integration with their blade product line. However, Sun blades are heavily focused on high-performance computing--even to the point of integrating the HPC-centric InfiniBand interconnect on some of its products.

Rather, microservers hark back to the days of RLX Technologies, the company that did the most to promote blade servers during the Internet boom of circa 2000. Microservers are simply thin servers--compact, cheap, and simple. They provide cable simplification. They let hosting providers allocate low-cost physical servers to customers who don't want to share using virtualization.

Microservers bring blades back to their roots. Everything old is new again.

One of the sure signs that a new technology is having some real impact on the industry, as a whole, is when it starts changing other technologies, products, and processes that touch it. In part, this is a simple reflection that a vendor or a small group of vendors aren't the only ones who care about their new shiny-ness. Press releases, consortia, and partnerships are all well and good. But the real proof of acceptance is when other companies and customers start spending real money and changing their own plans and products.

We're seeing this happening with server virtualization. IT shops have started to rethink the processes that they use to allocate new computing resources to users. Some at the forefront have even made virtual servers, rather than physical ones, their default unit of computing.

We're also seeing changes in the way that servers are designed and built. In the x86 processor world, Intel VT and AMD-V attacked some of the most fundamental difficulties of virtualizing x86 hardware. Both companies are continuing to introduce hardware virtualization enablers to address things like I/O performance, virtualization's handling of memory, and compatibility of virtual machines across multiple generations of hardware.

We're also seeing changes in the way that servers are designed and built. Fundamentally, the issue is this. Server virtualization's first big win was in providing a path to consolidate x86 servers that were otherwise very lightly utilized--5 percent or below in many cases. Virtualization can up that figure closer to 50 percent.

So how does this change server design? Servers are designed and configured to be "balanced." This means that processing speed, memory performance and capacity, and I/O ideally don't limit each other. (In practice, fundamental technology limits dictate certain inequalities, but the system designer's job is to work around these as much as possible.) Consider if you put the latest quad-core screamer in a PC configured with only 256MB of memory and a slow serial port coming out the back. It wouldn't run most applications well--however speedy the processor.

Virtualization doesn't actually make the processor faster. But it does tend to make the processor do more work and thereby makes the other system components do more work as well. In practice this means that virtualized servers need correspondingly more memory and more network connections. And that's exactly the sort of thing that we're seeing.

To pick just one recent announcement, consider this September 10 press release from Dell:

The PowerEdge M905 delivers the ultimate four-socket blade-based virtualization performance and is the first blade server to support 11 tiles and 66 Virtual Machines (VM) in VMmark testing. The PowerEdge M805 delivers the same number of DIMM slots in a two-socket blade that requires a four-socket blade from either HP or IBM. With a choice of hypervisors including Citrix XenServer, VMware, and now Microsoft Hyper-V, PowerEdge servers can deliver the optimal platform for virtualized environments. The Dell PowerEdge M805 and M905 servers are now available worldwide with a starting price of $1,699 and $4,999 USD respectively...In addition to the new servers, Dell announced full, high-speed 10Gb Ethernet and 8Gb Fibre Channel switches and mezzanine cards designed to provide customers increased bandwidth and performance.

I'll just note a few things here:

• More memory in servers is all the rage. That's because there's a fairly strong correlation between how much work a processor is doing and how much memory it needs to store the associated data and instructions. Memory requirements have been going up forever, of course, but server virtualization has accelerated the process.
• 10Gb Ethernet may not yet be needed for many single workloads--especially in the volume server world. But as a pipe for an aggregated group of virtual machines? It's not mainstream yet, but it's clearly an early use case for high-bandwidth networking on x86.
• Finally, observe that even performance claims are couched in virtualization terms. Sure, this is partly about using virtualization to lend a little dazzle to what might otherwise be taken as a just-another-server announcement. But the fact remains that system performance running a mix of workloads is increasingly a more important metric than how fast a system runs a single database application.

Enough proof that server virtualization is starting to change everything (or at least an awful lot) about the data center?

Over the past year or so, IBM has been revamping its Systems and Technology Group (STG) organization in a major way.

We see those changes reflected in a major way with IBM's Power systems announcement Wednesday at its COMMON User Group Conference in Nashville.

Two aspects of the STG reorg are of particular interest here.

The first is the customer aspect. This announcement reflects its venue; COMMON is IBM's midrange user group--which at IBM historically more or less equated to System i (and its iSeries and AS/400 predecessors). However, this announcement pulls in multiple product threads--including blades. This reflects how the client-facing part of the new STG organization now breaks down by customer type, rather than technology base. STG's Business Systems Group (BSG) is chartered with selling to the midmarket--across product groups. This is essentially a return to the older IBM sales model that was subsequently replaced by a more specialist-led approach.

The announcement also reflects changes to the product side of the reorganization. Looking back, System i and System p (to use the product line names in use prior to this announcement), sprang from wholly different roots. System i, long known as the AS/400 (although its lineage actually goes back further to the System/36 and System/38), was long an independent thread of IBM systems development based in Rochester, Minn.

Midwinter trips to AS/400 headquarters were not eagerly sought! It was a competitor to low-end and midrange minicomputers from the likes of Digital Equipment and Wang Labs.

System p, on the other hand, was long called the RS/6000 and had its home base in Austin, Texas. Its competition was other RISC-based servers running Unix such as those from Hewlett-Packard and Sun Microsystems. Especially given that the "old IBM's" product divisions could be best described (however uncharitably) as warring fiefdoms, there was little sharing of technology or anything else between them.

However, IBM has been steadily tearing down the wall between the two lines. Both i and p have used the same Power-family processor for several years now. Still, this week's announcement represents the first time that the wall is truly gone. System i is System p and vice versa. They're now both Power systems.

What this means is that there's now one common set of system models that can run AIX, i, or Linux operating systems--or a combination thereof using the integrated server virtualization features that fall under the PowerVM umbrella. The specific server models covered in this announcement are:

• IBM Power 520 Express is an entry-level server with up to four Power6 cores. The Power 520 Express is available in AIX, Linux, and i editions.

• IBM Power 550 Express is a midrange server with up to eight Power6 cores. The Power 550 is also available in AIX, Linux, and i editions.

There's also an i Edition Express for BladeCenter S. This basically backfills i support to previously announced Power blades in the SMB-oriented version of its BladeCenter and also adds the JS12, a new single-socket blade.

This is both a major midrange product announcement and the final (or, at least, as final as such things ever are) coming together of a complex organizational and product integration task that's been going on for years.

With the ProLiant DL785 G5 Server, Hewlett-Packard has re-entered the 8-socket x86 server space. This system has twice the computing headroom of the quad-processor servers that are generally considered at the top end of the volume or so-called commodity server space.

HP isn't new to this market segment. In 1997, Intel bought a company by the name of Corollary that was in the process of developing a chipset that effectively "glued together" two standard quad-processor x86 busses into a single 8-way symmetrical multiprocessor (SMP). Intel not only completed development, it also gave the chipset legitimacy by giving it an Intel blaze. Then Microsoft provided the last major missing piece with Windows 2000, an OS that not only showed real progress in reliability and scalability over its predecessors, but also lent credibility to Microsoft's efforts to be perceived as a serious OS vendor for serious servers.

ProLiant, initially as a Compaq server brand and then after its acquisition by HP, used this chipset and its successors for a succession of server products--even after Intel decided to stop contributing to further development. (Intel had, at various points, planned to do a Xeon version of Itanium's 870 chipset, but this never ended up happening.) Compaq's own version, the "ProLiant F8" chipset, adapted Profusion for the architecture and bus speeds associated with newer Intel processors, but did not fundamentally alter the design. (Subscribers can read about more of the historical background here.)

However, HP eventually decided to pull the plug on in-house development of 8-way chipsets for Xeon. I've broached the question "Why?" with HP executives on a number of occasions over the past few years and their responses have been pretty consistent. They've boiled down to two basic rationales:

• The size of the 8-socket market does not justify the expense associated with custom chipset development.
• To the degree that there's a demand for larger ProLiants, it's mostly from customers wanting to run larger Microsoft SQL Server databases and associated enterprise applications--and those needs can be met by Windows running on HP Integrity (Itanium-based) servers.

So what's changed to bring ProLiant back into this space? From my perspective, there's probably not one single reason but rather a few different factors that collectively served the needle from "No" to "Yes."

It's easier. Rather than using Intel processors, the ProLiant DL785 G5 uses Advanced Micro Devices Opteron "Barcelona" quad-core processors. Unlike Xeons, the AMD processors can support up to 8-socket servers without the use of special server vendor-developed chips. A lot of effort (and therefore money) still goes into designing, qualifying, and supporting a system in this class. However, the costs associated with primarily integrating existing in-house and third-party components and technologies are still much less than if bespoke chipset design is added to the mix.

The market is larger. Dual-socket servers still make up the bulk of server unit sales. However, server virtualization, in particular, has kicked demand for larger boxes, which once seemed to be on an inevitable slide, up a notch. Server virtualization allows as many workloads (more or less) to run on a system as processor, memory, and I/O capacity can support. Given this, many users are starting to think that they're better off consolidating onto larger servers than smaller ones. This reduces the number of physical boxes to manage. In addition, larger servers often come with a more sophisticated array of reliability and management features. The market for scale-up x86 servers isn't going away either--for reasons including the increasing sophistication of Microsoft SQL Server or the growth of Solaris on x86.

Integrity is only a partial solution. From HP's perspective, the "buy Itanium" message was always logical enough. Most of the critical high-end Windows applications were available and Integrity, after all, was specifically optimized for that space. It's a good story, but the reality is that a lot of Windows customers don't want to support multiple processor architectures in their environments--even if the software is (mostly) the same. 

As a final point, the HP of today is a tightly managed and highly measured organization. And ProLiant is clearly one of the growth stars. Thus, it's not hard to imagine that politely leaving high-end Windows opportunities to Integrity came to be regarded as sub-optimal from the perspective of HP as a whole.

Whatever the precise balance of reasons, HP is back in the 8-socket Xeon game. It's a space that HP has largely ceded to IBM's X4 designs. Now HP is re-engaging aggressively as they did with blades and as they've done across so much of the x86 space.

x86 servers with a single processor (as in single socket) are hardly unusual. They anchor the entry point for most vendors' product lines. Furthermore, beyond those systems that are sold specifically to be used as servers, an untold number of PCs sit under desks or in closets functioning as impromptu file or print servers.

However, pretty much since the advent of mass-market multiprocessing--in the Windows NT 3.51 era or thereabouts--uniprocessor servers have been very much the penny-pinching server option. Yes, they have fewer processors than their dual-socket brethren; that much is obvious. However, uniprocessor boxes have also typically jettisoned all manner of other capacity or reliability upgrades from memory to power.

This makes IBM's release of the uniprocessor IBM System x3350 notable and, perhaps, a harbinger of broader changes accompanying the wide adoption of multicore processors. (For whatever reason, this appears to have been a rather sub rosa announcement--rolled in almost incidentally with a broadened Lenovo partnership.)

For the x3350 is not another uniprocessor in the usual vein. The 1U rackmount server's features include:

• Hot-swap, redundant power supplies
• Integrated management controller
• Up to 8GB PC2-5300 DDR II 667MHz memory, using 4 DIMM slots
• Choice of dual- and quad-core Intel Xeon processors
• SAS or SATA drives, optionally hot-swap and RAID

In short, specs that wouldn't look out of place in a compact 1U dual-socket server.

Historically, each step down in processor count has tended to come with a concomitant cutting of other features--not just I/O or memory capacity (as might be expected), but also various and sundry management, reliability, and redundancy features. However, the storyline is getting more complicated.

On the one hand, server virtualization does appear to be pumping up interest in larger, scale-up servers. Part of the reason is that larger servers offer a degree of simplification through physical consolidation. They also tend to have the ultimate features to protect against memory failures and other hardware glitches. And server virtualization helps separate workloads and thereby makes effective use of this relatively expensive hardware.

At the same time, with quad-core well on its way to being the mainstream in x86 servers, individual processors look a lot like whole multiprocessor complexes of a few years back. This seems to be driving at least some interest in uniprocessor servers that aren't just cheap, cheap, cheap. Beyond x86 servers, Sun Microsystems' octal-core UltraSparc "Niagara" and "Niagara 2" servers are currently "uniprocessor" only (though with numerous cores and hardware support for multithreading, and multi-socket "Victoria Falls" versions coming). Of course, processors have been getting faster for pretty much forever, but the purchasing dynamics seem to be shifting with Moore's Law now reflected more by core count than by frequency.

The System x3350 doesn't foreshadow a massive change in how rackmount servers are designed and consumed. But it does offer an interesting data point that suggests the landscape is shifting.

Dell 1.0 was a religious company. I suppose you could refer to it instead as merely an intense focus on low costs in all matters of its operations, but it really went deeper than that. Low cost was an article of faith that was the deep guiding principle underlying essentially everything that the company did. Dell didn't merely tilt toward a streamlined supply chain and lean R&D, they were a fundamental part of what it was as a company.

This is not a pedantic distinction. Focus can be adjusted and tweaked; it's that much harder to change your core. Yet that's what Dell had to do. It had to respond to a world where "cheap boxes" was no longer the guiding mantra for server buyers, which made Michael Dell's public pronouncements suggesting that "Dell 2.0" was mostly about better execution so wrongheaded. ... Read more