The nice thing about standards is that there are so many of them.
This old saw is arguably less true than in years past. Today, for a lot of reasons, there's more pressure to reach agreement on one way to do a certain thing. (Think the HD DVD vs. Blu-ray debacle for an example of what happens when vendors can't agree on a single approach.)
Standards aren't a single thing. Some have been blessed with the appropriate incantations by some official or quasi-official body. Others come from an industry consortium. And still others are "de facto" (or at least began life that way)--the result of a dominant company or just a default way of doing things.
The purist will argue that just being widely used doesn't make something a standard. I agree up to a point and only use the "standard" term in this case for things for are truly ubiquitous. Contrariwise, a rigorous formal ratification process is no guarantee of success.
But some standards do win big and become part of just how IT gets done. Here are some of them.
Like many other successful standards, Ethernet has remained a fixture in local area networks for so many years in part by adapting to many successive waves of technology. First developed in the famous Xerox PARC labs in the mid-1970s, it initially ran over coaxial cable but soon moved to twisted pair cable with the 10 Mbit/second generation. 10 Gbit/second Ethernet is now starting to roll out along with a variety of additions to the specification that make it more suitable as a high-performance unified fabric.
Ethernet's initial success resulted in no small part from coordinated standardization efforts beginning in the IEEE. This helped it beat out alternatives, most notably IBM's Token Ring. Over time, Ethernet's ubiquity and the cost benefits provided by this volume helped it largely stave off server interconnect challengers. InfiniBand has had wins in high-performance computing and certain other clustering applications, but it didn't displace Ethernet as a "server area network" as early promoters had hoped.
PCI, Peripheral Component Interconnect, had its beginnings as an Intel-developed bus for connecting internal cards within systems. The version 1.0 spec came out in 1992. Given the ubiquity of PCI these days, it's easy to forget that it only replaced a plethora of other busses both standardized and proprietary in x86 and, later, large Unix servers based on other processors over the course of nearly a decade.
Nor was the process steady. Although PCI was initially introduced in part to replace the VESA Local Bus for graphics cards--which it eventually did--PCI was itself replaced by AGP (Accelerated Graphics Port) for a time prior to the PCI Express generation.
PCI Express makes for an interesting case study in the marketing of standards. With technology bumping up against the limits of parallel I/O busses like conventional PCI, the Arapahoe Working Group--spearheaded by Intel--started pushing a new serial interconnect standard in about 2001. Arapahoe's success was by no means pre-ordained. AMD's HyperTransport was one alternative among several.
Arapahoe required hardware that was largely different from PCI but it was compatible with PCI's software model in a number of respects. And this was enough to get Arapahoe adopted by the keeper of the PCI standard, the PCI-SIG, and get the SIG's imprimatur on what would now be called PCI Express. And that helped make it the obvious heir to PCI. Names matter. (Here's a more detailed accounting of PCI Express and its history.)
It's easy to forget just how painful it could be, in the years before USB (Universal Serial Bus), to connect external peripherals to a computer system. RS-232, a long-used and successful standard in its own right, was the most common way. It was also a way that could easily devolve into examinations of cable pin-outs, interrupt channels, and memory addresses.
USB was a cooperative effort by a group of large technology vendors who founded a non-profit corporation to manage the specification. Version 1.0 was introduced in 1996. Now up to version 3.0, USB has become the standard way to connect external peripherals to PCs; it's also commonly used on servers for devices such as printers.
There's a spec for wireless USB but, like other standards intended to connect peripherals to computers wirelessly, it's never taken off. The current such "personal area network" getting the most buzz is My WiFi from Intel.
USB's primary competition has been FireWire, Apple's name for IEEE 1394. Unlike USB, it does not need a host computer and is faster than the USB 2.0 generation. However, it didn't catch on widely in the computer industry outside of Apple (which is phasing it out in favor of USB) and video equipment.
TCP/IP refers to the combination of two protocols: Transmission Control Protocol and Internet Protocol. Together, they are among the most important pieces of software underpinning the Internet which transitioned to using TCP/IP in 1983. Work on TCP began under the auspices of the Defense Advanced Research Projects Agency (DARPA) a decade earlier but, along the way, the software stack was re-architected to add IP as the early Internet grew.
Like many of the Internet's building blocks, TCP/IP was firmly entrenched before commercial interests got involved to any significant degree and, indeed, before most of the world at large had any real notion of the Internet's existence. The general public came to know the Internet through the World Wide Web, an outgrowth of Tim Berners-Lee's development of HTML at CERN, in the 1990s. Thus HTML, as well, is a key standard.
At the time that TCP/IP was gaining momentum, the International Organization for Standardization (ISO) spearheaded a large project to standardize networking. The "OSI model" remains the standard way to think about layers of the networking stack. If you talk about a switch being "Layer 4," you're using OSI terminology. But the specific protocols developed to go with the model were never widely used. (TCP/IP largely maps to the layers defined in the OSI model.)
The x86 architecture is perhaps the canonical example of a de facto standard driven primarily by a single vendor: Intel. Microsoft Windows is also in the running, but it was very arguably x86's ubiquity in a segment of the market open to relatively low-cost packaged software that made the rise of Windows possible. Over the past decade, AMD has also driven x86 innovations--most notably 64-bit extensions. However, it was Intel that had the biggest hand in shifting the industry from a structure in which each company did everything from fabricating processors to writing operating systems to developing databases to one in which different companies tend to specialize in one part of the technology ecosystem.
x86 emerged as a dominant chip architecture for a variety of reasons. IBM designed Intel's 8088 into the first important business PC. It got this win and others at a time when the world was rapidly computerizing. And Intel optimized itself to ride key technology trends while divesting itself of businesses, such as memory, as they commoditized.
Finally, here are a few others that could make a list like this one:
Wi-Fi played a big role in making personal computers more mobile--which is why Intel pushed it so hard.
VGA is the computing video standard that finally helped merge a rather splintered landscape and had a good long reign. (The latest video interconnect trend is a shift to HDMI--representing a coming together of computing and consumer electronics standards.)
SCSI was the first storage interconnect to merge in a big way a disparate set of existing connection schemes, both proprietary and more or less standardized. However, storage has remained an area where different standards are used for different purposes. That's changing to a degree with SATA, however, which we now see in both PCs and data centers.
This is the time of year to take stock in where high-performance computing (HPC) sits and where it is headed. That's because the SC09 conference is taking place in Portland, Ore., this week and it's the biggest HPC conference around.
SC is an odd duck as conferences go. Last year it had more than 10,000 attendees and, yet, it's a largely volunteer-organized event in a world where trade shows of this scope are packaged by conference specialists or some specific corporation. Think the much-renamed LinuxWorld (run by IDG) or VMworld (run by VMware).
"SC" comes from supercomputing. Today's large computer complexes are typically not supercomputers in the sense of a specialized architecture only suitable for a specific type of technical computing. Rather, as Ashlee Vance notes in The New York Times, "The supercomputing world was long dominated by systems that required specialized chips, memory systems and networking technology. But about 10 years ago, researchers realized they could link thousands of cheaper machines running on mainstream chips and achieve pretty solid performance."
Thus an HPC event is no longer about supercomputers per se (although the term is still used as a convenient moniker for a collection of resources managed as a single entity in a single location). Rather it's about the computing components, the interconnects, the storage, and the software that ties everything together and the applications that run on top.
The Top500 nicely illustrates the evolution of HPC over time. This list, released twice annually, ranks the largest publicly acknowledged supercomputers--as the term is used today--on the basis of a somewhat simplistic, but objective, benchmark. The Top500 entries are certainly not typical of mainstream HPC; they're the biggest of the big. But they nonetheless provide some quantitative insight into important trends.
The newest iteration of the list was released Friday. There were no striking departures from the trends of the last few years, but there was some continued evolution that's worth taking note of.
The continued rise of InfiniBand. InfiniBand is a system interconnect that offers a higher performance alterative to the ubiquitous Ethernet. Although its initial backers envisioned a broader role for the technology, it's settled nicely into HPC and, to a lesser degree, back-end commercial data center functions like database clusters where low latency and high bandwidth are also paramount. (The Sun/Oracle Exadata appliance uses InfiniBand for example.)
(Credit:
TOP500.org)
InfiniBand's initial growth in HPC wasn't so much about displacing Ethernet as it was displacing the fractured collection of high-performance interconnects that preceded it. Myricom's Myrinet and Quadrics' QsNet were the most common of these, but there were many. This year InfiniBand is deployed on 181 of the Top500, a 28 percent increase from a year ago.
That's a striking increase clearly. But what is perhaps more striking is that about half that increase came at the expense of Ethernet rather than mopping up a variety of older or proprietary connection technologies. This shift started between 2007 and 2008 but was even more pronounced this year.
It's certainly possible that the next 10GbE generation of Ethernet, which today is essentially absent from the list, could again push Ethernet's numbers higher. However, whatever the specific technology, the message that I take away is that large computer clusters are starting to favor more optimized interconnects even if they and the components they connect are largely off-the-shelf.
And we see an analogous trend with the proliferation of blade servers as well. Blades, a more modular and pluggable approach to system design, have proven popular in many enterprises and midmarket companies, in part, because they help bring together computing, storage, and networking technologies into a single integrated whole. That type of integration isn't of much interest in HPC. Rather, blades play to HPC by offering high densities and reducing cable count and complexity.
In fact, among x86 servers at any rate, dominance is not too strong a word to describe the presence blades in the Top500. Consider just one vendor, Hewlett-Packard. HP has 208 ProLiant systems on the list. A full 203 of these, almost 98 percent, are ProLiant c-Class blades.
Collectively, these trends suggest what might be thought of as a trend toward building optimization around standardization. In the main, especially as one moves down from the very top of the list, the Top500 is composed mostly of systems using mainstream technologies such as x86, Linux, and standard interconnects. Clusters are the dominant architecture.
But we're increasingly not seeing mere rackmount servers connected by Gigabit Ethernet. As the systems on the Top500 list grow in capability, we're seeing more focus on how they're packaged, powered, and connected.
In October of 2000, I hopped a Las Vegas-bound flight to attend a developers' event being thrown by the InfiniBand Trade Association.
By way of background, InfiniBand was one of the hot technology properties of the pre-bubble-bursting days. It was touted as a better (faster, more efficient) way to connect servers than the ubiquitous Ethernet. Its more vocal backers, of which there were many, went so far as to position it as a "System Area Network"--a connective fabric for data centers. A whole mini-industry of silicon, software, host bus adapter, and switch vendors supported InfiniBand. One sizable cluster resided in Austin, Texas, but there were many of them scattered around the U.S. and elsewhere--to say nothing of significant InfiniBand initiatives at companies such as IBM and Intel.
I don't remember all the details of that past InfiniBand event but it filled a decent-sized hall at the Mandalay Bay and was followed by a party that took over the hotel's "beach" on a balmy Vegas evening.
Last week, I attended another InfiniBand event, Techforum '08. It was also in Las Vegas. More modest digs at Harrah's reflected that InfiniBand hasn't exactly lived up to those past hopes. However, the fact that there even was a TechForum '08 also reflects that InfiniBand is still with us--primarily as a server connect for high performance computing (HPC) applications where low latency and high bandwidth are especially important.
Given that I've been following InfiniBand since its early days, this seems like a good opportunity to reflect on where InfiniBand stands today and where it may be going.
As with another Big "I" technology, Intel's Itanium processor, it's tempting to glibly dismiss InfiniBand as a failure because it failed to live up to early (probably unrealistic) hopes and promises. In fact, InfiniBand now dominates performance sensitive connections between servers in HPC. It's largely taken the place of a plethora of competing alternatives, most notably Myricom's Myrinet and Quadrics' QsNet. Plain old Gigabit Ethernet has successfully held onto its position of default data center interconnect and FibreChannel has remained the default for storage area networks. But InfiniBand has actually been quite successful at establishing itself as the standard interconnect for optimized clusters.
One also finds InfiniBand technology beneath the covers in a variety of products. Among other products, a variety of blade chassis use InfiniBand in their backplanes. This may not exactly be InfiniBand the standard, but it is InfiniBand the technology. And this type of use contributes to InfiniBand component volumes--which tends to drive down prices.
But, what of 10 Gigabit Ethernet? Isn't it inevitable that 10 GbE will replace InfiniBand? Indeed, most InfiniBand component suppliers, such as Mellanox, are covering their bets by embracing both technologies.
But 10 GbE, after many years in development, remains in early days. Costs are still high. The converged 10 GbE that is most relevant to InfiniBand's future sometimes called "Data Center Ethernet" isn't even a single thing. It's at least six different standards initiatives from the IEEE and IETF (not including the related FibreChannel over Ethernet efforts). In many cases, 10 GbE will also require that data centers upgrade their cable plant to optical fiber.
In short, although 10 GbE will certainly emerge as an important component of data center infrastructures, lots of technical work (and political battles) remain.
So does Ethernet conquer all? Maybe. Someday. A lot happens someday. InfiniBand may not ever markedly expand on the sorts of roles that it plays. But 10 GbE is far from ready to take over when latency has to be lowest and bandwidth has to be highest.
- prev
- 1
- next
