Repairs under way for server speed tests

(continued from previous page)

support 135 different types of queries compared with TPC-H's 25, a change that will make it harder to build systems optimized just for the benchmark test that produced artificially high scores.

TPC-C then and now
TPC-C has survived tumultuous years in the history of servers. In 1992, mainframes topped the server pecking order and Unix servers were just catching on. Since then, the server market has been remade by the advent of inexpensive machines using Intel processors, the demise of Digital Equipment Corp. and Compaq Computer, the arrival of Microsoft Windows and Linux, and the emergence of behemoths with as many as 128 processors.

The first TPC-C scores were tiny compared with today's scores. It wasn't until 1998 that servers produced scores of about 100,000 transactions per minute. In 2001, a Fujitsu system arrived to dominate the rankings with a score of 456,000, about the time Sun withdrew after objecting that the test no longer represented reality. In 2004, though, IBM blew the roof off the test with a score of 3.2 million transactions per minute.

The price performance ratio--the system's cost divided by its throughput, or the amount of work a computer can do during a given time period--also has changed dramatically as server prices dropped. The ratio was more than $200 per unit of throughput for systems released before 1996--and nearly $1,200 in one case.

IBM published its first TPC-C result in May 1994, with a score of 485 transactions per minute and a price performance ratio of 654, the consortium said. Today, the ratio for IBM's p5-575 is $5.19.

TPC-C essentially simulates a computer database that manages a warehouse's inventory, processing basic transactions such as order placement, Buggert said. TPC-E simulates an electronic brokerage and includes much more sophisticated processes.

One such process is "two-phase commit" transactions in which a database operation can't be completed until a related operation on another database is completed, Buggert said. Another is "referential integrity," which makes sure a database isn't thrown off if one element is changed or deleted by one process while another process is using that element.

TPC-C's flaws
One problem with TPC-C was the ease with which certain servers could generate unrealistically high scores using hardware and software configurations that are highly improbable in the real world. For example, high TPC-C scores come from servers with colossal numbers of hard drives--6,548 in the case of IBM's top score.

Another problem hinged on the fact that TPC-C's test was easily distributed among relatively independent servers linked in a cluster. That gave the impression that a number of inexpensive machines were as good as a single multiprocessor behemoth, leading the consortium to list results separately for clustered and non-clustered systems.

Real-world database tasks today don't divide so easily, a fact that the upcoming TPC-E will reflect, Buggert said. "It should be a more realistic representation of what you get when you cluster things in the real world," he said.

A real comparison between clustered and non-clustered results should be useful to customers evaluating clustered databases, which are becoming more powerful with the gradual maturation of technology such as Oracle's Real Application Clusters and the InfiniBand high-speed communication link hardware.

Buggert said TPC-E is being created by the major sellers of databases, processors and servers--including Sun.