Database servers get a new yardstick

After 14 years measuring the world's most powerful servers, the TPC-C speed test is being supplanted by a new benchmark.

The Transaction Processing Performance Council has approved the new standard, called TPC-E, which is designed to be easier and less expensive to run and to be more representative of what databases are used for these days. The first results of the test should begin arriving this summer once auditors are trained to make sure computer makers know and adhere to test guidelines.

TPC-E simulates traders and brokers interacting with a stock exchange. Information is stored in 33 separate database tables, with actions in one area often triggering actions in another. In comparison, TPC-C dealt with simpler data and operations stored in nine tables that simulated inventory management at a warehouse.

There's no such thing as benchmarks that measure all aspects of a computer's performance, but they still can be a useful way to compare different companies' products. TPC-C was widely used for this task--but servers and databases have changed a lot since its debut in July 1992.

"TPC-C is at the end of its life. It's time for a new benchmark," said Andreas Hotea, who leads the effort to publicize TPC-E. The council, whose members include all the major server and database companies, spent five years creating the benchmark, said TPC administrator Michael Majdalany.

One problem with TPC-C is that high scores require an inordinate amount of storage--not just raw capacity, but numerous hard drives. IBM's and Hewlett-Packard's top TPC-C scores each used more than 7,000 hard drives, Majdalany said. TPC-E will probably need only a tenth of that capacity, he said.

It's typical for a TPC-C test to cost $4 million to run, Hotea said, mostly through hardware costs. An IBM machine tested in 2006 and currently No. 2 in the TPC-C rankings with a score of 4.03 million transactions per minute cost $12 million. HP topped that score with 4.09 million transactions per minute with another $12 million machine.

With TPC-C, Majdalany said, "only rich companies can do it."

The new test will be cheaper to run partly because hardware demands are more reasonable, but also because TPC will supply source code for the software instead of requiring testers to write their own, Hotea said.

[dargon19888]

Maybe Yes and Maybe No

While the TPC-C benchmark got too expensive, there were other ways to look at the results. Could you build an inexpensive machine that would let you focus on the cost per transactions rather than the maximum number of potential transactions?

You could tweak the TPC-C benchmark requiring of the shelf hardware and place a budget on the hardware that could be used.

But looking at the TPC-E, if the TPC Org is writing the applications, is there anything to ensure that the testing software will be vendor neutral?

Note that we all know that vendors will "cheat". Just looking at the TPC-C configurations is proof enough. My concern is that will the new(er) TPC-E actually reduce this to a more controllable level?

[mn39202]

Yep, it's going to be a big scramble

for the DBMS vendors to re-optimize for the new benchmark. I guess it's an improvement; personally I think it's a joke for an RDBMS benchmark to use simple queries and limited numbers of joins.