March 18, 2007 9:00 PM PDT

Database servers get a new yardstick

After 14 years measuring the world's most powerful servers, the TPC-C speed test is being supplanted by a new benchmark.

The Transaction Processing Performance Council has approved the new standard, called TPC-E, which is designed to be easier and less expensive to run and to be more representative of what databases are used for these days. The first results of the test should begin arriving this summer once auditors are trained to make sure computer makers know and adhere to test guidelines.

TPC-E simulates traders and brokers interacting with a stock exchange. Information is stored in 33 separate database tables, with actions in one area often triggering actions in another. In comparison, TPC-C dealt with simpler data and operations stored in nine tables that simulated inventory management at a warehouse.

There's no such thing as benchmarks that measure all aspects of a computer's performance, but they still can be a useful way to compare different companies' products. TPC-C was widely used for this task--but servers and databases have changed a lot since its debut in July 1992.

"TPC-C is at the end of its life. It's time for a new benchmark," said Andreas Hotea, who leads the effort to publicize TPC-E. The council, whose members include all the major server and database companies, spent five years creating the benchmark, said TPC administrator Michael Majdalany.

One problem with TPC-C is that high scores require an inordinate amount of storage--not just raw capacity, but numerous hard drives. IBM's and Hewlett-Packard's top TPC-C scores each used more than 7,000 hard drives, Majdalany said. TPC-E will probably need only a tenth of that capacity, he said.

It's typical for a TPC-C test to cost $4 million to run, Hotea said, mostly through hardware costs. An IBM machine tested in 2006 and currently No. 2 in the TPC-C rankings with a score of 4.03 million transactions per minute cost $12 million. HP topped that score with 4.09 million transactions per minute with another $12 million machine.

With TPC-C, Majdalany said, "only rich companies can do it."

The new test will be cheaper to run partly because hardware demands are more reasonable, but also because TPC will supply source code for the software instead of requiring testers to write their own, Hotea said.

See more CNET content tagged:
TPC-C, benchmark, database server, database, IBM Corp.

2 comments

Join the conversation!
Add your comment
Maybe Yes and Maybe No
While the TPC-C benchmark got too expensive, there were other ways to look at the results. Could you build an inexpensive machine that would let you focus on the cost per transactions rather than the maximum number of potential transactions?

You could tweak the TPC-C benchmark requiring of the shelf hardware and place a budget on the hardware that could be used.

But looking at the TPC-E, if the TPC Org is writing the applications, is there anything to ensure that the testing software will be vendor neutral?

Note that we all know that vendors will "cheat". Just looking at the TPC-C configurations is proof enough. My concern is that will the new(er) TPC-E actually reduce this to a more controllable level?
Posted by dargon19888 (412 comments )
Reply Link Flag
Yep, it's going to be a big scramble
for the DBMS vendors to re-optimize for the new benchmark. I guess it's an improvement; personally I think it's a joke for an RDBMS benchmark to use simple queries and limited numbers of joins.
Posted by mn39202 (32 comments )
Link Flag
 

Join the conversation

Add your comment

The posting of advertisements, profanity, or personal attacks is prohibited. Click here to review our Terms of Use.

What's Hot

Discussions

Shared

RSS Feeds

Add headlines from CNET News to your homepage or feedreader.