Repairs under way for server speed tests

Servers, the brawny computers that power network services, have changed radically since 1992. But since then, one thing that hasn't kept pace is the speed test most widely used to measure server performance.

That problem now is being addressed by an industry consortium called the Transaction Processing Performance Council, which plans a modernized successor to its 13-year-old TPC-C test.

The sequel has the working name of TPC-E and is expected to be available in 2006, said Jerrold Buggert, a Unisys representative to the consortium. It's designed to be more representative of modern database server work, less expensive to run and less susceptible to artificially high scores caused by oddball hardware and software configurations.

News.context

What's new:
Revisions are in the works for an outdated method that ranks the performance of server computers.

Bottom line:
The upcoming TPC-E test, due next year, is meant to be cheaper, more standardized and more cognizant of today's computing demands.

More stories on this topic

Benchmark scores aren't useful just for bragging rights or for engineers evaluating new designs. They're widely used by customers evaluating new purchases as a helpful, if imperfect, way to compare widely different collections of hardware and software.

The existing speed test, warts and all, is in fact critical to many customers' evaluations. "The most widely referenced result in RFPs"--the requests for proposals by which customers solicit bids--"is TPC-C," Buggert said. One reason it's useful compared with alternatives such as SAP's SD test is that it not only measures performance, but also provides a ratio of price to performance.

The importance of the test can be seen in the money and effort Hewlett-Packard invested in trying to find out why scores for its top-end Superdome server were lower than expected in 2001. The company spent more than $1 million trying to track down the problem, a figure it disclosed in a lawsuit that accused an employee of sabotaging the test.

The consortium isn't working just on an improvement to TPC-C. On Monday, it plans to launch a new benchmark for the midrange machines called application servers.

TPC-App and TPC-DS
The TPC-App test will measure how well these midrange machines perform typical tasks such as communicating with database servers, Web site servers and other application servers. Its score will be measured in Web-services interactions per second. And it will let people compare two dueling technologies used in application servers, .Net from Microsoft and Java from Sun Microsystems and several allies.

TPC-App will replace a test called TCP-W. That test had flaws: It was expensive, often requiring 25 servers to run, yet it tested not just application server performance but the performance of servers that cached information, as well as other ancillary machines. The process was so wide-ranging "it was hard to tell what you were measuring," Buggert said.

Companies involved in TCP-App's creation were IBM, Microsoft, HP, BEA Systems, Oracle, Dell, Unisys, Advanced Micro Devices and Intel, Buggert said.

A third test also is on the way, an alternative to the TPC-H test introduced in 1999 to measure "data warehouses," servers that process large amounts of data to extract information such as purchasing trends.

The new alternative, called, for the time being, TPC-DS, reflects more modern data warehouse usage. These machines might, for example, perform complex queries analyzing how different marketing campaigns affected sales in different regions, Buggert said. The new test also will