Microsoft has griped about oversimplified browser benchmarks before, but that didn't stopped it from prominently publishing results that now show it faster than Google's Chrome, Opera Software's Opera, Mozilla's Firefox, and Apple's Safari.
This idea involves scrutinizing software before it runs and eliminating parts that are mere busywork--flawed code programmers don't always realize they use. One aspect of SunSpider, a test called math-cordic, "is written exactly in a way that triggers this general optimization," Hachamovitch said.
His analysis showed that IE9 is using dead-code elimination for some tasks involving a number of mathematical operators. "Curiously, the IE9 team has chosen not to eliminate more expensive operations such as multiplication or division. The set of analyzed operators do happen to be the ones that appear in the math-cordic benchmark," Sayre said.
Then came his damning conclusion, essentially that Microsoft had tuned Chakra just for a good SunSpider result.
"I don't think it's possible to consider IE9's implementation of dead code elimination to be a serious general-purpose optimization. It seems clear that it hasn't been tested on anything other than SunSpider," Sayre said.
Added Mike Shaver, Mozilla's vice president of engineering, in a tweet, "Disappointed in it's based on real-world' DCE [dead code elimination] nonsense from MSFT; wish they'd just admitted starting with SS [SunSpider] benchmark and making mistakes." And he offered a challenge: "Treasure hunt: find a pattern in any site on the internet that is eliminated by the #IE9 DCE, other than math-cordic!"
Benchmark engineering is an ages-old practice in the computer industry. For example, with prominent server database test, TPC-C, server makers would attach absurdly large numbers of hard drives in an effort to achieve high scores. The design represent real-world constraints customers faced.
Alternatives to SunSpider exist--for example, Google's V8 benchmark and Mozilla's new Kraken. But even those are narrow tests that don't exercise many aspects of browser performance such as page-loading speed, graphics, and user interface responsiveness.
The situation frustrates Chris Wilson, who for years worked on IE but recently joined Google to become Web platform advocate and developer at Google.