Microsoft IE9 SunSpider JavaScript results raise questions
Summary: A Mozilla engineer has uncovered an oddity with Microsoft’s Internet Explorer 9 where one test that is part of the SunSpider JavaScript benchmark gives an odd, unexpected result.
A Mozilla engineer has uncovered an oddity with Microsoft’s Internet Explorer 9 where one test that is part of the SunSpider JavaScript benchmark gives an odd, unexpected result.
Mozilla engineer Rob Sayre set about benchmarking Firefox 4.0 beta against a selection of other browsers and found that IE9 was about ten times faster at one certain test (the math-cordic test) than the other browsers, with IE9 completing the test in around 1ms while Chrome and Opera took around 10ms.
Curiosity piqued, Sayre did some further investigating:
One last issue that can crop up has to do with over-specialization for a specific test. While I was running the SunSpider tests above, I noticed that IE9 got a score that was at least 10x faster than every other browser on SunSpider’s math-cordic test. That would be an impressive result, but it doesn’t seem to hold up in the presence of minor variations. I made a few variations on the test: one with an extra “true;” statement (diff), and one with a “return;” statement (diff). You can run those two tests along with the original math-cordic.js file here.
file here.
All three tests should return approximately the same timing results, so a result like the one pictured above would indicate a problem of some sort.
This effect shows up nicely in the raw benchmark tests results I carried out last week. Notice how the math-cordic test result for IE9 are consistent.
So what could be behind this. Three possibilities spring to mind:
- A bug in the JavaScript engine
- Deliberate optimization for the SunSpider test
- Accidental optimization for the SunSpider test
Can we put this down to cheating, as suggested by Digitizor (which was later picked up on by Slashdot)? Well, without access to the code it’s impossible to be sure, and we don’t have access to the code. The effect of this one aberration is quite small and tweaking the values from 1ms to 10ms in the tests I ran only drops the SunSpider score to 403.7ms per run, up from 394.7ms. But this is just one result out of many. It depends if there are other, more subtle, optimizations there.
I’m not ready to call this a cheat yet, but it’s certainly fishy. But even is there is some degree of optimization, I’m more likely to believe that it’s accidental rather than deliberate. The consistency of the result on IE9 is odd in that across multiple machines I get a consistent score of 1ms, which is not something I’d expect to see. Combine that with the fact that the change made to the benchmark code by Sayre should “functionally” make no difference, the fact that you can see wildly different results is again very odd and not something I’d expect to see.
Sayre has submitted this as a “bug” to Microsoft.
