Of Lies, Damned Lies, and Benchmarks
Out Of Date Warning
Languages change. Perspectives are different. Ideas move on. This article was published on August 11, 2009 which is more than two years ago. It may be out of date. You should verify that technical information in this article is still current before relying upon it for your own purposes.
It’s no secret that we like statistics. Human beings like being able to compare two things, apples to apples. We like the ability to say “this thing is better because…” and being able to back it up with a seemingly solid fact, rather than just our opinion. We build charts to show how things stack up side-by-side, and we even build software to make those charts, which are then compared using the charts, and on we go.
But benchmarks, for all their decision-making aid, fail under the best of circumstances for one simple reason: they’re not real life.
Never more is this true than in Joe Stagner’s blog post on whether Windows or Linux, and PHP or ASP was faster. It should really come as no surprise that a Microsoft employee would write an article that would conclude that Windows was faster, and that ASP was faster (I’m pretty sure if he’d found the other way and published it that Ballmer would have canned him).
Benchmarks come loaded with all sorts of problems. It doesn’t matter if it’s Microsoft doing them or Apple doing them; they don’t mimic real-world conditions, and any number of factors affect how the benchmarks are rendered (e.g. other processes running, configuration, installed modules, CLI or web, how the interpreter reads the code, etc).
Some of the problems I found with his benchmarks: there’s no discussion of configuration. Did he use the defaults? What extensions did he install alongside the core installation?
The code he used isn’t published. Update: It appears that he did include his code, though I don’t know when the links were added. They were not in the original draft of the article. How was the server configured? How does the engine render each loop? ASP.NET is closed-source; that’s to say, there’s no way for us to know if the interpreter sees an empty loop or function and decides to bail rather than actually execute it. What version of PHP are you using? Why isn’t there a Ubuntu test for PHP 5.3 (I actually know the answer to that question)?
I don’t think that Mr. Stagner is trying to insult anyone or assert that ASP is 2900% faster than PHP at every task. I do think that the benchmarks provided show that ASP is faster at the given tasks and that’s it.
So, if you need to iterate over 20 million empty for loops, ASP.NET is your language.
Update: After a conversation with Cal Evans and a read about ASP.NET it appears that ASP.NET comes with a built-in byte code cache that cannot be diabled. However, Mr. Stagner conducted his tests WITHOUT APC installed on PHP. While APC is not part of the PHP core (and bytecode caching appears to be part of the ASP.NET core), PHP without APC is a poor configuration choice.