Comments on: Of Lies, Damned Lies, and Benchmarks http://www.brandonsavage.net/of-lies-damned-lies-and-benchmarks/ The personal blog of Brandon Savage. Contains entries of a personal and professional nature focusing on PHP, Apple, LAMP, MySQL and Washington, DC. Fri, 03 Feb 2012 19:36:33 -0500 hourly 1 http://wordpress.org/?v=3.3.1 By: Dave http://www.brandonsavage.net/of-lies-damned-lies-and-benchmarks/#comment-911 Dave Thu, 13 Aug 2009 01:51:57 +0000 http://www.brandonsavage.net/?p=529#comment-911 Brandon, thanks for responding to Stagner's "drive-by comparison" in a productive manner. As Cal Evans wrote the other day -- "The PHP community has matured to the point where we don’t feel the need to be Zealots. We can respect other languages as good at what they do, even if the developers in other communities don’t offer the same respect back." Dave Brandon, thanks for responding to Stagner’s “drive-by comparison” in a productive manner. As Cal Evans wrote the other day –

“The PHP community has matured to the point where we don’t feel the need to be Zealots. We can respect other languages as good at what they do, even if the developers in other communities don’t offer the same respect back.”

Dave

]]> By: artur ejsmont http://www.brandonsavage.net/of-lies-damned-lies-and-benchmarks/#comment-906 artur ejsmont Wed, 12 Aug 2009 14:05:40 +0000 http://www.brandonsavage.net/?p=529#comment-906 LOL 1. Never believe benchmarks with agenda. 2. if your benchmark is 100 times faster on different machine you probably screwed up the setup ;- ) or .... you get errors that return instantly and you are not checking for that ;-) 3. shame he did all this stupid loop-the-loop instead of something more interesing like how much does it take to search the xml or parse it or do some more complex things. Same thing with massive inclusions etc. PHP is cool and all benchmarks lie. Anyone knows some extensive benchmark suite for PHP let me know :- ) I really dream of a nice benchmarking suite for PHP not only to compare platforms but for internal validation and configuration tweaks. Art LOL

1. Never believe benchmarks with agenda.

2. if your benchmark is 100 times faster on different machine you probably screwed up the setup ;- ) or …. you get errors that return instantly and you are not checking for that ;-)

3. shame he did all this stupid loop-the-loop instead of something more interesing like how much does it take to search the xml or parse it or do some more complex things. Same thing with massive inclusions etc.

PHP is cool and all benchmarks lie.

Anyone knows some extensive benchmark suite for PHP let me know :- )
I really dream of a nice benchmarking suite for PHP not only to compare platforms but for internal validation and configuration tweaks.

Art

]]> By: Thomas Bergheim http://www.brandonsavage.net/of-lies-damned-lies-and-benchmarks/#comment-905 Thomas Bergheim Wed, 12 Aug 2009 13:27:11 +0000 http://www.brandonsavage.net/?p=529#comment-905 @Duane Ah yes. Sometimes the stack can be partly blamed though :) For instance, there are a good number of the gazillion web-frameworks that bring in so much "bling" that it really takes up a lot in terms of bootstrapping (of course it was your own fault for using it in the first place!) (case in point: Zend Framework handled 29 res/sec, opcached of course, on my computer, while django doing the same handled almost 200! - they provide different things of course so this is a framework comparison). However usually most of the time spent in webapps these days ends up in SQL-statements so I agree that it is not as important as these "synthetic" benchmarks seem to imply. Still, I like knowing in which ballpark various languages lie when doing various primitive operations, that might just be me though.. @Duane
Ah yes. Sometimes the stack can be partly blamed though :) For instance, there are a good number of the gazillion web-frameworks that bring in so much “bling” that it really takes up a lot in terms of bootstrapping (of course it was your own fault for using it in the first place!) (case in point: Zend Framework handled 29 res/sec, opcached of course, on my computer, while django doing the same handled almost 200! – they provide different things of course so this is a framework comparison).

However usually most of the time spent in webapps these days ends up in SQL-statements so I agree that it is not as important as these “synthetic” benchmarks seem to imply.

Still, I like knowing in which ballpark various languages lie when doing various primitive operations, that might just be me though..

]]> By: Duane Gran http://www.brandonsavage.net/of-lies-damned-lies-and-benchmarks/#comment-904 Duane Gran Wed, 12 Aug 2009 12:52:43 +0000 http://www.brandonsavage.net/?p=529#comment-904 @tsb Sorry if I implied that raw speed wasn't important. My main point was that most people who experience speed issues look for failure in the application stack instead of in the application code. You do however bring up a good point that scalability and performance are different matters. My bad. @tsb
Sorry if I implied that raw speed wasn’t important. My main point was that most people who experience speed issues look for failure in the application stack instead of in the application code. You do however bring up a good point that scalability and performance are different matters. My bad.

]]> By: tsb http://www.brandonsavage.net/of-lies-damned-lies-and-benchmarks/#comment-902 tsb Wed, 12 Aug 2009 11:55:38 +0000 http://www.brandonsavage.net/?p=529#comment-902 @Duane I'm sorry, how are ORDERS OF MAGNITUDE not important, just because "evidently it will still work, as shown by Facebook and Wikipedia"? The article isn't about scalability, it is about raw performance. And php is s-l-o-w, that is no secret (however, I have no idea about how acurate these benchmarks are). Really, C approaches 3 orders of magnitude that of PHP for many things. This "throw more servers at it"-mentality really is not a good engineering-standpoint at all. And caching the hell out of your website because the language is horribly slow does not count in this context. @Joe: Use APC or xcache (http://xcache.lighttpd.net/) or something similar, also throw in Mono on Linux :) @Duane
I’m sorry, how are ORDERS OF MAGNITUDE not important, just because “evidently it will still work, as shown by Facebook and Wikipedia”?

The article isn’t about scalability, it is about raw performance. And php is s-l-o-w, that is no secret (however, I have no idea about how acurate these benchmarks are). Really, C approaches 3 orders of magnitude that of PHP for many things.

This “throw more servers at it”-mentality really is not a good engineering-standpoint at all. And caching the hell out of your website because the language is horribly slow does not count in this context.

@Joe: Use APC or xcache (http://xcache.lighttpd.net/) or something similar, also throw in Mono on Linux :)

]]> By: Joe Stagner http://www.brandonsavage.net/of-lies-damned-lies-and-benchmarks/#comment-896 Joe Stagner Tue, 11 Aug 2009 19:59:44 +0000 http://www.brandonsavage.net/?p=529#comment-896 Brandon - as I'm researching op code cahcing for my next set of tests. Is it really true that theONLY way to get APC for Windows / PHP 5.2 is to buildit from source ? Brandon – as I’m researching op code cahcing for my next set of tests. Is it really true that theONLY way to get APC for Windows / PHP 5.2 is to buildit from source ?

]]> By: Brandon Savage http://www.brandonsavage.net/of-lies-damned-lies-and-benchmarks/#comment-895 Brandon Savage Tue, 11 Aug 2009 19:51:31 +0000 http://www.brandonsavage.net/?p=529#comment-895 Now I know. ;-) Thanks for clearing that up. Now I know. ;-) Thanks for clearing that up.

]]> By: Joe Stagner http://www.brandonsavage.net/of-lies-damned-lies-and-benchmarks/#comment-893 Joe Stagner Tue, 11 Aug 2009 19:02:25 +0000 http://www.brandonsavage.net/?p=529#comment-893 And btw - the code was published at the same time as the orrigional article. And btw – the code was published at the same time as the orrigional article.

]]> By: Joe Stagner http://www.brandonsavage.net/of-lies-damned-lies-and-benchmarks/#comment-892 Joe Stagner Tue, 11 Aug 2009 19:00:52 +0000 http://www.brandonsavage.net/?p=529#comment-892 Just kidding Brandon :) PLEASE email me your suggestions as this was only the first of many perf tests I want to do and I - I have been a PHP developer longer than I've been a .NET developer and I'm not interested "engineered" results. If PHP on Windows is not ACTUALLY faster, and if ASP.NET is not ACTUALLY faster, I want to make it that way. -Joe Just kidding Brandon :)

PLEASE email me your suggestions as this was only the first of many perf tests I want to do and I – I have been a PHP developer longer than I’ve been a .NET developer and I’m not interested “engineered” results.

If PHP on Windows is not ACTUALLY faster, and if ASP.NET is not ACTUALLY faster, I want to make it that way.

-Joe

]]> By: Joe Stagner http://www.brandonsavage.net/of-lies-damned-lies-and-benchmarks/#comment-891 Joe Stagner Tue, 11 Aug 2009 18:50:33 +0000 http://www.brandonsavage.net/?p=529#comment-891 Samuel, Don't worry. Brandon is a tool and if I said eating dog shit was bad for you - he'd argue with me about it ! :) -Joe Samuel,

Don’t worry. Brandon is a tool and if I said eating dog shit was bad for you – he’d argue with me about it ! :)

-Joe

]]> By: Samuel Folkes http://www.brandonsavage.net/of-lies-damned-lies-and-benchmarks/#comment-888 Samuel Folkes Tue, 11 Aug 2009 17:05:25 +0000 http://www.brandonsavage.net/?p=529#comment-888 Brandon, it is not his intent that I doubt. It the pertinence of the tests and the environment in which they were conducted. PHP on Windows means a license but ASP.NET on Windows means 2 (or 3 if you count the inevitable inclusion of SQL Server). I will with no hesitation declare that I am a PHP advocate. I use PHP loving its strengths and accepting its weaknesses and limitations. As you stated in your article, as humans we love comparisons. We love numbers and statistics. However if one can truly accept the limitations of their chosen development tool then the numbers and the statistics should become almost irrelevant. I must admit, I am very interested in seeing the numbers (produced from fair, balanced tests of course), but that interest is purely academic. If we all adhere to the tried tested and proven concept of using the right tool for the job, in the end, the outcome of this debate will (should) have very little consequence. Brandon, it is not his intent that I doubt. It the pertinence of the tests and the environment in which they were conducted. PHP on Windows means a license but ASP.NET on Windows means 2 (or 3 if you count the inevitable inclusion of SQL Server). I will with no hesitation declare that I am a PHP advocate. I use PHP loving its strengths and accepting its weaknesses and limitations. As you stated in your article, as humans we love comparisons. We love numbers and statistics. However if one can truly accept the limitations of their chosen development tool then the numbers and the statistics should become almost irrelevant. I must admit, I am very interested in seeing the numbers (produced from fair, balanced tests of course), but that interest is purely academic. If we all adhere to the tried tested and proven concept of using the right tool for the job, in the end, the outcome of this debate will (should) have very little consequence.

]]> By: Brandon Savage http://www.brandonsavage.net/of-lies-damned-lies-and-benchmarks/#comment-887 Brandon Savage Tue, 11 Aug 2009 16:40:28 +0000 http://www.brandonsavage.net/?p=529#comment-887 Samuel, I think that Joe's intent was good and that while he may have known that ASP.NET has a built-in bytecode cache, I doubt he was trying to pull a once-over on anyone. Microsoft has invested considerably in the last few years to get PHP to run better on the Windows platform, and I applaud their effort. Even if developers don't use ASP.NET to build applications, getting them to use Windows still means a license. And I know that Microsoft is interested in remaining relevant, which they have to support PHP well in order to do. I work in a shop that uses a number of different languages, and I've learned that languages are tools that are used for a specific purpose. You don't use a hammer to dig a ditch, and likewise you don't use PHP to build desktop applications, for example. And though I make a dig at ASP.NET at the end of my article, I'm willing to admit that I'd use it if the situation called for it. To interface with Microsoft database tools, I don't think anything beats it. I want to make sure the debate remains civil; if our product is really better we can win the debate by proving it. Samuel, I think that Joe’s intent was good and that while he may have known that ASP.NET has a built-in bytecode cache, I doubt he was trying to pull a once-over on anyone.

Microsoft has invested considerably in the last few years to get PHP to run better on the Windows platform, and I applaud their effort. Even if developers don’t use ASP.NET to build applications, getting them to use Windows still means a license. And I know that Microsoft is interested in remaining relevant, which they have to support PHP well in order to do.

I work in a shop that uses a number of different languages, and I’ve learned that languages are tools that are used for a specific purpose. You don’t use a hammer to dig a ditch, and likewise you don’t use PHP to build desktop applications, for example.

And though I make a dig at ASP.NET at the end of my article, I’m willing to admit that I’d use it if the situation called for it. To interface with Microsoft database tools, I don’t think anything beats it. I want to make sure the debate remains civil; if our product is really better we can win the debate by proving it.

]]> By: Samuel Folkes http://www.brandonsavage.net/of-lies-damned-lies-and-benchmarks/#comment-886 Samuel Folkes Tue, 11 Aug 2009 16:31:10 +0000 http://www.brandonsavage.net/?p=529#comment-886 Brandon, I'm very glad you wrote this article. That ASP/PHP comparison stinks from the bottom up for a number of reasons in cluding but not limited to the fact that these tests were carried out in an environment that did not mimic a real world production evironment closely enough. Of course, Hardly anyone from Microsoft has given PHP the time of day before now. I wholeheartedly believe that Mr. Stagner's blog post is an indication that like linux, PHP is now a huge concern for Microsoft as formidable competitor. It is no longer prudent for them to ignore the strides that PHP as a viable and in my opinion preferable alternative to ASP has taken. I think we should expect more of these tainted assertions to surface in the near future. I say we use Facebook and Wikipedia as our benchmarks. Nothing makes a statement like some good old action. P.S. Nice work revealing that ASP.NET has a built in bytecode cache. That's something I'm almost sure Mr. Stagner was aware of but failed to mention in his article. Brandon, I’m very glad you wrote this article. That ASP/PHP comparison stinks from the bottom up for a number of reasons in cluding but not limited to the fact that these tests were carried out in an environment that did not mimic a real world production evironment closely enough. Of course, Hardly anyone from Microsoft has given PHP the time of day before now. I wholeheartedly believe that Mr. Stagner’s blog post is an indication that like linux, PHP is now a huge concern for Microsoft as formidable competitor. It is no longer prudent for them to ignore the strides that PHP as a viable and in my opinion preferable alternative to ASP has taken. I think we should expect more of these tainted assertions to surface in the near future. I say we use Facebook and Wikipedia as our benchmarks. Nothing makes a statement like some good old action.

P.S. Nice work revealing that ASP.NET has a built in bytecode cache. That’s something I’m almost sure Mr. Stagner was aware of but failed to mention in his article.

]]> By: Brandon Savage http://www.brandonsavage.net/of-lies-damned-lies-and-benchmarks/#comment-883 Brandon Savage Tue, 11 Aug 2009 14:57:13 +0000 http://www.brandonsavage.net/?p=529#comment-883 Sadly people do this, Steve. It's not recommended, though. Sadly people do this, Steve. It’s not recommended, though.

]]> By: Steve Clay http://www.brandonsavage.net/of-lies-damned-lies-and-benchmarks/#comment-882 Steve Clay Tue, 11 Aug 2009 14:55:08 +0000 http://www.brandonsavage.net/?p=529#comment-882 Lesson: PHP without a byte-code cache on a desktop OS is not ideal. Good thing <em>no one does this</em>. Lesson: PHP without a byte-code cache on a desktop OS is not ideal. Good thing no one does this.

]]>