Comments on: A Siege On Benchmarks

By: Paul M. Jones » Blog Archive » Comparing Benchmark Tools

Paul M. Jones » Blog Archive » Comparing Benchmark Tools — Wed, 27 Oct 2010 15:43:55 +0000

[...] point is that ab no longer appears to be over-reporting the baseline cases, as I noted in an earlier benchmark posting. There are two major changes between then and now: (1) the updated project uses Ubuntu 10.10 [...]

By: Khaoz

Khaoz — Wed, 11 Mar 2009 14:36:34 +0000

“Anyway, as the lead developer of the symfony framework, I never had objections or complaints. As a matter of fact, those benchmarks have some value. But, even if you take the time to describe the methodology and what you are trying to demonstrate with these benchmarks, people seem to sometimes make bad decisions based on these numbers only, without having a look at the whole ecosystem of the frameworks: community, features, documentation, activity, …”

Sry Fabien but

- Solar has community, features and activity. Need more docs, but it’s unreleased so ppl can wait.
- yii has community, features, documentation, activity.

Both are faster than symfony.

I can talk about django, wich have the same resources and is a lot faster but it’s python and are out of the scope here.

By: Paul Jones’ Blog: A Siege On Benchmarks : Dragonfly Networks

Paul Jones’ Blog: A Siege On Benchmarks : Dragonfly Networks — Sat, 07 Mar 2009 08:15:47 +0000

[...] the right word?) with keeping benchamrks on recent versions of several popular PHP frameworks, has posted another look at a slight change in his testing method – a move away from the Apache ab tool and towards seige (a [...]

By: Duke Ionescu

Duke Ionescu — Fri, 06 Mar 2009 17:43:56 +0000

Great work, it’s interesting to see how fast a framework can deliver “Hello World” compared to HTML/”just” PHP ()!

I have a few of suggestions to improve the usefulness of your work:

First, as other pointed out, include more frameworks. The two I would like to see (which I’ve actually already benched myself using your scripts) are CodeIgniter and Kohana. If you want to be on the bleeding edge, take FLOW3 for a spin.

Second, include DB connectivity in the frameworks’ tests. While you can argue this is outside the original scope of your work, nobody uses a framework [just] to output static text hardcoded in the source files. So even if it’s just retrieving the same row from the same table (which a RDBMS like MySQL should cache), it instantiates the framework’s DB components and makes a network/socket connection. This is important because a) some frameworks are smart enough not to load their DB classes if they’re not used and b) DB classes greatly vary in performance. Overall, it should be a step closer to real-world. Think of it as the Hello World of the web.

Lastly, consider measuring, and reporting, memory usage. Frameworks vary wildly in memory footprint, this variance being even larger when DB connectivity is used as some DB abstraction layers are more heavyweight than others.

Cheers,
-Duke

By: Fabien

Fabien — Fri, 06 Mar 2009 16:04:09 +0000

“Incidentally, Ivo, are you a Symfony developer, devotee, or advocate? I ask because it seems like the Symfony folks are the only ones who regularly have objections to and complaints about this benchmarking series.”

…and you seem to want to demonstrate that symfony is way slower than other frameworks:

“and Symfony trails with the lowest baseline responsiveness.”

even if the Zend Framework and symfony have quite similar numbers.

Anyway, as the lead developer of the symfony framework, I never had objections or complaints. As a matter of fact, those benchmarks have some value. But, even if you take the time to describe the methodology and what you are trying to demonstrate with these benchmarks, people seem to sometimes make bad decisions based on these numbers only, without having a look at the whole ecosystem of the frameworks: community, features, documentation, activity, …

Also, Symfony 2 will definitely be the fastest PHP framework around for a simple “Hello World” application ;) (http://fabien.potencier.org/talk/22/phpquebec-2009-symfony-2?position=29)

By: Tijs

Tijs — Fri, 06 Mar 2009 09:50:27 +0000

Love these stats as always and i switched to siege myself after reading your previous post. I would love to see you test cake 1.2.1 stable btw as opposed to the release candidate your using now as the latest version since i use Cake a lot myself.

By: Wei

Wei — Fri, 06 Mar 2009 01:45:32 +0000

Restart doesn’t always clean up APC correctly on some server settings. It may be best to stop, wait a little, and then start the apache server.

By: pmjones

pmjones — Fri, 06 Mar 2009 00:14:51 +0000

@Ivo: Fair enough — thanks for adding an actual data point to my otherwise anecdotal-only tally. :-)

By: Ivo

Ivo — Fri, 06 Mar 2009 00:02:23 +0000

@pmjones: no associations to symfony, nor do I use it (my company uses it in several projects, just like we use zend framework, code igniter, atk and others). My response was not Symfony driven. :)

By: pmjones

pmjones — Thu, 05 Mar 2009 23:31:46 +0000

@Anon — probably won’t include Drupal, as it is a CMS, not a framework.

The siege command, of course! I should have included that, very sorry for the oversight. It was similar to the ab command parameters of “-c 10 -t 60″; you can see the full siege script file here:

http://code.google.com/p/web-framework-benchmarks/source/browse/trunk/siege/siege.php

The important parts are:

‘protocol’ => ‘HTTP/1.0′,
‘chunked’ => ‘true’,
‘connection’ => ‘close’,
‘concurrent’ => ’10′,
‘time’ => ’60s’,
‘benchmark’ => ‘true’,

Hope that helps, and thanks for pointing it out.