Comments on: The Interesting Case of Diminishing Responsiveness

By: wluigi

wluigi — Wed, 03 Jan 2007 10:39:44 +0000

what would be interresting is to re-publish the result of your benchmark with this new parameters

By: Paul M. Jones » Blog Archive » New Year’s Benchmarks

Paul M. Jones » Blog Archive » New Year’s Benchmarks — Mon, 01 Jan 2007 19:36:17 +0000

[...] Paul M. Jones If it’s worth doing, it’s worth over-doing. « The Interesting Case of Diminishing Responsiveness [...]

By: gwoo

gwoo — Sat, 30 Dec 2006 19:06:28 +0000

Richard,
I agree a framework should do the bare minimum and only help the developer when they ask for it. In the case of Cake, most of the users seem to be using things like Session and Models, so they are expected by default. However, there is always an option to alter this default behavior. I am not so sure this is forcing tools on developers, but rather it is an attempt to reduce the amount of code the developer has to write. The key here is DRY. Having a standard convention and expected behavior allows DRY to extend from application to application and developer to developer. With this in mind, Cake requires nearly zero “setup”, which makes it development faster and allows things like code generation to become truly useful rather than a bloated mess. For instance, the bake utility in Cake, which generates all the code for a complete CRUD based application, only produces about 100 lines of controller code.

Paul,
If you are testing the speed of the front controller and render engine of each framework, which emulates the functionality of a static site, I would imagine sessions would rarely be used. Also, I would actually go so far as to say that any framework that does not allow for turning session off, should not be included in the test. Maybe I just need more clarification on the goals of your benchmark. I am not familar with how the other frameworks handle sessions, but in Cake, if the security is set to HIGH then the sessin id will be regenerated on each request and the information from the old session persisted and removed. Setting to MEDIUM or LOW will change this functionality, and result in a session with a longer life. So, you may see varying results based on this setting. The other thing to check is DEBUG setting, as I did not see it mentioned in your setup. In Cake it is set to 2 by default, while setting it to 0 should see changes in the benchmark as well.

I was also curious about other benchmarking methods. What is your take on ab vs. siege? Is there are particular reason you chose ab?

I think this is all a lot of fun. Thanks for putting in the effort to install these frameworks. I know some of them can be troublesome and I think your efforts to create the most consistent environment for comparison are commendable.

By: Travis Swicegood

Travis Swicegood — Sat, 30 Dec 2006 18:26:40 +0000

Very true.

I wonder if the same effect is found using an SQLite or MySQL DB to store the session data. With either of those (SQLite in particular), I guess you still get the same issues with loading that data to memory, but it would be interesting nonetheless.

By: pmjones

pmjones — Sat, 30 Dec 2006 18:14:44 +0000

Hi Travis — it’s not limited to Solar and Cake; *any* script that uses sessions would be susceptible.

By: Travis Swicegood

Travis Swicegood — Sat, 30 Dec 2006 18:08:13 +0000

Hey Paul. For a raw power comparison that’s definitely a handy tip. That does means that Solar and Cake both will have diminishing returns with a DDoS or /.’ing though.

It would be interesting to see if any solution can be created that would help mitigate this so you can hit 10k new sessions a minute and not kill server performance.

By: Luis

Luis — Sat, 30 Dec 2006 16:18:25 +0000

Another benchmarking tool I like is httperf. It’s very easy to install and use, and you could see if both results (from ab and httperf) are consistent:

http://www.hpl.hp.com/research/linux/httperf/docs.php

You can use it like:

httperf –server locahost –uri /path/to/testpage.php –num-conns 10000

You can add also –num-calls 10, for example.

Thanks (in advance) for the benchmarks !

By: Richard Thomas

Richard Thomas — Sat, 30 Dec 2006 02:51:28 +0000

Zend_Session will never be a “normal” part of the controller/action, ZF doesn’t try to bind and auto run anything, it requires a little bit more developer “setup”.

Thats part of the reason I like it so much, It provides all the tools but leaves it up to the developer to really tie them together, where as a lot of other frameworks try to force there own tools on the developer.

Neither approach is bad, its just a personal thing.

By: pmjones

pmjones — Fri, 29 Dec 2006 22:01:29 +0000

Hi Waldemar — it may be in the incubator, but it’s not part of the “normal” controller/action routine yet.

By: pmjones

pmjones — Fri, 29 Dec 2006 22:00:52 +0000

Hi Gwoo — in normal use, I think Cake reasonably expects a session to be available, so it seemed best to keep the session in place for the benchmarking. Also, other/future frameworks may not allow for turning sessions off, so this is also an attempt at future-proofing the benchmarking setup.