The Interesting Case of Diminishing Responsiveness

I’m working up a second run of PHP framework benchmarks on more powerful hardware than the last time. I’ll talk more about that in a future entry, but in the mean time, I wanted to share an experience that may help you out.

I was running the ab (Apache Benchmark) tool against Zend Framework, Solar, and Cake, using the minimalist “hello world” applications I described the previously. The ZF requests/minute were consistent between runs, always “Z” amount (plus or minus a couple percentage points).

But the Solar and Cake requests/minute declined considerably from run to run. For example, the first benchmark run would yield “X” requests/minute; the 2nd run would yield 0.93X, the 3rd 0.85X, then 0.75X, 0.64X, 0.52X, and so on. This was with the opcode caching turned on; without it, the decline was less severe, but it was still there. Very depressing.

Because I know Solar better than I know Cake, I started looking around the codebase for places to improve. Was it the way I was loading classes, thus causing the opcode cache to work inefficiently? Was it a memory leak in Solar, or PHP itself, or maybe in the web server somehow?

Nope: it was sessions.

Solar and Cake each call session_start() on new requests. Because the “ab” tool sends a new request each time, the frameworks start a new session each time. This means that 1000 requests will generate 1000 new session files. Because the benchmarking makes thousands of requests, the /tmp directory was overflowing with sess_* files (50,000 or more). Having a very large number of files in a single directory was causing the filesystem to drag, and so response time diminished exponentially over time.

Notice that Zend Framework did not have this problem, because it doesn’t have a session-management class; thus, no sessions were created, explaining why its performance stayed consistent from run to run.)

The solution (for benchmarking purposes) was to force the session ID value to ‘abc’ by calling session_id('abc') command at the top of the Solar and Cake bootstrap scripts. This makes the script think there’s already a session in place, and so it doesn’t create a new file. Once I did that, the diminishing responsiveness disappeared, and I got consistent results from run to run.

Are you stuck with a legacy PHP application? You should buy my book because it gives you a step-by-step guide to improving your codebase, all while keeping it running the whole time.
Share This!Share on Google+Share on FacebookTweet about this on TwitterShare on RedditShare on LinkedIn

14 thoughts on “The Interesting Case of Diminishing Responsiveness

  1. Hi Paul,
    the Zend Framework has a Session-Class called Zend_Session.
    AFAIK it’s in the incubator.


  2. Hmm. Doesn’t session_start() set the PHPSESSID cookie?
    Wouldn’t that cause the app to keep reusing the same sess_ file
    for all 50K requests?

    Or doesn’t ab do cookies? Can it be taught? Is that part of the issue?

    What other un-browser-like behaviors are implicit in your setup?

  3. Hi Micahel — ab does in fact “do cookies”; you can read the manual entry for it linked in the first paragraph. However, merely setting the cookie on the client side wasn’t enough in this case, although I don’t recall why just now. And regarding the session_start() and session_id() behaviors, you can see from the PHP manual that session_id() does override the session_start() behavior.

  4. Hi Gwoo — in normal use, I think Cake reasonably expects a session to be available, so it seemed best to keep the session in place for the benchmarking. Also, other/future frameworks may not allow for turning sessions off, so this is also an attempt at future-proofing the benchmarking setup.

  5. Zend_Session will never be a “normal” part of the controller/action, ZF doesn’t try to bind and auto run anything, it requires a little bit more developer “setup”.

    Thats part of the reason I like it so much, It provides all the tools but leaves it up to the developer to really tie them together, where as a lot of other frameworks try to force there own tools on the developer.

    Neither approach is bad, its just a personal thing.

  6. Another benchmarking tool I like is httperf. It’s very easy to install and use, and you could see if both results (from ab and httperf) are consistent:

    You can use it like:

    httperf –server locahost –uri /path/to/testpage.php –num-conns 10000

    You can add also –num-calls 10, for example.

    Thanks (in advance) for the benchmarks !

  7. Hey Paul. For a raw power comparison that’s definitely a handy tip. That does means that Solar and Cake both will have diminishing returns with a DDoS or /.’ing though.

    It would be interesting to see if any solution can be created that would help mitigate this so you can hit 10k new sessions a minute and not kill server performance.

  8. Very true.

    I wonder if the same effect is found using an SQLite or MySQL DB to store the session data. With either of those (SQLite in particular), I guess you still get the same issues with loading that data to memory, but it would be interesting nonetheless.

  9. Richard,
    I agree a framework should do the bare minimum and only help the developer when they ask for it. In the case of Cake, most of the users seem to be using things like Session and Models, so they are expected by default. However, there is always an option to alter this default behavior. I am not so sure this is forcing tools on developers, but rather it is an attempt to reduce the amount of code the developer has to write. The key here is DRY. Having a standard convention and expected behavior allows DRY to extend from application to application and developer to developer. With this in mind, Cake requires nearly zero “setup”, which makes it development faster and allows things like code generation to become truly useful rather than a bloated mess. For instance, the bake utility in Cake, which generates all the code for a complete CRUD based application, only produces about 100 lines of controller code.

    If you are testing the speed of the front controller and render engine of each framework, which emulates the functionality of a static site, I would imagine sessions would rarely be used. Also, I would actually go so far as to say that any framework that does not allow for turning session off, should not be included in the test. Maybe I just need more clarification on the goals of your benchmark. I am not familar with how the other frameworks handle sessions, but in Cake, if the security is set to HIGH then the sessin id will be regenerated on each request and the information from the old session persisted and removed. Setting to MEDIUM or LOW will change this functionality, and result in a session with a longer life. So, you may see varying results based on this setting. The other thing to check is DEBUG setting, as I did not see it mentioned in your setup. In Cake it is set to 2 by default, while setting it to 0 should see changes in the benchmark as well.

    I was also curious about other benchmarking methods. What is your take on ab vs. siege? Is there are particular reason you chose ab?

    I think this is all a lot of fun. Thanks for putting in the effort to install these frameworks. I know some of them can be troublesome and I think your efforts to create the most consistent environment for comparison are commendable.

  10. what would be interresting is to re-publish the result of your benchmark with this new parameters

Leave a Reply

Your email address will not be published. Required fields are marked *