Rasmus Lerdorf’s Laconic(a) Performance

As many of you know, I maintain a series of web framework benchmarks. The project codebase is here and the most recent report is here.

It was with some interest, then, that I viewed Rasmus Lerdorf’s slides on the subject of performance benchmarking. I’m beginning to think there’s something unexpected or unexamined in his testing methodology.

Note: see the update at the end of this entry.

On this slide, Rasmus notes the following:

  • Static HTML nets 611.78 trans/sec
  • Trivial PHP nets 606.77 trans/sec

This would seem to indicate that the mere invocation of PHP, on Rasmus’ setup, reduces Apache’s performance from serving static pages by less than 1%.

In my testing on Amazon EC2 small instances, I note somewhat different results:

  • Static HTML nets 2309.14 req/sec
  • Trivial PHP nets 1320.47 req/sec

The net reduction there is about 43%. Yes, this is with opcode caching turned on.

I then became really curious as to how Rasmus might have his system set up, to see only a 1% bit of “added overhead” from PHP being invoked. It would be nice if I could set up my own systems the same way.

When I asked about that at work, my colleague Elizabeth Smith opined that maybe Rasmus’ web server is running all requests through the PHP interpreter, not just PHP files. That sounded like a good intuition to me, so I set up an EC2 instance to try it out.

Per the setup instructions on this page I built an EC2 server the same as I’ve done for my own benchmarking reports. I didn’t check out the whole project, though; this time we just need the “bench”, “baseline-html”, and “baseline-php” directories.

As a reminder, the baseline index.html page is just the following plain text …

Hello World!

… and the baseline index.php page is the following single line of PHP code:

<?php echo 'Hello World!'; ?>

The php5.conf file for Apache looks like this by default …

<IfModule mod_php5.c>
  AddType application/x-httpd-php .php .phtml .php3
  AddType application/x-httpd-php-source .phps
</IfModule>

… and we’re going to leave it alone for a bit.

Using ab -c 10 -t 60 to benchmark baseline-html and baseline-php with the default php5.conf gives us these results (an average over 5 runs each):

               |      avg
-------------- | --------
baseline-html  |  2367.02
baseline-php   |  1270.15

That’s a 47% drop for invoking PHP. (That is itself 4 points different than the numbers I show above, so it appears there are some variables I have not controlled for, or maybe I just need to let this run longer than 5 minutes to smooth out the deviations.)

To test our hypothesis, we modify the php5.conf file to add .html to the list of files that get passed through PHP …

<IfModule mod_php5.c>
  AddType application/x-httpd-php .html .php .phtml .php3
  AddType application/x-httpd-php-source .phps
</IfModule>

… restart Apache, and run the same ab tests again:

framework      |      avg
-------------- | --------
baseline-html  |  1348.80
baseline-php   |  1341.31

That’s less than a 1% drop — close enough to make make me think that Rasmus might be pushing everything through the PHP interpreter, regardless of whether or not it’s a PHP file.

If that is true (and it’s a big “if”), then merely invoking PHP does appear to cause about a 45% drop (give or take) in Apache’s responsiveness, which is contrary to the point Rasmus makes on this slide about PHP “rarely” being a bottleneck — and I say that as someone who works with PHP almost exclusively. In fairness, I am depending only on the text of his slides here, so he may have said something to that effect in the presentation itself.

Failure modes on this analysis:

  • I am using XCache and not APC for the opcode cache. (Why? Because it works with both Lighttpd+FCGI and Apache+modphp, at least the last time I checked, and I’m interested in the differences between those two setups.)

  • I am using an EC2 server, which is more production-ish than Rasmus’ laptop.

  • I am using ab to benchmark with, not siege. I tried using Siege and did not notice any significant differences, so I’m sticking with the ab tools I’ve built for now.

I can’t imagine those three differences would lead to the kind of disparity in performance that I’m seeing, but it’s possible.

Has anyone else tried doing this?

Rasmus, if you have the time and inclination, would you care to shed some light on these prognostications?

Update: Is It EC2?

Rasmus replies below that he did not, in fact, have PHP running for all .html files.

For me, the next question was to see what the real difference on EC2 is between no cache, XCache, and APC.

No cache:


               |      avg
-------------- | --------
baseline-html  |  2339.25
baseline-php   |  1197.28

XCache (copied from above)


               |      avg
-------------- | --------
baseline-html  |  2367.02
baseline-php   |  1270.15

APC:


               |      avg
-------------- | --------
baseline-html  |  2315.83
baseline-php   |  1433.91

So on EC2, you get about 1200 req/sec without caching, about 1300 req/sec with XCache, and about 1400 req/sec with APC, in the “hello world” baseline scenarios.

Maybe this is all an artifact of how EC2 works, then? I have no idea. Next step is to test on non EC2 systems, if I can find one that others can reasonably build on themselves (since one of the goals here is for others to be able to duplicate the results).

Are you stuck with a legacy PHP application? You should buy my book because it gives you a step-by-step guide to improving your codebase, all while keeping it running the whole time.
Share This!Share on Google+Share on FacebookTweet about this on TwitterShare on RedditShare on LinkedIn

21 thoughts on “Rasmus Lerdorf’s Laconic(a) Performance

  1. The only other thing I can think of is the type of php install.

    Your using the precompiled mod_php comes with the OS where as Rasmus might be using a custom compile, maybe not even using mod_php/apxs but install compiled directly in with apache?

    Also by default centos/redhat have a lot of modules that get auto included based off the ini files in /etc/php.d/gd.ini for example.. I imagine this is a lot slower since it has to check for INI files and then load each module then a php where everything is compiled into a single binary!

    So, take a base centos version of php vs a hand compiled direct linked into apache with all used modules I suspect there might be a large difference in performance.. The question is if php is compiled directly into apache does every request suffer some “overhead” and therefore a html file vs a php file have are basically the same thing?

  2. Richard — entirely possible that he has it all compiled in. I only had a short time to experiment this morning, and what I did above was the simplest possible thing to try in under an hour. 😉 Even so, I would think that a file with no PHP in it would be served significantly faster than one with any PHP in it — but then, I could be wrong, and I’d love to know how to set up a system that does that.

  3. Honestly http://talks.php.net/show/froscon08/12 the change is so small it could be caulked up to standard deviation. Benchmarking from 45 rps to 46 rps on a laptop? is supposed to impress upon us that those changes caused the speed increase?

    A better number for him to base his “findings” on would be the 40% decrease in the number of system calls needed (just a rough guess based on diffing the 2 outputs)

  4. Did you configure APC to cache in memory? At least that is what Rasmus said he did in his presentation at FrOSCon.

  5. I recently did some basic tests with ab, comparing zero byte HTML and PHP files. The idea was to see what the highest number of requests per second was possible.

    http://www.slideshare.net/josephscott/wordpress-performance-scalability-presentation

    See slides 21 and 22. To sum up, zero byte HTML file was 1480 r/s and zero byte PHP file was 1359 r/s (using APC). Less than 10% drop and I was not having PHP parse .html files. I’m inclined to think that there’s something in your setup/test that is causing such a huge drop.

  6. Hi Joseph,

    You said, “I’m inclined to think that there’s something in your setup/test that is causing such a huge drop.”

    That’s certainly possible, although I am at a loss to think of what it might be. Perhaps the differences between XCache and APC really are that pronounced — I’ll have to check that out to make sure.

    I see from your slides that you were using Parallels VM on a Mac; while that’s perfectly reasonable for a development environment, I don’t think it’s representative of a production-style server. I know from personal experience that it’s better to run benchmarks on something close to the target hardware, to reduce variability.

    Perhaps you would be interested in trying your tests on a dedicated machine, or even a virtual private server like an EC2 instance, and reporting back?

  7. No, I didn’t screw up and had everything served through PHP.

    There are indeed less system calls for the PHP cached version. But more importantly there are 2 less disk-touching syscalls. We skip the open() and the read() by serving the script directly from memory. So what you lose from the PHP overhead you get back by by eliminating the disk access.

  8. I think the point Rasmus was trying to make is: “frameworks are slow”. The thing is, he doesn’t like object-oriented programming and wants to convince people that procedural programming is the next big thing. Come on Rasmus, that was 10 years ago. Web applications are now more complex. Rails changed the way we develop web applications. Django is becoming the next big thing and Guido supports it all the way. He even invites developers to talk about Pyhton frameworks at Google. PHP needs a Guido, not a Rasmus. Don’t you see, the “procedural is fast, object-oriented is slow” message is killing PHP.

  9. Running a zero-byte test off of a very small comercial VPS, I get the following results:

    HTML: 4944.03 [#/sec] (mean)
    PHP: 3584.90 [#/sec] (mean)

    So, PHP output is roughly 72%. This is with debian unstable, apache 2.2 prefork, dynamic PHP 5.2.6-3 w. Suhosin. 0.9.6.2, and Xcache.

    I can very easily imagine that with a bit more performance-friendly tweaking, like static PHP, not having a ton of extensions, etc, that number will come significantly closer to 90%, and very possibly past it.

    I have to agree that properly written PHP is almost never the bottleneck. The bottleneck is usually one of these three: 1) pointless overhead, 2) other forms of poor programming (whether procedural or object-oriented), or 3) the database.

  10. Hi Marcel —

    Note that Rasmus’ setup has less than a 1% difference between static HTML and a PHP equivalent, so what you’re seeing is about 28-times worse difference than Rasmus noted (and about 1.5-times better than the difference I’m seeing on Amazon EC2 ;-). I don’t know if Rasmus was using Suhosin or not.

    It appears to me that the operating environment has a huge impact on how much the opcode cache improves performance. It seems the lesson here is that opcode caching on Amazon EC2 in a “hello world” exercise does not gain you that much o of a benefit. An additional lesson is that benchmark numbers are simply not comparable between different operating environment (or, that some operating environments are more responsive than others, which intuitively makes sense to me).

    Per your last point (3), note that none of the “hello world” tests (either Rasmus’ or mine) involve the use of a database.

  11. I do not think Rasmus is saying no to OOP but he’s rather saying don’t over due it.

    An approach like “everything is a object” (where symfony is almost) is also in my oppinion not a good thing.

    We should be looking at ALL things with moderation… frameworks included… Even OOP…

  12. @Gabriel

    Looks like you missed the Drupalcon this year. Here’s the video:
    http://www.archive.org/details/simple_is_hard

    And a short transcript:

    1. Rasmus: “Tell me where the Web is going, and I’ll tell you what the future of PHP is”.

    I can only tell you that web applications are getting more an more complex, and that PHP is getting slower (unless you create a “hello world” application and tweak your laptop).

    2. Rasmus: “PHP is only useful for the presentation layer”.

    So, are you saying it’s kind of useless nowadays? Considering that Python and other scripting languages can do the same, and can be used not only for the presentation layer?

    3. Rasmus: “Facebook and Yahoo don’t scale because they use PHP, they scale because they use C++ libraries”.

    It can’t get worst than that… for us of course.

    4. Rasmus: “PHP frameworks are ridiculous slow”.

    Thanks for the tip Rasmus. I guess that’s why everyone is using Django now, right? According to you it’s faster than all the PHP framework together.

    5. Rasmus: “I don’t like frameworks”

    Yeah well, low-level programmers don’t like frameworks, but keep in mind that web developers do, and they need them.

    6. Rasmus: “The front-controller pattern goes against the web architecture”

    Yeah right, that’s why Rails and Django are becoming so popular right? Come on man, put yourself together and start supporting PHP developers. You are killing the language!

  13. @Rasmus

    Love your language, use it for everything web-related, but I have to say that you and I see a different PHP. For small sites, RAD is an ideal solution (where performance isn’t a big deal), and large sites can afford to optimize the site to match its growth.

    I’m not sure how you can keep from repeating yourself without the use of a Front Controller, which is really a combined router and bootstrap/common.inc file. Every application that wants routing to be more flexible than Apache can offer moves the responsibility to routing PHP code. The most necessary elements (first encountered and most frequently used) logically must be top-most parent element to avoid being called at all (which would require being called more than once).

Leave a Reply

Your email address will not be published. Required fields are marked *