Labor Day Benchmarks

By popular request, here is an update of my web framework benchmarks report. You can see previous result sets here:

Before you comment on this post, please have the courtesy to read at least the first two articles above; I am tired of refuting the same old invalid arguments about “hello world makes no sense”, “if you cache, it goes faster”, “the ORM systems are different”, and “speed isn’t everything” with people who have no understanding of what these reports actually say.

Full disclosure: I am the lead developer on the Solar Framework for PHP 5, and I was an original contributor to the Zend framework.

In the interest of putting to rest any accusations of bias or favoritism, the entire project codebase is available for public review and criticism here.

Flattered By Imitators

They say that imitation is the sincerest form of flattery. As such, I am sincerely flattered that the following articles and authors have adopted methodologies strikingly similar to the methodology I outlined in Nov 2006.

  • SellersRank here and here.
  • AVNet Labs here.
  • Rasmus Lerdorf here. I am considering writing a separate post about this talk by Rasmus.

Methodology, Setup, and Source Code

The methodology in this report is nearly identical to that in previous reports. I won’t duplicate that narrative here; please see this page for the full methodology.

The only difference from previous reports regards the server setup. Although I’m still using an Amazon EC2 instance, I now provide the full setup instructions so you can replicate the server setup as well as the framework setup. See this page for server setup instructions.

Finally, you can see all the code used for the benchmarking here.

Results, Part 1

Update: FYI, opcode caching is turned on for these results.

The “avg” column is the number of requests/second the framework itself can deliver, with no application code, averaged over 5 one-minute runs with 10 concurrent users. That is, the framework dispatch cycle of “boostrap, front controller, page controller, action method, view” will never go any faster than this.

The “rel” column is a percentage relative to PHP itself. Thus, if you see “0.1000” that means the framework delivers 10% of the maximum requests/second that PHP itself can deliver.

framework avg rel
baseline-html 2309.14 1.7487
baseline-php 1320.47 1.0000
cake-1.1.19 118.30 0.0896
cake-1.2.0-rc2 46.42 0.0352
solar-1.0.0alpha1 154.29 0.1168
symfony-1.0.17 67.35 0.0510
symfony-1.1.0 67.41 0.0511
zend-1.0.1 112.36 0.0851
zend-1.5.2 86.23 0.0653
zend-1.6.0-rc1 77.85 0.0590

We see that the Apache server can deliver 2300 static “hello world” requests/second. If you use PHP to echo "Hello World!" you get 1300 requests/second; that is the best PHP will get on this particular server setup.

Cake: After conferring with the Cake lead developers, it looks like the 1.2 release has some serious performance issues (more than 50% drop in responsiveness from the 1.1 release line). They are aware of this and are fixing the bugs for a 1.2.0-rc3 release.

Solar: The 1.0.0-alpha1 release is almost a year old, and while the unreleased Subversion code is in production use, I make it a point not to benchmark unreleased code. I might do a followup report just on Solar to show the decline in responsiveness as features have been added.

Symfony: Symfony remains the least-responsive of the tested frameworks (aside from the known-buggy Cake 1.2.0-rc1 release). No matter what they may say about Symfony being “fast at its core”, it does not appear to be true, at least not in comparison to the other frameworks here. But to their credit, they are not losing performance. (Could it be there’s not much left to lose? 😉 In addition, I continue to find Symfony to be the hardest to set up for these reports — more than half my setup time was spent on Symfony alone.

Zend: The difference between the 1.0 release and the 1.5 release is quite dramatic: a 25% drop in responsiveness. And then another 10% drop between 1.5 and 1.6.

To sum up, my point from earlier posts that “every additional line of code will reduce responsiveness” is illustrated here. Each of the newer framework releases has added features, and has slowed down as a result. This is neither good nor bad in itself; it is an engineering and economic tradeoff.

Results, Part 2

I have stated before that I don’t think it’s fair to compare CodeIgniter and Prado to Cake, Solar, Symfony, and Zend, because they are (in my opinion) not of the same class. Prado especially is entirely unlike the others.

Even so, I keep getting requests to benchmark them, so here are the results; the testing conditions are idential to those from the main benchmarking.

framework avg rel
baseline-html 2318.89 1.7710
baseline-php 1309.39 1.0000
ci-1.5.4 229.29 0.1751
ci-1.6.2 189.89 0.1450
prado-3.1.0 39.86 0.0304

CodeIgniter: Even the CI folks are not immune to the rule that “there is no such thing as a free feature”; between 1.5.4 and 1.6.2 releases they lost about 18% of their requests/second. However, they are still running at 14.5% of PHP’s maximum, compared with the 11.68% of Solar-1.0.0-alpha1 (the most-responsive of the frameworks benchmarked above), so it’s clearly the fastest of the bunch.

Prado: Prado works in a completely different way than the other frameworks listed here. Even though it is the slowest of the bunch, it’s simply not fair to compare it in terms of requests/second. If the Prado way of working is what you need, then the requests/second comparison will be of little value to you.

This Might Be The Last Time

Although I get regular requests to update these benchmark reports, it’s very time-consuming and tedious. It took five days to prepare everything, add new framework releases, make the benchmark runs, do additional research, and then write this report. As such, I don’t know when (if ever) I will perform public comparative benchmarks again; my thanks to everyone who provided encouragement, appreciation, and positive feedback.

php|works 2007 Teaser: Framework Benchmarks!

I’m giving two talks at php|works this year, and it turns out that they complement each other quite nicely. In a way, they are both about organization and architecture.

The first one is literally about how to organize your PHP project. (Yes, that link does show you all the slides for the talk – but you get a lot more info by attending. [Update: I have modified the presentation rather thoroughly, so those slides are no longer fully representative of the talk.]) Easy one-liner: “Project Planning In One Lesson.”

The second one dovetails nicely from “how to organize” to “how to measure your architectural decisions”. This talk is based on lessons from my benchmarking experiments combined with an extended riff on something Laura Thomson said about those experiments:

That, by the way, is an excellent, excellent article that displays a good methodology for researching design decisions.

(Thanks for the multiple compliments, Laura. 🙂

An Attendance Request

I would really like it if at least one experienced developer on each of the frameworks (Cake, Code Igniter, Prado, Symfony, Zend) could attend the benchmarking talk and critique it. I’m pretty sure there’s going to be a lot of questions and discussion during and after the presentation about each of the projects. Any takers?

About The Benchmarking Talk

Easy one-liner: “Statistical Histories for Fun and Profit.” And I do mean profit: the better your apps runs on a given stack, the more eyeballs you get on your ads (or, the more users you can handle).

  • First, I will talk briefly about why you should do benchmarking on your “whole app” (as opposed to profiling sub-components), give a short example on benching an application, and go from there to talk about discovering the limits imposed on you by the various portions of the server + software stack.
  • The second half of the talk is about how (and why) to benchmark a framework, and how to fairly compare different frameworks to each other on a level playing field.
  • At the end, I’ll give new benchmark numbers on Cake, Solar, Symfony, and Zend. Bonus: I include Code Igniter and Prado, too! But probably not in the context the Code Igniter guys want. You’ll have to attend the talk to see what I mean. 😉

A couple of slides from the talk:

Framework Benchmarking

Compare Like With Like

A Bit About Benchmarks

As the author of a relatively popular benchmarking article, I feel compelled to respond to this bit of misguided analysis from the Symfony camp about benchmarks.

Full disclosure: I am the lead developer on the Solar framework, and was a founding contributor to the Zend framework.

M. Zaninotto sets up a number of straw-man arguments regarding comparative benchmarks in general, although he does not link to any specific research. In doing so, he misses the point of comparative benchmarking almost entirely. Herein I will address some of M. Zaninotto’s arguments individually in reference to my previous benchmarking series.

All of the following commentary regards benchmarking and its usefulness in decision-making, and should not be construed as a general-purpose endorsement or indictment of any particular framework. Some frameworks are slower than others, and some are faster, and I think knowing “how fast is the framework?” is an important consideration when allocating scarce resources like time, money, servers, etc.

And now, on to a point-by-point response!

Continue reading

New Year’s Benchmarks

After the previous round of benchmarking, I received one very good criticism from Matthew Weier O’Phinney about it. He suggested that the hardware I was using, a PowerPC G4 Mac Mini, had an I/O system that was not representative of what a "regular" user would have as a web server. I have to agree with that.

As such, I have prepared a new set of benchmarks for Cake, Solar, Symfony, and Zend Framework using an almost identical methodology as last time.

Much of this article is copied directly from the previous one, as it serves exactly the same purpose, and little has changed in the methodology. Please consider the older article as superseded by this one.

Continue reading

The Interesting Case of Diminishing Responsiveness

I’m working up a second run of PHP framework benchmarks on more powerful hardware than the last time. I’ll talk more about that in a future entry, but in the mean time, I wanted to share an experience that may help you out.

I was running the ab (Apache Benchmark) tool against Zend Framework, Solar, and Cake, using the minimalist “hello world” applications I described the previously. The ZF requests/minute were consistent between runs, always “Z” amount (plus or minus a couple percentage points).

But the Solar and Cake requests/minute declined considerably from run to run. For example, the first benchmark run would yield “X” requests/minute; the 2nd run would yield 0.93X, the 3rd 0.85X, then 0.75X, 0.64X, 0.52X, and so on. This was with the opcode caching turned on; without it, the decline was less severe, but it was still there. Very depressing.

Because I know Solar better than I know Cake, I started looking around the codebase for places to improve. Was it the way I was loading classes, thus causing the opcode cache to work inefficiently? Was it a memory leak in Solar, or PHP itself, or maybe in the web server somehow?

Nope: it was sessions.

Solar and Cake each call session_start() on new requests. Because the “ab” tool sends a new request each time, the frameworks start a new session each time. This means that 1000 requests will generate 1000 new session files. Because the benchmarking makes thousands of requests, the /tmp directory was overflowing with sess_* files (50,000 or more). Having a very large number of files in a single directory was causing the filesystem to drag, and so response time diminished exponentially over time.

Notice that Zend Framework did not have this problem, because it doesn’t have a session-management class; thus, no sessions were created, explaining why its performance stayed consistent from run to run.)

The solution (for benchmarking purposes) was to force the session ID value to ‘abc’ by calling session_id('abc') command at the top of the Solar and Cake bootstrap scripts. This makes the script think there’s already a session in place, and so it doesn’t create a new file. Once I did that, the diminishing responsiveness disappeared, and I got consistent results from run to run.

How Fast Is Your Framework?

UPDATE (2007-01-01): I have conducted a new series of benchmarks; see them here. The numeric results below should be considered as outdated, although the history, methodology, and approach are still valid.

Wherein I relate the results of benchmark testing on three (plus one)four frameworks: Cake, Solar, Symfony, and Zend Framework(plus Cake). Anger-inducing broad-brush overview: Solar is 4x faster than Zend, and almost 2x faster than Symfony. Read on for all the nuances and caveats.

UPDATE: Have updated the Cake results, now that I’ve figured out how to turn off database access.

UPDATE 2: Added note about “views” in Zend Framework, also added list of loaded files and classes for each framework.

Continue reading

Easy Benchmarking with Solar

Comparative benchmarking is tedious work. It’s not hard to do, it’s just no fun to set up the same scaffold every time you want to figure out how long it takes different pieces of code to execute.

To do a benchmark, essentially you need only four things: a timing mechanism, a loop to run the first case, a loop to run the second case, and a way to show the time taken by each loop.


$loopTimes = 1000;

$before = microtime(true);
for ($i = 0; $i < = $loopTimes; ++$i) {
    // case #1:
    // this code gets run $loopTimes
}
$after = microtime(true);
$case1 = $after - $before;

$before = microtime(true);
for ($i = 0; $i <= $loopTimes; ++$i) {
    // case #2:
    // this code gets run $loopTimes
}
$after = microtime(true);
$case2 = $after - $before;

echo "Case 1 ran $loopTimes in $case1 seconds";
echo "Case 2 ran $loopTimes in $case2 seconds";

This is boring, and repetitive, and essentially the same every time (except for the code cases being benchmarked). This makes it a classic candidate for refactoring to a PHP class.

Enter Solar_Test_Bench. As with Solar_Test, you extend the base class, add your own code cases as bench*() methods, and a runner will execute each of those methods the same number of times and display how long each one took. (It uses Solar_Debug_Timer for the timing mechanism.)

Here’s an example of a real benchmark test from Solar to test the speed difference between require_once() and Solar::loadClass().


Solar::loadClass('Solar_Test_Bench');
class Bench_Solar_LoadClass extends Solar_Test_Bench {

    public $file = 'Solar/Sql.php';
    public $class = 'Solar_Sql';

    public function benchLoadClass()
    {
        Solar::loadClass($this->class);
    }

    public function benchRequireOnce()
    {
        require_once $this->file;
    }
}

To run it, we call the benchmark runner to execute each method in the class 1000 times, and get this output:


bash-2.05$ php bench.php Solar_FileExists 1000
name             : diff     : total
__start          : 0.000000 : 0.000000
benchLoadClass   : 0.005793 : 0.005793
benchRequireOnce : 0.034572 : 0.040365
__stop           : 0.000012 : 0.040377
bash-2.05$

You would want to run it a few times to get a better idea of the real speed, but the above is a representative result from my MacBook 2.0GHz. Looking at the “diff” column we can see that Solar::loadClass() is about 5 times faster than using require_once(). (Thanks to Clay Loveless for thinking up this particular benchmark.)

Here’s another one:

Currently, Solar::fileExists() uses an fopen() call to see if a file exists on the include_path. (The regular file_exists() and is_readable() PHP calls do not check the include path.) However, Clay Loveless wondered if it would be faster to explode() the include_path and see if the file_exists() on each of those individually.

I won’t copy the benchmarking class here; you can see it at Bench_Solar_FileExists. We actually test three different cases; the current fopen() case, and two new versions of using include_path. The results are:


bash-2.05$ php bench.php Solar_LoadClass 1000
name                   : diff     : total
__start                : 0.000000 : 0.000000
benchFopen             : 0.040105 : 0.040105
benchExplodeInclPath   : 0.015582 : 0.055687
benchNoExplodeInclPath : 0.020612 : 0.076299
__stop                 : 0.000004 : 0.076303
bash-2.05$

So the second case, where we explode() the include_path and then check file_exists() on each of the resulting paths, is about 3 times faster than using fopen(). The next release of Solar will use that code instead of the current fopen() code as a result.

(Note: the benchmarking classes are in Subversion only at this point, but will be in the next release of Solar when it comes out.)

Benchmarking call_user_func_array()

Andreas Korthaus wrote the Savant mailing list recently with some interesting benchmark numbers on Savant3 and the output escaping routines. He wrote up a quick template script (in PHP of course) and used the Savant eprint() method 300+ times to escape output (which itself uses only htmlspecialchars() by default). Then he ran the script; the time to run was …

Result: 161 ms

After that I looked at the cachegrind listings to find out
where some noticable time is lost. Looks like escaping costs a
lot of time, so I changed my templates not to use output
escaping:

Result: 12 ms!!!

No, I did not forget a digit 😉

So it seems the way escaping is implemented, hurts performance
very much!

That very dramatic difference in running times (14x!) looked mostly due to the fact that he wasn’t using htmlspecialchars() on the echoed output. Then he changed all the “echo” to “echo htmlspecialchars()” and claimed only a 1.5ms difference. Followup emails from his continued benchmarking noted the slow-points in Savant were related primarily to call_user_func_array() calling the escaping functions, although the output escaping itself also adds running time.

All this piqued my interest to see the speed difference between using native functions, user-defined functions, objects, variable-functions, and the call_user_func[_array]() functions. For an experiment, I wrote a user-defined function …

// mimics htmlentities() as a user-defined function
function html($value)
{
	return htmlentities($value);
}

… and a user-defined class:

class example {
	// mimics htmlentites() as a method
	function html($value)
	{
		return htmlentities($value);
	}
}

Then I wrote a script to make 100,000 calls to each of these:

  • htmlentities($value)
  • html($value)
  • $func($value) where $func = ‘html’
  • $object->html($value) where $object = new example()
  • $object->$func($value)
  • call_user_func($func, $value)
  • call_user_func(array($object, $func), $value)
  • call_user_func_array($func, array($value))
  • call_user_func_array(array($object, $func), array($value))

One set of representative results from my Mac Mini (PHP 5.1rc1), in seconds:

name            : diff     : description
native          : 0.614219 : htmlentities($value)
literal_func    : 0.745537 : html($value)
variable_func   : 0.826048 : $func($value)
literal_method  : 0.957708 : $object->html($value)
variable_method : 0.840837 : $object->$func($value)
call_func       : 1.006599 : call_user_func($func, $value)
call_object     : 1.193323 : call_user_func((array($object, $func), $value)
cufa_func       : 1.232891 : call_user_func_array($func, array($value))
cufa_object     : 1.309725 : call_user_func_array((array($object, $func), array($value)

Here’s another one:

name            : diff     : total
native          : 0.772733 : 0.772733
literal_func    : 0.737210 : 1.509943
variable_func   : 0.807990 : 2.317933
literal_method  : 0.820931 : 3.138864
variable_method : 0.821516 : 3.960380
call_func       : 1.006845 : 4.967225
call_object     : 1.210223 : 6.177448
cufa_func       : 1.114493 : 7.291941
cufa_object     : 1.307970 : 8.599911

And another, just for good measure:

name            : diff     : total
native          : 0.613295 : 0.613295
literal_func    : 0.733299 : 1.346594
variable_func   : 0.815782 : 2.162376
literal_method  : 0.965143 : 3.127519
variable_method : 0.842771 : 3.970290
call_func       : 1.023640 : 4.993930
call_object     : 1.221747 : 6.215677
cufa_func       : 1.104610 : 7.320287
cufa_object     : 1.449468 : 8.769755

So native calls to htmlentities() are twice as fast as doing effectively the same thing via call_user_func() with an object method, and using call_user_func_array() is 10-20% slower than using call_user_func(). I hadn’t realized the differences in speed of execution would be so distinct; of course, I never really thought about it before, either. Clearly PHP has to do a lot more work behind the scenes to map the variables to objects and parameters when using call_user_func_array().

You can download the very simple func_speed.php script here.

Um, you’ll need Solar for it (sorry about that, I used the debug-timer object for the testing).