New Year’s Benchmarks

After the previous round of benchmarking, I received one very good criticism from Matthew Weier O’Phinney about it. He suggested that the hardware I was using, a PowerPC G4 Mac Mini, had an I/O system that was not representative of what a "regular" user would have as a web server. I have to agree with that.

As such, I have prepared a new set of benchmarks for Cake, Solar, Symfony, and Zend Framework using an almost identical methodology as last time.

Much of this article is copied directly from the previous one, as it serves exactly the same purpose, and little has changed in the methodology. Please consider the older article as superseded by this one.

Introduction

If you like, you can go directly to the summary.

This report outlines the maximum requests-per-second limits imposed by the following frameworks:

• Cake
• Solar
• Symfony
• Zend Framework

The benchmarks show what the approximate relative responsiveness of the framework will be when the framework’s controller and view components are invoked.

Full disclosure: I am the lead developer of one of the frameworks under consideration (Solar), and I have contributed to the development of another (Zend Framework). I have attempted not to let this bias my judgment, and I outline my methodology in great detail to set readers at ease concerning this.

Why These Particular Frameworks?

In my opinion, the frameworks in this comparison are the most full-featured of those available for PHP. They make good use of design patterns, provide front and page controllers, allow for controller/view separation, and offer an existing set of plugins or added functionality like caching, logging, access control, authentication, form processing, and so on.

Some Code Igniter folks left comments previously asking that I include CI in the benchmark process. I don’t mean to sounds like a jerk, but after reviewing the CI code base, it is my opinion (note the small "o") that it is not in the same class as Cake, Solar, Symfony, and Zend. I won’t go into further detail than that; I think reasonable and disinterested observers will tend to agree. Code Igniter is indeed very fast, but as I note elsewhere, there is more to a framework than speed alone.

Regarding frameworks in languages other than PHP, I don’t have the expertise to set them up for a fair comparison. However, you can compare the results noted in this report with this other report. Perhaps by extrapolation, one can estimate how Rails and Django compare to the other PHP frameworks listed here.

Methodology

For each of the frameworks tested, I attempted to use the most-minimal controller and action setup possible, effectively a "Hello World!" implementation using the stock framework components and no configuration files (or as few as the framework would let me get away with). This was the only way I could think of to make sure the tests were identical on each framework.

The minimalist approach measures the responsiveness of the framework components themselves, not an application. There’s no application code to execute; the controller actions in each framework do the least possible work to call a view. This shows us the maximum possible throughput; adding application code will only reduce responsiveness from this point.

In addition, this approach negates most "if you cache, it goes faster" arguments. The "view" in each case is just a literal string "Hello World!" with no layout. It also negates the "but the database connection speed varies" argument. Database access is not used at all in these benchmarks.

Server

Previously, I ran the benchmarks on a Mac Mini G4; as noted by others, it is not exactly a "real" production server. With that in mind, these new benchmark tests were run on an Amazon EC2 instance, which is a far more reasonable environment:

• 1.7Ghz x86 processor
• 1.75GB of RAM
• 160GB of local disk
• 250Mb/s of network bandwidth.

Clay Loveless set up the instance with …

• Fedora Core 5
• Apache 2.2 and mod_php
• PHP 5.2.0 with Xcache (64M RAM)

Setup

Each framework benchmark uses the following scripts or equivalents …

• Bootstrap file
• Default configuration (or as close as possible)
• Front-controller or dispatcher
• Page-controller or action-controller
• One action with no code, other than invoking a View processor
• Static view with only literal text "Hello World!"

… so the benchmark application in each case is very small.

I did not modify any of the code in the frameworks, except for one condition. I modified the bootstrap scripts so they would force the session_id() to be the same every time. This is because the benchmarking process starts a new session with each request, and that causes responsiveness as a whole to diminish dramatically.

Benchmarking Tools

I wrote a bash script to automate the benchmarking process. It uses the Apache benchmark "ab" tool for measuring requests-per-second, on localhost to negate network latency effects, with 10 concurrent requests for 60 seconds. The command looks like this:

ab -c 10 -t 60 http://localhost/[path]

The benchmark script restarts the web server, then runs the "ab" command 5 times. This is repeated for each version of each framework in the test series, so that each gets a "fresh" web server and xcache environment to work with.

Benchmark Results

Baseline

First, we need to see what the maximum responsiveness of the benchmark environment is without any framework code.

framework	1	2	3	4	5	avg
baseline-html	2613.56	2284.98	2245.98	2234.94	2261.01	2328.09
baseline-php	1717.74	1321.49	1292.86	1511.40	1327.35	1434.17

The web server itself is capable of delivering a huge number of static pages: 2328 req/sec for a file with only "Hello World!" in it.

Invoking PHP (with Xcache turned on) to "echo ‘Hello World!’" slows things down a bit, but it’s still impressive: 1434 req/sec.

Cake

I ran benchmarks for Cake with the debug level set to zero. I edited the default bootstrap script to force the session ID with this command at the very top of the file…

<?php
session_id('abc');
?>

… but made no other changes to the distribution code.

The page-controller looks like this:

<?php
class HelloController extends AppController {
    var $layout = null;
    var $autoLayout = false;
    var $uses = array();
    function index()
    {
    }
}
?>

The page-controller automatically uses a related "index.thtml" view.

Running the benchmarking script against each version of Cake gives these results:

framework	1	2	3	4	5	avg
cake-1.1.10	85.65	85.87	85.71	85.66	85.93	85.76
cake-1.1.11	113.78	114.13	113.59	113.35	113.73	113.72
cake-1.1.12	114.62	114.26	114.55	113.89	114.64	114.39

So the most-recent release of Cake in a fully-dynamic mode has a top-limit of about 114 requests per second in the benchmarking environment.

Solar

The Solar bootstrap file looks like this; note that we force the session ID for benchmarking purposes.

<?php
session_id('abc');

error_reporting(E_ALL|E_STRICT);
ini_set('display_errors', true);

$dir = dirname(__FILE__) . DIRECTORY_SEPARATOR;

$include_path = $dir . 'source';
ini_set('include_path', $include_path);

require_once 'Solar.php';

$config = $dir . 'Solar.config.php';
Solar::start($config);

$front = Solar::factory('Solar_Controller_Front');
$front->display();

Solar::stop();
?>

The Solar page-controller code looks like this:

<?php
Solar::loadClass('Solar_Controller_Page');
class Solar_App_HelloMini extends Solar_Controller_Page {
    public function actionIndex()
    {
    }
}
?>

The page-controller automatically uses a related "index.php" file as its view script, and uses no layout.

Running the benchmarking script against each version of Solar gives these results:

framework	1	2	3	4	5	avg
solar-0.25.0	170.80	169.22	170.29	170.75	170.22	170.26
solar-svn	167.83	166.97	167.56	164.56	162.30	165.84

Clearly the current Subversion copy of Solar needs a little work to bring it back up the speed of the most recent release, but it can still handle 165 requests per second in the benchmarking environment.

Symfony

The Symfony action controller code looks like this:

<?php
class helloActions extends sfActions
{
  public function executeIndex()
  {
  }
}
?>

The action controller automatically uses a related "indexSuccess.php" file as its view script. However, there is a default "layout.php" file that wraps the view output and adds various HTML elements; I edited it to look like this instead:

<?php echo $sf_data->getRaw('sf_content') ?>

If there is some way to make Symfony not use this layout file, please let me know.

Finally, I added session_id('abc'); to the top of the web/index.php bootstrap script to force the session ID to be same for each request.

Running the benchmarking script against each version of Symfony gives these results:

framework	1	2	3	4	5	avg
symfony-0.6.3	53.10	53.15	52.27	52.12	52.10	52.55
symfony-1.0.0beta2	67.42	66.92	66.65	67.06	67.83	67.18

It looks like the most-recent beta version of Symfony can respond to about 67 requests per second in the benchmarking environment.

Zend Framework

As before, the Zend Framework involves some extra work. As others have noted, Zend Framework requires more putting-together than the other projects listed here.

The Zend 0.2.0 bootstrap script looks like this …

<?php
session_id('abc');

error_reporting(E_ALL|E_STRICT);
ini_set('display_errors', true);
date_default_timezone_set('Europe/London');

$dir = dirname(__FILE__) . DIRECTORY_SEPARATOR;

set_include_path($dir . 'source/library');

require_once 'Zend.php';
require_once 'Zend/Controller/Front.php';

Zend_Controller_Front::run($dir . 'application/controllers');
?>

… and the Zend 0.6.0 bootstrap looks like this:

<?php
session_id('abc');

error_reporting(E_ALL|E_STRICT);
ini_set('display_errors', true);
date_default_timezone_set('Europe/London');

$dir = dirname(__FILE__) . DIRECTORY_SEPARATOR;

set_include_path($dir . 'source/library');

require_once 'Zend.php';
require_once 'Zend/Controller/Front.php';

$front = Zend_Controller_Front::getInstance();
$front->setControllerDirectory($dir . 'application/controllers');
$front->setBaseUrl('/zend-0.6.0/');

echo $front->dispatch();
?>

The differences are minor but critical, since the front-controller internals changed quite a bit between the two releases.

Similarly, the page-controller code is different for the two releases as well. Zend Framework does not initiate a session on its own, whereas the other frameworks do. I think having a session available is a common and regular requirement in a web environment. Thus, I added a session_start() call to the page-controller for Zend; this mimics the session-start logic in the other frameworks. Also, the Zend Framework does not automatically call a view, so the page-controller code below mimics that behavior as well.

The page-controller code for Zend 0.2.0 looks like this …

<?php
require_once 'Zend/Controller/Action.php';
require_once 'Zend/View.php';
class IndexController extends Zend_Controller_Action 
{
    public function norouteAction()
    {
        return $this->indexAction();
    }

    public function indexAction()
    {
        // mimic the session-starting behavior of other application frameworks
        session_start();

        // now for the standard portion
        $view = new Zend_View();
        $view->setScriptPath(dirname(dirname(__FILE__)) . '/views');
        echo $view->render('index.php');
    }
}
?>

… and the page-controller code for Zend 0.6.0 looks like this:

<?php
require_once 'Zend/Controller/Action.php';
require_once 'Zend/View.php';
class IndexController extends Zend_Controller_Action 
{
    public function indexAction()
    {
        // mimic the session-starting behavior of other application frameworks
        session_start();

        // now for the standard portion
        $view = new Zend_View();
        $view->setScriptPath(dirname(dirname(__FILE__)) . '/views');
        echo $view->render('index.php');
    }
}
?>

Running the benchmarking script against each version of Zend Framework gives these results:

framework	1	2	3	4	5	avg
zend-0.2.0	215.91	208.50	207.84	210.43	211.73	210.88
zend-0.6.0	133.60	134.18	132.10	129.53	130.16	131.91

In the benchmarking environment, the most-recent Zend Framework can handle about 131 requests per second.

But look at Zend 0.2.0 — 210 req/sec! Richard Thomas noted this in a comment on the last set of benchmarks. If that’s a sign of what a future release might be capable of, it’s quite stunning.

Summary

To compare the relative responsiveness limits of the frameworks, I assigned the slowest average responder a factor of 1.00, and calculated the relative factor of the others based on that.

framework	avg	rel
cake-1.1.10	85.76	1.63
cake-1.1.11	113.72	2.16
cake-1.1.12	114.39	2.18
solar-0.25.0	170.26	3.24
solar-svn	165.84	3.16
symfony-0.6.3	52.55	1.00
symfony-1.0.0beta2	67.18	1.28
zend-0.2.0	210.88	4.01
zend-0.6.0	131.91	2.51

Thus, the Zend 0.2.0 framework by itself is about 4 times more responsive than Symfony 0.6.3. (My opinion is that this is likely due to the fact that the Zend code does less for developers out-of-the-box than Symfony does.)

For comparison of the currently-available options, we can highlight the most-recent releases of each framework, and place them in order for easy reading:

framework	avg	rel
solar-0.25.0	170.26	3.24
zend-0.6.0	131.91	2.51
cake-1.1.12	114.39	2.18
symfony-1.0.0beta2	67.18	1.28

As we can see, the framework with the greatest limits on its responsiveness is Symfony. Zend is about 15% faster than Cake. And as with the last benchmark series, Solar has the least limit on its responsiveness in a dynamic environment.

You can download the individual benchmark log files, including the report summary table, here.

Conclusion

It is important to reiterate some points about this report:

• The benchmarks note the top limits of responsiveness of the frameworks components themselves. This functions as a guide to maximum speed you will be able to squeeze from an application using a particular framework. An application cannot get faster than the underlying framework components. Even "good" application code added to the framework will reduce responsiveness, and "bad" application code will reduce it even further.

• Each of these frameworks is capable of building and caching a static page that will respond at the maximum allowed by the web server itself (about 2300 requests/second, as noted in the baseline runs). In such cases, the responsiveness of the framework itself is of no consequence whatsoever. But remember: the moment you hit the controller logic of a framework, responsiveness will drop to the levels indicated above.

• Dynamic responsiveness is only one of many critical considerations when choosing a framework. This benchmark cannot measure intangibles such as productivity, maintainability, quality, and so on.

Again, it is entirely possible that I have failed to incorporate important optimizations to one or more of the frameworks tested. If so, it is through ignorance and not maliciousness. If you are one of the lead developers on the frameworks benchmarked here and feel your project is unfairly represented, please contact me personally, and I’ll forward a tarball of the entire benchmark system (including the framework code as tested) so you can see where I may have gone wrong.

UPDATE (2007-01-05): I have created a repository of the benchmark project code for all to examine.