UPDATE (2007-01-01): I have conducted a new series of benchmarks; see them here. The numeric results below should be considered as outdated, although the history, methodology, and approach are still valid.

Wherein I relate the results of benchmark testing on three (plus one)four frameworks: Cake, Solar, Symfony, and Zend Framework(plus Cake). Anger-inducing broad-brush overview: Solar is 4x faster than Zend, and almost 2x faster than Symfony. Read on for all the nuances and caveats.

UPDATE: Have updated the Cake results, now that I've figured out how to turn off database access.

UPDATE 2: Added note about "views" in Zend Framework, also added list of loaded files and classes for each framework.

Introduction

"Figures don't lie, but liars can figure." -- Anon.

Recently I became curious how many requests-per-second the Solar framework could handle. This was related to the recent Mashery launch; we needed to be ready to receive a ton of visitors to the site (I won't disclose those numbers at this time ;-).

Most Mashery applications are built on top of the Solar framework, but when benchmarking those applications, I needed to know what the theoretical limit to responsiveness was, as imposed by the Solar framework.

After that, I figured it would be interesting to know the theoretical limits to responsiveness imposed by other frameworks, in particular Cake, Symfony, and the Zend Framework. This blog entry reports my findings.

Goals and Caveats

The purpose of this is not to say that one framework is better than another, or to engage in one-upmanship. The results do not say whether an application will run faster or slower on a particular framework; bad application code will slow you down dramatically on any framework. Similarly, use of full-page caching will bypass the framework completely, raising the responsiveness to something closer to the web-server's maximum.

Instead, the purpose of this is to tell you what the maximum possible responsiveness of the framework will be under the simplest and most straightforward conditions, while using the framework's controller and view components. Anything you add as application code will only reduce its responsiveness from there.

In addition, these results do not constitute an endorsement or indictment of any particular framework; they all have different strengths and weaknesses. I think the ones I tested are the most full-featured, in that they provide full-up MVC separation, make good use of development patterns, and offer an existing set of plugins or added functionality like caching, logging, access control, authentication, form processing, and so on.

Finally, it is entirely possible that I have missed some important speed-enhancing technique with one or more of the frameworks. If so, it is due to ignorance on my part, not maliciousness. I have done my best to honestly and fairly represent each of the frameworks here, and I would be very happy to hear what I can do to speed up any of the frameworks I benchmark below.

Methodology

For each of the frameworks tested, I attempted to use the most-minimal controller and action setup possible, effectively a "Hello World!" implementation using the stock framework components and no configuration files (or as few as the framework would let me get away with). This was the only way I could think of to make sure the tests were identical on each framework.

The minimalist approach has a number of effects:

  • It negates most "if you cache, it goes faster" arguments. The "view" in each case is just a literal string "Hello World!" with no layout wrapper. (For Symfony and Cake, this meant editing the layout files to only echo the view output.)

  • It negates the "but the database connection speed varies" argument. Database access is not used at all in these benchmarks. (I could not figure out how to turn off database access in Cake, which is why I don't include it as a peer in the results.)

  • There's no application code to execute; the controller actions in each framework are empty and only call the view, which itself is only the literal "Hello World!" text. This shows us the maximum possible throughput; adding application code will only reduce responsiveness from this point.

Technology and Setup

All the benchmark tests were run on this hardware (admittedly underpowered):

  • Macintosh Mini G4 with 1G RAM and 7200 RPM hard drive
  • Stock installation of Apache 1.3.33
  • Custom compiled PHP 5.2.0
  • APC 3.0.11 enabled with shm_size of 30 and stat turned on

Each framework benchmark uses the following scripts or equivalents ...

  • Bootstrap file
  • Default configs (or as close as possible)
  • Front-controller or dispatcher
  • Page-controller or action-controller
  • One action with no code, other than invoking a View processor
    • No authentication, no access control, no filters, no database, etc.
    • (although Cake kind of forces the database on you, so it's not a full peer in this report)
  • Static view with only literal text "Hello World!"
    • No PHP, helpers, layout, escaping, etc.

... so the benchmark application in each case is very small.

I used the Apache benchmark "ab" tool for measuring requests-per-second, on localhost to negate network latency effects, with 10 concurrent requests for 60 seconds. The command looks like this:

ab -c 10 -t 60 http://localhost/[path]

Benchmark Test Results

First, we need to see what the maximum responsiveness is without any framework code. The web server itself is capable of delivering a huge number of static pages: 808 req/sec (!!!) for a file with only "Hello World!" in it.

Document Path:          /helloworld.html
Document Length:        12 bytes
Requests per second:    808.95 [#/sec] (mean)
Time per request:       12.36 [ms] (mean)
Time per request:       1.24 [ms] (mean, across all concurrent requests)
Transfer rate:          238.69 [Kbytes/sec] received

Invoking PHP to "echo 'Hello World!'" slows things down a bit, but it's still impressive (remember, APC is turned on for all requests): 597 req/sec.

Document Path:          /helloworld.php
Document Length:        12 bytes
Requests per second:    597.61 [#/sec] (mean)
Time per request:       16.73 [ms] (mean)
Time per request:       1.67 [ms] (mean, across all concurrent requests)
Transfer rate:          107.00 [Kbytes/sec] received

Not too shabby for a consumer Macintosh Mini optimized for desktop use.

Cake

The Cake page-controller looks like this:

<?php
class HelloWorldController extends AppController {
    var $layout = null;
    var $autoLayout = false;
    var $uses = array();
    function index()
    {
    }
}
?>

The page-controller automatically uses a related "index.thtml" view script.

Respresentative result from three "ab" runs on Cake 1.1.10.3825: 17 req/sec.

Document Path:          /cake/helloworld/index
Document Length:        12 bytes
Requests per second:    17.13 [#/sec] (mean)
Time per request:       583.67 [ms] (mean)
Time per request:       58.37 [ms] (mean, across all concurrent requests)
Transfer rate:          5.09 [Kbytes/sec] received

Solar

The Solar bootstrap file looks like this:

<?php
error_reporting(E_ALL|E_STRICT);
ini_set('display_errors', true);

require_once 'Solar.php';
Solar::start();

$front = Solar::factory('Solar_Controller_Front');
$front->display();

Solar::stop();
?>

The Solar page-controller code looks like this:

<?php
Solar::loadClass('Solar_Controller_Page');
class Solar_App_HelloMini extends Solar_Controller_Page {
    public function actionIndex()
    {
    }
}
?>

The page-controller automatically uses a related "index.php" file as its view script, and uses no layout.

Representative result from three "ab" runs on Solar 0.25.0: 45 req/sec.

Document Path:          /index.php/hello-mini/index
Document Length:        12 bytes
Requests per second:    44.78 [#/sec] (mean)
Time per request:       223.32 [ms] (mean)
Time per request:       22.33 [ms] (mean, across all concurrent requests)
Transfer rate:          16.75 [Kbytes/sec] received

Symfony (Stable and Beta)

The Symfony action controller code looks like this:

<?php
class worldActions extends sfActions
{
  public function executeIndex()
  {
  }
}
?>

The action controller automatically uses a related "indexSuccess.php" file as its view script. However, there is a default "layout.php" file that wraps the view output and adds various HTML elements; I edited it to look like this instead:

<?php echo $sf_data->getRaw('sf_content') ?>

If there is some way to make Symfony not use this layout file, please let me know.

Representative result from three "ab" runs on Symfony 0.6.3-stable: 26 req/sec.

Document Path:          /symfony/web/world/index
Document Length:        12 bytes
Requests per second:    25.60 [#/sec] (mean)
Time per request:       390.70 [ms] (mean)
Time per request:       39.07 [ms] (mean, across all concurrent requests)
Transfer rate:          8.40 [Kbytes/sec] received

Representative result from three "ab" runs on Symfony 0.7.1914-beta: 24 req/sec.

Document Path:          /symfony/web/world/index
Document Length:        12 bytes
Requests per second:    23.94 [#/sec] (mean)
Time per request:       417.72 [ms] (mean)
Time per request:       41.77 [ms] (mean, across all concurrent requests)
Transfer rate:          7.86 [Kbytes/sec] received

Zend Framework (0.2.0 and incubator)

The Zend Framework bootstrap file looks like this:

<?php
error_reporting(E_ALL|E_STRICT);
date_default_timezone_set('Europe/London');

set_include_path(
    '.'
    . PATH_SEPARATOR . './incubator/library'
    . PATH_SEPARATOR . get_include_path()
);

require_once 'Zend.php';
require_once 'Zend/Controller/Front.php';

Zend_Controller_Front::run('./application/controllers');
?>

The Zend Framework action controller code looks like this:

<?php
require_once 'Zend/Controller/Action.php';
class IndexController extends Zend_Controller_Action
{
    public function indexAction()
    {
        include dirname(dirname(__FILE__)) . '/views/index.php';
    }
}
?>

Zend Framework has a couple of special cases:

  1. The front-controller can be used by itself or with a Rewrite Router. On advice of Matthew Weier O'Phinney, I used the front-controller without the router, which slows things down.

  2. Zend Framework is still in heavy development; again, on advice of Matthew Weier O'Phinney, I added the "incubator" directory to the include_path to be sure the most-recent code is being used.

  3. The Zend_View class apparently has some issues on PHP 5.2.0 that preclude its use, so I just do a plain "include" to get the view script. This is much faster than using a separate View class, but in normal use you would have a Zend_View instance here instead.

Even with that special treatment, the respresentative result from three "ab" runs on Zend Framework 0.2.0 is the lowest of the bunch: 11 req/sec.

Document Path:          /zf/index
Document Length:        12 bytes
Requests per second:    11.17 [#/sec] (mean)
Time per request:       895.28 [ms] (mean)
Time per request:       89.53 [ms] (mean, across all concurrent requests)
Transfer rate:          2.66 [Kbytes/sec] received

"This Is Stupid And Doesn't Mean Anything!"

"There are three kinds of lies: lies, damn lies, and statistics." -- variously attributed to B. Disraeli, M. Twain, and others.

"A man's got to know his limitations." -- Dirty Harry Callahan

I can hear the objections now:

  • "Not realistic!"
  • "Not comprehensive!"
  • "Doesn't account for features that I like!"
  • "Who cares, I don't need that level of responsiveness!"
  • "Doesn't matter if Framework X is slower, I'm more productive with it!"

Yes, yes, you're all correct. But when you get to the point when you have to scale across multiple servers, and you're looking to squeeze out every possible request you can, it'll be good to know what the realistic top-limit is.

Last caveat: they say "your mileage may vary." In these cases, your mileage is almost certain to vary. The absolute numbers will be different from server to server. Data and output caching can help in many cases, but the moment you invoke the front-controller and page-controller, you're going to hit these relative top limits no matter how much partial caching you use:

                  Req/Sec    % Relative
Cake              17         154%
Solar             45         409%
Symfony (avg)     25         227%
Zend Framework    11         100% (baseline)

And again, it only gets slower from there; these numbers were for the simplest possible "Hello World!" application in each framework.

Appendix: What Classes Get Loaded?

John Herren in the comments below asks how many files are loaded each time; below I give the counts as reported by get_included_files(), as well as the classes used by each framework in the benchmark application.

Cake (26)

34 files total, including these classes:

  1. Object
  2. CakeSession
  3. Security
  4. NeatArray
  5. Inflector
  6. Configure
  7. Dispatcher
  8. Router
  9. Controller
  10. Component
  11. View
  12. Helper
  13. ConnectionManager
  14. DataSource
  15. DATABASE_CONFIG
  16. Model
  17. ClassRegistry
  18. AppModel
  19. HelloWorld
  20. AppController
  21. HelloWorldController
  22. SessionComponent
  23. DboSource
  24. DboMysql
  25. HtmlHelper
  26. SessionHelper

Solar

18 files total, including these classes:

  1. Solar
  2. Solar_Base
  3. Solar_Controller_Front
  4. Solar_Uri
  5. Solar_Uri_Action
  6. Solar_Request
  7. Solar_Controller_Page
  8. Solar_App_HelloMini
  9. Solar_Session
  10. Solar_View
  11. Solar_Struct
  12. Solar_View_Helper
  13. Solar_Class_Stack
  14. Solar_Path_Stack
  15. Solar_View_Helper_GetTextRaw

Symfony (43)

42 files total, including these classes:

  1. sfConfig
  2. sfToolkit
  3. Symfony
  4. sfConfigCache
  5. sfException
  6. sfComponent
  7. sfAction
  8. sfActions
  9. sfActionStack
  10. sfActionStackEntry
  11. sfContext
  12. sfController
  13. sfFilter
  14. sfExecutionFilter
  15. sfRenderingFilter
  16. sfFilterChain
  17. sfRequest
  18. sfResponse
  19. sfStorage
  20. sfUser
  21. sfView
  22. sfWebController
  23. sfFrontWebController
  24. sfWebRequest
  25. sfSessionStorage
  26. sfPHPView
  27. sfRouting
  28. sfDatabaseManager
  29. myFrontWebController
  30. myWebRequest
  31. sfParameterHolder
  32. sfWebResponse
  33. sfBasicSecurityUser
  34. myUser
  35. sfSecurityFilter
  36. sfBasicSecurityFilter
  37. worldActions
  38. sfCommonFilter
  39. sfFlashFilter
  40. sfValidatorManager
  41. sfOutputEscaper
  42. sfOutputEscaperGetterDecorator
  43. sfOutputEscaperArrayDecorator

Zend (28)

34 files total, including these classes:

  1. Zend
  2. Zend_Exception
  3. Zend_Registry_Exception
  4. Zend_Registry
  5. Zend_Controller_Front
  6. Zend_Controller_Exception
  7. Zend_Controller_Plugin_Abstract
  8. Zend_Controller_Request_Abstract
  9. Zend_Controller_Response_Abstract
  10. Zend_Controller_Plugin_Broker
  11. Zend_Controller_Router_Exception
  12. Zend_Controller_Request_Http
  13. Zend_Http_Exception
  14. Zend_Uri
  15. Zend_Uri_Exception
  16. Zend_Uri_Http
  17. Zend_Filter
  18. Zend_Filter_Exception
  19. Zend_Uri_Mailto
  20. Zend_Http_Request
  21. Zend_Controller_Request_Exception
  22. Zend_Controller_Router
  23. Zend_Controller_Dispatcher_Exception
  24. Zend_Controller_Action
  25. Zend_Controller_Action_Exception
  26. Zend_Controller_Dispatcher
  27. Zend_Controller_Response_Http
  28. IndexController
Are you stuck with a legacy PHP application? You should buy my book because it gives you a step-by-step guide to improving you codebase, all while keeping it running the whole time.