Passing the Benchmarking Torch After 7 Years

Long-time readers will recall that I am interested in performance benchmarks as a tool to help discover the outer limits of framework responsiveness. See the blog category along with the most recent measurement report and the GitHub repo for replicating the results on your own.

Benchmarking is useful because it helps you decide if it makes more sense to work on improving your application, improving your framework, or improving your server. If the maximum dynamic responsiveness of the framework in question is 200 req/sec, and you need 300 req/sec, then there is no code you can add in your application to make it go faster in a dynamic scenario; you will have to look at the framework or the server. (Lazy reader who considers himself an unappreciated genius interjects: “But you can cache it!” I said a dynamic scenario, not a static one.)

There’s a lot of emotion and drama associated with benchmarking. The subjects that come in “first place” too often point to it as SCIENCE PROVES WE ARE BEST and the subjects that come in “last place” respond with variations of THIS IS STUPID AND PROVES NOTHING. (That is, until the last-placers run their own benchmarks where they come in first, and suddenly it’s a great marketing point. I’m looking at you, Symfony.) The point in benchmarking is to add information to your decision-making process so you can make better use of your limited resources of time and effort, and choose between competing tradeoffs in a more informed way. Speed alone over-and-above anything and everything is for suckers; it’s an important point, not the important point, when evaluating tradeoffs.

Leaving the elements of drama aside, benchmarking properly is difficult and time-consuming work. For my own limited benchmarks, it took three days or more to properly update, test, run, fix, and re-run to perform them well, even with automated scripts to do the setup and analysis. And that was for the most basic bare bones “hello world” that benchmarks only the dynamic dispatch cycle (bootstrap, front controller, page controller, action method, and view rendering).

Enter the guys at TechEmpower.

They’re doing a series of regular benchmarks that includes not just a double-handful of PHP frameworks, but 90 frameworks/languages/foundations across several languages. They do the basic “hello world” bench in addition to a few others, such as ORM/database speed. They appear to share an approach similar to the one I first published in Nov 2006 and improved with the help of Clay Loveless in Jan 2007. The TechEmpower motivations appear to the be similar to mine as well. They have equalled and then exceeded the efforts that I’ve been able to put forth on my own. From what I can tell it’s really good work.

With that, I am happy to say that I will be retiring my benchmarking project in favor of the TechEmpower one. Until futher notice, I’ll be combining my efforts (such as they may be) with the TechEmpower folks.

Comparing Benchmark Tools

As I noted last week, I have moved my framework benchmarking project to GitHub. As part of the move, I updated the project to allow benchmarking using any of three tools: Acme http_load, Apache ab, or Joedog siege. (For reference, the old project will remain at GoogleCode.)

I thought it might be interesting to see what each of them reports for the baseline “index.html” and “index.php” cases on the new Amazon EC2 setup (using a 64-bit OS on an m1.large instance). The results follow (all are at 10 concurrent users, averaged over 5 one-minute runs):

ab                       |      rel |      avg |
------------------------ | -------- | -------- |
baseline-html            |   1.2660 |  3581.54 |
baseline-php             |   1.0000 |  2829.11 |

http_load                |      rel |      avg |
------------------------ | -------- | -------- |
baseline-html            |   1.2718 |  4036.24 |
baseline-php             |   1.0000 |  3173.56 |

siege                    |      rel |      avg |
------------------------ | -------- | -------- |
baseline-html            |   1.2139 |  5060.25 |
baseline-php             |   1.0000 |  4168.76 |

They all show very different “absolute” numbers of requests/second: ab thinks the server delivers about 3600 req/sec, http_load reports about 4000, and siege says about 5000.

Note that the ab and http_load relative scores are in line with each other, reporting about a 26-27% slowdown for invoking PHP. Siege thinks PHP is more responsive than that, with only a 21% slowdown.

Which of these is the most accurate? I don’t know. I ran the benchmarking tool on the same server as was being benchmarked, so the differences may result from how much processing power was being consumed by the benchmarking tools themselves.

One interesting point is that ab no longer appears to be over-reporting the baseline cases, as I noted in an earlier benchmark posting. There are two major changes between then and now: (1) the updated project uses Ubuntu 10.10 instead of 8.10, which means the packaged ab binary might have been flawed earlier, or that the new OS otherwise corrects some other issue; (2) the updated project uses an m1.large 64-bit instance instead of an m1.small 32-bit instance. Either of those differences might be sufficient to account for the disparity in ab reporting previously.

PHP Framework Benchmarks on Github

As part of “trying new things,” I have moved my web frameworks benchmark project over to Git on Github and away from Subversion on Google Code.

This project is often imitated and occasionally adopted. For all you framework fans who want to compare their preferred systems to the ones officially included in the project, you can now fork the repo and add your favorite. Who knows, some may make their way back onto the officially-included list.

Additionally, I have modified the project so that you can use one of three different benchmarking tools: Apache Benchmark, JoeDog siege, or ACME http_load. After you follow the setup instructions, you can run benchmarks using each of the different tools against the same benchmark targets:

./bench/ab.php targets/baseline.ini
./bench/siege.php targets/baseline.ini
./bench/httpload.php targets/baseline.ini

Comments or questions? Leave a note below.

Running The Symfony 2 Benchmarks

Fabien Potencier released Symfony 2.0.0alpha1 last week, along with some benchmarks showing its performance. I am glad to see that Fabien used my benchmarking system and methodology, and am happy to see that he is paying attention to the performance of his framework. I take this as an acceptance on his part that my methodology is legitimate and valid, and that it has value when comparing framework responsiveness.

However, in attempting to reproduce the published Symfony 2 benchmarking results, I found Fabien’s reporting to be inaccurate (or at least incomplete). Read on for a very, very long post detailing my attempt to replicate his results for the “hello world” basic framework overhead comparison, and my conclusions.

For the impatient, here are my conclusions in advance:

  1. Fabien’s benchmark report, as shown at http://symfony-reloaded.org/fast, is inaccurate for the setup he describes. Lithium and Flow3 do not work in Fabien’s benchmark codebase at Github. Also, Symfony 2 is faster than Solar beta 3 by 5%, not 20%, on a “c1.xlarge” instance; to get a relative difference like Fabien describes, one has to use an “m1.large” instance. (It is entirely possible that the process Fabien used for benchmarking is incompletely described, and that the codebase is not fully updated, thus contributing to this disparity in results.)

  2. We should use Siege 2.69, not 2.66, for more accurate benchmarking of baseline responsiveness. If we notice that HTML is slower than PHP, it’s a sign that something is wrong.

  3. Symfony 2 preloads its foundation and application classes, something no other framework does in the benchmarked code. When we treat Solar and Symfony 2 the same way, by preloading the foundation classes for each, we find that Solar is roughly 28% faster than Symfony 2.

Continue reading

The Future of Zend Framework is Solar

I have said it before and I’ll say it again now: If you want to see the future of Zend Framework, look at the Solar Framework for PHP 5.

FYI: This is a cheerleading post. Some will be annoyed by it or find it “unfair”. Others may find it informative and enlightening. I make no apology for advocating what I think are the good things about Solar, especially when other projects appear to be mimicking those things.

I noticed that Matthew posted the ZF 2.0 roadmap yesterday:

http://framework.zend.com/wiki/display/ZFDEV2/Zend+Framework+2.0+Roadmap

Solar is already doing most of what is described as the future for Zend. Let’s go over this, shall we?

Unified constructor. All constructors will (optionally) accept an array of options or Zend_Config object as the first argument. This allows for more flexibility to add new arguments to the constructor, have variable numbers of arguments, and allow “named” arguments. Additionally, it’s a good technique for allowing Dependency Injection.

Solar has had a framework-wide unified constructor for years now. As far as I know, Solar is the first PHP framework to demonstrate this design pattern and give it a name. You can have a unified constructor right now with Solar.

In addition, we already have a unified dependency injection and service locator implementation, via the unified configuration mechanism and a lazy-loading registry. You can have DI/SL right now with Solar.

Options. In ZF 1.X, the various components which accept options accept a variety of formats: some expect underscore_separated keys, others expect camelCasedKeys, others expect UPPERCASEDKEYS, and some expect lowercasedkeys. This leads to confusion for many, and also leads to difficulties debugging. Our goal in ZF 2.0 is to standardize option keys to correct this situation.

Currently, we are leaning towards all_lowercase_underscore_keys.

Solar has already standardized on its config keys as lowercase-underscore. You can have standard config keys right now with Solar.

Exceptions. Each component will have an Exception marker interface, with exceptions defined for discrete exception types thrown by the component. The concrete exceptions will either extend the global Exception class or an SPL Exception, and also implement the component Exception interface.

Solar provides a unified exception mechanism already. This is not quite the same thing as described in the ZF2.0 document, but if the idea is to have a unified way of throwing and recognizing exceptions, Solar is already doing it.

Design By Contract.

OK, we don’t really do this. Solar tends to use adapters, plugins, class stacks, and duck typing, not interfaces.

Elimination of most singletons.

Solar doesn’t use singletons. We use the registry and, occasionally, unified static-method-collection classes. You want two or more database connections? Mark them for lazy-load in the registry, and set your config file to use the appropriate service locator keys for whatever objects may need them. You can have this right now with Solar.

Creation of components for general-purpose, cross-functional actions. A number of components duplicate code, and we want to push the duplication areas into discrete components that the original components may then consume. Some of these include:

  • Plugins/Helpers/Strategies (seen currently in Zend_Controller_Front, Zend_Controller_Action_Helper, Zend_View (helpers and filters), Zend_Form (validators, filters, and decorators), etc.). Plugin discovery and loading could benefit from a common API.
  • Decorators (seen currently in Zend_Form; other areas could benefit from the pattern)
  • Factories (seen currently in Zend_Db, Zend_Navigation_Page, Zend_Auth, Zend_Translate, Zend_Cache, etc.)
  • Caching (seen currently in Zend_Translate, Zend_Locale, Zend_Queue, Zend_Paginator, Zend_Feed_Reader, Zend_Db_Table, etc.)

Solar already has a standardized factory mechanism and a standardized adapter approach. With the lazy-loading registry and DI/SL mechanism we can inject a cache object into any other object that’s ready to receive it. You can have these right now with Solar.

(In fairness, we don’t use decorators much, and the plugin systems are more a standard approach using class stacks than an encapsulated system.)

Usage of new language features within plugin architectures.

Solar isn’t on 5.3 yet, but then, neither is Zend Framework.

Autoload-only. We will move to using autoloading throughout the framework. This solves a number of performance issues, as well as simplifies coding dependencies (particularly exceptions).

Solar is already autoload-only. You can be autoload-only too right now with Solar.

Namespaces. PHP namespaces benefit frameworks and libraries more than any other code bases, and ZF can benefit from it greatly, particularly with components such as Zend_Search_Lucene and Zend_Controller.

Solar already has first-class support for PHP 4/5 style namespaces (i.e., pseudo-namespaces) and supports mixing of multiple “vendor” prefixes in projects (and supports cross-vendor fallbacks across library hierarchies). Solar expects that you will be working within your own pseudo-namespace already, and so is poised to use 5.3 namespaces proper with little effort. You can have namespace-like support right now with Solar.

goto

As with “new language features” above, Solar isn’t on 5.3 yet, but then, neither is Zend Framework.

MVC Implementation: Our current MVC implementation is increasingly adding overhead to the dispatch cycle, slowing down the request cycle. While you can squeeze additional performance out of it via stripping require_once calls and using autoloading, the fastest requests are still far short of other, slimmer frameworks …

Solar‘s dynamic dispatch cycle is much faster than the one in Zend Framework, and has been for years; see these benchmarking posts:

You can have a much faster dispatch cycle right now by using Solar.

Models and Master-Slave

One more thing that Zend Framework doesn’t have, that Solar does, is an integrated model system. Zend Framework recently dropped the idea of having their own, and expect to depend on an external system for reasonable model/record/collection support. Solar has an integrated model system right now with support for single table inheritance, various relationships, unified saving of object graphs, automatic data filtering, automatic form generation, calculated columns, collection methods, automatic data caching and clearing, and much more.

Yet another thing: we also provide support out-of-the-box for master-slave MySQL replication, with no changes needed to the codebase. Change your config file from the single-server connection to the master-slave connection, and you are done. (Well, you might have to wrap some parts in transactions, but after that, it will work on both single-server and master-slave setups.)

You can have robust models and master-slave support right now with Solar.

Modesty

There is one significant thing that Zend has that Solar doesn’t: narrative documentation. Our API docs are second to none, but the tutorials are lacking. We’ve been spending our time making the framework great, not writing about the framework. You can help with that if you like by adding to the wiki.

Conclusion

The point of all this is not to belittle Zend, Zend Framework, its developers, or its contributors. The point is to ask Zend Framework adopters: why wait for ZF 2.0 when you can have all those features and more, right now, by using Solar?

(Let the ZF apologists begin their comments right now. ūüėČ

A Siege On Benchmarks

My regular readers (and perhaps the irregular ones as well ūüėČ know that I have been obsessed with baseline-responsiveness benchmarking of frameworks for years now. ¬†The idea has always been that, in order to know how far you can optimize your framework-based applications, you need to know the limits imposed by the framework itself. ¬†Only then can you have an idea of where to spend your limited resources on improvement. ¬†For example, if you need 200 dynamic requests/second, but the framework itself (with no application code in use) is capable only of 100, then you know that no amount of application or database optimization will help you — it’s time to start scaling, either horizontally or vertically.

To perform these benchmarks, I have only employed the ab tool provided by the Apache web server. ¬†It was easy to use, and relatively easy to parse the output to automate reporting. ¬†However, it turns out that ab over-reports responsiveness of Apache when serving static HTML files, and when serving minimal PHP scripts such as <?php echo "hello world"; ?>. ¬†I discovered this just recently when attempting to find out why PHP appeared to be faster than HTML, and then only with the assistance of Paul Reinheimer, whom I now owe a bottle of vodka for his trouble. ¬†ūüėČ

It turns out that the siege tool from JoeDog Software is more accurate in reporting static HTML and PHP responsiveness.  This is confirmed through Paul Reinheimer as well, who reported the expected responsiveness on other systems.

The over-reporting from ab means that all my previous reporting on benchmarks is skewed too low when comparing framework responsiveness to PHP’s maximum responsiveness. ¬†As such, I have re-run all the previously published benchmarks using siege instead of ab. ¬†Previous runs with ab are here …

… and below are the updated siege versions. ¬†As with previous attempts, these benchmarks are performed on an Amazon EC2 “small” instance. ¬†There is one difference to note: previous runs used Xcache for bytecode caching, but these use APC; I don’t suspect this change in caching engines has a significant effect, but I have not tested that assertion.

framework rel avg
baseline-html 1.1878 985.69
baseline-php 1.0000 829.82
cake-1.1.10 0.0938 77.84
cake-1.1.11 0.1277 105.96
cake-1.1.12 0.1288 106.84
cake-1.1.16 0.1166 96.77
cake-1.1.17 0.1165 96.70
cake-1.1.19 0.1298 107.69
cake-1.2.0-rc2 0.0516 42.79
solar-0.25.0 0.1852 153.66
solar-0.26.0 0.1789 148.43
solar-0.27.0 0.1734 143.93
solar-0.28.0 0.1671 138.64
solar-1.0.0alpha1 0.1706 141.58
symfony-0.6.3 0.0629 52.22
symfony-1.0.0beta2 0.0758 62.91
symfony-1.0.6 0.0746 61.91
symfony-1.0.6-dw 0.0820 68.03
symfony-1.0.6-fp 0.0853 70.78
symfony-1.0.17 0.0744 61.73
symfony-1.1.0 0.0745 61.84
zend-0.2.0 0.2176 180.56
zend-0.6.0 0.1998 165.78
zend-1.0.0 0.1268 105.25
zend-1.0.1 0.1263 104.80
zend-1.5.2 0.0951 78.93

Note the baseline-html and baseline-php numbers.  Using ab previously, these were reported as 2100-2400 requests/second and 1100-1400 requests/second, respectively.  The siege tool reports a much lower number for both, but the dropoff between static HTML and dynamic PHP is much smaller; with ab it looked like about 40-50%, but now with siege it looks like only about 15-18%.  This behavior is much more like what we would expect from a memory-based PHP script.

Note also the separate framework requests/second; they are very similar between ab and siege.  This means that the framework responsiveness numbers are almost unchanged.

Because the nearly-identical framework numbers are compared to a much smaller baseline PHP number, the frameworks now appear to be doing much better in relation to PHP’s maximum responsiveness. ¬†For example, Solar-1.0.0alpha1 with ab appeared to run at about 11% of PHP’s max, but with siege it looks close to 17%. ¬†All of the frameworks tested see this kind of comparative gain in their reporting.

However, when compared to each other, the framework rankings are the same as before:  Solar has the highest baseline responsiveness, followed by Cake and Zend (their respective releases are very close to each other in responsiveness), and Symfony trails with the lowest baseline responsiveness.

In summary, using ab skewed the “percentage of PHP” comparisons because it over-reported PHP’s maximum responsiveness, but the framework requests/second numbers and the framework comparative rankings are unchanged from previous reporting. ¬†The Google project for the benchmarking system has been updated to use siege, so all future reporting will reflect its results, not those of ab.

Lazyweb Request: Why would PHP be *faster* than HTML?

With the help of the great guys at Slicehost.com, I am attempting to run my benchmark series on a virtual private server, to compare with EC2. However, I’m seeing a very strange result for the baselines: a PHP page delivers more requests-per-second than a static HTML page.

The OS is a stock Ubuntu 8.10 installation; you can see the setup steps here.

The virtual private server has 2 gigs of RAM, and is on a box by itself, so there is no other external activity to skew the results.

I ran ab -c 10 -t 60 http://localhost/ on each of two files: index.html, which has only the static text “hello world”; and index.php, which has only the code <?php echo 'hello world'; ?>.

Here are the results without APC, averaged over 5 one-minute runs:


index.html : 7067.57 req/sec
index.php  : 7484.57 req/sec # faster???

Here are the results with APC, averaged over 5 one-minute runs:


index.html : 7013.50 req/sec
index.php  : 8041.06 req/sec # faster???

I haven’t seen this behavior on EC2. I’m not complaining, but it does seem unintuitive; invoking the PHP interpreter should be more expensive than just delivering a static HTML file. Does anyone have ideas as to why this might be happening?

UPDATE: With the help of Paul Reinheimer, we appear to have found the culprit: the ab tool itself seems to be at fault. Running similar tests by hand with siege returns much more reasonable and expected numbers (~4000 req/sec for HTML, ~3200 for PHP). I’m going to re-work the test scripts to use siege later and report back. Thanks to everyone who provided suggestions, and special thanks to Paul Reinheimer for working through it with me today.