Comparing Benchmark Tools

As I noted last week, I have moved my framework benchmarking project to GitHub. As part of the move, I updated the project to allow benchmarking using any of three tools: Acme http_load, Apache ab, or Joedog siege. (For reference, the old project will remain at GoogleCode.)

I thought it might be interesting to see what each of them reports for the baseline “index.html” and “index.php” cases on the new Amazon EC2 setup (using a 64-bit OS on an m1.large instance). The results follow (all are at 10 concurrent users, averaged over 5 one-minute runs):

ab                       |      rel |      avg |
------------------------ | -------- | -------- |
baseline-html            |   1.2660 |  3581.54 |
baseline-php             |   1.0000 |  2829.11 |

http_load                |      rel |      avg |
------------------------ | -------- | -------- |
baseline-html            |   1.2718 |  4036.24 |
baseline-php             |   1.0000 |  3173.56 |

siege                    |      rel |      avg |
------------------------ | -------- | -------- |
baseline-html            |   1.2139 |  5060.25 |
baseline-php             |   1.0000 |  4168.76 |

They all show very different “absolute” numbers of requests/second: ab thinks the server delivers about 3600 req/sec, http_load reports about 4000, and siege says about 5000.

Note that the ab and http_load relative scores are in line with each other, reporting about a 26-27% slowdown for invoking PHP. Siege thinks PHP is more responsive than that, with only a 21% slowdown.

Which of these is the most accurate? I don’t know. I ran the benchmarking tool on the same server as was being benchmarked, so the differences may result from how much processing power was being consumed by the benchmarking tools themselves.

One interesting point is that ab no longer appears to be over-reporting the baseline cases, as I noted in an earlier benchmark posting. There are two major changes between then and now: (1) the updated project uses Ubuntu 10.10 instead of 8.10, which means the packaged ab binary might have been flawed earlier, or that the new OS otherwise corrects some other issue; (2) the updated project uses an m1.large 64-bit instance instead of an m1.small 32-bit instance. Either of those differences might be sufficient to account for the disparity in ab reporting previously.

PHP Framework Benchmarks on Github

As part of “trying new things,” I have moved my web frameworks benchmark project over to Git on Github and away from Subversion on Google Code.

This project is often imitated and occasionally adopted. For all you framework fans who want to compare their preferred systems to the ones officially included in the project, you can now fork the repo and add your favorite. Who knows, some may make their way back onto the officially-included list.

Additionally, I have modified the project so that you can use one of three different benchmarking tools: Apache Benchmark, JoeDog siege, or ACME http_load. After you follow the setup instructions, you can run benchmarks using each of the different tools against the same benchmark targets:

./bench/ab.php targets/baseline.ini
./bench/siege.php targets/baseline.ini
./bench/httpload.php targets/baseline.ini

Comments or questions? Leave a note below.

Regarding Underscores

Today, PHPDeveloper.org referred to a post by Leszek Stachowski about underscore prefixes on non-public class elements.

The question which comes instantly to my mind is: why? Is there any reason why this convention should be kept when PHP object oriented programming has gone a long way since PHP 4 (when there was no access modifiers and such underscore was the only fast way to distinguish public from, hmm, not public methods and properties) ? Are, for instance (as one of major OOP languages), Java coding standards pushing towards such naming convention? No!

I think that we, as developers, should not stick to this silly convention. For the sake of progress, stop looking back (because that what in fact this convention is) and stop supporting this one, particular naming convention.

I think the underscore-prefix for protected and private is a good convention to keep. As with many things about programming, this convention is not for the program, it is for for the programmer. For example, limiting your line length to about 80 characters is a good idea, not for reasons of “tradition”, but because of cognitive limitations of human beings who have to read code.

Likewise, using the underscore prefix is an immediate and continuous reminder that some portions of the code are not public. Its purpose is as an aid to the programmer. The underscores make it obvious which parts of the program are internal, and which parts are externally available. (Note that I do not extend this argument to support the use of Hungarian Notation in PHP; if something like the underscore prefix is overused, it loses its obvious-ness and thus becomes less powerful.)

As an example, look at the following code:

<?php
class NoUnderscores
{
    protected $data = array(
        'item' => 'magic-data',
    );

    protected $item = 'property-value';

    public function __get($key)
    {
        return $this->data[$key];
    }

    protected function doSomething()
    {
        // do we want the magic public item,
        // or the internal protected item?
        return $this->item;
    }
}

Here we have magic __get() method that reads from the protected $data property. Any time you try to access a property that doesn’t exist, PHP will go to the __get() method and read from protected $data. Now look in the doSomething() method. Because the code executes inside the class, it has access ot the protected $item, so it’s not obvious if the programmer wanted the value of protected $item, or the magic $data['item'].

By way of comparison, take a look at the following modification to use the underscore prefix on private and protected elements:

<?php
class Underscores
{
    protected $_data = array(
        'item' => 'magic-data',
    );

    protected $_item = 'property-value';

    public function __get($key)
    {
        return $this->_data[$key];
    }

    protected function _doSomething()
    {
        // it is clear we want the internal protected item
        return $this->_item;
    }
}

Now the _doSomething() method is perfectly clear: the programmer wants the value of the internal protected property.