Paul M. Jones

Comparative benchmarking is tedious work. It's not hard to do, it's just no fun to set up the same scaffold every time you want to figure out how long it takes different pieces of code to execute.

To do a benchmark, essentially you need only four things: a timing mechanism, a loop to run the first case, a loop to run the second case, and a way to show the time taken by each loop.

$loopTimes = 1000;

$before = microtime(true);
for ($i = 0; $i < = $loopTimes; ++$i) {
    // case #1:
    // this code gets run $loopTimes
}
$after = microtime(true);
$case1 = $after - $before;

$before = microtime(true);
for ($i = 0; $i <= $loopTimes; ++$i) {
    // case #2:
    // this code gets run $loopTimes
}
$after = microtime(true);
$case2 = $after - $before;

echo "Case 1 ran $loopTimes in $case1 seconds";
echo "Case 2 ran $loopTimes in $case2 seconds";

This is boring, and repetitive, and essentially the same every time (except for the code cases being benchmarked). This makes it a classic candidate for refactoring to a PHP class.

Enter Solar_Test_Bench. As with Solar_Test, you extend the base class, add your own code cases as bench*() methods, and a runner will execute each of those methods the same number of times and display how long each one took. (It uses Solar_Debug_Timer for the timing mechanism.)

Here's an example of a real benchmark test from Solar to test the speed difference between require_once() and Solar::loadClass().

Solar::loadClass('Solar_Test_Bench');
class Bench_Solar_LoadClass extends Solar_Test_Bench {

    public $file = 'Solar/Sql.php';
    public $class = 'Solar_Sql';

    public function benchLoadClass()
    {
        Solar::loadClass($this->class);
    }

    public function benchRequireOnce()
    {
        require_once $this->file;
    }
}

To run it, we call the benchmark runner to execute each method in the class 1000 times, and get this output:

bash-2.05$ php bench.php Solar_FileExists 1000
name             : diff     : total
__start          : 0.000000 : 0.000000
benchLoadClass   : 0.005793 : 0.005793
benchRequireOnce : 0.034572 : 0.040365
__stop           : 0.000012 : 0.040377
bash-2.05$

You would want to run it a few times to get a better idea of the real speed, but the above is a representative result from my MacBook 2.0GHz. Looking at the "diff" column we can see that Solar::loadClass() is about 5 times faster than using require_once(). (Thanks to Clay Loveless for thinking up this particular benchmark.)

Here's another one:

Currently, Solar::fileExists() uses an fopen() call to see if a file exists on the include_path. (The regular file_exists() and is_readable() PHP calls do not check the include path.) However, Clay Loveless wondered if it would be faster to explode() the include_path and see if the file_exists() on each of those individually.

I won't copy the benchmarking class here; you can see it at Bench_Solar_FileExists. We actually test three different cases; the current fopen() case, and two new versions of using include_path. The results are:

bash-2.05$ php bench.php Solar_LoadClass 1000
name                   : diff     : total
__start                : 0.000000 : 0.000000
benchFopen             : 0.040105 : 0.040105
benchExplodeInclPath   : 0.015582 : 0.055687
benchNoExplodeInclPath : 0.020612 : 0.076299
__stop                 : 0.000004 : 0.076303
bash-2.05$

So the second case, where we explode() the include_path and then check file_exists() on each of the resulting paths, is about 3 times faster than using fopen(). The next release of Solar will use that code instead of the current fopen() code as a result.

(Note: the benchmarking classes are in Subversion only at this point, but will be in the next release of Solar when it comes out.)