A Bit About Benchmarks

As the author of a relatively popular benchmarking article, I feel compelled to respond to this bit of misguided analysis from the Symfony camp about benchmarks.

Full disclosure: I am the lead developer on the Solar framework, and was a founding contributor to the Zend framework.

M. Zaninotto sets up a number of straw-man arguments regarding comparative benchmarks in general, although he does not link to any specific research. In doing so, he misses the point of comparative benchmarking almost entirely. Herein I will address some of M. Zaninotto’s arguments individually in reference to my previous benchmarking series.

All of the following commentary regards benchmarking and its usefulness in decision-making, and should not be construed as a general-purpose endorsement or indictment of any particular framework. Some frameworks are slower than others, and some are faster, and I think knowing “how fast is the framework?” is an important consideration when allocating scarce resources like time, money, servers, etc.

And now, on to a point-by-point response!

Continue reading