M. Zaninotto sets up a number of straw-man arguments regarding comparative benchmarks in general, although he does not link to any specific research. In doing so, he misses the point of comparative benchmarking almost entirely. Herein I will address some of M. Zaninotto’s arguments individually in reference to my previous benchmarking series.
All of the following commentary regards benchmarking and its usefulness in decision-making, and should not be construed as a general-purpose endorsement or indictment of any particular framework. Some frameworks are slower than others, and some are faster, and I think knowing “how fast is the framework?” is an important consideration when allocating scarce resources like time, money, servers, etc.
And now, on to a point-by-point response!