Bad practices exposed
By eoin on Sep 01, 2009
It surveys nine years worth of papers on file system and storage benchmarks and makes for sobering reading:
We found that most popular benchmarks are flawed, and many research papers used poor benchmarking practices and did not provide a clear indication of the system's true performance.and:
Finally, only about 45% of the surveyed papers included any mention of statistical dispersion.We can only hope that the paper will raise awareness of the important of rigor and thoroughness in performance benchmarking.