Bad practices exposed

There is an interesting (and frightening!) study recently published in the ACM Transactions on Storage: A nine year study of file system and storage benchmarking together with a short summary of the results and a set of recommendations.

It surveys nine years worth of papers on file system and storage benchmarks and makes for sobering reading:

We found that most popular benchmarks are flawed, and many research papers used poor benchmarking practices and did not provide a clear indication of the system's true performance.
and:
Finally, only about 45% of the surveyed papers included any mention of statistical dispersion.
We can only hope that the paper will raise awareness of the important of rigor and thoroughness in performance benchmarking.
Comments:

Post a Comment:
  • HTML Syntax: NOT allowed
About

Snippets of code, useful tips, and observations of working with Sun.

Search

Categories
Archives
« April 2014
MonTueWedThuFriSatSun
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    
       
Today