Freitag Mai 29, 2009

FileBench ported to Mac OS X

I recently ported the benchmarking utility FileBench to Mac OSX. The chief attraction of FileBench is, that one can define the I/O operations put on the filesystem in something called the "Workload Definition Language". So - it's not a fixed benchmark, burden the system with the very same load pattern over and over again but an amazingly flexible Software which can simulate various workloads like file servers, databases, web servers etc.

FileBench was created by Richard McDougall who is a Solaris and Systems programming guru and one of the authors of the famous Solaris Internals book. More information about FileBench could be found at the official touch base here. FileBench is now included in OpenSolaris. Take a look here for the sources. There is also a SourceForge-Version landing page for sources which build on Linux.

Here's an example, how this definition language looks like:

set $dir=/tmp
set $nfiles=1000
set $meandirwidth=1000000
set $filesize=16k
set $nthreads=16
set $meaniosize=16k
 
define fileset name=bigfileset,path=$dir,size=$filesize,entries=$nfiles,dirwidth=$meandirwidth,prealloc=80
 
define process name=filereader,instances=1
{
  thread name=filereaderthread,memsize=10m,instances=$nthreads
  {
    flowop deletefile name=deletefile1,filesetname=bigfileset
    flowop createfile name=createfile2,filesetname=bigfileset,fd=1
    flowop appendfilerand name=appendfilerand2,iosize=$meaniosize,fd=1
    flowop fsync name=fsyncfile2,fd=1
    flowop closefile name=closefile2,fd=1
    flowop openfile name=openfile3,filesetname=bigfileset,fd=1
    flowop readwholefile name=readfile3,fd=1
    flowop appendfilerand name=appendfilerand3,iosize=$meaniosize,fd=1
    flowop fsync name=fsyncfile3,fd=1
    flowop closefile name=closefile3,fd=1
    flowop openfile name=openfile4,filesetname=bigfileset,fd=1
    flowop readwholefile name=readfile4,fd=1
    flowop closefile name=closefile4,fd=1
  }
}

On the FileBench homepage and in the distributions "workloads" directory one can find many more examples how to write these definitions.

Here is an console-log of loading and running the varmail benchmark:

$ /opt/filebench/bin/go_filebench

FileBench Version 1.4.4
filebench> load varmail
1364: 10.634: Varmail Version 2.1 personality successfully loaded
1364: 10.634: Usage: set $dir=<dir>
1364: 10.634: set $filesize=<size> defaults to 16384
1364: 10.634: set $nfiles=<value> defaults to 1000
1364: 10.634: set $nthreads=<value> defaults to 16
1364: 10.634: set $meaniosize=<value> defaults to 16384
1364: 10.634: set $readiosize=<size> defaults to 1048576
1364: 10.634: set $meandirwidth=<size> defaults to 1000000
1364: 10.634: (sets mean dir width and dir depth is calculated as log (width, nfiles)
1364: 10.634: dirdepth therefore defaults to dir depth of 1 as in postmark
1364: 10.634: set $meandir lower to increase depth beyond 1 if desired)
1364: 10.634:
1364: 10.634: run runtime (e.g. run 60)
filebench> set $dir=/tmp
filebench> run 10
1364: 26.250: Creating/pre-allocating files and filesets
1364: 26.252: Fileset bigfileset: 1000 files, 0 leafdirs avg dir = 0, avg depth = 0.0, mbytes=4602678819172646912
1364: 26.310: Removed any existing fileset bigfileset in 1 seconds
1364: 26.314: making tree for filset /tmp/bigfileset
1364: 26.314: Creating fileset bigfileset...
1364: 27.009: Preallocated 807 of 1000 of fileset bigfileset in 1 seconds
1364: 27.009: waiting for fileset pre-allocation to finish
1364: 27.009: Starting 1 filereader instances
1364: 27.009: Starting 16 filereaderthread threads
1364: 28.010: Running...
1364: 38.011: Run took 10 seconds...
1364: 38.012: Per-Operation Breakdown
closefile4 95ops/s 0.0mb/s 0.0ms/op 0us/op-cpu
readfile4 95ops/s 1.3mb/s 16.8ms/op 0us/op-cpu
openfile4 95ops/s 0.0mb/s 0.0ms/op 0us/op-cpu
closefile3 95ops/s 0.0mb/s 0.1ms/op 0us/op-cpu
fsyncfile3 96ops/s 0.0mb/s 62.1ms/op 0us/op-cpu
appendfilerand3 96ops/s 0.7mb/s 4.1ms/op 0us/op-cpu
readfile3 96ops/s 1.3mb/s 17.1ms/op 0us/op-cpu
openfile3 96ops/s 0.0mb/s 0.0ms/op 0us/op-cpu
closefile2 96ops/s 0.0mb/s 0.1ms/op 0us/op-cpu
fsyncfile2 97ops/s 0.0mb/s 59.7ms/op 0us/op-cpu
appendfilerand2 97ops/s 0.7mb/s 0.1ms/op 0us/op-cpu
createfile2 97ops/s 0.0mb/s 0.8ms/op 0us/op-cpu
deletefile1 97ops/s 0.0mb/s 3.2ms/op 0us/op-cpu

1364: 38.012:
IO Summary: 12515 ops, 1251.4 ops/s, (192/194 r/w) 4.2mb/s, 3587919346114us cpu/op, 40.9ms latency
1364: 38.012: Shutting down processes
filebench> stats dump "stats.varmail"
1364: 69.662: Stats dump to file 'stats.varmail'
1364: 69.662: in statsdump stats.varmail
filebench> quit
$ more stats.varmail
Flowop totals:
closefile4 95ops/s 0.0mb/s 0.0ms/op 0us/op-cpu
readfile4 95ops/s 1.3mb/s 16.8ms/op 0us/op-cpu
openfile4 95ops/s 0.0mb/s 0.0ms/op 0us/op-cpu
closefile3 95ops/s 0.0mb/s 0.1ms/op 0us/op-cpu
fsyncfile3 96ops/s 0.0mb/s 62.1ms/op 0us/op-cpu
appendfilerand3 96ops/s 0.7mb/s 4.1ms/op 0us/op-cpu
readfile3 96ops/s 1.3mb/s 17.1ms/op 0us/op-cpu
openfile3 96ops/s 0.0mb/s 0.0ms/op 0us/op-cpu
closefile2 96ops/s 0.0mb/s 0.1ms/op 0us/op-cpu
fsyncfile2 97ops/s 0.0mb/s 59.7ms/op 0us/op-cpu
appendfilerand2 97ops/s 0.7mb/s 0.1ms/op 0us/op-cpu
createfile2 97ops/s 0.0mb/s 0.8ms/op 0us/op-cpu
deletefile1 97ops/s 0.0mb/s 3.2ms/op 0us/op-cpu

IO Summary: 12515 ops 1251.4 ops/s, 192/194 r/w 4.2mb/s, 3587919346114uscpu/op

I've created a Diskimage including an Apple Installer package which installs the software into /opt/filebench. Just download the image file and you could install the program in an Apple fashion:


As a next step the process of creating the Mac-version from the OpenSolaris sources should be automated like this:


Here's the first version of the script, the skeleton and a set of patches to automate this.

About

schmidtm

Search

Archives
« Juli 2014
MoDiMiDoFrSaSo
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
   
       
Heute