I'm happy to report that FileBench has gone over a significant overhaul and we're happy to release that updated version. Bits will be posted to sourceforge tonight. I'm also happy to report that FileBench is now included in OpenSolaris. You can find it in our new "/usr/benchmarks" path.
Ok, great, just what the industry needed - FileBench is just another simple benchmark. right? Nope.
First let me give you, dear reader, a taste of what we get internally and externally here at the ZFS team:
"I ran bonnie, dd, and tar'd up myfav_linuxkernel_tarball.tar. Your file system sucks."
Though sometimes i'm happy to note we get:
"I ran bonnie, dd, and tar'd up myfav_linuxkernel_tarball.tar. Your file system rulz!."
What is FileBench?
It is nice to hear that your file system does in fact "rule", but the problem with the above is that bonnie, dd, and tar are (obviously) not a comprehensive set of applications that can completely measure a file system. IOzone is quite nice but it only tests basic I/O patterns. And there are many other file system benchmarks (mkfile, fsstress, fsrandom, mongo, iometer, etc). The problem with all of these benchmarks is that they only measure a specific workload (or a small set of workloads). None of them actually let you measure what a real application does. Yes part of what Oracle does is random aligned reads/writes (which many of the pre-mentioned benchmarks can measure), but the key is how the random read/writes interact with each other \*and\* how they interact with the other parts of what Oracle does (the log writer as a major example). None of the pre-mentioned benchmarks can do that.
So how does FileBench differ? FileBench is a framework of file system workloads for measuring and comparing file system performance. The key is in the workloads. FileBench has a simple .f language that allows you to describe and build workloads to simulate applications. You can create workloads to replace all the pre-mentioned benchmarks. But more importantly, you can create workloads to simulate complex applications such as a database. For instance, i didn't have to buy an Oracle license nor figure out how to install it on my system to find out if my changes to the vdev cache for ZFS helped database performance or not. I just used FileBench and its 'oltp.f' workload.
How do i Start using FileBench?
The best place to start is via the quick start guide. You can find out lots more at our wiki. Lots of good information and a great place to contribute. For trouble shooting see the gotchas section.
How do i Contribute to FileBench?
If you'd like to write your own workloads, check out the very nice documentation Drew Wilson wrote up. This is actually where we (where we is the FileBench community, not just us Sun people) would love the most contribution. We would really like to have people verify the workloads we have today and build the workloads for tomorrow. This is a great opportunity for industry types and academics. We very much plan to incorporate new workloads into FileBench.
If you would like to help with the actual code of the filebench binaries, find us at firstname.lastname@example.org.
FileBench is done, right?
Um, nope. There's actually lots of interesting work up ahead for FileBench. Besides building new workloads, the two other major focuses that need help are: multi-client support and plug-ins. The first is pretty obvious - we need to have support for multiple clients to benchmark NFS and CIFS. And we need that to work on multiple platforms (OpenSolaris, linux, \*BSD, OSX, etc). The second is where experts in specific communities can help out. Currently, FileBench goes through whatever client/initiator implementation you have on your machine. But if you wanted to just do a comparison of server/target implementations, then you need a plug-in built into FileBench that both systems utilize (even if they are different operating systems). We started prototyping a plug-in for NFSv3. We've also thought about NFSv4, CIFS, and iSCSI. A community member suggested XAM. This is a very interesting space to explore.
So What does this all Mean?
If you need to do file system benchmarking, try out FileBench. Let us know what you like and what needs some love.
If you're thinking about developing a new file system benchmark, consider creating a new workload for FileBench instead. If that works out for you, please share your work. If for some reason it doesn't, please let us know why.
We really believe in the architecture of FileBench and really want it to succeed industry wide (file system and hence storage). We know it works quite well on OpenSolaris and would love other developers to make sure it works just as well on their platforms (linux, \*BSD, OSX, etc.).
Happy Benchmarking and long live RMC!