Testing ZFS reminds me of when I was working for IBM at NASA on the Space Shuttle Flight Software Project. There we had 10,000 people around the world testing code being produced by 100 developers. The code had to get the crew safely into orbit, keep them in orbit and get them safely back to earth, plus perform a myriad of supporting system management tasks, like monitoring oxygen levels or deploying a satellite. On top of this, the interface had to be easy for the astronauts to use. Most of the time they needed just a high level "GO" button that hid all the complexity and performed the big jobs for them, like getting them into orbit. But, when adjustments were needed, they needed clear status information, and straight forward ways to change the system to meet their specific needs. Our job was to verify it all worked, and make sure the astronauts liked the look and feel. We were dedicated to this effort, and because the technology was so interesting and advanced, it was never hard to find people willing to help us.
20 years later, enter ZFS. Instead of protecting and managing systems for a crew of astronauts, ZFS protects and manages data for everyone. The scale of the development and test groups is a little different, but the task of verifying that it all works is just as critical and the team is just as dedicated. I joined the ZFS project team and became the test lead a few months after the first prototype became available. Immediately after this a core test team was formed, and we began developing tests and updating others to work with ZFS. The test team, which is part of the OPG Engineering Operations Quality Engineering organization is tasked with organizing all aspects of ZFS testing including developing automated functional and stress tests and performing daily test runs. In addition, other areas include regression, conformance, interoperability, memory leak, documentation and internationalization testing. The core ZFS test team alone has put over 20 people years of time into verifying ZFS so far. And, because our team is spread across the world from Dublin to Beijing and points in between, tests are running all the time.
But that's only part of the story. Like the Space Shuttle Flight Software, it was easy to find people interested in testing and supporting testing ZFS everywhere. Which is good, because file system and volume management software is complex and important to everyone, and needs verification from every angle. Some of the groups within Sun involved in this effort include, Central Storage Systems Test, Solaris Pre-Integration Test, Solaris Performance Test, Performance and Availability Engineering, Solaris Standards Conformance Test, Sun Cluster Test, the Enterprise Technology Center, OPG Engineering Operations Labs, the Data Management Group, Sun-On-Sun and the Diagnostics Engineering Group. These groups combined contributed another 30 people years of effort into improving the quality of ZFS. It's amazing! ZFS is not even released officially yet and we have over 50 years of testing behind us, and this doesn't even include all the contributions made by customers in the beta test program. Thanks to everyone who has made ZFS what it is today.
There's still work to do (we engineers always want more), but ZFS is definitely ready for initial blast off!