HPC Consortium: Storage
By Josh Simons on Jun 15, 2008
Storage was the theme of the first set of talks at the Sun HPC Consortium meeting here in Dresden.
Peter Braam, Vice President for Lustre, spoke first about Open Storage, which we at Sun believe marks an important shift within our industry comparable to the shift we've seen towards Open Servers and that we expect to see in the future with networking. Open Storage in a nutshell: an approach that leverages open source software, open architectures, common components, and the interoperability of open standards used to create innovative storage products, with breakthrough economics. For example, we do not believe expensive, closed, hardware RAID controllers are a part of the open storage future. Instead, data integrity will be delivered with software like Sun's ZFS filesystem with its end-to-end data integrity model using inexpensive disks, and the considerable capabilities of increasing powerful standard compute servers.
Peter Bojanic, Director of the Lustre Group, spoke next. He started with some fun facts about Lustre, Sun's parallel cluster file system, which joined Sun's portfolio with the acquisition of Cluster File System, Inc. Some of the superlatives he mentioned: 25000 clients accessing a single Lustre file system on Red Storm at Sandia National Laboratory and CEA achieving an aggregate 100 GB/sec transfer rate with their Lustre configuration. He also pointed out that Lustre is used on 7 of the 10 largest supercomputers in the world (ref. Nov 2007 TOP500.)
Peter spent the bulk of his time discussing the Lustre roadmap, which my fingers were not nimble enough to capture in any detail. I expect the slides to be posted at some point on the Consortium website, so watch for them there.
Harriet Coverston, Sun Distinguished Engineer, spoke about Shared QFS and the Storage Archive Manager (SAM), two storage products that are well-known in the HPC community. Perhaps less well-known is that QFS has in an increasing footprint with non-HPC enterprise customers who need its scalability, performance, and reliability. Home Box Office (HBO) is a great example of this. Read about their use of QFS here.
After giving an overview of Shared QFS and SAM, Harriet spoke about her group's plans to move to what she called intelligent storage. The intent is to move from a traditional SAN approach to one that embraces T10-based object storage mechanisms, which will greatly increase QFS scalability from its current limit of 256 clients to the range of thousands of clients. Read more about object-based storage here.
Our cluster of storage-related talks ended with a presentation by Chris Wood, CTO for Sun's Storage and Data Management Practice. Chris focused on how Sun plans to deliver complete, modular, and scalable storage solutions rather than merely pieces of excellent product and technology. The fundamental problem is how to best satisfy a wide range of potentially conflicting user requirements, the most of common of which are high performance, low cost, high capacity, and a single architectural approach for all workloads. Sun's approach uses a modular architecture that can grow and shrink based on customer requirements and which leverages Sun's hardware and software. For example, SAM-QFS for high performance and data archiving, Lustre for additional scalability and performance, the x4500 storage server, high performance network interface cards (for example, Neptune) and current and future x86 and T2-based servers.