SPEC awards, virtualization
By walterbays on May 14, 2009
More 2009 SPECtacular awards. SPEC's forthcoming virtualization benchmark will provide meaningful metrics of hardware and software performance in data center consolidation. As complex as this benchmark is, running several different benchmarks together in virtual machines on a host system under test, the code is only half the story. As with all benchmarks the workload is vital, to represent realistic usage scenario(s) so that performance improvements made on the benchmark will also benefit real world users. And the run rules are vital, needing to accommodate technology improvements over the lifetime of the benchmark, while precluding unrepresentative optimizations exploiting rule loopholes. (Or what the layman might call “cheating”) There is spirited debate from companies representing rather diverse user communities, all with an interest in seeing that their customers' needs are addressed by the benchmark. In the end when this group of top engineers reaches a consensus you know they've come up with a benchmark that is as rock solid as is possible to make. From among this great team of partners and competitors, three were singled out for SPECtacular awards:
Andrew Bond of HP always steps forward when a person is needed to test new code, features, parameter tuning. He performed many experiments whose results showed the committee the sensitivity of the benchmark to various parameters, sizes, and configuration options, so that the right choices could be made for fair benchmark comparisons. He also created scripts to set up and configure new guest VMs for each workload.
Chris Floyd of IBM improved and tailored the mail server and application server workloads for the new benchmark. He's revamped these workloads several times to improve the I/O profiles and add burstiness to the application server transaction injection. He helps the other developers at regular on-line coding sessions, explaining new features, and resolving problems. He even helps out when on vacation.
Greg Kopczynski of VMware developed a (necessarily) complex and feature extensive harness for the benchmark. He responds to countless pleas for help, assistance, debugging, etc., in true SPEC fashion without asking whether the help is for a partner or a competitor. He added burstiness to the web server workload. And he integrates new code and changes from all the developers for each development kit revision.
Thanks for your great efforts!