By user12601629 on Jan 25, 2007
It looks like SPEC is working on a virtualization benchmark. This article makes it sound like it's been going on a while and Sun even seems to be involved. I may need to find out who our rep is on the commitee. At first glance this seems a bit silly (what would a virtualization benchmark even mean?), but experience says this will probably be a good things for customers in the long-term.
Back in 2001 and 2002 I was involved in the great Java benchmarking wars. I was the manager of the Java performance team back then and Sun was launching a new line of servers around the UltraSPARC III chip. Marketing needed to prove that these were the fastest servers on the planet and Java was a hot technology on the server side. They asked us to produce a world record benchmark for the launch. We spent a huge amount of effort on this (including rewriting the HotSpot garbage collector almost from scratch) and produced a world record on the SPECjbb benchmark. We partied! A few weeks later, IBM released a score that trumped ours and two years of benchmarking wars erupted that included players from all parts of the computer hardware industry.
There are some hugely entertaining stories in here (including the time one of the vendors secretly hired a small consulting company to run benchmarks on old Sun hardware and submit the scores to SPEC in order to make us look bad!). In addition, people on all sides implemented short-lived hacks the produced benchmark results, but didn't really benefit customers.
However, in the end, the wars over SPECjbb, ECperf, and SPECjAppServer led to major improvements in JVMs and Application Servers from all vendors. We tuned the JVM to run on huge numbers of processors -- more than we ever thought reasonable, actually. However, SPECjbb later became one of the benchmarks the engineers used to design Sun's super-scalable Niagara chips, which have many characteristics in common with a 32-CPU super computer, and Java runs like crazy on them now. Go figure!
I'll be really curious to see what this virtualization benchmark turns into. It's probably a couple years away, but virtualization will be here for a long while.