Why Does Sun Publish 64-bit SPECjbb2000?
By dagastine on Sep 23, 2005
Sun has chosen to only submit 64-bit SPECjbb2000 results because of limitations of the benchmark which have allowed our competition to make special case optimizations in the 32-bit case. JVM optimizations for this benchmark can only really apply to customer workloads using 64-bit JVMs with huge heaps. A good example of the special case optimizations is the SPECjbb2000 call for a full system GC during the initialization stage of the benchmark and before every measurment interval, effective removing Full GC performance from the benchmark. Notice the flags used for the JRockit and IBM JVMs for this benchmark: BEA: -XXfullsystemgc IBM: -Xcompactexplicitgc (see: http://www.spec.org for additional flag details.) These flags suggest special case optimizations for the unrealistic use case of SPECjbb2000. The BEA flags suggests enabling "a full system gc", and the IBM flags suggests "compact at a explicit gc call". Both special case for this benchmark. SPECjbb2000 is an old, unrealistic benchmark that is ripe for "benchmark special" optimizations. Sun is not interested in optimizing for benchmarks that have no real effect on our customers. We publish with 64-bit because it is a better match for environments where full gc's will be avoided (by using huge heaps), and therefore optimizations made in that space yield benefits for our customers as well. SPECjbb2005 on the other hand has addressed many of the benchmarks limitations. Explicit GC calls have been removed, allowing Full GC's to effect the performance during the measurement interval. XML and J2SE API's have replaced hand-rolled code. All in all the new benchmark has been successfully updated to model modern Java server applications. Please notice as well that BEA has only 1 public submission for SPECjbb2005 and IBM has none. Also notice that Sun Hotspot is the performance leader with SPECjbb2005. Why? Because the new benchmark actually attempts to model real customer applications through API use and GC behavior, and Sun's concentration has alway been on customer performance, not stunt JVM's for SPECjbb2000.