Drinking the Cool-aid, and SPECjbb2005

A colleague of mine pointed me to a comment to BMSeer's  by Frank.  I was a bit late reading this comment,
so I'm going to respond with a posting of my own.


Here's the text from Frank's comment:
"Hi BMSeer,

I'm just confused with your statement saying T5440 is the best
specjbb2005 performance machine. In your table I can see total
throughput is 692736 and probably the highest number now. This
benchmark specifically for T5440 uses 32 JVMs and the performance (not
througput) is only 21648/JVM whic is the lowest number in your table.
By comparing to Dell R900 (6 core Xeom) the throughput is 508240 and
performance is 132917/JVM which means use only 4 JVMs. This benchmark
will give us two important information : CPU/Core performance and JVM
performance. Unfortunately you didn't put SUN Intel/AMD machine to be
compared here. However we know that SUN X4600M2 (8 chips, 32 cores)
will give 683542 and the performance is 85443/JVM by runnning 8 JVMs.
So it is very clear nothing so special with T5440 to run Java
application because it has the lowest performance/JVM and SUN JVM
performance on Intel/AMD is lower than Jrockit JVM which is used on
Dell R900 benchmark. The other important thing that we have to know, by
running more JVM means more memory will be needed. By looking at T5440
compared to Dell R900 (32 JVM vs 4 JVM), it is very clear that Intel
wins in Java performance benchmark (the most efficient) and Jrockit is
much faster than SUN JVM."

First, I'd like to thank Frank for his comment to BMSeer's blog.  It give me a chance to talk about Sun SPARC Enterprise T5440, and the Sun Fire X4450.

Now, the golden rule when smearing your competition or making competitive performance claims is to do your homework.  Frank didn't do his homework.

The Sun Fire X4450 is a 4-Socket Intel Xeon X7460 and is very similarly configured to the Dell R900 mentioned by Frank.  The have identical memory and CPU configurations, and are running the same number of JVMs, which more a less allows a head to head software performance comparision. The Sun result is a solid 5% faster than the Dell result.  Why?  The Sun result is running Sun Java 6 Update 6 Performance release, the Dell result is running Oracle JRockit.  

Dell PowerEdge R900: SPECjbb2005 bops = 508240, SPECjbb2005 bops/JVM = 127060

Sun Fire X4450: SPECjbb2005 bops = 531669, SPECjbb2005 bops/JVM = 132917

Seems that Frank's closing comment is a bit unfounded.  Perhaps another colleague from Dell could help Frank out.

Sun also published a single JVM result on the Sun Fire X4450 with a score of 448,262 SPECjbb2005 bops.  Using Frank's logic this would make Sun JDK 6 Update 6-P 3.37x faster than Oracle JRockit!  Clearly this is not a correct comparison, as is Frank's broken AMD v. Intel, Sun v. Oracle comparisons. 

Sun Fire X4450: SPECjbb2005 bops = 448,262, SPECjbb2005 bops/JVM = 448,262

Now as for the Sun SPARC Enterprise T5440 compared to the Dell PowerEdge R900, the proper comparison to gauge system throughput when comparing SPECjbb2005 multi-JVM result is to look at the SPECjbb2005 bops metric.  You're treading on thin ice when attempting to compare SPECjbb2005 bops/JVM using multi-JVM results.  When running multi-JVM the JVM count is used as a tunable configuration option, and the only reason 4-JVMs were chosen when running on the Dell R900 or the Sun Fire X4450 is because it was the fastest configuration.  The previous generation of the Intel Xeon MP ran with twice as many JVMs.  So, comparing the 4-Socket, 24-core Dell PowerEdge R900 to the 4-Socket, 32-core Sun SPARC Enterprise T5440, the Sun T5440 is 36% faster.  We plan to submit a single JVM result on the Sun Enterprise T5440 and it seems that we may need to push a little harder to get that out.   Stay tuned.

Sun Fire T5440: SPECjbb2005 bops = 692,736 SPECjbb2005 bops/JVM = 21,648

Last word.  Do your homework when trashing your competition.  You're making it to easy.


Disclosure Statement:

Sun SPARC Enterprise T5440 (4 chips, 32 cores) 692736 SPECjbb2005 bops, 21648 SPECjbb2005 bops/JVM.   Dell R900 (4 chips, 24 cores) 508240 SPECjbb2005 bops, 127060 SPECjbb2005 bops/JVM. 
Sun X4450 (4 chips, 24 cores) 531669 SPECjbb2005 bops, 132917 SPECjbb2005 bops/JVM.
SPEC, SPECjbb reg tm of Standard Performance Evaluation Corporation.
Results from www.spec.org as of 10/27/08


Frank would be further surprised to find that a 1 HotSpot JVM just might beat the 508240 4-JVM Dell R900 SPECjbb2005 score. I'd invite Frank to do the comparison himself or anyone else who's interested in observing the performance a 5440 and HotSpot. Sun offers a "60 day trial evaluation" of the Sun 5440. See http://www.sun.com/tryandbuy/index.jsp to get your evaluation under way. I'm sure you (Dave) would be very happy to provide Frank and anyone else the necessary information to run SPECjbb2005 in a 1 JVM configuration on a 5440. :-) Btw, Dell might realize better SPECjbb2005 performance using a HotSpot JVM. Wonder how I would know? ;-)

Posted by Robert Boyd on November 02, 2008 at 10:28 AM EST #

Post a Comment:
Comments are closed for this entry.



« April 2014