No Tuning Required: Java SE Out-of-Box Vs. Tuned Performance

In my last entry titled Java SE Out-of-Box Competitive Performance I stressed the importance of out-of-box performance to customers and developers and how it is a passionate focus for Sun HotSpot and JVM Performance engineering. The following is a comparison of out-of-box and hand-tuned performance. The charts below are run on the same system as my previous entry and the charts are normalized to the same baseline, therefore the two sets of charts are directly comparable.

I have to say the numbers are quite impressive (hence the "No Tuning Required" in the title). My colleagues are going to say I'm blogging us out of a job :-).

  • On SPECjbb2005 the numbers are impressive. JDK 5.0_08 is ~22% faster when tuned compared to JDK 5.0_08 right out of the box. JDK 6 is ~11% faster when tuned versus right out of the box, and JDK 6 out of the box is only ~7% slower than a highly tuned JDK 5.0_08. Very impressive indeed!
  • On Scimark, tuning only improved slightly when running JDK 5.0_08. JDK 6 is more or less a wash.
  • On Volano, except when running JDK 5.0_08 64-bit, the out-of-the-box configuration seems to work well, and doesn't require any explicit tuning.
The system under test is a 2-way dual-core Opteron 280 Processors (2 CPUs, 4 cores, 2.4 Ghz) and 8GB of RAM. The Operating System is Red Hat EL 4.0 AS Update 4. The kernel version is unmodified from the base install, which is 2.6.9-42.ELsmp. The charts are statistical comparisons. No less than 10 samples were performed, and a T-test (single-tailed) was used to ensure confidence in the significance of the result. The data is normalized to the 32-bit Sun JDK 1.5.0_08 out-of-box result.

The following JVMs were tested:

  • Sun JDK 1.5.0_08
    • 32-bit: java version "1.5.0_08" Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_08-b03) Java HotSpot(TM) Server VM (build 1.5.0_08-b03, mixed mode)
    • 64-bit: java version "1.5.0_08" Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_08-b03) Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_08-b03, mixed mode)
  • Sun Java SE 6 build 99
    • 32-bit: java version "1.6.0-rc" Java(TM) SE Runtime Environment (build 1.6.0-rc-b99) Java HotSpot(TM) Server VM (build 1.6.0-rc-b99, mixed mode)
    • 64-bit: java version "1.6.0-rc" Java(TM) SE Runtime Environment (build 1.6.0-rc-b99) Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-rc-b99, mixed mode)
The following command line arguments were used:
  • SPECjbb2005
    • J2SE 5.0_08 32-bit: -Xmn1g -Xms1500m -Xmx1500m -XX:+UseBiasedLocking -XX:+AggressiveOpts -XX:+UseLargePages -XX:+UseParallelOldGC -Xss128k
    • J2SE 5.0_08 64-bit: -Xmn2g -Xms3g -Xmx3g -XX:+UseBiasedLocking -XX:+AggressiveOpts -XX:+UseLargePages -XX:+UseParallelOldGC -Xss128k
    • Java SE 6 RC1 32-bit: -Xmn1g -Xms1500m -Xmx1500m -XX:+UseLargePages -XX:+UseParallelOldGC -Xss128k
    • Java SE 6 RC1 64-bit: -Xmn2g -Xms3g -Xmx3g -XX:+UseLargePages -XX:+UseParallelOldGC -Xss128k
  • SciMark2
    • J2SE 5.0_08 32-bit: -XX:+UseBiasedLocking
    • J2SE 5.0_08 64-bit: -XX:+UseBiasedLocking
    • Java SE 6 RC1 32-bit: -XX:+DoEscapeAnalysis
    • Java SE 6 RC1 64-bit: -XX:+DoEscapeAnalysis
  • Volano 2.5.0.9
    • J2SE 5.0_08 32-bit: -XX:CompileThreshold=1500
    • J2SE 5.0_08 64-bit: -XX:CompileThreshold=1500
    • Java SE 6 RC1 32-bit: -XX:CompileThreshold=1500 -XX:-UseBiasedLocking
    • Java SE 6 RC1 64-bit: -XX:CompileThreshold=1500 -XX:-UseBiasedLocking
The SPECjbb2005 numbers are impressive. JDK 5.0_08 is ~22% faster tuned compared to JDK 5.0_08 out-of-box. JDK 6 is only ~11% faster tuned vs. JDK 6 out-of-box, and JDK 6 out-of-box is only ~7% slower than highly tuned JDK 5.0_08. Nice.

Tuning only improved Scimark slightly when running 5.0_08. When running JDK 6 it's more or less a wash. The JDK 6 64-bit difference is statistically insignificant.

Tuning seems to hurt Volano, except when running 5.0_08 64-bit. Come to find out the negative differences are statistically insignificant so tuning is a wash with Volano as well.

In summary, meeting or exceeding tuned performance is the end game for out-of-box performance engineering. The above results make me quite proud of our accomplishments. Yes, every application is different and in some cases we'll find ourselves needing to tune. But chances are if you let us know the issues you're facing a release or two down the line you won't need to tune. Eventually it will just be us geeks who can't help it :-). Next step is a Solaris x86 vs Linux comparison. Stay tuned.

SPEC(R) and the benchmark name SPECjbb(TM) are trademarks of the Standard Performance Evaluation Corporation. Competitive benchmark results stated above reflect experiments performed by Sun Microsystems, Inc. For the latest SPECjbb2005 benchmark results, visit http://www.spec.org/osg/jbb2005.

Comments:

Next step is a Solaris x86 vs Linux comparison. Stay tuned.

Hehe.. No, I'll stay out of the box! =)

Dmitri

Posted by Dmitri Trembovetski on October 16, 2006 at 11:51 AM EDT #

Hi,

I'm interested in uour use of -XX:CompileThreshold=1500

Presumably this implies that tiered compilation, and also a run-to-run memory of which routines are hot during start-up, would be especially helpful! Go and prod FatCatAir and see how his wedding cake is getting on!

Rgds

Damon

Posted by Damon Hart-Davis on October 16, 2006 at 11:12 PM EDT #

hi there,

cool!

"Next step is a Solaris x86 vs Linux comparison. Stay tuned." -- that sounds very tasty!!

BR,
~A

Posted by anjan bacchu on October 17, 2006 at 07:32 PM EDT #

i think that team's Sun is buildding quality software

Posted by guest on October 18, 2006 at 03:25 AM EDT #

yea men

Posted by guest on October 18, 2006 at 01:38 PM EDT #

I heard about Swing performance gains alot in JDK 6.0. Can you do the benchmark to show us?

Posted by deans4j on October 18, 2006 at 07:36 PM EDT #

Yes, Swing performance is quite a bit faster. Thanks for the feedback, I'll work on gettting Swing performance data posted.

Posted by dagastine on October 19, 2006 at 01:51 AM EDT #

Post a Comment:
Comments are closed for this entry.
About

dagastine

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today