Take Two: Comparing JVMs on ARM/Linux

Although the intent of the previous article, entitled Comparing JVMs on ARM/Linux, was to introduce and highlight the availability of the HotSpot server compiler (referred to as c2) for Java SE-Embedded ARM v7,  it seems, based on feedback, that everyone was more interested in the OpenJDK comparisons to Java SE-E.  But there were two main concerns:

  • The fact that the previous article compared Java SE-E 7 against OpenJDK 6 might be construed as an unlevel playing field because version 7 is newer and therefore potentially more optimized.
  • That the generic compiler settings chosen to build the OpenJDK implementations did not put those versions in a particularly favorable light.

With those considerations in mind, we'll institute the following changes to this version of the benchmarking:

  1. In order to help alleviate an additional concern that there is some sort of benchmark bias, we'll use a different suite, called DaCapo.  Funded and supported by many prestigious organizations, DaCapo's aim is to benchmark real world applications.  Further information about DaCapo can be found at http://dacapobench.org.
  2. At the suggestion of Xerxes Ranby, who has been a great help through this entire exercise, a newer Linux distribution will be used to assure that the OpenJDK implementations were built with more optimal compiler settings.  The Linux distribution in this instance is Ubuntu 11.10 Oneiric Ocelot.
  3. Having experienced difficulties getting Ubuntu 11.10 to run on the original D2Plug ARMv7 platform, for these benchmarks, we'll switch to an embedded system that has a supported Ubuntu 11.10 release.  That platform is the Freescale i.MX53 Quick Start Board.  It has an ARMv7 Coretex-A8 processor running at 1GHz with 1GB RAM.
  4. We'll limit comparisons to 4 JVM implementations:
    • Java SE-E 7 Update 2 c1 compiler (default)
    • Java SE-E 6 Update 30 (c1 compiler is the only option)
    • OpenJDK 6 IcedTea6 1.11pre 6b23~pre11-0ubuntu1.11.10.2 CACAO build 1.1.0pre2
    • OpenJDK 6 IcedTea6 1.11pre 6b23~pre11-0ubuntu1.11.10.2 JamVM build-1.6.0-devel

Certain OpenJDK implementations were eliminated from this round of testing for the simple reason that their performance was not competitive.  The Java SE 7u2 c2 compiler was also removed because although quite respectable, it did not perform as well as the c1 compilers.  Recall that c2 works optimally in long-lived situations.  Many of these benchmarks completed in a relatively short period of time.  To get a feel for where c2 shines, take a look at the first chart in this blog.

The first chart that follows includes performance of all benchmark runs on all platforms.  Later on we'll look more at individual tests.  In all runs, smaller means faster.  The DaCapo aficionado may notice that only 10 of the 14 DaCapo tests for this version were executed.  The reason for this is that these 10 tests represent the only ones successfully completed by all 4 JVMs.  Only the Java SE-E 6u30 could successfully run all of the tests.  Both OpenJDK instances not only failed to complete certain tests, but also experienced VM aborts too.

One of the first observations that can be made between Java SE-E 6 and 7 is that, for all intents and purposes, they are on par with regards to performance.  While it is a fact that successive Java SE releases add additional optimizations, it is also true that Java SE 7 introduces additional complexity to the Java platform thus balancing out any potential performance gains at this point.  We are still early into Java SE 7.  We would expect further performance enhancements for Java SE-E 7 in future updates.

In comparing Java SE-E to OpenJDK performance, among both OpenJDK VMs, Cacao results are respectable in 4 of the 10 tests.  The charts that follow show the individual results of those four tests.  Both Java SE-E versions do win every test and outperform Cacao in the range of 9% to 55%.

For the remaining 6 tests, Java SE-E significantly outperforms Cacao in the range of 114% to 311%

So it looks like OpenJDK results are mixed for this round of benchmarks.  In some cases, performance looks to have improved.  But in a majority of instances, OpenJDK still lags behind Java SE-Embedded considerably.

Time to put on my asbestos suit.  Let the flames begin...

Comments:

No flames here, thanks for the comparison. The more JVMs we have, the more Oracle will be forced to keep performance as a priority in HotSpot. :)

Posted by Jonathan Fisher on March 20, 2012 at 02:42 PM EDT #

Of course its worth noting that all three working ARM variants of OpenJDK + Zero are left out in the above test:

1. The optimized interpreter + Thumb2 JIT Zero *mixed mode* jvm that performed the best of the openjdk variants in the last study. ( icedtea6-1.11.1 )
https://blogs.oracle.com/jtc/entry/comparing_jvms_on_arm_linux
2. The assembler-optimized Zero *interpreted mode* ( also found in icedtea6-1.11.1 -Xint ) this one was also included in the last study.
https://blogs.oracle.com/jtc/entry/comparing_jvms_on_arm_linux
3. The pure c++ Zero interpreter from OpenJDK upstream <- the zero variant shipped in oneiric, i can understand that you left this one out of this study above.

Nice to see the some big speedup especially for CACAO when switching from Debian armel armv4+ to Ubuntu armel armv7 thumb2 tuning.

Please note that in the next Ubuntu 12.04 long term support release, targeting servers, the userspace EABI have switched from the armel "softfp" to armhf "hard" for extra performance. All OpenJDK JVM variants, JamVM, CACAO and Zero have been updated to support this armhf EABI hard in order to be ready to best serve the ARM server market.

Posted by Xerxes Rånby on March 29, 2012 at 01:03 AM EDT #

The blog states that certain OpenJDK implementations were left out because they were not competitive. When running the DaCapo benchmark on these OpenJDK + Zero variants, performance was 5x slower (and sometimes worse) than the Java SE-E versions. I was trying to be nice by including only the best of the OpenJDK variants.

Posted by Jim Connors on March 29, 2012 at 04:01 AM EDT #

I tried to run the latest Embedded Java SE 7u4 on my Pandaboard running Ubuntu Server 12.04 LTS but so far I couldn't get it working.

The first problem was the hardcoded library path "/lib/ld-linux.so.3".
I switched to /lib and ran the following command to fix that issue:
sudo ln -s arm-linux-gnueabihf/ld-2.15.so ld-linux.so.3

The other libs where found but it still doesn't work.
Installation details and my attempts to get it working can be found here:
http://groups.google.com/group/pandaboard/t/6dbe77b4457918a0

Any ideas? Linux Kernel 3.2.0 issue?

Posted by J.U. on May 03, 2012 at 07:59 PM EDT #

J.U., Ubuntu 12.04 uses the new hard-float ABI (aka armhf or gnueabihf), whereas 7u4 only supports the soft-float ABI (aka armel or gnueabi). You need to remove the symlink you created and install the soft-float libc and gcc support libraries:

sudo apt-get install -y libc6-armel libsfgcc1

Posted by Trevor Robinson on July 17, 2012 at 03:50 PM EDT #

Hello Trevor,

many thanks I'll try that.
Still it would be great if the next Embedded Java (7u5+) would support the hard-float ABI (armhf) as this will speed things up even more.

Posted by J.U. on July 18, 2012 at 08:30 AM EDT #

Post a Comment:
  • HTML Syntax: NOT allowed
About

Jim Connors

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today