Java SE Embedded Performance Versus Android 2.2

INTRODUCTION

Having recently released our latest version of Java SE Embedded 1.6.0_21 on the web, I thought I'd take a breather and have a little fun and do something completely different.   You most likely have seen the barrage of TV commercials advertising the droid phones which run the Android Operating System from Google.  Android is a Java-like platform that runs a Java Virtual Machine called Dalvik.  Until recently, this Virtual Machine (VM) ran interpreted only which means that its Java execution speed was substantially slower than most Java runtimes available on the web.   Google's latest version of Android (version 2.2 the Froyo Release), now contains a Just-In-Time compiler (or JIT for short).   As a result, the speed at which Android 2.2 can now execute Java bytecodes has increase.  There are claims on the web that Android 2.2 can now process Java bytecodes from 3 to 6 times faster than the previous release.   Now that Android has at least gotten in the ballpark of other mature VMs, I thought it would be fun to see how well it stacks up against our own Java SE Embedded Hotspot Virtual Machine.

METHODOLOGY

In order to have an apples-to-apples comparison, I had to overcome a few obstacles.   First, Dalvik doesn't directly execute Java class files, it processes dex files.   During the process of creating Android applications, Java class files are converted to an Android specific dex format with a "dx" utility provided in the Android SDK.  It's these "dex" files that Android runs.  Second, Android and Java SE don't share the same Graphical Interface APIs so I can't run any GUI based benchmarks.   Finally, Java SE doesn't run on Android's version of Linux since it doesn't run the standard glibc "C" library.

In order to overcome these obstacles, I configured my test hardware platforms to run both Android and a standard Linux distribution.  I selected some benchmarks that didn't require GUI APIS and I converted each benchmark to "dex" format for the runs on Android.  I was then able to run the Java class file version of the benchmarks on standard Linux and the "dex" version of the benchmark was run on Android.    I booted up Android, ran the benchmarks, then rebooted the same system and then ran Java SE Embedded.   

I performed these benchmark runs on two different ARM based devices - a  Beagleboard and an Nvidia Tegra2 device.  The Beagleboard is a single core ARM device and the Tegra2 is a dual-core ARM Cortex-A9 based device.  

Here are some notes on how I ran these benchmarks on Android 2.2:

The dex conversion command I used was:

% dx --dex --output=benchmark.dex benchmark.jar

% mv benchmark.jar benchmark.jar.save

% mv benchmark.dex benchmark.jar

To copy the converted benchmark to Android, I used their adb tool:

% adb push benchmark.jar /sdcard/benchmark.jar

To run the benchmark on Android, I used:

% adb shell dalvikvm -cp /sdcard/benchmark.jar  Mainclassname

Note: substitute the real  benchmark name for benchmark.jar and the main application name for Mainclassname.


DEVICES AND SOFTWARE DISTRIBUTIONS

Here are the specifications and links to the boards and operating system platforms that I installed and ran on these two devices.

BEAGLEBOARD  (Rev C3)  Specifications

Texas Instruments OMAP3530 ARM Cortex-A8™ Processor 600Mhz, 256MB RAM

Hardware specification URL:  

http://beagleboard.org/hardware

Android 2.2 distribution for BeagleBoard was downloaded from here:

http://code.google.com/p/android-development-environment/wiki/FroyoOnBeagleBoard

The Angstrom Linux Distribution for BeagleBoard was downloaded from here: 

http://www.angstrom-distribution.org/demo/beagleboard

The main Angstrom beagle board project page is here:

http://beagleboard.org/project/angstrom/

 

NVIDIA TEGRA-2  250 Developer Kit Specifications

Dual-core ARM® Cortex-A9 MPCore™ processor,  1.0 GHz, 1024MB RAM

URL:   http://developer.nvidia.com/tegra/tegra-devkit-features

The Android 2.2 and Linux Distributions for Tegra-2 can be downloaded from the Nvidia developer site at :

http://tegradeveloper.nvidia.com/tegra/downloads



BENCHMARKS

Here are the benchmarks that I ran on these two devices.  I had to select benchmarks that did not require any Java graphical APIs and could run on the APIs that are implemented in Android.

PENDRAGON CAFFEINEMARK (tm) ver 3.0  * (http://www.benchmarkhq.ru/cm30)

Description: The CaffeineMark is a series of tests that measure the speed of Java programs running in various hardware and software configurations.

* The CaffeineMark benchmark was run without independent verification by Pendragon Software.  Pendragon Software makes no representations or warranties as to the result of the test.  CaffeineMark is a trademark of Pendragon Software.

SCIMARK 2.0   (http://math.nist.gov/scimark2)

Description:  SciMark 2.0 is a Java benchmark for scientific and numerical computing.  It measures several computational kernels and reports a composite score in approximate Mflops (Millions of floating point operations per second).

KBENCH (Sun/Oracle Internal Benchmark)

Description: kBench is a collection of small Java programs used to determine the relative performance of Java Virtual Machines.

 

BEAGLEBOARD BENCHMARK RESULTS

 

image001.png

 

 

 

TEGRA2 BENCHMARK RESULTS

 

image007.png

image009.png

image011.png


CONCLUSION


The results show that although Androids new JIT is an improvement over its interpreter only implementation, Android is still lagging behind the performance of our Hotspot enabled Java SE Embedded.   As you can see from the above results, Java SE Embedded can execute Java bytecodes from 2 to 3 times faster than Android 2.2.



Comments:

As the trend towards multi-core embedded processors accelerates, another embedded performance factor that heretofore may have been given less attention is scalability. A tremendous amount of history is on the side of the HotSpot JVM when it comes to dealing with massively threaded applications that run on a large number of processors/cores. With the introduction of just a second core, you can see that HotSpot gains further performance advantages. It will be real interesting to see how the benchmarks play out on the upcoming 4-core Arm processors. The type of optimization work done to HotSpot does not take place overnight. Android has its work cut out for it.

Posted by Jim Connors on November 23, 2010 at 07:32 AM EST #

I wish Oracle would stop stuffing around, take the bull by the horns and release Java SE Embedded, including LWUIT for mobiles complete with Java ME CLDC emulation. Java SE Embedded has the very important Isolates API MVM. Just imagine the possibilities for distributed computing.

Posted by Peter on November 23, 2010 at 09:40 PM EST #

so what about memory consumption and battery life? And why does HS lose the "Strings" test?

Posted by alsdkfj on November 24, 2010 at 11:08 PM EST #

It would be interesting to see the effect of having multiple instances of each test running at the same time (since the Dalvik VM is supposed to excel in this regard).

Posted by Rob Dickens on November 24, 2010 at 11:23 PM EST #

It is for sure interesting to know that Dalvik is slower and how much in a specific set of microbenchmarks, but how says the the goal of dalvik was to optimize the byte code execution speed *only*? I doubt that seriously. With any performance optimization problem you have more than one dimension you need to take into account. For example how important is startup performance versus long time running performance? How much memory is consumed? How much time is spend in native code? Therefore at the end this single KPI measured using an artificial micro benchmarks don't really tell you whether dalvik's or Java SE Embedded design is better in practice Regards, Markus

Posted by Markus Kohler on November 24, 2010 at 11:30 PM EST #

Honestly, who cares? I should apologize for sounding like a troll, but if oracles answer to the iphone is java me or se, they lost years ago. Java se is far superior to dalvik. Rather than suing google, they should have pressured google into a licensing agreement. if I walk into a sprint store, how many java se/me phones are there compared to android? Wake up oracle, offer an olive branch to google. With your company plus google, you could whip apple and microsoft into shape. Divided you've just increased their market share.

Posted by Jonathan on November 25, 2010 at 08:36 AM EST #

Interesting results Bob. We have found an open source java application benchmark suite (dacapobench.org) you may want to consider for future testing. In our benchmarking, DaCapo shows significant performance gains for u21 over the previous release, on beagle board and atom x86.

Posted by Tim B on November 29, 2010 at 05:47 AM EST #

Thanks for the tip. We just started looking at DaCapo and in fact a Java SE Embedded customer just reported great results against our previous release as well as other JREs. I haven't tried this yet, but since Android doesn't provide the complete Java SE API set, I seriously doubt if I could run DaCapo on Android. I did try to convert specJVM98 but the Android dex converter failed the conversion process. Thanks again, Bob.

Posted by bobvandette on December 02, 2010 at 06:39 AM EST #

In this blog, I decided to focus on pure JIT performance. In future blogs, I plan on taking a look at other aspects such as memory usage, startup time, compatibility, scalability and reliability. Thanks, Bob.

Posted by bobvandette on December 02, 2010 at 06:43 AM EST #

The CaffeineMark string test boils down to a tight loop performing string concatenation. We have analyzed other JVM's that combine the concatenation operations in order to avoid the array bounds checks. These JVMs are not conforming to the Java language specification. We take the conservative path in the regard. I do not know that Dalvik is doing this. It requires further investigation. Bob.

Posted by bobvandette on December 02, 2010 at 06:49 AM EST #

Based on such a benchmark, could we consider as a feasible near-future scenario an application running a JavaFX 2.0 user interface, in a JavaSE Embedded JVM on a Linux OS? Would such a platform be a viable substitute to the same application being developped against the Android-specific GUI APIs? My company is now developing such an application where the client part has to be a mobile device with touch screen. We do not really care about it to be Oracle- or Google-based. We do care about really rich user interface (not RIA). Is there any intent from Oracle to take the move onto the mobile platform? For real. Alain PS: the String lack of performance has been a real drawback of the Java platform for already more than a decade... Most user interface require some kind of text to be played with.

Posted by Alain Béarez on December 09, 2010 at 12:36 AM EST #

Post a Comment:
  • HTML Syntax: NOT allowed
About

bocadmin_ww

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today