Benchmarking the iPhone, and lots of other devices too

Benchmarking has been around for years, and I started playing around with writing them back in college.  I recall an assembly language class were the assignment was to implement the Sieve of Eratosthenes in as few bytes as possible. I "won" the assignment by doing it in 24 (or was it 25, too many years ago!) bytes, but that got me started on benchmarking as I instrumented the code to time how long it took and then proceeded to run it on all the different machines I could to see which one was the fastest.

Clearly though, having a benchmark written in 8086 assembly language had limited usefulness, so in later years I rewrote it in C and added several other standard algorithms to the stable that were executed including calculating PI, a selection sort, and computing Ackermann's function to test recursive performance. I proceeded to run it on every machine I could from PCs running MS-DOS to workstations running Solaris to PPC machines running LynxOS and AIX. In later years, I ran the same benchmark on UltraSPARC, SGI MIPS, and DEC Alpha machines as well as today's dual-core Intel chips. Somewhere along the line I starting writing more in Java than C and I ported the benchmark to Java to see how it compared, and several years ago I began using BlackBerry's and wrote a Java ME version.

As you can imagine given the wide range of machines I've run it on in the past, having an iPhone was just begging to run it there too. While I have creating a proper iPhone app version on my list of things to do, thankfully a jailbroken iPhone makes it pretty easy to run the C and Java versions I already had. A sampling of results appears below, but the summary for the iPhone is that the native version is about 5 times faster than the Java one and ever so slightly faster than a 75 MHz Pentium 5 of yesteryear.

 Machine Language Math Array Sort Sieve Ackermann Total
Motorola L6 Java ME 0.11 0.19 0.23 0.13 0.19 0.17
BlackBerry Curve (OS 4.5) Java ME 0.95 2.21 2.96 1.72 2.96 1.81
iPhone 3G Java SE 12.4 4.79 4.64 3.28 5.01 4.95
Intel 486DX2-66 (Linux) C 10 10 10 10 10 10
SPARC-20 (Solaris) C 66.4 20.3 17.6 20.1 4.1 12.1
Pentium 5 75 MHz (Linux) C 21.3 19.6 28.6 23.2 33.4 24.2
iPhone 3G C 65.9 21 23.4 19.2 46.7 27.9
DEC Alpha 275 MHz (OSF/1) C 111.4 93.6 63.1 41.2 76.1 68.6
Pentium 600 MHz (Linux) Java SE 191.9 106.1 99.1 68.9 78.2 96.1
Pentium 600 MHz (Linux) C 216.4 319.3 219.8 171.5 131.2 194.1
MacBook Pro 2.33 GHz Java SE 655 1096 794 549 334 587
MacBook Pro 2.33 GHz C 1089 2413 664 800 1356 1037

You may also view the full list that I have maintained over the years.  You'll notice that I did my initial calibration on a 66 MHz 486 DX-2 so that it scores 10 in each of the tests.

Now, I'll be the first to admit that my benchmark is pretty simplistic and thus shouldn't be be the end all be all testing to compare machines or platforms. The results can also be significantly affected by the quality of the C compiler or JVM being used. However, the results can provide a rough comparison and are at least fun to look at.

For example a SPARC-20 has terrible performance with deep recursion. And as noted above the JVM on the iPhone is about 5 times slower than a native app whereas on an x86 box the JVM is about half as fast as native.

While I've run the benchmark on quite a few machines, I'd love to add more to the list.  If you'd like to run it, here is the binaries in a variety of formats:

  • Linux on x86 - Download the binary, set the executable flag, and execute with "./sci <iterations>". On most of today's faster machines iterations should be 10,000 to have reasonable results.
  • OS X on x86 - Download the binary, set the executable flag, and execute with "./sci <iterations>". On most of today's faster machines iterations should be 10,000 to have reasonable results.
  • OS X on ARM/iPhone - Download the binary, use ssh/scp to get it on your jailbroken iPhone, and execute with "./sci <iterations>". The default iterations should be fine.
  • Solaris on x86 - Download the binary, set the executable flag, and execute with "./sci <iterations>". On most of today's faster machines iterations should be 10,000 to have reasonable results.
  • Solaris on SPARC - Download the binary, set the executable flag, and execute with "./sci <iterations>". On most of today's faster machines iterations should be 10,000 to have reasonable results.
  • AIX on RS/6000 - Download the binary, set the executable flag, and execute with "./sci <iterations>". On most of today's faster machines iterations should be 10,000 to have reasonable results.
  • Java SE - Download the jar and execute it on any machine you have a JVM with "java -jar SCIJ.jar <iterations>". On most of today's faster machines iterations should be at least 5,000.
  • Java ME devices - Point your phone/device browser at the link and it should prompt to install the application. Run the app and if it executes in less than a few seconds change the iterations to a larger number to make it more accurate in calculating the benchmark.

If you do run the benchmark, please post a comment on this blog entry with the chipset, clock-speed, operating system, and results from each of the tests that the program displays. As soon as someone jailbreaks an iPhone 3G S, it would be great to add that result to the list.

Enjoy!

 

Comments:

Hi, I have been doing embedded Java benchmarks for a while - the latest version is here (along with source): http://www.practicalembeddedjava.com/benchmark/11a/benchmark_11c.html

Can you post the source code for yours?

I will try to run yours on my Sony phone K850i.

I'm just starting a major new effort to benchmark the latest crop of embedded (industrial, not phones) Java systems from aJile, Imsys, Sun (Squawk) and others, including the Android G1 (I just ordered the dev unlocked HTC G1). I'm trying to get some support from the Java.net Mobile and Embedded Community and other places since this will be a significant effort (including documentation of each platforms dev tools and environment).

best regards
Bruce

Posted by Bruce Boyes on June 29, 2009 at 10:44 PM PDT #

"Invalid application" when trying to run the ME jad on Sony Ericsson K850i (last year's JavaOne show phone).

Posted by Bruce Boyes on June 30, 2009 at 12:00 AM PDT #

IBM T43 XPPro SP3, 1.86 GHz Pentium M, 2 GB RAM

C:\\Documents and Settings\\bab\\My Documents\\Incoming\\java\\KevinSchmidt>java -jar SCIJ.jar 5000
Math Index: 504.750593824228
Array Index: 590.5292479108635
Sort Index: 482.8767123287672
Sieve Index: 366.78200692041526
Recursion Index: 251.7772511848341
Schmidt Computing Index: 401.70454545454544,439.3431624338217

C:\\Documents and Settings\\bab\\My Documents\\Incoming\\java\\KevinSchmidt>java -jar SCIJ.jar 5000
Math Index: 485.1598173515982
Array Index: 590.5292479108635
Sort Index: 482.8767123287672
Sieve Index: 366.78200692041526
Recursion Index: 238.76404494382024
Schmidt Computing Index: 392.3418423973363,432.822365891093

C:\\Documents and Settings\\bab\\My Documents\\Incoming\\java\\KevinSchmidt>java -jar SCIJ.jar 5000
Math Index: 400.1883239171375
Array Index: 588.8888888888889
Sort Index: 483.98169336384444
Sieve Index: 315.47619047619054
Recursion Index: 242.85714285714286
Schmidt Computing Index: 368.8695652173913,406.27844790064086

C:\\Documents and Settings\\bab\\My Documents\\Incoming\\java\\KevinSchmidt>java -jar SCIJ.jar 10000
Math Index: 453.09168443496804
Array Index: 602.2727272727274
Sort Index: 491.86046511627916
Sieve Index: 371.603856266433
Recursion Index: 247.09302325581396
Schmidt Computing Index: 395.48760022375535,433.1843512692443

C:\\Documents and Settings\\bab\\My Documents\\Incoming\\java\\KevinSchmidt>java -jar SCIJ.jar 10000
Math Index: 438.5964912280702
Array Index: 590.5292479108635
Sort Index: 474.7474747474748
Sieve Index: 361.7747440273038
Recursion Index: 251.92649673977473
Schmidt Computing Index: 390.10483722641163,423.5148909306974

Posted by Bruce Boyes on June 30, 2009 at 12:02 AM PDT #

Thanks for the comments Bruce. I've added your results to the online spreadsheet and I'll post the source in another blog entry this evening.

And apologies for the Java ME version not working. I wrote it a long time ago and know it worked on a few BlackBerry's and a Motorola L6.

Posted by Kevin Schmidt on June 30, 2009 at 04:15 AM PDT #

Post a Comment:
  • HTML Syntax: NOT allowed
About

kevinschmidt

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today