Monday Jul 06, 2009

Update: Benchmarking the iPhone, and lots of other devices

I had several comments and requests stemming from my original blog on benchmarking so what better way to respond to them than another entry.  I also have an update from starting to create a full fledged app rather than the console one I wrote about earlier.

First, thanks for the comments and keep them coming.  I had one comment that the Java ME version didn't work on Sony K850i.  I created a new version created with the latest NetBeans 6.7 for the latest CLDC/MIDP versions but it too didn't work simply saying "Invalid application".  I know it worked on several BlackBerry's and a Motorola L6 so suggestions are welcome.

Another comment provided some numbers for an IBM T43 which I've added to the spreadsheet.

As promised, I started fiddling with creating a proper iPhone app using Xcode and have gotten an ugly but functional version of the app working and initial indications are that it scores better than the one compiled with gcc on the iPhone.  Where the gcc version had a score of 27.9 the Xcode version scores double that at 65.7.  This is running on the same phone/hardware so the difference has to be the compiler and Xcode must generate a lot better code.  I'm still researching to see what I can find out.

Last, as promised, I've made some source code available.  Take a look at the C version and I'll work on getting the others up as soon as I know they are stable.

I welcome more feedback and results from running the benchmark.

Monday Jun 29, 2009

Benchmarking the iPhone, and lots of other devices too

Benchmarking has been around for years, and I started playing around with writing them back in college.  I recall an assembly language class were the assignment was to implement the Sieve of Eratosthenes in as few bytes as possible. I "won" the assignment by doing it in 24 (or was it 25, too many years ago!) bytes, but that got me started on benchmarking as I instrumented the code to time how long it took and then proceeded to run it on all the different machines I could to see which one was the fastest.

Clearly though, having a benchmark written in 8086 assembly language had limited usefulness, so in later years I rewrote it in C and added several other standard algorithms to the stable that were executed including calculating PI, a selection sort, and computing Ackermann's function to test recursive performance. I proceeded to run it on every machine I could from PCs running MS-DOS to workstations running Solaris to PPC machines running LynxOS and AIX. In later years, I ran the same benchmark on UltraSPARC, SGI MIPS, and DEC Alpha machines as well as today's dual-core Intel chips. Somewhere along the line I starting writing more in Java than C and I ported the benchmark to Java to see how it compared, and several years ago I began using BlackBerry's and wrote a Java ME version.

As you can imagine given the wide range of machines I've run it on in the past, having an iPhone was just begging to run it there too. While I have creating a proper iPhone app version on my list of things to do, thankfully a jailbroken iPhone makes it pretty easy to run the C and Java versions I already had. A sampling of results appears below, but the summary for the iPhone is that the native version is about 5 times faster than the Java one and ever so slightly faster than a 75 MHz Pentium 5 of yesteryear.

 Machine Language Math Array Sort Sieve Ackermann Total
Motorola L6 Java ME 0.11 0.19 0.23 0.13 0.19 0.17
BlackBerry Curve (OS 4.5) Java ME 0.95 2.21 2.96 1.72 2.96 1.81
iPhone 3G Java SE 12.4 4.79 4.64 3.28 5.01 4.95
Intel 486DX2-66 (Linux) C 10 10 10 10 10 10
SPARC-20 (Solaris) C 66.4 20.3 17.6 20.1 4.1 12.1
Pentium 5 75 MHz (Linux) C 21.3 19.6 28.6 23.2 33.4 24.2
iPhone 3G C 65.9 21 23.4 19.2 46.7 27.9
DEC Alpha 275 MHz (OSF/1) C 111.4 93.6 63.1 41.2 76.1 68.6
Pentium 600 MHz (Linux) Java SE 191.9 106.1 99.1 68.9 78.2 96.1
Pentium 600 MHz (Linux) C 216.4 319.3 219.8 171.5 131.2 194.1
MacBook Pro 2.33 GHz Java SE 655 1096 794 549 334 587
MacBook Pro 2.33 GHz C 1089 2413 664 800 1356 1037

You may also view the full list that I have maintained over the years.  You'll notice that I did my initial calibration on a 66 MHz 486 DX-2 so that it scores 10 in each of the tests.

Now, I'll be the first to admit that my benchmark is pretty simplistic and thus shouldn't be be the end all be all testing to compare machines or platforms. The results can also be significantly affected by the quality of the C compiler or JVM being used. However, the results can provide a rough comparison and are at least fun to look at.

For example a SPARC-20 has terrible performance with deep recursion. And as noted above the JVM on the iPhone is about 5 times slower than a native app whereas on an x86 box the JVM is about half as fast as native.

While I've run the benchmark on quite a few machines, I'd love to add more to the list.  If you'd like to run it, here is the binaries in a variety of formats:

  • Linux on x86 - Download the binary, set the executable flag, and execute with "./sci <iterations>". On most of today's faster machines iterations should be 10,000 to have reasonable results.
  • OS X on x86 - Download the binary, set the executable flag, and execute with "./sci <iterations>". On most of today's faster machines iterations should be 10,000 to have reasonable results.
  • OS X on ARM/iPhone - Download the binary, use ssh/scp to get it on your jailbroken iPhone, and execute with "./sci <iterations>". The default iterations should be fine.
  • Solaris on x86 - Download the binary, set the executable flag, and execute with "./sci <iterations>". On most of today's faster machines iterations should be 10,000 to have reasonable results.
  • Solaris on SPARC - Download the binary, set the executable flag, and execute with "./sci <iterations>". On most of today's faster machines iterations should be 10,000 to have reasonable results.
  • AIX on RS/6000 - Download the binary, set the executable flag, and execute with "./sci <iterations>". On most of today's faster machines iterations should be 10,000 to have reasonable results.
  • Java SE - Download the jar and execute it on any machine you have a JVM with "java -jar SCIJ.jar <iterations>". On most of today's faster machines iterations should be at least 5,000.
  • Java ME devices - Point your phone/device browser at the link and it should prompt to install the application. Run the app and if it executes in less than a few seconds change the iterations to a larger number to make it more accurate in calculating the benchmark.

If you do run the benchmark, please post a comment on this blog entry with the chipset, clock-speed, operating system, and results from each of the tests that the program displays. As soon as someone jailbreaks an iPhone 3G S, it would be great to add that result to the list.

Enjoy!

 

About

kevinschmidt

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today