SPECjbb2005: Single Instance vs. Multiple Instance Competitive Comparisons

SPECjbb2005 can be run in single and multiple-instance modes. Single instance is where one JVM runs the benchmark on a single system. Multiple instance is where n JVMs run in parallel, with the benchmark load distributed between the separate JVM processes. SPECjbb2005 also has two equally important metrics. SPECjbb2005 bops (business operations per second) is a measure of overall system throughput, and SPECjbb2005 bops/JVM, which is a measure of JVM performance and scalability. Single instance results target hardware, OS, and highlight JVM performance and scalability. The multiple instance results target hardware, OS, JVM performance and scalability, and highlight total system throughput. Both single and multi instance configurations of SPECjbb2005 can provide a sense of hardware, OS, and JVM performance and scalability. However, single instance configurations put more focus on the throughput delivered by the JVM, where as multi instance configurations put more focus on total throughput delivered by the system. When multiple instance configurations demonstrate higher throughput than single instance configurations, it's usually an indication that there's either a JVM limitation, such as maximum heap size or 64-bit JVM performance, or that there's some hardware architectural aspect of the system that multiple JVMs can take advantage of, such as a NUMA memory architecture. A SPECjbb2005 performance comparison between two hardware platforms is a comparison of the highest bops score as a measure of overall system throughput. Whe comparing hardware platforms the comparison can be made regardless of the benchmark configuration, but its important that you choose a configuration type that matches the deployment characteristics of your system as deployed in production. Most large MP servers with greater than 16 hardware threads are deployed with many, many JVM (or OS) instances, and customers are concerned with complete system throughput and scalability. The comparison is system throughput, not necessarily software component performance, but often JVM scalability is a factor considering each JVM must scale to 8 hardware threads or more. In this case the fastest results by hardware vendor A should be compared to the fastest results by hardware vendor B, with an eye to JVM scalability as measured by the bops/JVM metric. Small x86 or x64 systems with 8 or less cores are not typically deployed with more than one JVM. Customers are concerned with total system throughput but also efficient system utilization by their Java server software and the JVM. The SPECjbb2005 single instance configuration is a good match for small systems with less than 8 hardware threads. SPECjbb2005 multiple instance results should not be used to compare systems with less than 8 hardware threads simply because those systems are not typically deployed in production in that fashion. Its the responsibility of the hardware and JVM vendor along with the benchmark submitter to hold the line on SPECjbb2005 configuration types and to ensure that the configuration type matches the system under test and more importantly how they are deployed in production. JVM performance comparisions using SPECjbb2005 are a bit different. In this case JVM performance and scalability are the concentration and are best demonstrated using the single instance SPECjbb2005 configuration. When comparing JVMs, multiple instance results can only be compared to other multiple instance results, and its best if each result was run with the same number of JVM instances. Single instance SPECjbb2005 results on large SMP systems can help give insight into performance capabilities of the JVM within given instruction set and the potential scalability characteristics on other supported platforms. The latest SPECjbb2005 score can be found at http://www.spec.org/jbb2005
Comments:

Post a Comment:
Comments are closed for this entry.
About

dagastine

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today