IBM "per core" comparisons for SPECjEnterprise2010

I recently stumbled upon a blog entry from Roman Kharkovski (an IBM employee) comparing some SPECjEnterprise2010 results for IBM vs. Oracle.

Mr. Kharkovski's blog claims that SPARC delivers half the transactions per core vs. POWER7.

Prior to any argument, I should say that my predisposition is to like Mr. Kharkovski, because he says that his blog is intended to be factual; that the intent is to try to avoid marketing hype and FUD tactic; and mostly because he features a picture of himself wearing a bike helmet (me too).

Therefore, in a spirit of technical argument, rather than FUD fight, there are a few areas in his comparison that should be discussed.

  1. Scaling is not free

    For any benchmark, if a small system scores 13k using quantity R1 of some resource, and a big system scores 57k using quantity R2 of that resource, then, sure, it's tempting to divide: is
     13k/R1 > 57k/R2 ?

    It is tempting, but not necessarily educational.

    The problem is that scaling is not free. Building big systems is harder than building small systems. Scoring  13k/R1  on a little system provides no guarantee whatsoever that one can sustain that ratio when attempting to handle more than 4 times as many users.

  2. Choosing the denominator radically changes the picture

    When ratios are used, one can vastly manipulate appearances by the choice of denominator. In this case, lots of choices are available for the resource to be compared (R1 and R2 above).

    IBM chooses to put cores in the denominator. Mr. Kharkovski provides some reasons for that choice in his blog entry. And yet, it should be noted that the very concept of a core is:

    1. arbitrary: not necessarily comparable across vendors;
    2. fluid: modern chips shift chip resources in response to load;
    3. and invisible: unless you have a microscope, you can't see it.

    By contrast, one can actually see processor chips with the naked eye, and they are a bit easier to count. If we put chips in the denominator instead of cores, we get:

       13161.07 EjOPS / 4 chips  = 3290 EjOPS per chip for IBM
        vs
       57422.17 EjOPS / 16 chips = 3588 EjOPS per chip for Oracle
    

    The choice of denominator makes all the difference in the appearance.

    Speaking for myself, dividing by chips just seems to make more sense, because:

    • I can see chips and count them; and
    • I can accurately compare the number of chips in my system to the count in some other vendor's system; and
    • Tthe probability of being able to continue to accurately count chips over the next 10 years of microprocessor development seems higher than the probability of being able to accurately and comparably count "cores".
  3. SPEC Fair use requirements

    Speaking as an individual, not speaking for SPEC and not speaking for my employer, I wonder whether Mr. Kharkovski's blog article, taken as a whole, meets the requirements of the SPEC Fair Use rule www.spec.org/fairuse.html section I.D.2.

    For example, Mr. Kharkovski's footnote (1) begins

    Results from http://www.spec.org as of 04/04/2013 Oracle SUN SPARC T5-8 449 EjOPS/core SPECjEnterprise2010 (Oracle's WLS best SPECjEnterprise2010 EjOPS/core result on SPARC). IBM Power730 823 EjOPS/core (World Record SPECjEnterprise2010 EJOPS/core result)

    The questionable tactic, from a Fair Use point of view, is that there is no such metric at the designated location. At www.spec.org,

    SPEC says that you can, under its fair use rule, derive your own values; but it emphasizes: "The context must not give the appearance that SPEC has created or endorsed the derived value."

  4. Substantiation and transparency

    Although SPEC disclaims responsibility for non-SPEC information (section I.E), it says that non-SPEC data and methods should be accurate, should be explained, should be substantiated.

    Unfortunately, it is difficult or impossible for the reader to independently verify the pricing:

    • Were like units compared to like (e.g. list price to list price)?
    • Were all components (hw, sw, support) included?
    • Were all fees included? Note that when tpc.org shows IBM pricing, there are often items such as "PROCESSOR ACTIVATION" and "MEMORY ACTIVATION".

    Without the transparency of a detailed breakdown, the pricing claims are questionable.

  5. T5 claim for "Fastest Processor"

    Mr. Kharkovski several times questions Oracle's claim for fastest processor, writing

    You see, when you publish industry benchmarks, people may actually compare your results to other vendor's results.

    Well, as we performance people always say, "it depends". If you believe in performance-per-core as the primary way of looking at the world, then yes, the POWER7+ is impressive, spending its chip resources to support up to 32 threads (8 cores x 4 threads).

    Or, it just might be useful to consider performance-per-chip. Each SPARC T5 chip allows 128 hardware threads to be simultaneously executing (16 cores x 8 threads).

    The Industry Standard Benchmark that focuses specifically on processor chip performance is SPEC CPU2006. For this very well known and popular benchmark, SPARC T5:

    • provides better performance than both POWER7 and POWER7+,
    • for 1 chip vs. 1 chip,
    • for 8 chip vs. 8 chip,
    • for integer (SPECint_rate2006) and floating point (SPECfp_rate2006),
    • for Peak tuning and for Base tuning.

    For example, at the 8-chip level, integer throughput (SPECint_rate2006) is:

    • 3750 for SPARC
    • 2170 for POWER7+.

    You can find the details at the March 2013 BestPerf CPU2006 page


    SPEC is a trademark of the Standard Performance Evaluation Corporation, www.spec.org. The two specific results quoted for SPECjEnterprise2010 are posted at the URLs linked from the discussion. Results for SPEC CPU2006 were verified at spec.org 1 July 2013, and can be rechecked here.
Comments:

Counting chips will make sense when vendors start licensing software using that same metric

Posted by Nacho on July 01, 2013 at 05:25 AM EDT #

Hi Nacho,
Yes, there can be different reasons for choice of denominator. I do not claim that their choice is without merit; I simply think that my choice has MORE merit :-)
-john

Posted by jhenning on July 01, 2013 at 12:00 PM EDT #

On the matter of scaling not being free, certainly that is the case, though based on http://www.spec.org/cgi-bin/osgresults?conf=default&op=fetch&pattern=T5&field=CPU it would seem that T5-based systems scale rather well. On SPECcpu2006 anyway. (Not sure why the SPECjEnterprise2010 result didn't appear in that search result...)

I have to agree with Nacho - if (most?) software is licensed per-core then a performance/core comparison makes a great deal of sense.

Posted by rick jones on July 03, 2013 at 09:30 PM EDT #

IBM discussion has some merits if they talk about the fastest core. But Oracle is not talking about fastest cores, or fastest ALU, or whatever. Oracle is claiming they have the highest performing cpu in the world, and that is correct. This is something that IBM can not question, therefore IBM shift the focus to "fastest core in the world". And when Oracle also has the fastest core, IBM will shift focus to having the "fastest ALU in the world", or "highest clocked in the world", or whatever.

Let IBM talk about cores or GHz, or whatever. Just say: "whatever, Oracle has the fastest cpu in the world, so a 8-socket Oracle server is much faster than a 8-socket IBM server. That is the metric to beat". Dont fall into IBM's trap about cores. Just say "our Oracle servers are fastest in the world".

Are you discussing fastest cpus, or fastest cores? "We are discussing fastest cpus, not something else".

Posted by Kebabbert on July 08, 2013 at 04:36 AM EDT #

Hi Rick, thank you for your comment. As with Nacho's comment, I would not presume to claim that the argument has zero merit. Nor would I claim that pricing per core does not exist. I just think that cores are arbitrary, fluid, and invisible; and that the implied metric clearly does not exist at the SPEC website. -john

Posted by guest on July 08, 2013 at 12:09 PM EDT #

Hi Kebabbert, Well, it's not that I'm willing to "fall into IBM's trap". It's simply that I had hopes - based on Mr. Kharkovski's claimed intent "to avoid marketing hype and FUD tactic" - that we might engage in a bit of technical argument (non-FUD). Perhaps Roman Kharkovski is on vacation, or perhaps he is ignoring me; my comment on his page has been awaiting approval for a week; see below -john

>Your comment is waiting for approval.
>John L Henning
>July 1, 2013 at 1:33 am
>I think there are few things to talk about: scaling, choice of
>of denominators, SPEC Fair Use, and substantiation
>for the “fastest processor”.
> https://blogs.oracle.com/jhenning/entry/ibm_transactions_per_core
>
>Roman, thank you for leaving comments open on this page. I have
>done the same on mine.

Posted by jhenning on July 08, 2013 at 12:22 PM EDT #

Post a Comment:
  • HTML Syntax: NOT allowed
Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today