The Ten Percent Solution
By Jsavit-Oracle on Mar 03, 2008
How do you make a slow, expensive platform look like it's competitive with any other platform? This can be difficult, especially if you don't have (or don't dare have) public open benchmarks such as those at SPEC (Standard Performance Evaluation Corporation) or TPC (Transaction Processing Performance Council). This is the problem facing IBM with the announcement of their new System z10 mainframe. How do you make costly, proprietary hardware look like it can compete on a price/performance basis?
One way is to play tricky games with numbers. Unfortunately, that's what it seems IBM has done with in its z10 announcement. If you look at http://www-03.ibm.com/press/us/en/pressrelease/23592.wss you will see the following text
The z10 also is the equivalent of nearly 1,500 x86 servers, with up to an 85% smaller footprint, and up to 85% lower energy costs. The new z10 can consolidate x86 software licenses at up to a 30-to-1 ratio. \*3and a pointer to footnote 3:
3 Source: On Line Transaction Processing Relative Processing Estimates (OLTP-RPEs): Derivation of 760 Sun X2100 2.8 Opteron processor cores with average OLTP-RPEs per Ideas International of 3,845 RPEs and available utilization of 10% and 20 RPEs equating to 1 MIPS compared to 26 z10 EC IFLs and an average utilization of 90%.s
This even comes up on an IBM blog at http://www-128.ibm.com/developerworks/blogs/page/benchmarking?entry=back_to_the_future_with#comments, where a reader asks how the calculations work, and is told:
The 30 to 1 claim is based on 760 X2100 cores to 26 z10. The 760 to 26 is based on 3845 RPEs at 10% = 384.5 RPEs is approximately equivalent to the number of z10 RPEs at 90% when you use 20 RPEs equal to 1 MIP where MIPS are based on the LSPR curve for the z10.
Games People Play
When I saw this, I thought "what kind of nonsense is this?" The whole idea of benchmarks is to determine the capacity of a system under load - not to say "assume 100% of the capital and operational costs of the server, and then only use 1/10th of it". That's crazy. Or crazy like a fox. (Not to mention, there's no such thing as a "MIP". All true mainframe performance guys know that).
Let's illustrate how this works: they take a consultancy's benchmark (instead of an open one like those at SPEC and TPC), and then specify that the competitor's machines (that's ours, by the way) should be evaluated at 10% CPU busy, while they specify that their own machines are 90% busy. With this "just because we say so" trick they use 9 times as much of their machine as they do of ours. Play games like this, and they can make yourself look a lot better than they are. On top of that, they base their figures on extrapolating an IBM-only benchmark (the "LSPR" referred to) to estimate the difference between the z9 and z10. That's a really shaky limb to go out on. There's not a lot of science here.
Work the numbers
Let's do some math and convert that into prices. Assume for the moment that the price of a z10 IFL is the same as the $125,000 per IFL on the z9. It's not easy to tell, since IBM isn't exactly transparent about pricing and I couldn't find it on the site. So, 26 times $125,000 is $3,250,000. They say 760 x2100 CPU cores, which I can price on sun.com at $1,189 for 2 cores and 1GB of RAM. I can add 500GB of disk for $359, but let's hold that for a moment. Now, using IBM's "ten percent solution", 760 cores requires 380 servers at $1,189, for a cost of $451,820. Hey, that's not bad. Even with the bogus 10% trick, Sun still has a 7 to 1 price/performance advantage over the IBM z10 mainframe. Fantastic, huh? Even with that little deceptive trick, the z10 is trailing in the dust. You did notice that IBM didn't actually put prices on their web pages, even with the 10% trick, right?
Now let's make it more realistic :-)
Database servers are just as likely to be 90% CPU busy on one platform as on another - that's just a question of not overprovisioning. So let's work the numbers fairly so the x2100 systems are no more overprovisioned than the z10. Instead of 760 cores at 10% busy, that would be 84.4 cores at 90%, which I'll round up to 86. Two cores per server: 43 servers. A rack. So, a set of Sun x2100s provisioned for actual capacity is 43 servers times $1,189 for a total of $51,127. That leaves Sun with a 63 to 1 price performance advantage over the z10. That's actually pretty consistent with mainframe-to-SPARC and mainframe-to-x64 conversions I've been involved in. Do you wonder why I got out of the mainframe side of computing after becoming an expert there? This is part of the reason.
And it gets worse (for IBM, that is). I didn't include the disk drives, which will be much more expensive on the mainframe side. Nor did I include the operating systems: For the mainframe you will need 26 CPU licenses of z/VM, which will be about $20,000 per CPU, for a total of $520,000. There's a formula for this, but that's close enough for a quick estimate, and excludes subsequent maintenance. We provide Solaris "right to use" for free, and I assure you our software maintenance is a heck of a lot less. Plus, on the mainframe, you'll license Linux at a cost of $15,000 or more per CPU for a one-year license (see Novell and Red Hat pricing) for another $390,000 per year.
Your first year cost for the IBM solution is $3,250,000 for the server (which by the way, just gave you a single point of failure!), $520,000 for z/VM, and an annual $390,000 for Linux. Ours is a few orders of magnitude less expensive, starting with $51K for the servers, and continues that way.
It's late and I may have made a few mistakes. This is all "back of the envelope" calculation. But, when there's orders-of-magnitude differences in platform price, a few percent error one way or the other really doesn't matter. If I made a mistake - it's happened! - I apologize for it in advance, and will correct as needed.
Integrity in what you say
Let me make this clear: I made my living for years as a mainframe system performance expert. I ran mission critical, performance-sensitive systems, consulted with major companies, wrote books, and taught classes on mainframe performance. I stay current. I've run mainframe Linux, and taught a performance class on that, too. I know what is a fair comparison and what is "benchmarketing" and trying to game the system. If the numbers worked out differently, I would report them and let the chips (no pun intended) fall where they may.
Which reminds me. I Am Not A Lawyer, but I have to imagine that IDEAS (the consultancy whose benchmark IBM used) specifies contracted terms and conditions that state how their benchmarks can be reported and used. I expect that there's a contract stipulating that RPE is a proprietary metric and restricted in use. IANAL, but I would be unsurprised if IBM used it without IDEAS' permission, and may be in violation of IDEAS copyright or usage terms.
The moral of the story
There are several:
- Use open, standard benchmarks, such as those from SPEC and TPC.
- Read and understand what they measure, instead of just accepting them uncritically.
- Get the price-tag associated with the system used to run the benchmark.
- Relate benchmarks to reality. Nobody buys computers to run Dhrystone.
- Don't permit games like "assume the other guy's system is barely loaded while ours is maxed out". That distorts price/performance dishonestly.
- Don't compare the brand-new machine to the competitor's 2 year old machine
- Insist that your vendors provide open benchmarks and not just make stuff up.
- Be suspicious!