By jmeyer on Jun 04, 2007
A Look at IBM's TPC-C Results
One has to wonder what IBM was thinking when they published results on the fifteen-year-old TPC-C benchmark as part of their drive to impress the world about the performance characteristics of the single POWER6TM system they announced the other day. IBM's reasons couldn't possibly have included helping their customers make intelligent buying decisions by reflecting a modern, real-world workload with a reasonable database architecture. Criticisms of TPC-C over the years are legion, and it's probably not useful to repeat them here. Suffice to say that Gartner, IDC, and Oracle have weighed in, and this is one of the reasons that the Transaction Processing Performance Council (TPC) announced a new benchmark in March, TPC-E, which goes much further to reflect 21st-century OLTP workloads.
According to TPC-C's description: "TPC-C simulates a complete environment where a population of terminal operators executes transactions against a database. The benchmark is centered around the principal activities (transactions) of an order-entry environment. These transactions include entering and delivering orders, recording payments, checking the status of orders, and monitoring the level of stock at the warehouses." It should be noted that there is no set piece of code to execute: the submitter is allowed to architect the system and write the SQL code however he wishes. And this is where the fun begins.
Let's take a look at IBM's configuration, which you can see for yourself in the full disclosure report (FDR) that they submitted to the TPC.
Pretty impressive database engine, isn't it? But did I mention that according to the FDR (page 27), the total database table size was about 13TB? Let me say that again, so you'll know it's not a misprint: IBM used almost 120 terabytes worth of disks (3,482 of them, to be exact) to store 13 terabytes of table data. When was the last time you asked your boss to give you money for 42 4Gb fibre channel connections to access 13TB of data? Or funds for 3,482 spindles so you could use only 10% of the space on them? Where do you think you'll be able to dig up 3,312 36GB disk drives? Are you even running DB2 in your datacenter?
I should mention that nothing IBM did here was in violation of the rules of the TPC, but it strains the imagination to see how their result could possibly be useful to anyone making OLTP database performance comparisons. This is why Sun hasn't published a TPC-C result since 2001, preferring instead iGEN OLTP or running real applications such as Oracle Apps or SAP.
If you still believe that IBM's TPC-C results actually tell you something useful, then take a look at the following chart, compiled by my colleague Pavel Anni using IBM's previous FDR:
|IBM's TPC-C RESULTS||System p 570
|System p 570
|Number of CPUs:||8||8||0%|
|Number of Cores:||16||16||0%|
|Number of Threads:||32||32||0%|
|Amount of RAM, GB:||512||768||50%|
|List price, server:||$3,407,192||$2,141,159||-37%|
|List price, complete:||$7,621,279||$9,419,485||24%|
|Price, including 3-Year TCO:||$4,544,268||$5,713,181||26%|
|Database Software:||DB2 UDB 8.2||DB2 Enterprise 9||N/A|
|Availability Date:||May 31, 2006||November 21, 2007||N/A|
At least IBM has given you something to compare the performance of the old and new processors, right? Think again. If that were IBM's benign intent, why did they add 50% more memory? Why did they use a different version of DB2? Customers looking to upgrade their POWER5+ systems to POWER6 have no idea how much of the 58% performance improvement was due to better performance of the new processor and how much was due to the larger memory configuration and newer software. And considering a 114% increase in clock rate (and a quadrupling of their level-2 cache sizes), why did they achieve only a 58% performance boost? And where are IBM's power consumption and cooling numbers? TPC-C does not require reporting such data, but I'd be willing to bet that POWER6's much-ballyhooed power reduction features were inactive for this run. Especially since their press release said that customers could choose higher performance or lower power consumption, but not both.
So IBM (1) wasn't interested in running a real-world configuration on a modern-day workload, even though TPC-E is available; and (2) wasn't interested in giving customers a valid apples-to-apples comparison with the old benchmark. So why did they do this?
Giving customers useful information is not what this is about. If you don't believe me, here's what IBM Fellow Bruce Lindsay said (page 73): "Well, the benchmarking business is dirty work. The idea is to get the numbers by hook and by crook. ... [M]uch of what we're doing in the TPC-C realm these days is in a performance range that goes beyond what any user is doing." There's a word for this in the industry: benchmarketing. And IBM is the best in the business.
They understand that customers are hungry for a number, any number, however irrelevant, because it's much harder for a customer to take the time to benchmark his own code on a try-and-buy system or in a benchmark center. I can understand this; I used to be in the exact same situation in a previous job. I know how hard it is to be under the gun to finish a proposal or make a project deadline with no time to do the kind of rigorous performance bake-off that is truly necessary to make an intelligent purchasing decision. But IBM is well aware of your predicament too, has been taking advantage of it for years, and will continue to do so until their customers speak up.
TPC, TPC Benchmark C, TPC-C, tpmC are trademarks of the TPC. Please see www.tpc.org for more details.