No, there isn't a Santa Claus
By jsavit on Jun 15, 2008
But, once again into the fray. I receive Mainframe Executive magazine, and the May/June issue's closing column by Jon Toigo of Toigo Partners had some incorrect statements that I just had to correct, saying:
- that LPARs support up to 1,500 virtual environments. Actual maximum is 60.
- that z/VM and z/Linux make use of IBM's Sysplex, Workload Manager (WLM), and Intelligent Resource Director (IRD). No, those are only useable in a z/OS environment, which doesn't host virtual machines. That's z/VM's job, and not only does it not have those features (frequently claimed to have magic properties!) but it also has substantial costs that affect the cost-per-server figures that Toigo cited.
- that z/Linux can use a feature called DFHSM to reduce disk space needs. No, that also is a z/OS-only feature.
- that VMware systems (why were no others mentioned?) can only support about 20 guests on a high end server. Too low by a factor of 4 or 5 (Sun virtualization technologies like Solaris Containers and Logical Domains were not mentioned, alas) Besides, as I mentioned in my Don't Keep Your Users Hostage blog, user counts without reference to service levels is the wrong way to think about capacity.
I don't mean to pick on Mr. Toigo. I e-mailed him, and he said that he wanted to be accurate, and would contact IBM to verify facts. You can't ask for more than that from a journalist. I don't know if he'll come to see the light regarding the exaggerations I point out in the Ten Percent Solution (he is after all writing for a mainframe publication), but at least we can straighten out errors of indisputable fact - stuff you can look up in vendor manuals.
(This confusion about system features is very common, even among IBMers, because so many people think that "mainframe" implies "z/OS function set", when z/OS is only one of several operating systems that run on z. When you are not running an operating system, you don't get to use its features - for good or bad!).
All this was inspired by IBM's recent claims, which I have refuted at length on this blog. I won't repeat my points in full, because the material is here and here. but IBM makes the absurd claim that customers run database servers at 10% busy, and through the magic wand of a few buzzwords, you can run any collection of workloads at 90% CPU utilization, and somehow you can only do this using features that only exist on z. All complete rubbish, including mistakes about which IBM products have which features. It's silly that I have to correct IBM employees about IBM technology.
Here's the comment I sent to the blog of IBMer Tony Pearson:
I'm sorry you didn't take the opportunity to challenge my blog, cited as "some might question, dispute or challenge this ten percent". That would have been a good time to expose errors, if they exist, in my refutation of IBM claims.
However, I see errors of fact in your blog:
(1) You say WLM and IRD make it possible to run mainframes at 90% utilization. This is impossible: z/VM and z/Linux do not implement these z/OS-only functions. See, for example http://publib.boulder.ibm.com/infocenter/eserver/v1r1/en_US/index.htm?info/veicinfo/eicartechzseries.htm or http://www-03.ibm.com/servers/eserver/zseries/zos/wlm/
(2) David Boyes' Test Plan Charlie ran no workload other than booting OS images. It cannot be used to extrapolate capacity for doing any actual work. It also used Linux kernel customizations to reduce overhead that you could not use "in real life".
(3) You say you can define z/VM LPARs in a Sysplex. Sysplex is a System z feature only available with z/OS, so what you suggest is impossible. You cannot use Sysplex for coordinating times or recovery with z/VM or z/Linux. z/VM only supports guest Sysplex within a single z/VM instance, and only for z/OS guests.
(4) Actual cost per IFL is $125,000, not $100,000, and that doesn't count the cost for RAM.
You are right in suggesting that you would have to add up actual software and hardware costs of both platforms for a fair comparison. I've done so, and even using IBM's "10% solution on x86, 90% busy on z" argument that I dispute, and the server counts at http://www-03.ibm.com/press/us/en/pressrelease/23592.wss, IBM z cost about 7 times as much as the distributed solution IBM compares it to.
Your figures disagree with the IBM press announcement, which had claimed that 26 IFLs had sufficient capacity to do the job of 760 x86 cores (which is 380 servers, not 1,500). The page http://www-03.ibm.com/press/us/en/pressrelease/23592.wss used to have a footnote number 3 with the math, which now has been removed. In your analysis, a 64-CPU z10 E64 would be needed. That costs about $26 million dollars, excluding RAM, disks, and software licenses. That is over 14 times more expensive than 1,500 x2100s (the Sun price includes the RAM and pre-installed OS). If the CPUs are configured as IFLs, then they cost $125,000 each, totalling $8 million dollars. With the minimum RAM configuration of 160GB, it still costs 5.38 times as much as the x2100s.
I will address several other errors and points of contention in my blog. The most important mistake, though, is the implication that it is hard to consolidate or virtualize servers on x86 (or SPARC) servers at high utilization, or simply to share assets among production, test, and disaster recovery for reduced costs. Nobody need run at 10% busy, or pay a high premium to get higher utilization. If the x86 servers are managed for higher utilization, far fewer will be needed, and the price difference will be even higher.
That's the comment I placed on the blog. It will be interesting to see what response it generates, if any. It's very interesting to me that IBM removed the "justification" in footnote number 3 that used to exist on the announcement page I referred to above. Also interesting, in my previous blog I linked to another IBM blog which had the "basis" of their claims. That blog has also had the content I referred to expurgated! Curiouser and curiouser!
There are other mistakes in the blog: There is no rule of thumb saying you can reduce by 15% the capacity needed to run consolidated workloads. I'm interested in learning where that came from. I should mention again that IBM's LSPR figures don't run the RPE benchmark anyway - they run proprietary IBM benchmarks, so all of the IBM projections are specious reasoning, comparing their servers running one workload to other servers running different workloads.
I did enjoy the part where he talks about the poor scalability of System z. That part was accurate. Sun's high end SPARC servers not only are "bigger" in every capacity metric than z10, but they also are much better at vertical scale - we don't suffer from the problem he describes. That's why IBM doesn't produce LSPR for above 32-CPU systems, and they truncate results for "Single Image" capacity before that. They just don't scale as well. That's another reason not to believe these projections: they assume they understand the scalability of the application as more CPUs are added! The original IBM press release used fuzzy math based on linear scale - which System z doesn't achieve (as the IBM blog says).
I guess I should mention that Sun uses NPIV too, and that Sun also has hierarchical storage management capabilities. ZFS, a free feature of the Solaris 10 operating environment, also provides on-disk data compression to reduce disk space needs.
The 90% fallacy
In fact, the whole "90% utilization" premise is completely flawed, regardless of vendor.
Let's reason this through: If you have 90% average utilization, you probably have periods higher than 90% and lower, unless you are incredibly lucky enough to have absolutely static, predictable resource demand. That rarely happens, especially with interactive or on-line systems, which is the kind of workload under discussion.
You do not want to run on-line systems at close to 100% busy due to queueing effects that would raise response times, regardless of platform. Queuing theory applies to everybody!
The only way you can do this is if you have an individual workload with predictable characteristics on a platform with enough capacity to serve it. No workload manager in the world helps you run on systems that don't have enough capacity to meet service level objectives. Workload managers shift resources between workloads to meet service level objectives. When there is only one application, or insufficient capacity to meet service levels, there's nothing a workload manager can do.
On the other hand, when you are consolidating workloads, then you can run at extremely high utilizations levels only if the different workloads have different service levels and priorities which would permit you to starve the lower-priority workloads while giving preferential service to the high-priority workloads. If the aggregate capacity requirements of the high-priority workloads are less than the server capacity, and you're willing to starve low priority workloads (we sometimes call these "cycle soakers", since they soak up whatever capacity is left over) then you can run your systems at or near 100% busy.
There's absolutely no magic to this, and nothing that makes it possible on one platform and impossible on another. As long as you have a resource manager: Solaris Resource Manager on Solaris (There is no technical obstacle to running Solaris systems at high utilization. Sun runs compute farms for circuit design close to 100% busy for months at a time), System Resource Manager on z/VM (same initials!), Workload Manager (WLM) on z/OS, then you can run flat out (Linux doesn't seem to have anything suitable), But you also require a workload you can shed or run slower if needed, and a way to throttle it while running more important work. (The real difference here is that mainframe systems have a tradition of being run fulling loaded, due to the high acquisition costs, while distributed systems have less economic pressure, and frequently are purchased by individual lines of business who didn't want to share!)
Even this is an over-simplification, as "capacity" consists of many facets (CPU, I/O bandwidth, I/O operations per second, network bandwidth and latency) - many applications, such as the OLTP example touted by IBM, are more likely to be I/O bound than CPU bound. Throwing more CPU capacity at such systems is just a way to go idle sooner, and they naturally run I/O bound no matter what you do!.
The moral of the story: don't believe the hype. High utilization isn't the answer to all questions (the right question is "how do I minimize the cost of computing: acquisition costs, operational costs, energy, real-estate, staff and license costs - while maintaining service levels and meeting business needs".) High utilization is helpful, of course - idle machines are an expense. But you can only get "close to 100%" with fortunate combinations of workload and business priorities.