SPEC releases CPU2006 benchmark suite
By walterbays on Aug 24, 2006
Bid farewell to CPU2000, arguably the world's most widely used benchmark suite with more than 6,000 results published to date. Today SPEC announced its replacement, CPU2006. The old suite, formidable in its day, was being overtaken by modern systems. Originally some of the benchmark run times had been well in excess of an hour, so that running three repetitions of each of the 14 floating point benchmarks and each of the 10 integer benchmarks took a considerable time. But some of today's systems run some of the benchmarks in less than 15 seconds.
The new CPU2006 suite has 14 integer benchmarks and 17 floating point benchmarks, covering a wider range of application areas. Some of the best old benchmarks like gcc remain, but with considerably bigger more demanding workloads. And there are many new benchmarks, some of which came from a wide ranging search program that paid "bounties" to authors to contribute code to the suite and work with SPEC to adapt the applications into suitable benchmarks.
SPEC member companies, universities, and individuals spent many hours (and sleepless nights) working on these benchmarks, starting soon after CPU2000 was released. They tested and performed detailed performance analysis to ensure that the benchmarks test a wide range of application areas and coding styles, that they run on any standard conforming system, and that they are not biased towards any particular operating system, compiler, processor pipeline, or cache organization. For every benchmark included in the suite, one or two benchmarks was worked on for months only to drop out along the way. Equally important as the benchmarks are the run rules and lots of work also went into these to ensure that tests are run under appropriate conditions for fair and accurate comparisons. All this work required that the vendor representatives temporarily set aside their competitive instincts to share comparative performance analyses, to help their competitors port code, and to compromise on technical and run rule issues.
CPU2006 is off to a fast start with already 66 results published from a wide range of vendors. This ensures that competition will heat up quickly and we should see many new results in coming months.
Congratulations to all the members of SPEC's CPU committee!