wwwww.jtreg: The Who What Where When Why of jtreg

Executive summary for those in a hurry

Who? Originally, the JavaSoft SQE team.
What? The JDK regression test harness
Where? Cupertino, CA, USA
When? November 7, 1997
Why? Those who ignore history (or bugs) are doomed to repeat it (or them).


And for those with time for a cup of coffee...

Why jtreg, why Yet Another Harness? Well, actually, at the time, jtreg was quite the opposite. According to the SCCS history, we started jtreg back in the fall of 1997, which means that JDK 1.1 had shipped and the JDK team were starting on 1.2. There were limited choices for a test platform that would run on all the platforms that JDK ran on, and the obvious in-house choice was JavaTest, which in time got trademarked and became the JavaTest™ harness. But, JavaTest was designed for running the tests in the Java Compatibility Kit (JCK) which had a different set of design goals. JCK tests were API tests, or low-level byte-code tests, or compiler tests. More specifically, because they were designed to be able to run the tests on everybody else's version of the Java platform, as well as our own, JCK tests could not rely on specific command line options, output file formats, etc, except where defined by the specifications, as defined by JLS, JVMS, and the API javadoc. And because the tests didn't test that sort of stuff, the harness was not designed to allow that. But that was what we wanted to test, when writing regression tests.

So rather than write Yet Another Harness, instead, we adapted JavaTest to use a plug-in architecture, so that instead of running just JCK tests, it could run JCK tests, or regression tests, or whatever.

Why are regression tests different, you might ask? Good question.

JCK tests from the outset were designed to be able to run in the full spectrum of Java platforms. At the time, that meant across a spectrum of software vendors, and across different types of platforms, from command-line environments to browsers (remember applets?), and on systems that could support multiple instances of a virtual machine, to those that could only support one at a time, to those where the OS was the virtual machine and vice versa. Because of this wide spectrum, tests could not be procedural, in the sense of "execute this command and check the output"; instead, tests were given keywords to describe attributes of the test (it's a runtime test, or, it's a negative compiler test) etc, and it was up to the harness to figure out how to run each test on any given platform. Today, we still use many of those same tests, to test everything from Java-enabled phones to Web Application Servers.

But, regression tests were different. People filed bugs saying "I did this, deleted that, edited one file and recompiled another, and Something Bad happened". Writing a regression test to make sure that never happened again could not be done by tagging a test with a few keywords and letting the test harness figure it out. And while we didn't have to worry about running tests on every platform for which there was a Java virtual machine, we did have to be able to run tests on all platforms for which Sun had a virtual machine.

Thus was born the need for a test harness with a different focus -- less platforms, more procedural. This time, we did want to write tests that said "execute this command, with these options, and check the results". Thus was born the JDK Test Framework Tag Language Specification, and the JDK regression test suite. These days, it's used for unit tests as well as regression tests, and as of October 2006, there are over 7400 tests in one workspace alone.


Post a Comment:
Comments are closed for this entry.

Discussions about technical life in my part of the OpenJDK world.


« February 2015