Speeding up tests: a langtools milestone
By jjg on Oct 07, 2008
This has been a background activity ever since I joined the JDK Language Tools group. It has required work on a number of fronts.
- We've had to fix a number of bugs in jtreg itself. There was a chicken and egg problem: there was no demand for the feature since it didn't work work enough, and with no demand, there were no engineering cycles to fix it. But, I wanted to run the javac tests fast, and I took over jtreg, so the chicken and egg became one.
- We converted a lot of javac tests so that they could be run in the same VM. These tests mostly started out as "shell tests" (i.e. written in ksh.) They executed javac and then compared the output against a golden file. By fixing jtreg, and by adding hidden switches to javac, we converted the tests to use jtreg built-in support for golden files.
This was enough to get the javac tests to run in samevm mode, which has been the state for a while now, but there was still the issue of the other related tools, such as apt, javadoc, javap, javah. And, the fact that not all the tests could be run in samevm mode meant that anyone wanting to run all the tests had to use lowest common denominator, slow mode.
- javap was the next to get fixed: it needed a rewrite anyway, and when that happened, it was natural to fix those tests to run fast as well.
- javah was similar: there's a partial rewrite underway (to decouple javah from javadoc), and so those tests have been fixed.
- javadoc has always been the tough one, and has the second largest group of tests after javac. A while back, I took the time out to figure out why the tests failed in samevm mode. It turns out to be a number of factors, mostly relating to the fact that javadoc is a fairly old program, and is somewhat showing its age. Internally it was using static instances a lot, and as a result, was neither reentrant nor reusable. In addition, there were classloader issues when creating the classloader for the doclet: javadoc was not following the recommended "parent classloader" pattern. Having identified those issues, we've been working to fix them, and it is the result of fixing the last of those issues that gives the milestone today.
- In parallel with the work on javadoc, and with the milestone in sight, we checked out the final tool, apt. There were 24 test failures there when using samevm mode, and since the tool is scheduled to be decommissioned in Java 8, we "fixed" those tests simply by marking them as requiring othervm mode. That doesn't make them run any faster, but it does mean they don't fail if you run the tests with samevm mode as the default mode.
What does all this mean? It means the tests run is less than a quarter of the time than before. Using jtreg samevm mode, and using my laptop, 1421 tests run and pass in a little over 5 minutes, compared to 22 minutes in the standard othervm mode. As a developer waiting to move on the next step, that's a big saving :-) Although all the supporting work has not yet made it back to the master workspace, that's all underway, and at some point we'll throw the switch so that samevm mode is the default for the langtools repository. So far, we've been targeting the JDK 7 repository, but Joe Darcy is keen to see as much of this work as possible in the OpenJDK 6 repository as well.
Can we do the same for the main jdk repository? In principle yes, but it will take someone to do it. The good news is that jtreg now supports the samevm feature well, and the payoff is clear. It does not have to be done all at once: as we did in the langtools workspace, all it takes is someone interested to work on it section by section. The payoff is worthwhile.