Monday May 16, 2011

Speeding up tests again: another milestone

A while back, I reported a milestone about being able to run all the langtools tests in jtreg's samevm mode. And more recently I reported on a new feature in jtreg called "agentvm" mode. Now, we can report more progress on being able to make the tests run even faster, or put another way, being able to run even more tests in the same amount of time.

With a number of folk beginning to look at improving (read: speeding up) the OpenJDK build infrastructure, it seems appropriate to take a corresponding look at working on the test infrastructure as well. With that in mind, we investigated the possibility of running jtreg tests in parallel.

jtreg was originally built on top of the JavaTest test harness, now available via the open source "jtharness" project. From the very beginning, JavaTest was designed to support concurrent test execution, and so all the necessary hooks for concurrent test execution are available in the underlying harness. But, while it would have been possible to leverage those hooks for any tests running in othervm mode, executing tests concurrently in samevm mode has never been a realistic option, because of the risk of interference between tests in the same virtual machine. Tests assume write access to global state, such as system properties, the security manager, and the standard input and output streams. (In short, lots of stuff in java.lang.System.) And, in the early days, it seemed better to focus on getting tests to run in samevm mode than it was to focus on running tests in parallel in othervm mode. That was then, but this is now.

Nowadays, multicore machines are more common, and given that we've taken samevm tests about as far as we can go, it's time to look at running tests in parallel again. I typically describe the new "agentvm" mode as "like samevm mode, but better." Up to now, that has primarily meant that the agentvm mode is more robust in the face of really bad tests (that don't clean up after themselves), and it allows the possibility of different instances of JDK for compiling and executing tests (e.g. using different JVM options.) But, Maurizio has been investigating using multiple agents at once as a way of running tests in parallel, and he has come up with a small but elegant patch for jtreg to make it all work, just like that.

The idea is simple. The underlying harness supports parallel test execution in the obvious way: threads are created to run tests, these threads take tests from a queue of tests to be executed, and do whatever necessary to execute the test, until all the tests are done. Separately, "agentvm" mode works by using a pool of virtual machines, with specific individual characteristics, such as the version of JDK being used, and the options used to create the VM. Virtual machines are taken from the pool as needed for each test, creating them as needed if there is not a suitable machine already available. When the test completes, the virtual machines are returned to the pool for reuse in another test, provided they can be reset to a known reasonably clean state. So if you run tests in parallel with agentvm mode, it just means that more requests will be made on the pool of virtual machines, and that more virtual machines will be instantiated as needed to meet the demand.

At the user level, the work is as simple as providing a new option, -concurrency:N. Since parallel test execution is incompatible with samevm mode, that combination is forbidden, but that still leaves othervm mode and agentvm mode. By itself, parallel test execution in othervm mode is not so interesting, but it is important that parallel test execution should not prevent individual tests from specifying othervm mode as needed, even when the rest of the tests are running in agentvm mode.

At the implementation level, it is almost as easy as passing the required concurrency down to the underlying harness. However, there are a few gotchas to take care of as well. Each jtreg test is run in an empty scratch directory, and by default, all tests use the same scratch directory, which is emptied before each test begins. Likewise, the tests share a classes hierarchy, which can be problematic if multiple copies of javac are using it at the same time. The solution to both of these is to create subdirectories of the scratch and classes directory, one for each thread running tests. Finally, the agent code in jtreg maintains a pool of JVMs, and since each JVM is configured to run in a specific scratch directory, it is necessary to update the agent pool to take the requisite scratch directory into account when allocating a JVM for a test.

And that is pretty much all there is to it!

Does it work? Yes. One langtools test had to be fixed up because it was incorrectly relying on another test having already been run, because the former did not correctly specify a full set of dependencies to be compiled.

Did it work well? Yes. So far it seems to scale well with the number of processors available. However, it is worth noting that when using Hotspot, Hotspot itself uses additional threads to do its work, so it may not be effective to run as many tests in parallel as you have processors available.

Machine samevm agentvm
Concurrency Change
Intel Core 2 Duo
real 11m20.444s user 9m21.795s sys 2m08.220s real 4m55.814s user 6m38.905s sys 1m48.327s 4 x2.3
2x Quad core
lab machine
real 12m23.199s user 10m15.857s sys 2m51.223s real 2m35.136s user 9m50.262s sys 4m36.250s 8 x4.8

The langtools test suite currently has just over 1900 tests, and by instrumenting javac, we know that running the entire langtools test suite performs about 30895 compilations (actually, it creates that many instances of the internal JavaCompiler class.) So from the table we can infer that on a suitable machine, we can run 1905 tests or 30895 compilations in 2m 35s, which works out to about 12 tests per second or 200 compilations per second! And even just being able to run all the tests in under 5 minutes on what these days is an average engineering laptop is not bad going either. Woohoo! That's definitely worth declaring as a milestone!


This feature will be available soon, in an update to jtreg 4.1. Watch here or on the jtreg project page for more details.

Thanks to Maurizio Cimadamore for working on this feature and for his feedback on this note.

Discussions about technical life in my part of the OpenJDK world.


« August 2016