Monday May 16, 2011

Speeding up tests again: another milestone

A while back, I reported a milestone about being able to run all the langtools tests in jtreg's samevm mode. And more recently I reported on a new feature in jtreg called "agentvm" mode. Now, we can report more progress on being able to make the tests run even faster, or put another way, being able to run even more tests in the same amount of time.

With a number of folk beginning to look at improving (read: speeding up) the OpenJDK build infrastructure, it seems appropriate to take a corresponding look at working on the test infrastructure as well. With that in mind, we investigated the possibility of running jtreg tests in parallel.

jtreg was originally built on top of the JavaTest test harness, now available via the open source "jtharness" project. From the very beginning, JavaTest was designed to support concurrent test execution, and so all the necessary hooks for concurrent test execution are available in the underlying harness. But, while it would have been possible to leverage those hooks for any tests running in othervm mode, executing tests concurrently in samevm mode has never been a realistic option, because of the risk of interference between tests in the same virtual machine. Tests assume write access to global state, such as system properties, the security manager, and the standard input and output streams. (In short, lots of stuff in java.lang.System.) And, in the early days, it seemed better to focus on getting tests to run in samevm mode than it was to focus on running tests in parallel in othervm mode. That was then, but this is now.

Nowadays, multicore machines are more common, and given that we've taken samevm tests about as far as we can go, it's time to look at running tests in parallel again. I typically describe the new "agentvm" mode as "like samevm mode, but better." Up to now, that has primarily meant that the agentvm mode is more robust in the face of really bad tests (that don't clean up after themselves), and it allows the possibility of different instances of JDK for compiling and executing tests (e.g. using different JVM options.) But, Maurizio has been investigating using multiple agents at once as a way of running tests in parallel, and he has come up with a small but elegant patch for jtreg to make it all work, just like that.

The idea is simple. The underlying harness supports parallel test execution in the obvious way: threads are created to run tests, these threads take tests from a queue of tests to be executed, and do whatever necessary to execute the test, until all the tests are done. Separately, "agentvm" mode works by using a pool of virtual machines, with specific individual characteristics, such as the version of JDK being used, and the options used to create the VM. Virtual machines are taken from the pool as needed for each test, creating them as needed if there is not a suitable machine already available. When the test completes, the virtual machines are returned to the pool for reuse in another test, provided they can be reset to a known reasonably clean state. So if you run tests in parallel with agentvm mode, it just means that more requests will be made on the pool of virtual machines, and that more virtual machines will be instantiated as needed to meet the demand.

At the user level, the work is as simple as providing a new option, -concurrency:N. Since parallel test execution is incompatible with samevm mode, that combination is forbidden, but that still leaves othervm mode and agentvm mode. By itself, parallel test execution in othervm mode is not so interesting, but it is important that parallel test execution should not prevent individual tests from specifying othervm mode as needed, even when the rest of the tests are running in agentvm mode.

At the implementation level, it is almost as easy as passing the required concurrency down to the underlying harness. However, there are a few gotchas to take care of as well. Each jtreg test is run in an empty scratch directory, and by default, all tests use the same scratch directory, which is emptied before each test begins. Likewise, the tests share a classes hierarchy, which can be problematic if multiple copies of javac are using it at the same time. The solution to both of these is to create subdirectories of the scratch and classes directory, one for each thread running tests. Finally, the agent code in jtreg maintains a pool of JVMs, and since each JVM is configured to run in a specific scratch directory, it is necessary to update the agent pool to take the requisite scratch directory into account when allocating a JVM for a test.

And that is pretty much all there is to it!

Does it work? Yes. One langtools test had to be fixed up because it was incorrectly relying on another test having already been run, because the former did not correctly specify a full set of dependencies to be compiled.

Did it work well? Yes. So far it seems to scale well with the number of processors available. However, it is worth noting that when using Hotspot, Hotspot itself uses additional threads to do its work, so it may not be effective to run as many tests in parallel as you have processors available.

Machine samevm agentvm
Concurrency Change
Intel Core 2 Duo
real 11m20.444s user 9m21.795s sys 2m08.220s real 4m55.814s user 6m38.905s sys 1m48.327s 4 x2.3
2x Quad core
lab machine
real 12m23.199s user 10m15.857s sys 2m51.223s real 2m35.136s user 9m50.262s sys 4m36.250s 8 x4.8

The langtools test suite currently has just over 1900 tests, and by instrumenting javac, we know that running the entire langtools test suite performs about 30895 compilations (actually, it creates that many instances of the internal JavaCompiler class.) So from the table we can infer that on a suitable machine, we can run 1905 tests or 30895 compilations in 2m 35s, which works out to about 12 tests per second or 200 compilations per second! And even just being able to run all the tests in under 5 minutes on what these days is an average engineering laptop is not bad going either. Woohoo! That's definitely worth declaring as a milestone!


This feature will be available soon, in an update to jtreg 4.1. Watch here or on the jtreg project page for more details.

Thanks to Maurizio Cimadamore for working on this feature and for his feedback on this note.

Monday May 05, 2008

jtharness vs jtreg

Now that the source code for the OpenJDK Regression Test Harness (jtreg) is available, this provides an overview of how jtreg relates to JT Harness.

In the beginning, there was a need to run compatibility tests on all Java plaforms, and thus was born the "Java Compatibility Kit" (JCK), and the test harness to run those tests (JavaTest). Not long after, there was a need to run a different type of test, regression tests, for JDK. They are fundamentally a different style of test (think white box and black box), but rather than write another, different harness, we adapted JavaTest, and thus was born "jtreg". Eventually, we needed to be able to run tests on different types of platforms, such as Personal Java, a precursor to the J2ME platform. JavaTest evolved to fill the need, and gave rise to the basic architecture you see today.

As you can see, JavaTest, now open sourced under the name JT Harness, is composed of a number of distinct components

  • The core harness. This provides the general ability to represent tests, test results, and to sequence through the execution of a series of tests.
  • Utilities. These provide support to the other JT Harness components.
  • The user interface. JT Harness provides a rich graphical user interface, as well as a command line interface.

And, deliberately, there's a hole. By itself, JT Harness cannot directly run tests in any test suite. It needs to be given a plugin for each test suite, that define how to locate and read tests, how to configure and execute tests, and other additional information, often used by the GUI, to provide documentation specific to the test suite. The plugin code is not complicated and is often a direct use or simple extension of the utility classes provided by JT Harness. Note: most test suites do not need to use all of the JT Harness utility classes, which explains why JT Harness may have more build-time dependencies than are required to build and run any individual test suite.

Thus, the complete picture is more as follows:

This is the form used for most test suites, such as JCK and other TCKs. The code to plugin to the harness is bundled in the test suite, along with a copy of JavaTest or JT Harness itself. These test suites all leverage the user interfaces provided by the harness, both the CLI and GUI. When the user runs JT Harness, and tells it to open a test suite, the harness looks inside the test suite for details of the plugin to be used in conjunction with that test suite.

Because jtreg evolved from JavaTest early on, and specifically, before the JavaTest user interfaces evolved to what you see today, jtreg has a slightly different architecture.

jtreg still utilizes the harness' plugin architecture to configure how to read and execute the tests in the test suite, but it provides its own custom command line interface for setting up a test run. The jtreg CLI provides options tailored to running the JDK regression test suite. It can also invoke the JT Harness GUI, but this is just for monitoring a test run, and does not (yet) provide much support for reconfiguring a test run with different options from within the GUI, as you can do for other test suites.

jtreg is also different from other test suites in that jtreg is distributed separately from the tests that it gets to execute. TCKs are generally built and distributed as self-contained products in their own right, whereas the JDK regression tests do not get prebuilt: they exist and are executed in their source form in each JDK developer's repository. Per the The JDK Test Framework: Tag Language Specification, the tests are compiled and executed as needed.

For more information, see ...

Thursday May 01, 2008

OpenJDK Regression Test Harness (jtreg) - open source

The OpenJDK Regression Test Harness, also known as "jtreg", is now available with an open source license.[Read More]

Discussions about technical life in my part of the OpenJDK world.


« April 2014