Thursday Jul 21, 2011

jtreg update, 2011

There is a new update for jtreg 4.1 available. The most notable change is the addition of support for concurrent test execution: great for use on modern multi-core machines! Other changes have been driven by the goal to help cleanup and improve the JDK unit and regression test suite.

jtreg can be downloaded from the OpenJDK jtreg page.

Concurrent test execution

As reported earlier, jtreg now supports concurrent test execution in the agentvm mode and othervm modes. Agentvm mode was introduced last year, and is "like samevm mode, but better." To run tests currently, use the new -concurrency:N option. Start off with a value equal to the number of processors on your system, but depending on the tests you are running, you may be able to raise the number higher.

Multi-run mode

Previously, jtreg could only run tests under a single root directory, identified by a TEST.ROOT file. Now, for various reasons, the JDK unit and regression tests are being split across multiple repositories, into separate directory trees, each with their own test root and TEST.ROOT fie. For example, the "langtools" tests, in langtools/test. are separate from the main set of "jdk" tests, in jdk/test. So, the restriction on running tests under a single root directory has been relaxed. Under the covers, jtreg automatically groups the tests according to their test root directory, runs each group of tests in turn, and then aggregates the results.

More result and report details

The hostname on which a test is run is recorded in the *.jtr file. This is useful when test runs have been distributed across a set of machines.

The test results can now also be written to XML files in a format recognized by Hudson, which makes it easier to track the test results over time. Use the -xml option to enable this feature. The data for each test.jtr file is written to a corresponding XML file named test.jtr.xml in the same directory. Thanks to Kumar Srinivasan for this contribution.

The time taken to run a test is now reported in the .jtr file. Previously, the start and end times were available, but they had to be read separately and the elapsed time computed. The elapsed time is now available directly, in both milliseconds and hh:mm:ss.millis.

The distribution of times to run each of the tests in a test run is now reported in a new file text/timeStats.txt written to the report directory at the end of the run. The mean and standard deviation are given as well.

You can now configure the format of the status message written by jtreg to the console at the end of the test run. See the online help for details of the system property to set and the escape sequences that are recognized. The message is also written to the file text/stats.txt in the report directory.

-limittime:N

There is a new option -limittime:N that can be used to filter out tests which may take a long time to run. jtreg examines the test's declared timeout value to determine whether the test should be run or not.

ProblemList.txt

jtreg now provides direct support for the ProblemList.txt file used to identify problematic tests in the jdk/test/ regression test suite. Previously, it was processed by test/Makefile into an exclude list; now, the fie can be given directly to the -exclude option.

javac exit codes

Since JDK 6, javac has used a small set of fixed exit codes to identify different exit conditions. jtreg now recognizes those codes. In particular, the @compile/fail option will now only succeed if the compilation exits normally, after generating diagnostics according to bad source files. It will not succeed if javac exits for a more serious reason, such as a javac crash.

JUnit support

For licensing reasons, we cannot ship a copy of JUnit with jtreg. If you want to use the jtreg support for running JUnit tests, install a copy of junit.jar in the jtreg lib/ directory or set the system property junit.jar to a place where it can be found.


Thanks to Joe and Maurizio for their feedback on this note.
About

Discussions about technical life in my part of the OpenJDK world.

Search

Archives
« July 2011 »
SunMonTueWedThuFriSat
     
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
22
23
24
25
26
27
28
29
30
31
      
Today