Monday Apr 01, 2013

jtreg to embrace Perl, PHP and Python

Now that OpenJDK has an improved new build system, it is time to re-examine our test infrastructure with a view to gaining similar improvements for writing and running tests.

We have an ongoing problem with test reliability. Part of the issue is that we write our tests in Java, which apparently, and regrettably, has bugs in it. (Otherwise, why would we be testing it?) With the recent progress in Java scripting technology over the past couple of years, we should convert all our tests to use a scripting language. Although, we currently specify the use of Bourne shell, there are too many evolutionary variants of that, and so it is proposed we instead use Perl for all our test code. Of course, there are different versions of Perl, and so users will have to run a configure script ahead of time to determine if they have enough versions of Perl available on their system, and to recommend how to compile additional versions if necessary.

From early days, JavaTest, jtharness and jtreg have embraced the web, providing HTML reports, support for an HTTP server to be available while tests are running, and servlets to display test results in full color, eliminating the need for thousands of words. We should build on those ideas and embrace web-based test execution and reporting using a new PHP-based back-end for jtreg.

Finally, we should rewrite jtreg itself in Python. This will facilitate easy integration into our work flow as a Mercurial extension. Initially, this can be provided as "hg test" but the long term goal is to integrate jtreg functionality into jcheck, so that all appropriate tests are run automatically before any code can be pushed to a jcheck-controlled forest.

Thursday Dec 13, 2012

jtreg update, December 2012

There is a new version of jtreg available. The primary new feature is support for tests that have been written for use with TestNG, the popular open source testing framework. TestNG is supported by a variety of tools and plugins, which means that it is now possible to develop tests for OpenJDK using those tools, while still retaining the ability to have the tests be part of the OpenJDK test suite, and run with a single test harness, jtreg.

jtreg can be downloaded from the OpenJDK jtreg page: http://openjdk.java.net/jtreg.

TestNG support

jtreg supports both single TestNG tests, which can be freely intermixed with other types of jtreg tests, and groups of TestNG tests.

A single TestNG test class can be compiled and run by providing a test description using the new action tag:

    @run testng classname

The test will be executed by using org.testng.TestNG. No main method is required.

A group of TestNG tests organized in a standard package hierarchy can also be compiled and run by jtreg. Any such group must be identified by specifying the root directory of the package hierarchy. You can either do this in the top level TEST.ROOT file, or in a TEST.properties file in any subdirectory enclosing the group of tests. In either case, add a line to the file of the form:

   TestNG.dirs = dir ...

Directories beginning with '/' are evaluated relative to the root directory of the test suite; otherwise they are evaluated relative to the directory containing the declaring file. In particular, note that you can simply use "TestNG.dirs = ." in a TEST.properties file in the root directory of the test group's package hierarchy.

No additional test descriptions are necessary, but test descriptions containing information tags, such as @bug, @summary, etc are permitted.

All the Java source files in the group will be compiled if necessary, before any of the tests in the group are run. The selected tests within the group will be run, one at a time, using org.testng.TestNG.

Library classes

The specification for the @library tag has been extended so that any paths beginning with '/' will be evaluated relative to the root directory of the test suite.

In addition, some bugs have been fixed that prevented sharing the compiled versions of library classes between tests in different directories. Note: This has uncovered some issues in tests that use a combination of @build and @library tags, such that some tests may fail unexpectedly with ClassNotFoundException. The workaround for now is to ensure that library classes are listed before the test classes in any @build tags.

To specify one or more library directories for a group of TestNG tests, add a line of the following form to the TEST.properties file in the root directory of the group's package hierarchy:

   lib.dirs = dir ...
As before, directories beginning with '/' are evaluated relative to the root directory of the test suite; otherwise they are evaluated relative to the directory containing the declaring file. The libraries will be available to all classes in the group; you cannot specify different libraries for different tests within the group.

Coming soon ...

From this point on, jtreg development will be using the new jtreg repository in the OpenJDK code-tools project. There is a new email alias jtreg-dev at openjdk.java.net for discussions about jtreg development. The existing alias jtreg-use at openjdk.java.net will continue to be available for questions about using jtreg.

For more information ...

An updated version of the jtreg Tag Language Specification is being prepared, and will be made available when it is ready. In the meantime, you can find more information about the support for TestNG by executing the following command:

   $ jtreg -onlinehelp TestNG

For more information on TestNG itself, visit testng.org.

Thursday Jul 21, 2011

jtreg update, 2011

There is a new update for jtreg 4.1 available. The most notable change is the addition of support for concurrent test execution: great for use on modern multi-core machines! Other changes have been driven by the goal to help cleanup and improve the JDK unit and regression test suite.

jtreg can be downloaded from the OpenJDK jtreg page.

Concurrent test execution

As reported earlier, jtreg now supports concurrent test execution in the agentvm mode and othervm modes. Agentvm mode was introduced last year, and is "like samevm mode, but better." To run tests currently, use the new -concurrency:N option. Start off with a value equal to the number of processors on your system, but depending on the tests you are running, you may be able to raise the number higher.

Multi-run mode

Previously, jtreg could only run tests under a single root directory, identified by a TEST.ROOT file. Now, for various reasons, the JDK unit and regression tests are being split across multiple repositories, into separate directory trees, each with their own test root and TEST.ROOT fie. For example, the "langtools" tests, in langtools/test. are separate from the main set of "jdk" tests, in jdk/test. So, the restriction on running tests under a single root directory has been relaxed. Under the covers, jtreg automatically groups the tests according to their test root directory, runs each group of tests in turn, and then aggregates the results.

More result and report details

The hostname on which a test is run is recorded in the *.jtr file. This is useful when test runs have been distributed across a set of machines.

The test results can now also be written to XML files in a format recognized by Hudson, which makes it easier to track the test results over time. Use the -xml option to enable this feature. The data for each test.jtr file is written to a corresponding XML file named test.jtr.xml in the same directory. Thanks to Kumar Srinivasan for this contribution.

The time taken to run a test is now reported in the .jtr file. Previously, the start and end times were available, but they had to be read separately and the elapsed time computed. The elapsed time is now available directly, in both milliseconds and hh:mm:ss.millis.

The distribution of times to run each of the tests in a test run is now reported in a new file text/timeStats.txt written to the report directory at the end of the run. The mean and standard deviation are given as well.

You can now configure the format of the status message written by jtreg to the console at the end of the test run. See the online help for details of the system property to set and the escape sequences that are recognized. The message is also written to the file text/stats.txt in the report directory.

-limittime:N

There is a new option -limittime:N that can be used to filter out tests which may take a long time to run. jtreg examines the test's declared timeout value to determine whether the test should be run or not.

ProblemList.txt

jtreg now provides direct support for the ProblemList.txt file used to identify problematic tests in the jdk/test/ regression test suite. Previously, it was processed by test/Makefile into an exclude list; now, the fie can be given directly to the -exclude option.

javac exit codes

Since JDK 6, javac has used a small set of fixed exit codes to identify different exit conditions. jtreg now recognizes those codes. In particular, the @compile/fail option will now only succeed if the compilation exits normally, after generating diagnostics according to bad source files. It will not succeed if javac exits for a more serious reason, such as a javac crash.

JUnit support

For licensing reasons, we cannot ship a copy of JUnit with jtreg. If you want to use the jtreg support for running JUnit tests, install a copy of junit.jar in the jtreg lib/ directory or set the system property junit.jar to a place where it can be found.


Thanks to Joe and Maurizio for their feedback on this note.

Monday May 16, 2011

Speeding up tests again: another milestone

A while back, I reported a milestone about being able to run all the langtools tests in jtreg's samevm mode. And more recently I reported on a new feature in jtreg called "agentvm" mode. Now, we can report more progress on being able to make the tests run even faster, or put another way, being able to run even more tests in the same amount of time.

With a number of folk beginning to look at improving (read: speeding up) the OpenJDK build infrastructure, it seems appropriate to take a corresponding look at working on the test infrastructure as well. With that in mind, we investigated the possibility of running jtreg tests in parallel.

jtreg was originally built on top of the JavaTest test harness, now available via the open source "jtharness" project. From the very beginning, JavaTest was designed to support concurrent test execution, and so all the necessary hooks for concurrent test execution are available in the underlying harness. But, while it would have been possible to leverage those hooks for any tests running in othervm mode, executing tests concurrently in samevm mode has never been a realistic option, because of the risk of interference between tests in the same virtual machine. Tests assume write access to global state, such as system properties, the security manager, and the standard input and output streams. (In short, lots of stuff in java.lang.System.) And, in the early days, it seemed better to focus on getting tests to run in samevm mode than it was to focus on running tests in parallel in othervm mode. That was then, but this is now.

Nowadays, multicore machines are more common, and given that we've taken samevm tests about as far as we can go, it's time to look at running tests in parallel again. I typically describe the new "agentvm" mode as "like samevm mode, but better." Up to now, that has primarily meant that the agentvm mode is more robust in the face of really bad tests (that don't clean up after themselves), and it allows the possibility of different instances of JDK for compiling and executing tests (e.g. using different JVM options.) But, Maurizio has been investigating using multiple agents at once as a way of running tests in parallel, and he has come up with a small but elegant patch for jtreg to make it all work, just like that.

The idea is simple. The underlying harness supports parallel test execution in the obvious way: threads are created to run tests, these threads take tests from a queue of tests to be executed, and do whatever necessary to execute the test, until all the tests are done. Separately, "agentvm" mode works by using a pool of virtual machines, with specific individual characteristics, such as the version of JDK being used, and the options used to create the VM. Virtual machines are taken from the pool as needed for each test, creating them as needed if there is not a suitable machine already available. When the test completes, the virtual machines are returned to the pool for reuse in another test, provided they can be reset to a known reasonably clean state. So if you run tests in parallel with agentvm mode, it just means that more requests will be made on the pool of virtual machines, and that more virtual machines will be instantiated as needed to meet the demand.

At the user level, the work is as simple as providing a new option, -concurrency:N. Since parallel test execution is incompatible with samevm mode, that combination is forbidden, but that still leaves othervm mode and agentvm mode. By itself, parallel test execution in othervm mode is not so interesting, but it is important that parallel test execution should not prevent individual tests from specifying othervm mode as needed, even when the rest of the tests are running in agentvm mode.

At the implementation level, it is almost as easy as passing the required concurrency down to the underlying harness. However, there are a few gotchas to take care of as well. Each jtreg test is run in an empty scratch directory, and by default, all tests use the same scratch directory, which is emptied before each test begins. Likewise, the tests share a classes hierarchy, which can be problematic if multiple copies of javac are using it at the same time. The solution to both of these is to create subdirectories of the scratch and classes directory, one for each thread running tests. Finally, the agent code in jtreg maintains a pool of JVMs, and since each JVM is configured to run in a specific scratch directory, it is necessary to update the agent pool to take the requisite scratch directory into account when allocating a JVM for a test.

And that is pretty much all there is to it!

Does it work? Yes. One langtools test had to be fixed up because it was incorrectly relying on another test having already been run, because the former did not correctly specify a full set of dependencies to be compiled.

Did it work well? Yes. So far it seems to scale well with the number of processors available. However, it is worth noting that when using Hotspot, Hotspot itself uses additional threads to do its work, so it may not be effective to run as many tests in parallel as you have processors available.

Machine samevm agentvm
Concurrency Change
Intel Core 2 Duo
laptop
real 11m20.444s user 9m21.795s sys 2m08.220s real 4m55.814s user 6m38.905s sys 1m48.327s 4 x2.3
2x Quad core
lab machine
real 12m23.199s user 10m15.857s sys 2m51.223s real 2m35.136s user 9m50.262s sys 4m36.250s 8 x4.8

The langtools test suite currently has just over 1900 tests, and by instrumenting javac, we know that running the entire langtools test suite performs about 30895 compilations (actually, it creates that many instances of the internal JavaCompiler class.) So from the table we can infer that on a suitable machine, we can run 1905 tests or 30895 compilations in 2m 35s, which works out to about 12 tests per second or 200 compilations per second! And even just being able to run all the tests in under 5 minutes on what these days is an average engineering laptop is not bad going either. Woohoo! That's definitely worth declaring as a milestone!

Availability

This feature will be available soon, in an update to jtreg 4.1. Watch here or on the jtreg project page for more details.


Thanks to Maurizio Cimadamore for working on this feature and for his feedback on this note.

Thursday May 27, 2010

jtreg: old and new

jtreg 4.1 is now available, and one of the new features is a new "agentvm" mode. Here, by way of a short historical review, is a comparison of the new agentvm mode against the existing othervm and samevm modes.
In the beginning...

The spec for jtreg calls for tests to execute in a scratch directory which can be cleared before every test. Early versions of Java™ did not provide the ability to change the current directory when executing a command in a sub process, and so the first versions of jtreg were split into a complicated shell script and a Java program using the JavaTest™ test framework. The shell script analyzed the command line arguments, determined the scratch directory to be used, and set it as the current directory before finally executing the test harness itself.

Even back then, there was support for two modes: "othervm", being the ability to run every test in its own JVM, different from (other than) the JVM used to run the test harness, and "samevm", being the ability to run every test in the same JVM used to run the test harness. However, to begin with, the use of "othervm" was strongly recommended, because the risk of tests affecting one another was quite high, and the abilility to recover from over-zealous tests was quite limited. And, although "samevm" was conceptually well defined for running API tests (@run main Foo, etc) it was less well defined for compilation tests (@compile, etc) and there is still some legacy code in javac that was added to work around the limitations (e.g. -XDstdout, to force the compiler to write to the stream used by the jtreg @compile/ref= option.)

And then ...

As jtreg evolved, the maintenance of the wrapper script became increasingly difficult. Eventually, the script was translated into Java code, and merged with the harness itself. However, the issue of the current directory remained an issue, and so after analyzing the command line arguments, jtreg performed a check to see if the JVM being used was acceptable for the execution of the test run itself. This is primarily an issue for samevm mode, when the correct current directory must be set, the correct version of Java must be in use, and so on. If any of the checks failed, jtreg restarted itself using the correct parameters -- the correct version of Java, the correct current directory and so on. In this case, the Java code was behaving as "a better wrapper script", and although there were now potentially two JVMs involved, making the name "samevm" somewhat ambiguous, the name of the mode stuck. (Well, the tests themselves were mostly all executing in the "same VM"...)

And along the way, samevm mode has been improved such that it is now the recommended mode to use for all the tests in the OpenJDK langtools repositories. To be fair, making that happen has involved changes to the test harness, the tests and in a couple of cases, to the tools themselves. But the payoffs have been substantial, and we can now execute over 1700 tests in just one sixth of the time it would using using othervm mode. (11 minutes vs. 66 minutes on my laptop)

But ...

While we have been able to fix the tests in the langtools repositories to use samevm mode, it has not been so easy to find the resources to do the same for the tests in the jdk repositories. Common problems are tests not closing open files when they exit, and tests trying to set a security manager. These conditions are not an issue when running tests in othervm mode, but both are problematic in samevm mode. If files are left open in the scratch directory, on Windows that will prevent the files being deleted, which in turn will cause problems for all the subsequent tests in the test run. Setting a security manager always been forbidden in samevm because originally it could only be set once anyway, and even now, it is possible to set a security manager that you cannot remove. Together, these and other problems, significantly reduce the number of tests in the jdk repositories that can be run in samevm mode, and that can therefore benefit from the corresponding improvement in performance.

Ideally, it would be possible to make sure that most tests can be run in samevm mode. But updating tests is a risky business at the best of times, and even harder when the original authors are no longer available. And so, in practice, it has been easier to improve the test harness...

And so...

The new agentvm mode provides a way to work around these problems. It's still a problem if a test leaves open files when run on Windows, or if it sets a security manager that cannot be unset, but with the new agentvm mode, such problems do not affect the rest of the tests in the test run.

In agentvm mode, jtreg runs in one JVM and creates JVMs with the required characteristics (version of JDK, directory, VM options) as needed to run the tests. More significantly, when each test completes, jtreg will do some housekeeping to restore the JVM involved and the scratch directory to a standard initial state. If any of the housekeeping fails for any reason, the JVMs are allowed to exit. But if the housekeeping is successful, the JVMs are kept available for reuse by any subsequent tests that need JVMs with the same characteristics.

Thus, for "well-behaved" tests, agentvm mode will provide similar performamce to samevm mode, because there will not be any overhead to create a new JVM for each test. But, for less well behaved tests, agentvm mode will automatically degrade to something more like othervm mode, starting new JVMs as necessary for each test.

How many JVMs does jtreg use in agentvm mode at any time? Typically, just two: one to run the test harness, and one to run the tests. That's the same as samevm mode, except it's better, insofar as the JVM used to run the tests will be restarted if there are any problems. There may be three, because agentvm mode allows us to relax the restrictions inherent in samevm mode about not allowing different JVM options or even versions of Java for the compilation and execution of tests. It can do this while still retaining the performance of samevm mode, by creating and using different agent JVMs with the appropriate parameters.

Thus, with agentvm mode, it is now possible to compile tests using a standard JVM but to execute the compiled classes with special VM options, such as for profiling. Previously, this combination was only possible with othervm mode.

And, with agentvm mode, it is now possible to compile tests with a standard version of JDK, and to execute the tests using a version of the Java platform that does not include javac. Previously, jtreg could only test versions of the Java platform that included javac.

Note: if any tests are explicitly marked /othervm, they will obviously be run in their own JVM, separate from the reusable JVMs used for the rest of the tests.

To try out the new agentvm mode, just use the -agentvm option instead of -samevm or -othervm. Or, if you have scripts or Makefiles already set up to run jtreg in samevm mode, there's a backdoor environment variable you can set to use agentvm mode instead, until you're ready to commit any change to your script or Makefile: just set JTREG_USE_AGENTVM_FOR_SAMEVM to true.

Finally...

Note that when you use agentvm mode, it doesn't make the problems of running a test in samevm mode go away -- it just makes jtreg better able to identify and then tolerate such issues. When issues occur, the performance will degrade to similar to that of othervm mode. To keep the performance up to the level of samevm mode, the issues in the tests reported by jtreg will still need to be fixed. But that's a different story...


Thanks to Joe Darcy and Alan Bateman for their feedback on this note.

jtreg 4.1

There is a new version of jtreg available, 4.1, with a number of useful new features. Most of these are to do with the way that jtreg runs tests, but one feature, limited support for JUnit tests, involves a spec change for the test descriptions, which means that this new version of jtreg will be required to run any testsuites that include such tests.

jtreg can be downloaded from the OpenJDK jtreg page.

New "agentvm" mode

Up to now, jtreg has provided two ways of running tests: in "othervm" mode and in "samevm" mode. In "othervm" mode, a new JVM is created for each action that needs it. This provides the maximum isolation between tests, at a very high cost in performance. In "samevm" mode, all Java actions are run in the same VM, which provides high performance at the risk of tests being able to affect the behavior of any subsequent tests. In some cases, a bad test can prevent all subsequent tests from being able to execute.

Problems can arise when setting a security manager such that it cannot be removed when a test completes, or failing to close any open files in the scratch directory, which on Windows means they cannot be deleted before the next test runs.

"Agentvm" mode is a new mode that is "like samevm mode, but better". JVMs created to run tests will be reused for subsequent tests provided that the jtreg harness can successfully perform a certain amount of cleanup after each test. If ever the cleanup fails, the corresponding JVMs will be terminated, and new ones will be started if necessary for any subsequent tests.

In "samevm" mode, because the same JVM is used for both test compilation and test execution, it is not possible to specify JVM options to be used just when executing a test. This restriction is not necessary in "agentvm" mode, and separate JVMs will be created if needed for any actions requiring different JVM characteristics. Assuming the cleanup after a test is successful, all the JVMs used by a test will be saved for reuse by later tests that may need JVMs with those same characteristics. However, if any of the cleanup fails, all the JVMs used by the test will be terminated, because in general it is not possible to determine which of the JVMs may have been at fault.

The new mode can be selected with the -agentvm option. For those situations where "samevm" mode is currently being used, it is also possible to select the new mode by setting the environment variable JTREG_USE_AGENTVM_FOR_SAMEVM to true. This may be convenient if you don't have easy access to the command line used to invoke jtreg.

Better links in HTML report files

When jtreg writes HTML report files, it creates links from the report to files in the work directory, and these links could easily be broken if the files in these directories were moved. jtreg now checks if the report directory and work directory are "closely related", meaning they are the same, or one is a parent of the other, or if they have a common parent. In these cases, jtreg will use relative links from the report files to the test result files in the work directory, so that if the two directories are moved together and their relative relationship is retained, the links will remain valid.

JUnit tests

Limited support has been added for running JUnit tests. Tests still require a standard jtreg test description, but you can use a new action "@run junit classname", which will invoke the test in the manner of JUnit 4. The class will be compiled if necessary, or it can be compiled explicitly in the standard way with the "@build" and "@compile" actions. (Thanks to John Rose for providing this feature.)

Separate compilation: -compilejdk:jdk

Up to now, jtreg has only been able to test versions of JDK, or more accurately, versions of Java that include the compiler javac. jtreg now has a new option to specify an alternate JDK to be used for any @compile actions in a test. This includes all implicit @compile actions as well. This means that jtreg is now able to test versions of the Java platform that do not include javac.

Notes:

  • If the -compilejdk option is not used, the same version of Java used to compile and run the tests, as now.
  • Previously, if no version of Java to test was specified on the command line, the value defaulted to the version of Java used to run jtreg. If this value was a JRE embedded in a JDK, the JDK was used instead. Now, if -compilejdk is specified, the preference to use a JDK instead of the embedded JRE is no longer required. If in doubt, always do one of the following:
    • specify -testjdk for the JDK used to compile and run the tests
    • specify -compilejdk for the JDK used to compile the tests, and -testjdk for the JDK or other version of Java to run the tests
  • Some shell tests invoke the compiler directly, using the value in the TESTJAVA environment variable. (For example, ${TESTJAVA}${FS}bin${FS}javac -d classes HelloWorld.java.) jtreg provides a new environment variable, COMPILEJAVA, which will be set to the value of -compilejdk if it was specified, and $TESTJAVA otherwise. If the tests in a testsuite might be run with -compilejdk set, any shell tests using $TESTJAVA to access the compiler should be updated to use $COMPILEJAVA instead.
Improved path handling

In various situations, jtreg creates or modifies search paths, such as a classpath or sourcepath, for use in the actions of a test. Previously, jtreg did not check whether the components it was adding to a path actually existed, and this could cause problems for some tests that checked the contents of these paths (for example, with the javac -Xlint or -Xlint:all options.) Components to be added to a path are now checked to verify they identify an existing file or directory.

Future source

The @compile action now checks the modification time of source files to be compiled, and gives a warning if the last modified time is in the future.

New option -allowSetSecurityManager[:value]

jtreg now allows tests in samevm mode and the new agentvm mode to set the security manager. (It was always possible to set the security manager in othervm mode.) If the security manager cannot be reset when the test completes, then in samevm mode, verbose messages will be printed directly to the console output, and in agentvm mode, the test's JVM(s) will be terminated. Acceptable values are yes, on or true and no, off or false. The feature is enabled by default; use -allowSetSecurityManager:off to disable it.

New suboption for -retain

The -retain option allows you to specify which files in the scratch directory should be retained when a test completes. The default (when the option is not specified) is to leave the files in the scratch directory until the beginning of the next test (if any.) This is convenient when running a single test, but is not convenient when there is a problem cleaning up any files at the beginning of the next test -- by then, the identity of the test that created the files has been lost.

A new suboption is now available for -retain: -retain:none. With this option, all files remaining in the scratch directory when a test completes will be removed immediately. Any problems that may arising in deleting these files will cause the test to be reported with an ERROR result.

-classpathappend in samevm mode

The -classpathappend option did not work in samevm mode, and has now been fixed.


Thanks to Joe Darcy for his feedback on this note.

Wednesday Jul 01, 2009

Minor updates for jtreg

jtreg 4.0 b03 is now available, and fixes a number of minor issues. You can find it on the standard jtreg download page

It includes the following minor fixes and updates:

  1. Do not follow symlinks when cleaning a directory.
  2. Avoid potential NPE when cleaning a directory.
  3. Allow annotation processing of class files in jtreg directives.
    New option /process for @compile tag disables requirement that one or more \*.java files must be given to @compile.
  4. Set test.src and test.classes properties for @compile tasks.
    These properties can now be used by annotation processors invoked by the @compile task.
  5. Recategorize -e option as a general option instead of a JDK one.
  6. Set MKS to use case-sensitive compare in launcher script.
  7. Remove obsolete reference to JAVA_HOME from launcher script.
  8. (jtdiff) Avoid NPE in standard jtdiff if no output file given.

Tuesday Oct 21, 2008

jtreg update

I've been working on some bug fixes and features for the OpenJDK Regression Test Harness, jtreg. This new version has just been released in the standard place on the OpenJDK website. Here is a quick summary of the changes.
  1. New option -noreport.

    An issue with the underlying JT Harness is that it always generates reports about all the tests in the test suite, instead of just the tests you selected to run. This means that there may be a noticeable pause when the tests have completed while jtreg writes out reports. For those folk that never read the reports anyway, you can now disable the reports with this new option.

  2. Fix -exclude option in -samevm mode.

    This combination of options did not work correctly. Now, it should.

  3. Improve version detection in jtreg.sh script.

    The wrapper shell script tries to verify it is running on JDK 1.5 or later. The version detection used to be confused if you were running JDK in headless mode, when a spurious message could appear on System.out. Such messages are now filtered out.

  4. Allow System.setIO calls in -samevm mode.

    jtreg tries to insulate tests from each other, so that side effects in one test cannot affect the functioning of another. Previously, tests were not allowed to call System.setIO when running in samevm mode. This prevented a number of tests from running in samevm mode. However, since jtreg only supports running one test at a time, there is no reason to prohibit this call.

  5. Fix the initialization of jtreg-specific system properties for samevm tests to use a space separated list, not the default List.toString() format ("[a,b,c]").

    It was an oversight that a suitable toString() method was not provided on an internal class, resulting in the default toString() for a List being used, instead of the intended format of a space-separated list.

  6. Set the classpath for -samevm tests to include standard jtreg directories, the same as for -othervm tests.

    This is to reduce any unnecessary differences between othervm mode and samevm mode.

  7. Enable XML report via system property.

    While some folk don't want reports at all, other folk want more reports. The system property javatest.report.kinds can be used to select which report types are generated by the underlying JT Harness report facilities. The property can be set to a space separated list of any of the values "text", "html", "xml". The default is "text html". The XML format will be useful for a possible upcoming new plugin for Hudson.

  8. Reduce false positives when setting keywords from test description.

    Some users reported problems with -ignore:quiet and -k:!ignore incorrectly ignoring tests with -XDignore.symbol.file=true. This fixes that issue.

  9. Recategorize the -e option in the "Main" group instead of the "JDK" group.

  10. Update FAQ with reference to -e as a way to override the PATH setting.

  11. (jtdiff) Add convenience 'jtdiff' script.

  12. (jtdiff) Refactor and add new "super-diff" mode.

    "Super-diff" mode is a way of analyzing a matrix of test runs, organized by platform and date. It runs a standard jtdiff for each row in the matrix, thus comparing test results across platforms for each date, and for each column in the matrix, thus comparing the history of the test results for each platform. See jtdiff -help -super for more details.

Monday May 05, 2008

jtharness vs jtreg

Now that the source code for the OpenJDK Regression Test Harness (jtreg) is available, this provides an overview of how jtreg relates to JT Harness.

In the beginning, there was a need to run compatibility tests on all Java plaforms, and thus was born the "Java Compatibility Kit" (JCK), and the test harness to run those tests (JavaTest). Not long after, there was a need to run a different type of test, regression tests, for JDK. They are fundamentally a different style of test (think white box and black box), but rather than write another, different harness, we adapted JavaTest, and thus was born "jtreg". Eventually, we needed to be able to run tests on different types of platforms, such as Personal Java, a precursor to the J2ME platform. JavaTest evolved to fill the need, and gave rise to the basic architecture you see today.

As you can see, JavaTest, now open sourced under the name JT Harness, is composed of a number of distinct components

  • The core harness. This provides the general ability to represent tests, test results, and to sequence through the execution of a series of tests.
  • Utilities. These provide support to the other JT Harness components.
  • The user interface. JT Harness provides a rich graphical user interface, as well as a command line interface.

And, deliberately, there's a hole. By itself, JT Harness cannot directly run tests in any test suite. It needs to be given a plugin for each test suite, that define how to locate and read tests, how to configure and execute tests, and other additional information, often used by the GUI, to provide documentation specific to the test suite. The plugin code is not complicated and is often a direct use or simple extension of the utility classes provided by JT Harness. Note: most test suites do not need to use all of the JT Harness utility classes, which explains why JT Harness may have more build-time dependencies than are required to build and run any individual test suite.

Thus, the complete picture is more as follows:

This is the form used for most test suites, such as JCK and other TCKs. The code to plugin to the harness is bundled in the test suite, along with a copy of JavaTest or JT Harness itself. These test suites all leverage the user interfaces provided by the harness, both the CLI and GUI. When the user runs JT Harness, and tells it to open a test suite, the harness looks inside the test suite for details of the plugin to be used in conjunction with that test suite.

Because jtreg evolved from JavaTest early on, and specifically, before the JavaTest user interfaces evolved to what you see today, jtreg has a slightly different architecture.

jtreg still utilizes the harness' plugin architecture to configure how to read and execute the tests in the test suite, but it provides its own custom command line interface for setting up a test run. The jtreg CLI provides options tailored to running the JDK regression test suite. It can also invoke the JT Harness GUI, but this is just for monitoring a test run, and does not (yet) provide much support for reconfiguring a test run with different options from within the GUI, as you can do for other test suites.

jtreg is also different from other test suites in that jtreg is distributed separately from the tests that it gets to execute. TCKs are generally built and distributed as self-contained products in their own right, whereas the JDK regression tests do not get prebuilt: they exist and are executed in their source form in each JDK developer's repository. Per the The JDK Test Framework: Tag Language Specification, the tests are compiled and executed as needed.

For more information, see ...

Thursday May 01, 2008

OpenJDK Regression Test Harness (jtreg) - open source

The OpenJDK Regression Test Harness, also known as "jtreg", is now available with an open source license.[Read More]

Monday May 14, 2007

Unit Testing Wisdom From An Ancient Software Start-up

JavaOne contained many gems of wisdom, of varying sizes. This one indirectly came my way during the week, and is fun enough to pass along. It is self-described as "Good advice on developer and unit testing, packaged as twelve cryptic bits of ancient Eastern wisdom." You can find it here. It is also available in PDF. Enjoy -- and thanks to Alberto Savoia for making it available.

Thursday Nov 16, 2006

com.sun.javatest.\*: JT Harness API

JavaTest provides some useful APIs for specialized usage ...[Read More]

Wednesday Nov 15, 2006

javatest.png: JavaTest in pictures

A look at the evolution of the JavaTest harness, over the years.[Read More]

Tuesday Nov 14, 2006

Way to go, JT Harness

A brief background overview of the harness underlying jtreg: the JavaTest harness, which is now available as JT Harness. [Read More]

Sunday Nov 12, 2006

wwwww.jtreg: The Who What Where When Why of jtreg

A brief background overview of jtreg, the JDK regression test harness. [Read More]
About

Discussions about technical life in my part of the OpenJDK world.

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today