Friday Jun 28, 2013

Friday Stats

As some of you may have noticed, we've recently opened a new repository in the Code Tools project for small utilities which can be used to gather info about the OpenJDK code base and builds. 1

The latest addition is a utility for analyzing the class file versions in a collection of class files. I've posted an example set of results from analyzing the class files in an OpenJDK build on Linux. 2. Most of the files are version 52 files as you would expect, but there is a surprising number of version 51 and 50 files, as well as a handful of v45.3 files as well. Digging deeper, it turns out that Nashorn is still using version 51 class files, and the Serviceability Agent is still using version 50 class files and one 45.3 class file, leaving the remainder of the 45.3 class files coming from RMI.

For more info on the different class file versions, see Joe Darcy's class file version decoder ring.


Thanks to Stuart Marks for planting the seed for the class file version tool.

  1. See the project page, repo, and mail archive.
  2. http://cr.openjdk.java.net/~jjg/cfv-summary/open/

Friday Jun 14, 2013

Testing javac: more, faster

Getting numbers is so much fun, and rarely fails to disappoint.[Read More]

Wednesday May 29, 2013

javadoc TLC

The recent series of patches1 for javadoc completes the work started during JDK 7 to change the internal data model for standard javadoc doclets from strings to a document tree. As a result of this work, there should no longer be any unnecessary internal conversion from tree nodes to strings and back again.

As a side-effect of this work, some bugs were uncovered and fixed, such as not using entities for literal use of '<', '>', and '&', and conversely, treating some HTML fragments as plain text, and then incorrectly replacing those characters with entities. Oops. Also, the indentation of method signatures should now be fixed, so that parameters and exceptions thrown should be vertically aligned, as used to be the case.

A more important side-effect is that the code to generate HTML content has been consolidated within the com.sun.tools.doclets.formats.html package, leaving the main internal taglet API to be more format-neutral. This should make it easier to provide doclets that write to alternate formats.


1. hg: jdk8/tl/langtools: 17 new changesets

Friday May 17, 2013

Shelling Tests

jtreg provides support for various different types of tests. API tests can use "@run main", simple compiler tests can use "@compile", and simple GUI or applet tests can use "@run applet". And then there's the ubiquitous "misc" or "other" category, for which "@run shell" is provided. However, there are a number of reasons why not to use "shell tests", and recently we made another step forward to reduce the use of shell tests in the langtools repository.[Read More]

Monday Apr 01, 2013

jtreg to embrace Perl, PHP and Python

Now that OpenJDK has an improved new build system, it is time to re-examine our test infrastructure with a view to gaining similar improvements for writing and running tests.

We have an ongoing problem with test reliability. Part of the issue is that we write our tests in Java, which apparently, and regrettably, has bugs in it. (Otherwise, why would we be testing it?) With the recent progress in Java scripting technology over the past couple of years, we should convert all our tests to use a scripting language. Although, we currently specify the use of Bourne shell, there are too many evolutionary variants of that, and so it is proposed we instead use Perl for all our test code. Of course, there are different versions of Perl, and so users will have to run a configure script ahead of time to determine if they have enough versions of Perl available on their system, and to recommend how to compile additional versions if necessary.

From early days, JavaTest, jtharness and jtreg have embraced the web, providing HTML reports, support for an HTTP server to be available while tests are running, and servlets to display test results in full color, eliminating the need for thousands of words. We should build on those ideas and embrace web-based test execution and reporting using a new PHP-based back-end for jtreg.

Finally, we should rewrite jtreg itself in Python. This will facilitate easy integration into our work flow as a Mercurial extension. Initially, this can be provided as "hg test" but the long term goal is to integrate jtreg functionality into jcheck, so that all appropriate tests are run automatically before any code can be pushed to a jcheck-controlled forest.

Thursday Dec 13, 2012

jtreg update, December 2012

There is a new version of jtreg available. The primary new feature is support for tests that have been written for use with TestNG, the popular open source testing framework. TestNG is supported by a variety of tools and plugins, which means that it is now possible to develop tests for OpenJDK using those tools, while still retaining the ability to have the tests be part of the OpenJDK test suite, and run with a single test harness, jtreg.

jtreg can be downloaded from the OpenJDK jtreg page: http://openjdk.java.net/jtreg.

TestNG support

jtreg supports both single TestNG tests, which can be freely intermixed with other types of jtreg tests, and groups of TestNG tests.

A single TestNG test class can be compiled and run by providing a test description using the new action tag:

    @run testng classname

The test will be executed by using org.testng.TestNG. No main method is required.

A group of TestNG tests organized in a standard package hierarchy can also be compiled and run by jtreg. Any such group must be identified by specifying the root directory of the package hierarchy. You can either do this in the top level TEST.ROOT file, or in a TEST.properties file in any subdirectory enclosing the group of tests. In either case, add a line to the file of the form:

   TestNG.dirs = dir ...

Directories beginning with '/' are evaluated relative to the root directory of the test suite; otherwise they are evaluated relative to the directory containing the declaring file. In particular, note that you can simply use "TestNG.dirs = ." in a TEST.properties file in the root directory of the test group's package hierarchy.

No additional test descriptions are necessary, but test descriptions containing information tags, such as @bug, @summary, etc are permitted.

All the Java source files in the group will be compiled if necessary, before any of the tests in the group are run. The selected tests within the group will be run, one at a time, using org.testng.TestNG.

Library classes

The specification for the @library tag has been extended so that any paths beginning with '/' will be evaluated relative to the root directory of the test suite.

In addition, some bugs have been fixed that prevented sharing the compiled versions of library classes between tests in different directories. Note: This has uncovered some issues in tests that use a combination of @build and @library tags, such that some tests may fail unexpectedly with ClassNotFoundException. The workaround for now is to ensure that library classes are listed before the test classes in any @build tags.

To specify one or more library directories for a group of TestNG tests, add a line of the following form to the TEST.properties file in the root directory of the group's package hierarchy:

   lib.dirs = dir ...
As before, directories beginning with '/' are evaluated relative to the root directory of the test suite; otherwise they are evaluated relative to the directory containing the declaring file. The libraries will be available to all classes in the group; you cannot specify different libraries for different tests within the group.

Coming soon ...

From this point on, jtreg development will be using the new jtreg repository in the OpenJDK code-tools project. There is a new email alias jtreg-dev at openjdk.java.net for discussions about jtreg development. The existing alias jtreg-use at openjdk.java.net will continue to be available for questions about using jtreg.

For more information ...

An updated version of the jtreg Tag Language Specification is being prepared, and will be made available when it is ready. In the meantime, you can find more information about the support for TestNG by executing the following command:

   $ jtreg -onlinehelp TestNG

For more information on TestNG itself, visit testng.org.

Friday Sep 28, 2012

Experimental new utility to detect issues in javadoc comments

I have posted a preview of an experimental new utility to detect issues in javadoc comments. For more details, see my announcement to the compiler-dev mailing list.

Thursday Mar 22, 2012

jtreg update, March 2012

There is a new update for jtreg 4.1 available. The primary changes have been to support faster and more reliable test runs, especially for tests in the jdk/ repository. [Read More]

Thursday Jul 21, 2011

jtreg update, 2011

There is a new update for jtreg 4.1 available. The most notable change is the addition of support for concurrent test execution: great for use on modern multi-core machines! Other changes have been driven by the goal to help cleanup and improve the JDK unit and regression test suite.

jtreg can be downloaded from the OpenJDK jtreg page.

Concurrent test execution

As reported earlier, jtreg now supports concurrent test execution in the agentvm mode and othervm modes. Agentvm mode was introduced last year, and is "like samevm mode, but better." To run tests currently, use the new -concurrency:N option. Start off with a value equal to the number of processors on your system, but depending on the tests you are running, you may be able to raise the number higher.

Multi-run mode

Previously, jtreg could only run tests under a single root directory, identified by a TEST.ROOT file. Now, for various reasons, the JDK unit and regression tests are being split across multiple repositories, into separate directory trees, each with their own test root and TEST.ROOT fie. For example, the "langtools" tests, in langtools/test. are separate from the main set of "jdk" tests, in jdk/test. So, the restriction on running tests under a single root directory has been relaxed. Under the covers, jtreg automatically groups the tests according to their test root directory, runs each group of tests in turn, and then aggregates the results.

More result and report details

The hostname on which a test is run is recorded in the *.jtr file. This is useful when test runs have been distributed across a set of machines.

The test results can now also be written to XML files in a format recognized by Hudson, which makes it easier to track the test results over time. Use the -xml option to enable this feature. The data for each test.jtr file is written to a corresponding XML file named test.jtr.xml in the same directory. Thanks to Kumar Srinivasan for this contribution.

The time taken to run a test is now reported in the .jtr file. Previously, the start and end times were available, but they had to be read separately and the elapsed time computed. The elapsed time is now available directly, in both milliseconds and hh:mm:ss.millis.

The distribution of times to run each of the tests in a test run is now reported in a new file text/timeStats.txt written to the report directory at the end of the run. The mean and standard deviation are given as well.

You can now configure the format of the status message written by jtreg to the console at the end of the test run. See the online help for details of the system property to set and the escape sequences that are recognized. The message is also written to the file text/stats.txt in the report directory.

-limittime:N

There is a new option -limittime:N that can be used to filter out tests which may take a long time to run. jtreg examines the test's declared timeout value to determine whether the test should be run or not.

ProblemList.txt

jtreg now provides direct support for the ProblemList.txt file used to identify problematic tests in the jdk/test/ regression test suite. Previously, it was processed by test/Makefile into an exclude list; now, the fie can be given directly to the -exclude option.

javac exit codes

Since JDK 6, javac has used a small set of fixed exit codes to identify different exit conditions. jtreg now recognizes those codes. In particular, the @compile/fail option will now only succeed if the compilation exits normally, after generating diagnostics according to bad source files. It will not succeed if javac exits for a more serious reason, such as a javac crash.

JUnit support

For licensing reasons, we cannot ship a copy of JUnit with jtreg. If you want to use the jtreg support for running JUnit tests, install a copy of junit.jar in the jtreg lib/ directory or set the system property junit.jar to a place where it can be found.


Thanks to Joe and Maurizio for their feedback on this note.

Thursday Jun 02, 2011

What's Up, JavaDoc?

The Java documentation tool, javadoc, has been somewhat neglected in recent releases, but in JDK 7, it's been getting some amount of long-overdue TLC, albeit mostly under the covers.

Internally, the biggest change has been to rewrite much of the internals of the standard doclet with respect to the way it generates pages. Previously, javadoc worked by processing data structures modelling the API and then generating the HTML files with a combination of using strings and by writing directly to an output stream, which means you need to know sequentially everything that needs to be written. As anyone who has tried to do this knows, this is hard to get right, and in a number of places javadoc got it wrong, and as a result it generated invalid HTML.  Ooops.  Now, the doclet works by creating an HTML "document tree" using a family of internal, new HTMLTree classes. This allows pages to be created non-linearly when necessary, and allows the page to be written by simply walking the document tree. There's a special node that is used to provide user-provided HTML fragments, which may come from documentation comments or from command line options. For now, these fragments are not checked for validity, but given valid input, javadoc now generates valid compliant HTML as output.

As part of the work to make sure that javadoc generates valid HTML output, we have updated the output to meet the Section 508 accessibility guidelines as well. This has caused some minor changes in the visual appearance, such as ensuring tables have captions, headings, and so on.

Also as part of this work, we have updated javadoc to use CSS and a stylesheet. This means that it is reasonably easy to change the appearance of the generated documentation by simply replacing the stylesheet in the generated documentation.

There have been some other more subtle changes as well. javadoc used to be such that it could only be executed once in any VM. This was not a significant restriction as long as javadoc was run using the command line tool, which started a new VM for each invocation of javadoc, but it was a significant impediment for speeding up test execution in order to be able to test javadoc more, and more often. We have also started the work to convert javadoc to use the javax.tools Compiler API, although more work in this area is required.

What's Next?

Now that javadoc uses a better foundation to build and generate compliant HTML, we are much better placed going forward to consider more radical changes to the contents of the pages, including the possible use of JavaScript for search and menu operations.

Having turned our attention to the way javadoc writes its output, it's now time to also turn our attention to the way it reads its input. In order to be sure we are generating truly compliant (X)HTML, we need to be able to detect issues in any user-provided (X)HTML fragments, in documentation comments or options. One way to facilitate this will be to extend the com.sun.source Compiler Tree API to provide structured access to the content of documentation comments.

Finally, the com.sun.javadoc API has largely been superseded by the new javax.lang.model Language Model API, and so with new language features on the horizon that will require javadoc support, it may be appropriate to migrate the standard doclet onto the newer API.


Thanks to Bhavesh Patel for providing feedback on this entry, and for working on the features described here.

Monday May 16, 2011

Speeding up tests again: another milestone

A while back, I reported a milestone about being able to run all the langtools tests in jtreg's samevm mode. And more recently I reported on a new feature in jtreg called "agentvm" mode. Now, we can report more progress on being able to make the tests run even faster, or put another way, being able to run even more tests in the same amount of time.

With a number of folk beginning to look at improving (read: speeding up) the OpenJDK build infrastructure, it seems appropriate to take a corresponding look at working on the test infrastructure as well. With that in mind, we investigated the possibility of running jtreg tests in parallel.

jtreg was originally built on top of the JavaTest test harness, now available via the open source "jtharness" project. From the very beginning, JavaTest was designed to support concurrent test execution, and so all the necessary hooks for concurrent test execution are available in the underlying harness. But, while it would have been possible to leverage those hooks for any tests running in othervm mode, executing tests concurrently in samevm mode has never been a realistic option, because of the risk of interference between tests in the same virtual machine. Tests assume write access to global state, such as system properties, the security manager, and the standard input and output streams. (In short, lots of stuff in java.lang.System.) And, in the early days, it seemed better to focus on getting tests to run in samevm mode than it was to focus on running tests in parallel in othervm mode. That was then, but this is now.

Nowadays, multicore machines are more common, and given that we've taken samevm tests about as far as we can go, it's time to look at running tests in parallel again. I typically describe the new "agentvm" mode as "like samevm mode, but better." Up to now, that has primarily meant that the agentvm mode is more robust in the face of really bad tests (that don't clean up after themselves), and it allows the possibility of different instances of JDK for compiling and executing tests (e.g. using different JVM options.) But, Maurizio has been investigating using multiple agents at once as a way of running tests in parallel, and he has come up with a small but elegant patch for jtreg to make it all work, just like that.

The idea is simple. The underlying harness supports parallel test execution in the obvious way: threads are created to run tests, these threads take tests from a queue of tests to be executed, and do whatever necessary to execute the test, until all the tests are done. Separately, "agentvm" mode works by using a pool of virtual machines, with specific individual characteristics, such as the version of JDK being used, and the options used to create the VM. Virtual machines are taken from the pool as needed for each test, creating them as needed if there is not a suitable machine already available. When the test completes, the virtual machines are returned to the pool for reuse in another test, provided they can be reset to a known reasonably clean state. So if you run tests in parallel with agentvm mode, it just means that more requests will be made on the pool of virtual machines, and that more virtual machines will be instantiated as needed to meet the demand.

At the user level, the work is as simple as providing a new option, -concurrency:N. Since parallel test execution is incompatible with samevm mode, that combination is forbidden, but that still leaves othervm mode and agentvm mode. By itself, parallel test execution in othervm mode is not so interesting, but it is important that parallel test execution should not prevent individual tests from specifying othervm mode as needed, even when the rest of the tests are running in agentvm mode.

At the implementation level, it is almost as easy as passing the required concurrency down to the underlying harness. However, there are a few gotchas to take care of as well. Each jtreg test is run in an empty scratch directory, and by default, all tests use the same scratch directory, which is emptied before each test begins. Likewise, the tests share a classes hierarchy, which can be problematic if multiple copies of javac are using it at the same time. The solution to both of these is to create subdirectories of the scratch and classes directory, one for each thread running tests. Finally, the agent code in jtreg maintains a pool of JVMs, and since each JVM is configured to run in a specific scratch directory, it is necessary to update the agent pool to take the requisite scratch directory into account when allocating a JVM for a test.

And that is pretty much all there is to it!

Does it work? Yes. One langtools test had to be fixed up because it was incorrectly relying on another test having already been run, because the former did not correctly specify a full set of dependencies to be compiled.

Did it work well? Yes. So far it seems to scale well with the number of processors available. However, it is worth noting that when using Hotspot, Hotspot itself uses additional threads to do its work, so it may not be effective to run as many tests in parallel as you have processors available.

Machine samevm agentvm
Concurrency Change
Intel Core 2 Duo
laptop
real 11m20.444s user 9m21.795s sys 2m08.220s real 4m55.814s user 6m38.905s sys 1m48.327s 4 x2.3
2x Quad core
lab machine
real 12m23.199s user 10m15.857s sys 2m51.223s real 2m35.136s user 9m50.262s sys 4m36.250s 8 x4.8

The langtools test suite currently has just over 1900 tests, and by instrumenting javac, we know that running the entire langtools test suite performs about 30895 compilations (actually, it creates that many instances of the internal JavaCompiler class.) So from the table we can infer that on a suitable machine, we can run 1905 tests or 30895 compilations in 2m 35s, which works out to about 12 tests per second or 200 compilations per second! And even just being able to run all the tests in under 5 minutes on what these days is an average engineering laptop is not bad going either. Woohoo! That's definitely worth declaring as a milestone!

Availability

This feature will be available soon, in an update to jtreg 4.1. Watch here or on the jtreg project page for more details.


Thanks to Maurizio Cimadamore for working on this feature and for his feedback on this note.

Thursday May 27, 2010

jtreg: old and new

jtreg 4.1 is now available, and one of the new features is a new "agentvm" mode. Here, by way of a short historical review, is a comparison of the new agentvm mode against the existing othervm and samevm modes.
In the beginning...

The spec for jtreg calls for tests to execute in a scratch directory which can be cleared before every test. Early versions of Java™ did not provide the ability to change the current directory when executing a command in a sub process, and so the first versions of jtreg were split into a complicated shell script and a Java program using the JavaTest™ test framework. The shell script analyzed the command line arguments, determined the scratch directory to be used, and set it as the current directory before finally executing the test harness itself.

Even back then, there was support for two modes: "othervm", being the ability to run every test in its own JVM, different from (other than) the JVM used to run the test harness, and "samevm", being the ability to run every test in the same JVM used to run the test harness. However, to begin with, the use of "othervm" was strongly recommended, because the risk of tests affecting one another was quite high, and the abilility to recover from over-zealous tests was quite limited. And, although "samevm" was conceptually well defined for running API tests (@run main Foo, etc) it was less well defined for compilation tests (@compile, etc) and there is still some legacy code in javac that was added to work around the limitations (e.g. -XDstdout, to force the compiler to write to the stream used by the jtreg @compile/ref= option.)

And then ...

As jtreg evolved, the maintenance of the wrapper script became increasingly difficult. Eventually, the script was translated into Java code, and merged with the harness itself. However, the issue of the current directory remained an issue, and so after analyzing the command line arguments, jtreg performed a check to see if the JVM being used was acceptable for the execution of the test run itself. This is primarily an issue for samevm mode, when the correct current directory must be set, the correct version of Java must be in use, and so on. If any of the checks failed, jtreg restarted itself using the correct parameters -- the correct version of Java, the correct current directory and so on. In this case, the Java code was behaving as "a better wrapper script", and although there were now potentially two JVMs involved, making the name "samevm" somewhat ambiguous, the name of the mode stuck. (Well, the tests themselves were mostly all executing in the "same VM"...)

And along the way, samevm mode has been improved such that it is now the recommended mode to use for all the tests in the OpenJDK langtools repositories. To be fair, making that happen has involved changes to the test harness, the tests and in a couple of cases, to the tools themselves. But the payoffs have been substantial, and we can now execute over 1700 tests in just one sixth of the time it would using using othervm mode. (11 minutes vs. 66 minutes on my laptop)

But ...

While we have been able to fix the tests in the langtools repositories to use samevm mode, it has not been so easy to find the resources to do the same for the tests in the jdk repositories. Common problems are tests not closing open files when they exit, and tests trying to set a security manager. These conditions are not an issue when running tests in othervm mode, but both are problematic in samevm mode. If files are left open in the scratch directory, on Windows that will prevent the files being deleted, which in turn will cause problems for all the subsequent tests in the test run. Setting a security manager always been forbidden in samevm because originally it could only be set once anyway, and even now, it is possible to set a security manager that you cannot remove. Together, these and other problems, significantly reduce the number of tests in the jdk repositories that can be run in samevm mode, and that can therefore benefit from the corresponding improvement in performance.

Ideally, it would be possible to make sure that most tests can be run in samevm mode. But updating tests is a risky business at the best of times, and even harder when the original authors are no longer available. And so, in practice, it has been easier to improve the test harness...

And so...

The new agentvm mode provides a way to work around these problems. It's still a problem if a test leaves open files when run on Windows, or if it sets a security manager that cannot be unset, but with the new agentvm mode, such problems do not affect the rest of the tests in the test run.

In agentvm mode, jtreg runs in one JVM and creates JVMs with the required characteristics (version of JDK, directory, VM options) as needed to run the tests. More significantly, when each test completes, jtreg will do some housekeeping to restore the JVM involved and the scratch directory to a standard initial state. If any of the housekeeping fails for any reason, the JVMs are allowed to exit. But if the housekeeping is successful, the JVMs are kept available for reuse by any subsequent tests that need JVMs with the same characteristics.

Thus, for "well-behaved" tests, agentvm mode will provide similar performamce to samevm mode, because there will not be any overhead to create a new JVM for each test. But, for less well behaved tests, agentvm mode will automatically degrade to something more like othervm mode, starting new JVMs as necessary for each test.

How many JVMs does jtreg use in agentvm mode at any time? Typically, just two: one to run the test harness, and one to run the tests. That's the same as samevm mode, except it's better, insofar as the JVM used to run the tests will be restarted if there are any problems. There may be three, because agentvm mode allows us to relax the restrictions inherent in samevm mode about not allowing different JVM options or even versions of Java for the compilation and execution of tests. It can do this while still retaining the performance of samevm mode, by creating and using different agent JVMs with the appropriate parameters.

Thus, with agentvm mode, it is now possible to compile tests using a standard JVM but to execute the compiled classes with special VM options, such as for profiling. Previously, this combination was only possible with othervm mode.

And, with agentvm mode, it is now possible to compile tests with a standard version of JDK, and to execute the tests using a version of the Java platform that does not include javac. Previously, jtreg could only test versions of the Java platform that included javac.

Note: if any tests are explicitly marked /othervm, they will obviously be run in their own JVM, separate from the reusable JVMs used for the rest of the tests.

To try out the new agentvm mode, just use the -agentvm option instead of -samevm or -othervm. Or, if you have scripts or Makefiles already set up to run jtreg in samevm mode, there's a backdoor environment variable you can set to use agentvm mode instead, until you're ready to commit any change to your script or Makefile: just set JTREG_USE_AGENTVM_FOR_SAMEVM to true.

Finally...

Note that when you use agentvm mode, it doesn't make the problems of running a test in samevm mode go away -- it just makes jtreg better able to identify and then tolerate such issues. When issues occur, the performance will degrade to similar to that of othervm mode. To keep the performance up to the level of samevm mode, the issues in the tests reported by jtreg will still need to be fixed. But that's a different story...


Thanks to Joe Darcy and Alan Bateman for their feedback on this note.

jtreg 4.1

There is a new version of jtreg available, 4.1, with a number of useful new features. Most of these are to do with the way that jtreg runs tests, but one feature, limited support for JUnit tests, involves a spec change for the test descriptions, which means that this new version of jtreg will be required to run any testsuites that include such tests.

jtreg can be downloaded from the OpenJDK jtreg page.

New "agentvm" mode

Up to now, jtreg has provided two ways of running tests: in "othervm" mode and in "samevm" mode. In "othervm" mode, a new JVM is created for each action that needs it. This provides the maximum isolation between tests, at a very high cost in performance. In "samevm" mode, all Java actions are run in the same VM, which provides high performance at the risk of tests being able to affect the behavior of any subsequent tests. In some cases, a bad test can prevent all subsequent tests from being able to execute.

Problems can arise when setting a security manager such that it cannot be removed when a test completes, or failing to close any open files in the scratch directory, which on Windows means they cannot be deleted before the next test runs.

"Agentvm" mode is a new mode that is "like samevm mode, but better". JVMs created to run tests will be reused for subsequent tests provided that the jtreg harness can successfully perform a certain amount of cleanup after each test. If ever the cleanup fails, the corresponding JVMs will be terminated, and new ones will be started if necessary for any subsequent tests.

In "samevm" mode, because the same JVM is used for both test compilation and test execution, it is not possible to specify JVM options to be used just when executing a test. This restriction is not necessary in "agentvm" mode, and separate JVMs will be created if needed for any actions requiring different JVM characteristics. Assuming the cleanup after a test is successful, all the JVMs used by a test will be saved for reuse by later tests that may need JVMs with those same characteristics. However, if any of the cleanup fails, all the JVMs used by the test will be terminated, because in general it is not possible to determine which of the JVMs may have been at fault.

The new mode can be selected with the -agentvm option. For those situations where "samevm" mode is currently being used, it is also possible to select the new mode by setting the environment variable JTREG_USE_AGENTVM_FOR_SAMEVM to true. This may be convenient if you don't have easy access to the command line used to invoke jtreg.

Better links in HTML report files

When jtreg writes HTML report files, it creates links from the report to files in the work directory, and these links could easily be broken if the files in these directories were moved. jtreg now checks if the report directory and work directory are "closely related", meaning they are the same, or one is a parent of the other, or if they have a common parent. In these cases, jtreg will use relative links from the report files to the test result files in the work directory, so that if the two directories are moved together and their relative relationship is retained, the links will remain valid.

JUnit tests

Limited support has been added for running JUnit tests. Tests still require a standard jtreg test description, but you can use a new action "@run junit classname", which will invoke the test in the manner of JUnit 4. The class will be compiled if necessary, or it can be compiled explicitly in the standard way with the "@build" and "@compile" actions. (Thanks to John Rose for providing this feature.)

Separate compilation: -compilejdk:jdk

Up to now, jtreg has only been able to test versions of JDK, or more accurately, versions of Java that include the compiler javac. jtreg now has a new option to specify an alternate JDK to be used for any @compile actions in a test. This includes all implicit @compile actions as well. This means that jtreg is now able to test versions of the Java platform that do not include javac.

Notes:

  • If the -compilejdk option is not used, the same version of Java used to compile and run the tests, as now.
  • Previously, if no version of Java to test was specified on the command line, the value defaulted to the version of Java used to run jtreg. If this value was a JRE embedded in a JDK, the JDK was used instead. Now, if -compilejdk is specified, the preference to use a JDK instead of the embedded JRE is no longer required. If in doubt, always do one of the following:
    • specify -testjdk for the JDK used to compile and run the tests
    • specify -compilejdk for the JDK used to compile the tests, and -testjdk for the JDK or other version of Java to run the tests
  • Some shell tests invoke the compiler directly, using the value in the TESTJAVA environment variable. (For example, ${TESTJAVA}${FS}bin${FS}javac -d classes HelloWorld.java.) jtreg provides a new environment variable, COMPILEJAVA, which will be set to the value of -compilejdk if it was specified, and $TESTJAVA otherwise. If the tests in a testsuite might be run with -compilejdk set, any shell tests using $TESTJAVA to access the compiler should be updated to use $COMPILEJAVA instead.
Improved path handling

In various situations, jtreg creates or modifies search paths, such as a classpath or sourcepath, for use in the actions of a test. Previously, jtreg did not check whether the components it was adding to a path actually existed, and this could cause problems for some tests that checked the contents of these paths (for example, with the javac -Xlint or -Xlint:all options.) Components to be added to a path are now checked to verify they identify an existing file or directory.

Future source

The @compile action now checks the modification time of source files to be compiled, and gives a warning if the last modified time is in the future.

New option -allowSetSecurityManager[:value]

jtreg now allows tests in samevm mode and the new agentvm mode to set the security manager. (It was always possible to set the security manager in othervm mode.) If the security manager cannot be reset when the test completes, then in samevm mode, verbose messages will be printed directly to the console output, and in agentvm mode, the test's JVM(s) will be terminated. Acceptable values are yes, on or true and no, off or false. The feature is enabled by default; use -allowSetSecurityManager:off to disable it.

New suboption for -retain

The -retain option allows you to specify which files in the scratch directory should be retained when a test completes. The default (when the option is not specified) is to leave the files in the scratch directory until the beginning of the next test (if any.) This is convenient when running a single test, but is not convenient when there is a problem cleaning up any files at the beginning of the next test -- by then, the identity of the test that created the files has been lost.

A new suboption is now available for -retain: -retain:none. With this option, all files remaining in the scratch directory when a test completes will be removed immediately. Any problems that may arising in deleting these files will cause the test to be reported with an ERROR result.

-classpathappend in samevm mode

The -classpathappend option did not work in samevm mode, and has now been fixed.


Thanks to Joe Darcy for his feedback on this note.

Thursday Jan 28, 2010

O Tree, Where Art Thou

One of the more subtle aspects of javac syntax trees is that every tree node has position information associated with it. This information is used to identify the location of errors with the source text, and is used by IDEs when refactoring or reformatting code. Ensuring the information is accurate is tricky, and with a number of projects ongoing to update the Java language, and hence the trees used by the compiler, the time has come for some better test support in this area.

The new test is called TreePosTest, and can either be run by the jtreg test harness or as a standalone utility. It can read one or more files or directories given on the command line, looking for source files. It reads each file in turn, ignoring those with syntax errors (there's a lot of those in the javac test directories!) For each file, it scans the tree, checking invariants for the position information for every tree node. Any errors that are found are reported.

Each tree node has three nominal positions, identified as character offsets from the beginning of the file. (Older versions of javac used line-number/char-on-line coordinates, packed into a single integer.) The three positions are the start of the node within the source text, the end of the node within the source text, and a position used to identify that node -- typically the first non-white character in the source text that is unique to that node. The last of these is stored directly in every tree node. The start position is always available and is recursively computed by TreeInfo.getStartPos. The end position requires a table to maintained on the side; for performance reasons, this table is not normally enabled; it must be enabled if end positions are going to be required. When enabled, the end position for a node is recursively computed by TreeInfo.getEndPos, using the endPosTable. (Certain nodes also store an end position directly, when such a position may be required for an error message.)

Given these three positions, we can identify various invariants.

  • For any node: start <= pos <= end
  • Any node must be enclosed in its parent node: parent.start <= start && end <= parent.end
  • The position of a parent node should not be within any of its childen: parent.pos <= start || end <= parent.pos

The first surprise was that the test program found a number of faults within its own source text. Ooops. Running the test program over the source files in the langtools test/ directory, it found 6000 errors in over 2000 files. More ooops. Fortunately, many of those errors are repetitions, but what started as a proactive exercise to test new compiler code was turning out to have more payoff than expected.

I don't know about you, but I tend not to think in characters positions very easily, and error messages like the following leave a little to be desired:

test/tools/javac/treepostests/TreePosTest.java: encl.start <= start: encl:WILDCARD[start:9232,pos:9232,end:9246] this:TYPEBOUNDKIND[start:9224,pos:9224,end:9231]
? extends 
But, you know what they say: a picture is worth a thousand numbers, so the test program now has an optional GUI mode, in which it becomes clearer that the reported range for the parent wildcard node (in red) incorrectly omits the type bound kind (in green). In fact, the type bound kind and therefore the enclosing wildcard node both actually begin at the preceding '?'.

Here is another example. Here, it becomes clear that the position for the parent AND node is incorrectly within the expression on the right hand side of the &&, instead of at the && operator. In fact, this is an instance of a previously reported bug, 6654037.

Issues

Most of the issues that have arisen have been reasonably easy to fix, and bug fixes are already underway. However, there are some problem cases.
Enum declarations
These are desugared right in the parser into equivalent declarations of static final fields within the enclosing class. The question then becomes, what position information should be recorded for these desugared nodes. On the one hand, one might argue to use the "invalid position" constant, NOPOS, since these nodes do not directly correspond to source text, but on the other hand, it is important to record a position in case of errors. (See 6472751.)
Array declarations
Array declarations are complicated by support for legacy syntax, that allows constructions like:
int[] array[];
int[] f()[] { return null; }
Annotations
A number of issues have been observed with the positions recorded for annotations, but which have not yet been fully investigated.
Currently, these issues are addressed by making allowances within the test program.

Summary

The test program can easily be applied to large code bases, such as JDK or JDK test suites. Despite some outstanding issues within javac, the test program has proved its worth in identifying errors within the existing javac code, and should prove useful in future to ensure that any support for new language features will also satisfy the expected invariants for tree positions. And even if the bar is not currently at 100%, at least we know where the bar actually is, by virtue of the specific allowances made in the test program.

Friday Nov 20, 2009

Building javac for JDK 7

Back in August, Kelly posted a blog entry about the Anatomy of the JDK build. However, upcoming new features for javac mean that building the JDK is about to get more interesting. More specifically, building the langtools component of the JDK is about to get a whole lot more challenging.

Background

Currently, it is a requirement that we can build a new version of JDK using the previous version. In other words, we can use JDK 6 to build images for JDK 7. However, we also want to be able to use new JDK features, including new language features, throughout most of the JDK source code. This means we need to be able to use the new version of javac to compile the Java code in the new version of JDK, which in turn imposes a restriction that we must at least be able to compile the new version of javac with the previous version of javac.

In practice, this means langtools is built using the boot JDK, which it uses to build bootstrap versions of javac, javah and javadoc, which understand the latest version of Java source code and class files, but which can be run by the boot JDK. These bootstrap tools will be used through the rest of the JDK build. In addition, the langtools build uses the new bootstrap javac to compile all of the langtools code for eventual inclusion in the new version of JDK. This is shown here, in Figure 1. This directly corresponds to step 1 in Anatomy of the JDK build.

Figure 1: Building langtools today
Building langtools today

The main body represents the langtools build; inputs are on the left, and deliverables (for the downstream parts of the build) are shown on the right.

Problem

In a recent blog entry, JSR 199 meets JSR 203, I described a new file manager for use with javac that can make of of the new NIO APIs now available in JDK7. Separately, Project Jigsaw is working to provide a Java module system that is available at both compile time and runtime, with consequential changes to javac. These two projects together have one thing in common: they both require the new version of javac should be able to access and use new API that is only available in JDK 7, which is at odds with the restriction that we should be able to compile the new javac with the previous version of javac.

The problem, therefore, is, How do we build javac for JDK 7?

Using the source path

One might think we could simply put the JDK 7 API source files on the source path used to compile javac. If only it were that simple! Various problems get in the way: some of the new API already uses new language features which will not be recognized by earlier versions of javac -- for example, some of the Jigsaw code already uses the new diamond operator. Also, javac ends up trying to read the transitive closure of all classes it reads from the source path, and when you put all of JDK on the source path, you end up reading a whole lot of JDK classes! Even though the new javac may just directly reference the NIO classes, to compile those classes, the transitive closure eventually leads you to AWT (really!) and to a couple of show stoppers: some of the classes are platform specific (i.e. in src/platform/classes instead of src/share/classes) and worse, some of the source files do not even exist at the time javac is being compiled -- they are generated while building the jdk repository, which happens much later in the JDK build process. (Step 6 in Anatomy of the JDK build.) So, simply putting the JDK 7 API source files on the source path is not a viable solution -- and reorganizing the build to generate the automatically generated source code earlier would be a very big deal indeed.

Using an import JDK

So, clearly, you can no longer build all of a new javac using the previous version of javac. But, we could leave out the parts of the new javac that depend on the new API, provided that we can build a bootstrap javac that functions "well enough" to be able to build the rest of javac and the JDK. However, we would still need to be able to build the new version of javac to be included in the final JDK image.

If you temporarily ignore chickens and eggs and their temporal relationships, the problems would all go away if you could put the classes for JDK 7 on the (boot) class path used to compile javac. This is very similar to the use of an "import JDK" used elsewhere by the JDK build system when performing partial JDK builds: an import JDK is used to provide access to previously built components when they are not otherwise part of the current build environment, which is somewhat the case here.

This is shown here, in Figure 2, and is not so different from what we are currently doing.

Figure 2: Building langtools with an import JDK
Building langtools with an import JDK

Stub files

In a full JDK build, we cannot compile against the JDK source code on the source path, and we cannot assume the availability of an import JDK to use on the (boot) class path. The solution is to provide stub files for the necessary JDK 7 API, which are sufficient for the purpose of compiling javac. Stub files have the same public signature as the files they represent, but none of the implementation detail, so they do not suffer from the same extensive transitive closure problem as occurred when trying to compile against the real JDK 7 API source code. And, we only need stub files for those classes required by javac that are either new or changed from their JDK 6 counterparts. This also simplifies the problem substantially.

The number of files involved, and the rate at which some of the files are changing, makes it impractical to create and maintain such stub files manually. The solution is to generate the stub files automatically from the latest JDK 7 API that would otherwise be used instead. The stub generator is built from parts of javac -- it reads in the JDK 7 source files to create javac ASTs, it rewrites the ASTs by removing as many implementation details as possible, then writes out the modified AST in Java source form to be used in place of the original. And, as a minor added complication, although the output stub files must be readable by a JDK 6 compiler, the input source files may contain JDK 7 artifacts (remember I said that the Jigsaw code already uses the diamond operator), so the stub generator must be built on top of the new javac -- or at least, those parts of the new javac that can be compiled by the old javac.

The final result is shown here, in Figure 3.

Figure 3: Building langtools using generated stubs
Building langtools using generated stubs

Implementation details

The langtools build.xml file uses three new properties. Two are statically defined in build.properties, and specify the langtools source files that depend on new JDK 7 API, and the API that is depended upon; the third is provided by the user and can specify the location of either an import JDK or a jdk repository.

  • When building a full JDK, the langtools build.xml must be given the location of the jdk/ repository. The langtools build will create and compile against stubs files generated from the necessary JDK source code. [Figure 3, above.] In a full JDK control build, the location of the jdk/ repository is passed in automatically by the Makefile from the JDK_TOPDIR make variable, which exists for this purpose.

  • When building langtools by itself, a developer may choose to pass in the location of an import JDK. In this case, the langtools build will compile against rt.jar in the import JDK, thus precluding the need to generate and use stub files. [Figure 2, above.]

  • If no value is passed in for the jdk/ repository or import JDK, the langtools build will not build those classes that require the use of JDK 7 API. This allows a developer to create a compiler that is just "a better JDK 6" compiler. [Figure 1, above.]

It is also worth noting the compiler options are quite tricky for these different cases, and specifically, for the boxes in the diagrams labelled "compile product classes".

  • javac itself is run using the bootstrap javac classes on the JVM boot class path (-J-Xbootclasspath/p:).

  • If being used, the stub files go on the compiler's source path (-sourcepath), together with -implicit:none and -Xprefer:source. Together, these mean that the stub files are used in preference to any files from the boot JDK, and that class files are not generated for the stub files. Other JDK API comes from the normal boot class path. Note that unlike other situations when overriding the standard JDK API, the stub files cannot go on the boot class path because source files are not read from that path.

  • If an import JDK is being used, it is used together with the javac output directory for the compiler's boot class path (-Xbootclasspath). This completely replaces the normal boot class path used by the compiler, so all JDK classes are read from the import JDK.

  • Unless an import JDK is being used, the javac output directory is prefixed to the normal boot class path (-Xbootclasspath/p:). This means that langtools classes are used in preference to classes on the normal boot class path, while not hiding any classes not defined by langtools.

Summary

With these build changes, it is possible to allow limited references from javac into new JDK 7 API, which are forward references in terms of the normal build process. Furthermore, this can be done without changing the overall structure of the JDK build pipeline.

Acknowledgements

Thanks for Kelly and Maurizio for reviewing the work here.

See Also

Changesets:

About

Discussions about technical life in my part of the OpenJDK world.

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today