Monday Jan 31, 2011

Project Coin: How to Terminate try-with-resources

In addition to mulling over nulling in the try-with-resources statement, the JSR 334 expert group has decided to slightly amend the syntax of try-with-resources statement: an optional trailing semicolon will now be allowed to terminate the list of resources.

Previously, a semicolon could only be used as a separator between two resources as in

try(Resource r0 = new Resource(); Resource r1 = new Resource(); Resource r2 = new Resource()) {...}

or reformatted as

try(Resource r0 = new Resource();
    Resource r1 = new Resource();
    Resource r2 = new Resource()) {...}

However, including an extraneous semicolon at the end of the list would be rejected as a syntax error:

try(Resource r0 = new Resource();
    Resource r1 = new Resource();
    Resource r2 = new Resource();) {...}  // Illegal under JSR 334 EDR!

While requiring a semicolon at the end of a list of resources would be excessive, especially when there is only a single resource being managed, optionally allowing a terminating resource offers several advantages. First, when adding a resource to the list of resources being managed, there are fewer necessary edits when cutting and pasting. More importantly, programmatic generation of code is simplified since the code generation logic does not need to know ahead of time whether or not a particular resource will be the last one when the declaration for that resource is generated. The simple rule "terminate a resource declaration with a semicolon" will result in acceptable code, even if the code is not ideal style-wise. Finally, allowing an optional trailing semicolon is consistent with the handling of commas in array initializers,

int[] values = {1,
                3,  // Legal

and the handling of commas in the declaration of enum constants.

enum PrimaryColor {
    BLUE,  // Legal

The full amended grammar for try-with-resources which allows the optional trailing semicolon is:

try Block Catches
try Block Catchesopt Finally
try ResourceSpecification Block Catchesopt Finallyopt
( Resources ;opt )
Resource ; Resources
VariableModifiersopt Type VariableDeclaratorId = Expression

The necessary compiler changes to implement the revised grammar have been pushed into a JDK 7 integration repository and will appear in a promoted build in due course.

Monday Jan 24, 2011

Project Coin: Safe Varargs in JDK Libraries

Back for JDK 7 build 123, the language support for the Project Coin's safe varargs feature was pushed; the time has come to update the libraries to take advantage of this feature.

Following the same general methodology used to systematically flush out types that should be made Closeable or AutoCloseable, I wrote an annotation processor to identify candidate varargs methods and constructors where adding a @SafeVarargs annotation might be appropriate.

Several JDK library methods were known candidates for @SafeVarargs; running the annotation processor found another one. The complete list of methods to be annotated in a java.\* or javax.\* package is:

  • public static <T> List<T> java.util.Arrays.asList(T... a)
  • public static <T> boolean java.util.Collections.addAll(Collection<? super T> c, T... elements)
  • public static <E extends Enum<E>> java.util.EnumSet<E> EnumSet.of(E first, E... rest)
  • protected final void javax.swing.SwingWorker.publish(V... chunks)

After this update, many fewer spurious unchecked warnings will be reported when calling core library classes.

Friday Jan 21, 2011

OpenJDK 6: b21 regression test results

Running with the usual jtreg flags, "-a -ignore:quiet" in all repositories, using generally accessible hosts for the network configuration file, and adding "-s -ea -esa" for langtools, the basic regression test results on Linux for OpenJDK 6 build 21 are:

  • HotSpot, 97 tests passed and 1 failed.

  • Langtools, 1,391 tests passed.

  • JDK, 3,300 tests pass, 29 fail, and 3 have errors.

More HotSpot tests were added and all but one pass; the new failing test is improperly platform-specific:

0: b20-hotspot/summary.txt  pass: 85
1: b21-hotspot/summary.txt  pass: 97; fail: 1

0      1      Test
---    pass   compiler/6431242/
---    pass   compiler/6894807/
---    pass   compiler/6932496/
---    pass   compiler/6946040/
---    pass   compiler/6958485/
---    pass   compiler/6973329/
---    pass   compiler/6982370/
---    pass   compiler/7002666/
---    pass   gc/6581734/
---    pass   runtime/6626217/
---    pass   runtime/6888954/
---    pass   runtime/6925573/
---    fail   runtime/6929067/

13 differences

In langtools some tests were added and all the tests continue to pass:

0: b20-langtools/summary.txt  pass: 1,365
1: b21-langtools/summary.txt  pass: 1,391

0      1      Test
---    pass   tools/javac/6508981/
---    pass   tools/javac/6734819/
---    pass   tools/javac/6734819/
---    pass   tools/javac/6734819/
---    pass   tools/javac/6889255/
---    pass   tools/javac/
---    pass   tools/javac/
---    pass   tools/javac/
---    pass   tools/javac/
---    pass   tools/javac/api/6411310/
---    pass   tools/javac/api/6440333/
---    pass   tools/javac/api/6733837/
---    pass   tools/javac/api/
---    pass   tools/javac/api/
---    pass   tools/javac/api/
---    pass   tools/javac/api/
---    pass   tools/javac/api/
pass   ---    tools/javac/policy/
pass   ---    tools/javac/policy/
pass   ---    tools/javac/policy/
pass   ---    tools/javac/policy/
pass   ---    tools/javac/policy/
pass   ---    tools/javac/policy/
pass   ---    tools/javac/policy/
pass   ---    tools/javac/policy/
---    pass   tools/javac/policy/test1/
---    pass   tools/javac/policy/test1/
---    pass   tools/javac/policy/test1/
---    pass   tools/javac/policy/test1/
---    pass   tools/javac/policy/test1/
---    pass   tools/javac/policy/test1/
---    pass   tools/javac/policy/test1/
---    pass   tools/javac/policy/test1/
---    pass   tools/javac/policy/test1/
---    pass   tools/javac/policy/test1/
---    pass   tools/javac/policy/test1/
---    pass   tools/javac/policy/test1/
---    pass   tools/javac/policy/test2/
---    pass   tools/javac/policy/test2/
---    pass   tools/javac/policy/test2/
---    pass   tools/javac/policy/test2/
---    pass   tools/javac/policy/test3/

42 differences

And in jdk, a small percentage of tests were added and the existing tests have generally consistent results:

0: b20-jdk/summary.txt  pass: 3,273; fail: 33; error: 2
1: b21-jdk/summary.txt  pass: 3,300; fail: 29; error: 3

0      1      Test
---    pass   com/sun/java/swing/plaf/gtk/
fail   pass   java/awt/Focus/NonFocusableWindowTest/
---    pass   java/awt/Frame/FrameSize/
fail   pass   java/awt/Frame/MaximizedToIconified/
fail   pass   java/awt/Multiscreen/LocationRelativeToTest/
fail   pass   java/awt/TextArea/UsingWithMouse/SelectionAutoscrollTest.html
---    pass   java/awt/font/TextLayout/
pass   error  java/lang/management/MemoryMXBean/
---    pass   java/math/BigDecimal/
pass   fail   java/net/InetAddress/
pass   fail   java/net/URL/
pass   fail   java/net/URLClassLoader/
pass   fail   java/rmi/transport/rapidExportUnexport/
---    pass   java/util/logging/
---    pass   java/util/logging/
---    pass   javax/imageio/plugins/png/
---    pass   javax/imageio/plugins/png/
---    pass   javax/swing/JPopupMenu/6675802/
---    pass   javax/swing/JPopupMenu/6691503/
---    pass   javax/swing/Security/6938813/
---    pass   javax/swing/UIDefaults/6622002/
---    pass   javax/swing/UIDefaults/6795356/
---    pass   javax/swing/UIDefaults/6795356/
---    pass   javax/swing/UIDefaults/6795356/
fail   pass   javax/swing/plaf/synth/
fail   pass   sun/nio/cs/
---    pass   sun/security/pkcs11/SecureRandom/
---    pass   sun/security/pkcs11/Signature/
---    pass   sun/security/ssl/com/sun/net/ssl/internal/ssl/SSLSocketImpl/
---    fail   sun/security/ssl/com/sun/net/ssl/internal/www/protocol/https/HttpsURLConnection/
---    pass   sun/security/ssl/javax/net/ssl/NewAPIs/
---    pass   sun/security/ssl/javax/net/ssl/NewAPIs/SSLEngine/
---    pass   sun/security/ssl/javax/net/ssl/NewAPIs/SSLEngine/
---    pass   sun/security/ssl/javax/net/ssl/NewAPIs/SSLEngine/
fail   pass   sun/security/ssl/sun/net/www/protocol/https/HttpsURLConnection/
fail   pass   sun/security/ssl/sun/net/www/protocol/https/HttpsURLConnection/
fail   pass   sun/security/ssl/sun/net/www/protocol/https/HttpsURLConnection/
fail   pass   sun/security/ssl/sun/net/www/protocol/https/HttpsURLConnection/
pass   fail   sun/security/validator/
---    pass   sun/tools/common/

40 differences

In addition to the basic results, the hotspot and jdk regression test suites were rerun with assertions and system assertions enabled, -ea -esa. (The langtools tests are already run with assertions enabled.) The results for hotspot were identical to running without assertions; in jdk a few additional tests failed:

0: b21-jdk/summary.txt   pass: 3,300; fail: 29; error: 3
1: b21-jdka/summary.txt  pass: 3,295; fail: 33; error: 4

0      1      Test
pass   error  com/sun/jdi/
pass   fail   java/awt/Focus/ActualFocusedWindowTest/
pass   fail   java/awt/Focus/ActualFocusedWindowTest/
pass   fail   java/awt/Focus/TranserFocusToWindow/
fail   pass   java/awt/xembed/server/
pass   fail   java/net/Authenticator/
pass   fail   java/net/CookieHandler/
fail   pass   java/rmi/transport/rapidExportUnexport/
pass   fail   java/util/ResourceBundle/

9 differences

Going forward, the baseline regression test results for all OpenJDK 6 repositories will be reported with assertions and system assertions enabled.

OpenJDK 6 b21 Source Bundle Published

On January 21, 2011 the source bundle for OpenJDK 6 b21 was published.

Major changes include the latest round of security fixes and, courtesy Andrew John Hughes, syncing in HotSpot 19. In addition, jaxp was upgraded to 1.4.4, Kelly removed the binary plug logic, some stray rebranding issues were corrected, Jon Gibbons backported a set of JavaFileManager fixes from JDK 7, and the usual sorts of small backports of fixes from JDK 7 occurred too.

A detailed list of all the changes in b21 is also available.

Project Coin: JSR 334 Expert Group Update

Besides working to address issues identified in the EDR draft, such as refining the diamond specification, the JSR 334 expert group has been considering other matters as well. One change being contemplated is removing the ability to have multiple underscores between digits of a literal; under that possible change, no underscore or a single underscore would be allowed. The primary consideration here is to prevent abuses of the underscores in literal feature that would obscure program meaning; on the other hand, there may be reasonable uses of repeated underscores that are desirable to allow.

The expert group has agreed to one significant change from the EDR draft: as called out as a possibility in the draft, support has been dropped for the general expression form of the try-with-resources statement. That is, support has been removed for passing a resource as a general expression without an accompanying explicit variable declaration.

Several technical problems were identified with allowing a general expression:

  • Syntactic ambiguity: In the parser, it was not always possible to distinguish with one-token look-ahead between the start of an Expression and the start of a Type. Consider code like

      try(i < j // Ternary operator on variables i and j
          ? new Resource1() :
            new Resource2()) {...}

    compared to code like

      try(Box < Resource // Simple generic wrapper around a resource
          > resourceBox = Box<>(new Resource1())) {...}

    A natural grammatical fallback short of banning Expression would be to only allow a more restricted expression, such as Identifier. However, that restricted grammar would require compiler changes to alert programmers to some surprising legal code, as given in the following examples.

  • Usability issues: Consider a try-with-resources statement being used to manage an existing variable where the variable is mutated inside the try block:

    public class TwrExamples implements AutoCloseable {
       public static void main(String... args) {
           TwrExamples twrEg1 = new TwrExamples();
           try(twrEg1) {
               twrEg1 = new TwrExamples();  // Mutating the variable!
       public void close() {

    As try-with-resources was previously specified, this would cause close to be called on the original value, not the value twrEg1 pointed to at the time the try block finishes. In this case, the printed output of the program may be something like:
    which indicates that while close was called on the original value, close was not called on the new TwrExamples object created inside the try-with-resources block. Either policy of calling code on the original value or the value on exit of the block could be problematic. The compiler did not issue any warnings about this situation and warnings should be added if this feature were to be kept. (Mutating a resource variable declared as part of the try-with-resources statement is illegal since such variables are implicitly or explicitly final).

Other complications that stemmed from supporting a general expression as a resource were making sure the specification and implementation accepted both

   try(null) {...}
   try(myGenericMethodThatInfersTheTypeOfItsResult()) {}
as valid programs.

The main rationale for supporting general expressions was to allow non-Closeable objects, such as locks, to be easily wrapped in a suitable type to enjoy the call-method-on-block-exit guarantees of try-with-resources. When this is desirable, the same effect can still be achieved with an explicit resource declaration. As experience is gained with try-with-resources, extensions to support other usage patterns will be considered in future releases.

I'm working with the javac team to remove support for this feature in an upcoming JDK 7 build.

Tuesday Jan 11, 2011

Project Coin: JSR 334 EDR now available

The JSR 334 expert group has been hard at work and now the early draft review (EDR) is now available for your reading pleasure, enjoy.

Tuesday Dec 21, 2010

New javac warning for setting an older source without bootclasspath

To use javac from JDK N to cross-compiler to an older platform version, the correct practice is to:

  • Use the older -source setting.
  • Set the bootclasspath to compile against the rt.jar (or equivalent) for the older platform.

If the second step is not taken, javac will dutifully use the old language rules combined with new libraries, which can result in class files that do not work on the older platform since references to non-existent methods can get included.

Thanks to work by Jon Gibbons, in JDK 7 build 121 and later javac detects and warns about this suspicious situation; for example:

$ javac -source 6
warning: [options] bootstrap class path not set in conjunction with -source 1.6

One way to address the warning is to set the bootclasspath. If that is inappropriate, the warning can be disabled with a new suboption within the -Xlint family, -Xlint:-options.

With this change, a likely problematic combination of options to javac that can lead to subtle build errors are diagnosed by the compiler and can easily by either directly addressed, or documented as part of the build process via the new -Xlint suboption.

Monday Dec 20, 2010

Project Coin: The Mint is Sprouting

As planned, earlier today Stuart pushed the first set of changes to the JDK libraries to use the new Project Coin features, adding a bit of sparkle with diamonds mounted in, java.lang, java.util, and elsewhere. More diamonds and other shiny coins to follow!

Tuesday Dec 14, 2010

Project Coin: Minty Fresh Libraries

The JDK 7 build has been using -source 7 for some time, but to date use of new language features has been informal and incidental. Supporting Project Coin and JSR 334, Stuart Marks will be leading a "minting" exercise over the JDK code base to systematically update the JDK libraries to take advantage of the Project Coin language features. Efforts will be focused on the src and test portions of the jdk repository of the JDK 7 forest. The first features to be rolled into the code will be diamond and strings in switch.

Review requests for different areas will be sent to the appropriate OpenJDK aliases for review. The proposed changes will consist solely of conversions of existing code to use a new language feature. While use of the new features is encouraged, in areas of the platform where code is kept synchronized with earlier release trains, such code cannot be updated to use the Project Coin features at this time.

For several years, we've been collaborating with the NetBeans team to provide IDE support for the Project Coin features. NetBeans 7 Beta is the latest release to support Project Coin, as detailed in its release notes. We will continue to coordinate with the NetBeans team to update the Project Coin support in NetBeans as refinements are made to the Project Coin features. Additionally, to allow use of NetBeans for JDK development, we will defer updating the JDK to use a Coin feature until there is matching support in a NetBeans build, as there is now for diamond and strings in switch. (The NetBeans 7 Beta supports a slightly older version of multi-catch and try-with-resources.)

Monday Dec 13, 2010

Project Coin: Safe Varargs

Following up on earlier work, the javac team has pushed a new implementation of Project Coin's simplified varargs method invocation feature (jdk, langtools). The changes are scheduled to appear in the promotion of JDK 7 b123.

As envisioned previously, a new @Documented annotation type, java.lang.SafeVararags, can be used to suppress warnings related to unchecked warnings, both the new mandatory warnings at the declaration site of a varargs method/constructor with a non-reifiable element type and the existing unchecked warnings at the call sites of such methods. A systematic application of this annotation to appropriate declarations in the JDK libraries will follow as future work.

Since new unchecked warnings are being introduced, those diligently compiling with options like "-Xlint:unchecked -Werror" will see a build error under JDK 7 if any of the suspicious varargs method declarations are found. To address this, the @SafeVarargs annotation can be applied to the declarations, if appropriate, or the @SuppressWarnings({"unchecked", "varargs"}) annotation can be applied. Unlike @SafeVarargs, the @SuppressWarnings annotation will not squelch unchecked warnings at the call site of the annotated method.

The specification of the new SafeVarargs annotation type is below.

Annotation Type SafeVarargs

public @interface SafeVarargs
A programmer assertion that the body of the annotated method or constructor does not perform potentially unsafe operations on its varargs parameter. Applying this annotation to a method or constructor suppresses unchecked warnings about a non-reifiable variable-arity (vararg) type and suppresses unchecked warnings about parameterized array creation at call sites.

In addition to the usage restrictions imposed by its @Target meta-annotation, compilers are required to implement additional usage restrictions on this annotation type; it is a compile-time error if a method or constructor declaration is annotated with a @SafeVarargs annotation, and either:

  • the declaration is a fixed-arity method or constructor
  • the declaration is a variable-arity method that is neither static nor final.

Compilers are encouraged to issue warnings when this annotation type is applied to a method or constructor declaration where:

  • The variable-arity parameter has a reifiable element type, which includes primitive types, Object, and String. (The unchecked warnings this annotation type suppresses already do not occur for a reifiable element type.)
  • The body of the method or constructor declaration performs potentially unsafe operations, such as an assignment to an element of the variable-arity parameter's array that generates an unchecked warning.

    Future versions of the platform may mandate compiler errors for such unsafe operations.

Tuesday Nov 16, 2010

Project Coin: JSR Filed!

Keeping with the plan of Plan B for JDK 7 and 8, a Java Specification Request (JSR) proposal has been submitted to the JCP covering the language features explored by Project Coin. The proposal for Small Language Changes to Enhance Productivity is now being considered by the relevant JCP Executive Committee under an approval ballot which runs for two weeks. This proposal is one of four proposals submitted to the JCP to cover the full Plan B roadmap.

I'm looking forward to working with veterans of Project Coin and other expert group members to finish specifying, implementing, and testing the language changes to complete getting them into the platform.

Thursday Oct 28, 2010

Project Coin Improving Multi-catch and More Precise Rethrow

We've been working on some improvements to the multi-catch and more precise rethrow feature and are planning to push the changes soon. First, as long as a catch parameter is effectively final (in other words not reassigned inside the catch block), the more precise analysis will be enabled if the exception is rethrown. An explicit final modifier will no longer be needed to enable the more precise analysis.

As pointed out in the original proposal form for Improved Exception Handling for Java, the more precise exception analysis can cause programs that currently compile to stop compiling since more unreachable code is identified. While the general evolution policy of the JDK does not promise source compatibility across releases, gratuitously breaking compatibility should be avoided. Fortunately, after examining millions of lines of code in a diverse set of code bases, include the JDK, no instances where the more precise analysis would cause an actual source incompatibility were found. (Details of the analysis will be written up later.)

Second, the catch parameter of a multi-catch clause (catch(Exception1 | Exception2 e) {...}) will be regarded as implicitly final; the compiler will reject code that writes to such a catch parameter. Consequently, an explicit final modifier will no longer be needed in this case, although it will remain legal. This provides a more concise syntax for multi-catch in JDK 7 while preserving flexibility to more fully support disjunctive types in later JDK releases.

Monday Sep 20, 2010

Project Coin at JavaOne 2010

This morning and early afternoon Maurizio and I gave our JavaOne talk about Project Coin to a packed room; the slides for the talk are now available. The NetBeans demo of Coin support also went smoothly.

As announced earlier at JavaOne, we'll be following the "plan B" option for JDK 7 so the accepted Coin features that are not currently implemented will be reconsidered for JDK 8. In the meantime, we're interested to have more feedback on the Project Coin features

  • Improved numeric literals

  • Strings in switch

  • Reduced varargs warnings

  • Diamond operator

  • Multi-catch with more precise rethrow

  • try-with-resources statement

that are available to test out now in JDK 7 builds.

Friday Sep 10, 2010

Project Coin: JDBC 4.1 and try-with-resources

I'm happy to report that Lance Andersen and the JDBC expert group have decided to support the new try-with-resources statement in JDBC 4.1. Lance just pushed a change set for JDBC 4.1 that retrofits the AutoCloseable interface to the types:

  • java.sql.Connection

  • java.sql.ResultSet

  • java.sql.Statement

The change will be available in JDK 7 promoted builds in due course.

Wednesday Sep 08, 2010

Project Coin possibilities under JDK 7 plan A and plan B

Mark Reinhold has announced in various venues that Oracle is rethinking the plan for JDK 7.

In summary, the two plans under consideration with their ship dates are:

   Plan A:  JDK 7 (as currently defined)                      Mid 2012

   Plan B:  JDK 7 (minus Lambda, Jigsaw, and part of Coin)    Mid 2011
            JDK 8 (Lambda, Jigsaw, the rest of Coin, ++)     Late 2012

Oracle is leaning heavily toward plan B, but is also soliciting feedback on the plans, as indicated in Mark's announcement.

For Project Coin, the maximum likelihood outcome under plan B is that the currently implemented Coin features:

will be included in JDK 7, subject to future refinements and revisions (and even possible removals) as experience is gained with the features. The remaining accepted Coin features would then be reconsidered in the context of other language changes being planned for JDK 8.

Friday Aug 20, 2010

Project Coin: Try out try-with-resources

As of build 105 of JDK 7, the compiler and libraries have support for Project Coin's try-with-resources statement (ARM blocks).

We're curious to get feedback on how try-with-resources works in practice, so please download the build, use the feature to on your favorite Closeable type, and report back on how it goes.

Thursday Aug 05, 2010

Project Coin: Uniform Suppression in try-with-resources

As previously considered, the specification of the try-with-resources statement has been updated to require uniform suppression of all throwables. That is, the semantics of the code around the implicitly generated calls to close was previously

    try {
    } catch(Exception #suppressedException) {

and has been replaced with

    try {
    } catch(Throwable #suppressedException) {

(The initial implementation actually implemented the latter semantics; a test has been added to verify the behavior.)

Wednesday Aug 04, 2010

JVM Language Summit: Numbers big and fixed

Last week, I enjoyed attending the 2010 JVM Language Summit. Some of the requests from the "what the JVM needs" session could be addressed by a few additional numerical libraries in the platform.

The default integer arithmetic in many languages conceptually doesn't overflow, analogous to operating on Java's java.math.BigInteger objects rather than 32-bit int or 64-bit long values. However, small integers that enjoy hardware support on commodity CPUs are much more commonly operated on than big numbers which require multi-word support. Ideally, conceptually unbounded integers would still run very fast when they were small enough without consuming either excess CPU cycles or excess memory. In the limit, fixnums would result, where the transitions between small ↔ big integers were managed and optimized by the JVM. Until that happy day, internally BigDecimal manages longBigInteger transitions and potential future library additions could ease independently coding up similar logic.

Semantically a BigDecimal is a BigInteger significand/mantissa along with an int scale (negated exponent). If the scale/exponent is zero, then exact integer results can be computed. The BigDecimal implementation in the JDK internally often uses a long where possible to hold the significand value, failing over to allocating and operating on a BigInteger value when needed. The compact long representation is seamlessly supported in arithmetic and other operations. BigDecimal is not an ideal type for providing pure integer arithmetic since BigDecimal has many operations unneeded and inappropriate for that use and because each BigDecimal object has various fields other than the ones holding the integer value; however, the class comes with the JDK and may perform adequately out of the box.

The algorithms used for the long overflow checking in BigDecimal are adapted from the indispensable bit-twiddling tome Hacker's Delight. A few conference attendees expressed interest in an API around integer overflow detection. Represented as a Java API, the functionality might look like:

 \* Returns the sum of the arguments, throwing an ArithmeticException 
 \* if the exact sum is out of {@code int} range.
 \* @return the sum of the arguments, throwing an ArithmeticException 
 \* if the exact sum is out of {@code int} range.
 \* @throws ArithmeticException if the exact sum is out of {@code int} range
public static int addExact(int a, int b) throws ArithmeticException

The source code of such a method could use the Hacker's Delight techniques while a JVM could intrinsify the functionality to a processor-optimized idiom. There are some design choices even in this simple method. For example, instead of ArithmeticException, a new subclass of that exception called, say, IntegerOverflow, could be thrown instead. Such a subclass could offer the ability to store the original operands to facilitate continuing the computation in a wider type. (An unfriendly API design for the purposes at hand would be for addExact to have int parameters but return a java.lang.Integer, with null being used to indicate overflow occurred!)

While I don't recall it being explicitly mentioned at the conference, a piece of similar functionality is accessing the high-order 64 bits of a full 64 × 64 → 128 bit multiply (5100935).

Adding these low-level library operations to the platform would be a fun project!

Thursday Jul 22, 2010

Writing javac regression and unit tests for new language features

With Java language changes in progress for JDK 7, such as Project Coin's strings in switch, improved exception handling, and try-with-resources statement, writing effective regression and unit tests for javac is an integral component of developing the new language features.

Unit and regression tests differ from conformance tests. Unit and regression tests focus on achieving code coverage on an implementation while conformance tests focus on specification coverage, having tests that probe a high percentage of the testable assertions in the specification. The underlying functionality probed by both styles of testing is shared (source code is accepted or rejected, compiled class files do or do not implement the required semantics), but the different test methodologies differently categorize the space of concerns to be spanned. For more information on the methodology of conformance testing, see these blog entries about finding assertions in the Java language specification and on direct and derived assertions.

While there is overlap in the functionality testable by unit/regression tests and conformance tests, a key difference is that unit/regression tests can be written against implementation-specific functionality while conformance tests must be valid for any implementation (since all implementations are written against same specification). For example, a conformance test suite, like the JCK compiler test suite, can test that a particular source file is properly accepted or rejected by a compiler. Testing that the compiled class file(s) accurately implement the semantics of the source is also in-bounds for a conformance test. (The JCK test suite uses an interview process to configure the test suite to work with a particular compiler so that compiler-specific properties such as how acceptance/rejection of sources is indicated can be accommodated.) The JCK compiler test suite includes both positive tests (testing that programs that are in the language are properly accepted) and negative tests (testing that programs that are outside the language are properly rejected). Besides properties that can be verified in conformance-style testing, the regression and unit tests for a new language feature as implemented in javac also need to verify that various javac-specific properties hold.

For both positive and negative aspects of a language feature, unit and regression tests can cover all properties of interest for a particular compiler. A subset of those properties are also in the domain of conformance tests:

  • Negative Tests

    • Conformance and unit/regression: Invalid source files are rejected. This includes that sources only valid under, say, -source 7, are rejected when javac is run under -source 6 and earlier source settings.

    • Unit/regression only: The expected error messages are provided that reference the right source locations.

  • Positive Tests

    • Conformance and unit/regression: Valid source is compiled.

    • Conformance and/or unit/regression: Proper modeling of the new language construct. Depending on the language feature, the feature may be surfaced in the standard language model API javax.lang.model.\*, which can be tested with a conformance test. However, aspects of a language feature reflected in the javac tree API are only testable with regression or unit tests.

    • Conformance and unit/regressoin: Resulting class files are structurally well-formed. This includes passing verification and other checks done by the JVM upon loading and linking.

    • Unit/regression only: Resulting class files follow compiler-specific idioms. There are many ways a compiler can transform source code into valid class files which have the correct operational semantics.

    • Conformance and unit/regression: Resulting class file(s) when run have correct operational semantics. (Being able to run a class file implies the class file is well-formed.) Tests of operational semantics should be structured so that the code in question must positively run for the test to pass. For example, if a piece of code, like a string switch, was erroneously compiled to no byte codes, the test should fail.

An example of a negative test verifying both conformance and implementation-specific properties is a test included in the changeset for strings in switch:

 * @test  /nodynamiccopyright/
 * @bug 6827009
 * @summary Check for case labels of different types.
 * @compile/fail -source 6
 * @compile/fail/ref=BadlyTypedLabel1.out -XDrawDiagnostics
class BadlyTypedLabel1 {
    String m(String s) {
        switch(s) {
        case "Hello World":
        case 42:
            return ("Don't forget your towel!");

Decoding the initial comments as jtreg directives, @test indicates the file is a jtreg test; jtreg is the test harness used for JDK regression and unit tests. By default, jtreg builds the source file and run its main method; however, for compiler tests that combination of actions is often inappropriate. The directive
@compile/fail -source 6
means that jtreg should compile the indicated source and expect a failure, meaning the overall test will fail if the compile succeeds. In this case, the @compile/fail directive tests that a string switch is rejected under -source 6. The next directive
@compile/fail/ref=BadlyTypedLabel1.out -XDrawDiagnostics
is more specific. Not only must the compilation fail, but as specified in the ref= option to @compile/fail, the reported error messages must match the expected output in file BadlyTypedLabel1.out: compiler.err.prob.found.req: (compiler.misc.incompatible.types), int, java.lang.String
1 error 

The -XDrawDiagnostics option to javac turns on "raw diagnostics" where the source location and resource keys are output instead of the localized text of the error messages. (For more information about javac diagnostics see Improving javac diagnostics and Playing with formatters.) The effect of the second @compile/fail line is thus to verify the proper error message is generated at "case 42:" where an integer label erroneously occurs inside a string switch. Since this kind of test relies on checking messages reporting at particular code locations, no explicit copyright occurs in such test files so that the golden output files do not have to updated if the length of the copyright notice changes; "/nodynamiccopyright/" indicates the explicit copyright is omitted for this reason.

For positive tests, a @compile directive without /fail declares a compile must succeed for the overall test to pass. Modeling tests can generally be run as annotation processors over selected source code. Since annotation processing is built into javac as of JDK 6, @compile directives are one option for running annotation processing tests. The tree API can also be tested via annotation processors by using the Trees class to bridge between javax.lang.model.element.Element objects and the corresponding abstract syntax trees, a technique also used in some regression tests for ordinary javac bug fixes.

The regression tests for multi-catch include checks for generating idiomatic class files. There are a variety of ways the multi-catch aspect of improved exception handling could be implemented. One way to compile multi-catch would be to duplicate the code blocks for each exception, in source terms treating

try {...}
catch (A | B except) {Statements}


try {...}
catch (A except) {Statements}
catch (B except) {Statements}

However, for javac we do not consider this compilation strategy to result in an acceptable class file. (There are many implicit requirements for generating class files from Java source code, but generally no explicit specification for the compiler's behavior.) Instead, for javac we require catching the multiple exceptions to be represented in the class file as repeated entries in the table of exception ranges stored in the class file to map exceptions to their handling catch blocks. This check is implemented by using javap APIs to introspect on the structure of the exception table.

Testing operational semantics can be challenging due to the difficulty of computing a known-good result all code paths of interest. For strings in switch, testing the operational semantics of the generated code included comparing the control paths taken through a switch on an enum type with the control paths taken through an analogous switch on the names of the enum constants:

private static int 
enumSwitch(MetaSynVar msv) {
    int result = 0;
    switch(msv) {
    case FOO:
        result |= (1<<0);
        // fallthrough:

    case BAR:
    case BAZ:
        result |= (1<<1);

        switch(msv) {
        case QUX:
            result |= (1<<2);

        case QUUX:
            result |= (1<<3);

            result |= (1<<4);
        result |= (1<<5);

    case MUMBLE:
        result |= (1<<6);
        return result;

    case FOOBAR:
        result |= (1<<7);
    result |= (1<<8);
    return result;
private static int 
stringSwitch(String msvName) {
    int result = 0;
    switch(msvName) {
    case "FOO":
        result |= (1<<0);
        // fallthrough:

    case "BAR":
    case "BAZ":
        result |= (1<<1);

        switch(msvName) {
        case "QUX":
            result |= (1<<2);

        case "QUUX":
            result |= (1<<3);

            result |= (1<<4);
        result |= (1<<5);

    case "MUMBLE":
        result |= (1<<6);
        return result;

    case "FOOBAR":
        result |= (1<<7);
    result |= (1<<8);
    return result;

Matching code sections in the two switch statements are identified with the same bit position; executing a code section sets the corresponding bit. The test loops over all enum constants and verifies that the set of code sections run when the enum constant itself is switched on matches the set of code sections run when the name of the enum constant is switched on. Notice that if the string switch statement did nothing, the result would be set to 0, which would not match any proper run of the enum-based switch statement. Inside javac, enum switches and string switches have different implementations so checking the new string switch behavior against the old enum switch behavior is a reasonable approach to validate string switch functionality.

Analogous checks implemented with annotations were used to see if exceptions were caught in the proper catch blocks for improved exception handling.

Analogous approaches were employed to develop the unit and regression tests for try-with-resources.

Thanks to Alex, Maurizio, Jon, and Brian for comments on initial drafts of this entry.

Friday Jul 16, 2010

Project Coin ARM Implementation

I'm happy to announce that starting with a prototype written by Tom Ball, Oracle's javac team has produced and pushed an implementation of the try-with-resources statement, otherwise known as ARM blocks, into JDK 7.

Today the resourceful can apply a changeset to a copy of the JDK 7 langtools repository and do a build to get a compiler supporting this feature. Otherwise, following the integration process, support for try-with-resources statements will be available in the promoted JDK 7 builds in due course.

Besides possible refinements to the semantics of the translations, there will likely be other small adjustments to the implementation of the language feature in the future. For example, the lint category named "arm" will likely be renamed to something more like "resource."




« April 2014

No bookmarks in folder