Writing javac regression and unit tests for new language features

With Java language changes in progress for JDK 7, such as Project Coin's strings in switch, improved exception handling, and try-with-resources statement, writing effective regression and unit tests for javac is an integral component of developing the new language features.

Unit and regression tests differ from conformance tests. Unit and regression tests focus on achieving code coverage on an implementation while conformance tests focus on specification coverage, having tests that probe a high percentage of the testable assertions in the specification. The underlying functionality probed by both styles of testing is shared (source code is accepted or rejected, compiled class files do or do not implement the required semantics), but the different test methodologies differently categorize the space of concerns to be spanned. For more information on the methodology of conformance testing, see these blog entries about finding assertions in the Java language specification and on direct and derived assertions.

While there is overlap in the functionality testable by unit/regression tests and conformance tests, a key difference is that unit/regression tests can be written against implementation-specific functionality while conformance tests must be valid for any implementation (since all implementations are written against same specification). For example, a conformance test suite, like the JCK compiler test suite, can test that a particular source file is properly accepted or rejected by a compiler. Testing that the compiled class file(s) accurately implement the semantics of the source is also in-bounds for a conformance test. (The JCK test suite uses an interview process to configure the test suite to work with a particular compiler so that compiler-specific properties such as how acceptance/rejection of sources is indicated can be accommodated.) The JCK compiler test suite includes both positive tests (testing that programs that are in the language are properly accepted) and negative tests (testing that programs that are outside the language are properly rejected). Besides properties that can be verified in conformance-style testing, the regression and unit tests for a new language feature as implemented in javac also need to verify that various javac-specific properties hold.

For both positive and negative aspects of a language feature, unit and regression tests can cover all properties of interest for a particular compiler. A subset of those properties are also in the domain of conformance tests:

  • Negative Tests

    • Conformance and unit/regression: Invalid source files are rejected. This includes that sources only valid under, say, -source 7, are rejected when javac is run under -source 6 and earlier source settings.

    • Unit/regression only: The expected error messages are provided that reference the right source locations.

  • Positive Tests

    • Conformance and unit/regression: Valid source is compiled.

    • Conformance and/or unit/regression: Proper modeling of the new language construct. Depending on the language feature, the feature may be surfaced in the standard language model API javax.lang.model.\*, which can be tested with a conformance test. However, aspects of a language feature reflected in the javac tree API are only testable with regression or unit tests.

    • Conformance and unit/regressoin: Resulting class files are structurally well-formed. This includes passing verification and other checks done by the JVM upon loading and linking.

    • Unit/regression only: Resulting class files follow compiler-specific idioms. There are many ways a compiler can transform source code into valid class files which have the correct operational semantics.

    • Conformance and unit/regression: Resulting class file(s) when run have correct operational semantics. (Being able to run a class file implies the class file is well-formed.) Tests of operational semantics should be structured so that the code in question must positively run for the test to pass. For example, if a piece of code, like a string switch, was erroneously compiled to no byte codes, the test should fail.

An example of a negative test verifying both conformance and implementation-specific properties is a test included in the changeset for strings in switch:

/*
 * @test  /nodynamiccopyright/
 * @bug 6827009
 * @summary Check for case labels of different types.
 * @compile/fail -source 6 BadlyTypedLabel1.java
 * @compile/fail/ref=BadlyTypedLabel1.out -XDrawDiagnostics BadlyTypedLabel1.java
 */
class BadlyTypedLabel1 {
    String m(String s) {
        switch(s) {
        case "Hello World":
            return(s);
        case 42:
            return ("Don't forget your towel!");
        }
    }
}

Decoding the initial comments as jtreg directives, @test indicates the file is a jtreg test; jtreg is the test harness used for JDK regression and unit tests. By default, jtreg builds the source file and run its main method; however, for compiler tests that combination of actions is often inappropriate. The directive
@compile/fail -source 6 BadlyTypedLabel1.java
means that jtreg should compile the indicated source and expect a failure, meaning the overall test will fail if the compile succeeds. In this case, the @compile/fail directive tests that a string switch is rejected under -source 6. The next directive
@compile/fail/ref=BadlyTypedLabel1.out -XDrawDiagnostics BadlyTypedLabel1.java
is more specific. Not only must the compilation fail, but as specified in the ref= option to @compile/fail, the reported error messages must match the expected output in file BadlyTypedLabel1.out:

BadlyTypedLabel1.java:13:14: compiler.err.prob.found.req: (compiler.misc.incompatible.types), int, java.lang.String
1 error 

The -XDrawDiagnostics option to javac turns on "raw diagnostics" where the source location and resource keys are output instead of the localized text of the error messages. (For more information about javac diagnostics see Improving javac diagnostics and Playing with formatters.) The effect of the second @compile/fail line is thus to verify the proper error message is generated at "case 42:" where an integer label erroneously occurs inside a string switch. Since this kind of test relies on checking messages reporting at particular code locations, no explicit copyright occurs in such test files so that the golden output files do not have to updated if the length of the copyright notice changes; "/nodynamiccopyright/" indicates the explicit copyright is omitted for this reason.

For positive tests, a @compile directive without /fail declares a compile must succeed for the overall test to pass. Modeling tests can generally be run as annotation processors over selected source code. Since annotation processing is built into javac as of JDK 6, @compile directives are one option for running annotation processing tests. The tree API can also be tested via annotation processors by using the Trees class to bridge between javax.lang.model.element.Element objects and the corresponding abstract syntax trees, a technique also used in some regression tests for ordinary javac bug fixes.

The regression tests for multi-catch include checks for generating idiomatic class files. There are a variety of ways the multi-catch aspect of improved exception handling could be implemented. One way to compile multi-catch would be to duplicate the code blocks for each exception, in source terms treating

try {...}
catch (A | B except) {Statements}

as

try {...}
catch (A except) {Statements}
catch (B except) {Statements}

However, for javac we do not consider this compilation strategy to result in an acceptable class file. (There are many implicit requirements for generating class files from Java source code, but generally no explicit specification for the compiler's behavior.) Instead, for javac we require catching the multiple exceptions to be represented in the class file as repeated entries in the table of exception ranges stored in the class file to map exceptions to their handling catch blocks. This check is implemented by using javap APIs to introspect on the structure of the exception table.

Testing operational semantics can be challenging due to the difficulty of computing a known-good result all code paths of interest. For strings in switch, testing the operational semantics of the generated code included comparing the control paths taken through a switch on an enum type with the control paths taken through an analogous switch on the names of the enum constants:

private static int 
enumSwitch(MetaSynVar msv) {
    int result = 0;
    switch(msv) {
    case FOO:
        result |= (1<<0);
        // fallthrough:

    case BAR:
    case BAZ:
        result |= (1<<1);
        break;

    default:
        switch(msv) {
        case QUX:
            result |= (1<<2);
            break;

        case QUUX:
            result |= (1<<3);

        default:
            result |= (1<<4);
        }
        result |= (1<<5);
        break;

    case MUMBLE:
        result |= (1<<6);
        return result;

    case FOOBAR:
        result |= (1<<7);
        break;
    }
    result |= (1<<8);
    return result;
}
private static int 
stringSwitch(String msvName) {
    int result = 0;
    switch(msvName) {
    case "FOO":
        result |= (1<<0);
        // fallthrough:

    case "BAR":
    case "BAZ":
        result |= (1<<1);
        break;

    default:
        switch(msvName) {
        case "QUX":
            result |= (1<<2);
            break;

        case "QUUX":
            result |= (1<<3);

        default:
            result |= (1<<4);
        }
        result |= (1<<5);
        break;

    case "MUMBLE":
        result |= (1<<6);
        return result;

    case "FOOBAR":
        result |= (1<<7);
        break;
    }
    result |= (1<<8);
    return result;
}

Matching code sections in the two switch statements are identified with the same bit position; executing a code section sets the corresponding bit. The test loops over all enum constants and verifies that the set of code sections run when the enum constant itself is switched on matches the set of code sections run when the name of the enum constant is switched on. Notice that if the string switch statement did nothing, the result would be set to 0, which would not match any proper run of the enum-based switch statement. Inside javac, enum switches and string switches have different implementations so checking the new string switch behavior against the old enum switch behavior is a reasonable approach to validate string switch functionality.

Analogous checks implemented with annotations were used to see if exceptions were caught in the proper catch blocks for improved exception handling.

Analogous approaches were employed to develop the unit and regression tests for try-with-resources.

Thanks to Alex, Maurizio, Jon, and Brian for comments on initial drafts of this entry.

Comments:

Post a Comment:
Comments are closed for this entry.
About

darcy

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today
News

No bookmarks in folder

Blogroll