Sunday Dec 08, 2013

JSR 269 Maintenance Review for Java SE 8

The annotation processing API, both the processor-specific portion of the API in javax.annotation.processing and the language modeling portions in javax.lang.model.*, are being updated to support the new language features in Java SE 8. Procedurally, the proposed changes are covered by the second maintenance review of JSR 269: Maintenance Draft Review 2.

As summarized on on the maintenance review page, there are three categories of changes from the version of the API shipped with Java SE 7:

  1. Cleaning up the existing specification without changing its semantics (adding missing javadoc tags, etc.)
  2. API changes to support the language changes being made in Project Lambda /JSR 335. These includes adding javax.lang.model.type.IntersectionType as well as javax.lang.model.element.ExecutableElement.isDefault.
  3. API changes to support the language changes being made under JSR 308, Annotations on Java Types. These include javax.lang.model.AnnotatedConstruct and updating javax.annotation.processing.Processor.

The small repeating annotations language change, discussed on an OpenJDK alias, is also supported by the proposed changes.

A detailed specification difference is available. Please post any comments here or send them to me through email.

Monday May 20, 2013

javax.lang.model backed by core reflection

Back when the javax.lang.model API was being designed as part of JSR 269, while the API was primarily intended for use at compile-time with annotation processing, the expert group also wanted the API to be usable in other contexts as well, including at runtime.

JEP 119, javax.lang.model Implementation Backed by Core Reflection, proposed adding such an alternative runtime implementation of javax.lang.model to JDK 8. Such an implementation has recently been pushed as sample code in the JDK 8 langtools repository.

At a high level, the sample code is at its core repeated uses of the adapter pattern, translating between the core reflection API and the javax.lang.model API. However, there were a number of design issues in what interface the javax.lang.model implementation for core reflection should expose. First, should a core reflection specialization of javax.lang.model expose a more specific interface? There are some advantages to just implementing the base API defined in the existing interfaces; it is the simplest approach and would appear to maximize the ability to inter-operate with other implementations. However, javax.lang.model depends on hidden state to define concepts like equality, so there is intrinsically limited inter-operation between disjoint implementations. Therefore, especially in sample code, it was viewed as worthwhile to experiment with a more specific API in the core reflection case.

To make the javax.lang.model backed by core reflection more specific, for each FooElement interface in javax.lang.model.element, a ReflectionFooElement subinterface was defined, as shown in the diagram below.

The subinterfaces are specialized in a few ways:

  • If a method in the base interface was defined to return a FooElement, a covariant override in the subinteface was defined to return a ReflectionFooElement.
  • If a method in the base interface was defined to return a List< ? extends FooElement>, a covariant override in the subinteface was defined to return a List<ReflectionFooElement>.
  • For getBar(FooElement arg) methods defined in the Elements helper interface, add a getBar() method to the ReflectionFooElement subinterface.
  • Add a getSource method to the root ReflectionElement interface to return the underlying core reflection object being adapted and add covariant overrides in subinterfaces as appropriate.
  • The root ReflectionElement interface implements the full AnnotatedElement interface, not just the subset of its methods found in the base javax.lang.model.element.

A cost of specializing the API is the need to define a new visitor interface as well. Covariant overrides cannot be used as in the element modeling interfaces since the visitor methods take elements in the argument position rather than the return position.

Generally, working on the core reflection specialization proceeded as expected. Equality determinations were generally delegated to core reflection source object; in other words, two ReflectionElement objects are equal if they are instances of the same interface and if their sources are .equals. Writing the sample code highlighted several shortcomings of the core reflection API which have been addressed in Java SE 8, including the addition of an Executable type to abstract over the commonalities of Method and Constructor. The sample code also benefited from other reflection changes in Java SE 8, such as the introduction of a java.lang.reflect.Parameter class and retrofitting TypeVariable to extend AnnotatedElement.

While significantly more testing would be needed to productize javax.lang.model backed by core reflection, even as sample code it validates a number of the technological decisions made in JSR 269 and is an interesting demonstration of how one of the platform's reflective APIs can be bridged to another.

Friday Feb 10, 2012

The passing of apt

With a pair of changesets pushed recently, the time for apt to be included in the JDK has drawn to a close, nearly eight years after first being added to the platform. In the Mythical Man-Month sense, apt was always planned to be the one we threw away, we just didn't realize how slow the windup and pitch would be!

The API, but not implementation, of apt were among the first components of the JDK to be released as open source. I learned a lot about technologies and project managment while working on apt, and it was quite satisfying to carry those lessons over to the "second system" of annotation procesing in JSR 269.

Friday Dec 02, 2011

An apt ending draws nigh

I brought you into this world, and I'll take you out!
—Cliff Huxtable

The end of an era draws nigh! After being deprecated in JDK 7, the apt command line tool and the entirely of its associated API is on track to be removed from JDK 8 within the next few months. While apt was fine back in JDK 5, the time has come to transition annotation processing to the superior standardized annotation processing provided by javax.annotation.processing and javax.lang.model.*. These packages were added to Java SE 6 under JSR 269.

This removal effort was discussed in JEP 117: Remove the Annotation-Processing Tool (apt).

Portions of jax-ws in the JDK use apt, but those portions are being rewritten to use the JSR 269 APIs. Once that revised version of jax-ws is being used by the JDK builds, apt will be excised in short order.

As a com.sun.* API, apt is not part of Java SE; it is just a component of the JDK and is thus easier to remove from the platform. While I was the lead in creating apt, lo these many years ago, I'm looking forward to deleting the code from the JDK to encourage use of a better replacement API and to ease maintenance of javac.

Monday Apr 25, 2011

JSR 269 Maintenance Review Concludes

The previously discussed maintenance review of JSR 269, after a slight extension, has concluded. Compared to the start of the cycle, the only change made during the review period was to rename the "disjunction"/"disjunctive" API elements to "union."

Tuesday Mar 22, 2011

Project Coin EclipseCon 2011

This afternoon at EclipseCon I gave a talk on Project Coin: Small Language Changes in JDK 7; the slides have been posted. An earlier talk at the conference gave a demo of the current Eclipse support for the Project Coin features; my talk included a demo of the "quick fix" hints to introduce Coin features in code provided by NetBeans 7.0 Beta 2.

Monday Mar 14, 2011

JSR 269 Maintenance Review

As a planned part of Java SE 7, JSR 269, which standardized an API for annotation processing, is now undergoing maintenance review. In the JCP, a maintenance review is a process to take comments on small changes so that those small changes can be formally incorporated into an existing specification without running a whole new JSR. The changes being proposed in the JSR 269 maintenance review are the changes already implemented in the JSR 269 APIs in JDK 7. In summary, those proposed changes are:

  • Clarified interaction between the Filer and rounds.

  • Constructors explicitly added to the kinds of elements that can be returned by RoundEnvironment.getElementsAnnotatedWith.

  • New enum constant javax.lang.model.SourceVersion.RELEASE_7.

  • In the package description of javax.lang.model.element, requirements on when a model must be provided are loosened to remove the requirement in case of an "irrecoverable error that could not be removed by the generation of new types," a condition which includes but is not limited to syntax errors.

  • New exception type javax.lang.model.UnknownEntityException added as a common superclass for existing exception types UnknownAnnotationValueException, UnknownElementException, and UnknownTypeException.

  • New enum constant javax.lang.model.element.ElementKind.RESOURCE_VARIABLE.

  • New mixin interfaces Parameterizable and QualifiedNameable added to package javax.lang.model.element. ExecutableElement and TypeElement are retrofitted to extend Parameterizable; PackageElementand TypeElement are retrofitted to extend QualifiedNameable.

  • Behavior of getEnclosingElement method defined to return the generic element of a type parameter instead of null.

  • New interface javax.lang.model.type.DisjunctiveType to model disjunctive types.

  • New enum constant javax.lang.model.type.TypeKind.DISJUNCTIVE to mark disjunctive types.

  • New method visitDisjunctive added to visitor interface javax.lang.model.type.TypeVisitor. Utility visitor implementations updated accordingly.

  • In the package javax.lang.model.type, MirroredTypesException retrofitted to be the superclass of MirroredTypeException.

  • New utility visitors for release 7 in package javax.lang.model.util:

    • AbstractAnnotationValueVisitor7

    • AbstractElementVisitor7

    • AbstractTypeVisitor7

    • ElementKindVisitor7

    • ElementScanner7

    • SimpleAnnotationValueVisitor7

    • SimpleElementVisitor7

    • SimpleTypeVisitor7

    • TypeKindVisitor7

  • The visitors ElementKindVisitor6, ElementScanner6, and SimpleElementVisitor6, are updated to account for new element kind RESOURCE_VARIABLE.

  • The visitor AbstractTypeVisitor6 is updated to account for the possibility of visiting a DisjunctiveType.

  • Definition of documentation comment added to javadoc of javax.lang.model.util.Elements.getDocComment.

Monday Nov 15, 2010

Original apt API files

With the transition of the site to a new infrastructure, the time has come to retire the long-hosted but dormant apt mirror API project, The old source bundles in the project are just the JDK 5 apt API; the implementation was not included. This apt project predated OpenJDK; with OpenJDK, the apt API and implementation (and much, much more!) has been available under an open source license for several years. Additionally, annotation processing should transition away from the first-generation apt to the much-improved and standardized JSR 269 annotation processing available since JDK 6 and supported directly in javac. Moreover, the apt tool and API have been deprecated as of JDK 7.

Purely for historical interest, I'm placing an archive of the apt source bundles on this blog:

Friday Mar 12, 2010

Last Round Compiling

As of build 85 of JDK 7, bug 6634138 "Source generated in last round not compiled" has been fixed in javac. Previously, source code generated in a round of annotation processing where RoundEnvironment.processingOver() was true was not compiled. With the fix, source generated in the last round is compiled, but, as intended, while compiled such source still does not undergo annotation processing since processing is over. The fix has also been applied to OpenJDK 6 build 19.

Annotation Processor SourceVersion

In annotation processing there are three distinct roles, the author of the annotation types, the author of the annotation processor, and the client of the annotations. The third role includes the responsibility to configure the compiler correctly, such as setting the source, target, and encoding options and setting the source and class file destination for annotation processing. The author of the annotation processor shares a related responsibility: property returning the source version supported by the processor.

Most processors can be written against a particular source version and always return that source version, such as by including a @SupportedSourceVersion annotation on the processor class. In principle, the annotation processing infrastructure could tailor the view of newer-than-supported language constructs to be more compatible with existing processors. Conversely, processors have the flexibility to implement their own policies when encountering objects representing newer-than-supported structures. In brief, by extending version-specific abstract visitor classes, such as AbstractElementVisitor6 and AbstractTypeVisitor6, the visitUnknown method will be called on entities newer than the version in question.

Just as regression tests inside the JDK itself should by default follow a dual policy of accepting the default source and target settings rather than setting them explicitly like other programs, annotation processors used for testing with the JDK should generally support the latest source version and not be constrained to a particular version. This allows any issues or unexpected interactions of new features to be found more quickly and keeps the regression tests exercising the most recent code paths in the compiler.

This dual policy is now consistently implemented in the langtools regression tests as of build 85 of JDK 7 (6926699).

Wednesday Sep 02, 2009

Properties via Annotation Processing

The annotation processing APIs provided by apt in JDK 5 and javac in JDK 6 both present a read-only view of source code; by design directly modifying the input sources is not supported through either annotation processing API. Recently, Project Lombok has used the hook of being able to run an annotation processor inside javac to start an agent which rewrites javac internal classes to change the compiler's behavior; yikes! Such extreme measures are not needed to get much of the effect of modifying the input sources. As I've outlined in the annotation processing forum over the years, just using standard annotation processing to generate a subclass or superclass of a type being processed is a very powerful technique for controlling the ultimate semantics and behavior of the original class. For example, I've hacked up a proof-of-concept annotation type and matching processor to generate property-style getter and setter methods based on annotations on fields.

Concretely, the programmer writes something like

public class TestProperty extends TestPropertyParent {
    protected TestProperty() {}

    protected int property1;

    @ProofOfConceptProperty(readOnly = false) // Generate a setter too.
    protected long property2;

    protected double property3;
    public static TestProperty newInstance(int property1,
                                           long property2,
                                           double property3) {
        return new TestPropertyChild(property1, property2, property3);

and, after suitable annotation processing, using the TestProperty type as in

public class Main {
    public static void main(String... args) {
        TestProperty testProperty = TestProperty.newInstance(1, 2, 3);

    private static void output(TestProperty testProperty) {
        System.out.format("%d, %d, %g%n",

produces the expected output:

prompt$ java Main 
1, 2, 3.00000
1, 42, 3.00000

This approach does have limitations; primarily the annotated class like TestProperty needs to be written to allow its superclass and subclass(es) to be generated. Since it runs as part of the build, the annotation processor needs to be built separately beforehand. The javac command to run the annotation processor looks like:

javac -s ../gen_src/ -d ../bin -processor foo.PocProcessor -cp ../lib 

Good practice sets an output location for generated source code, TestPropertyParent and TestPropertyChild in this case, separate from the output location for class files. When compiling this, javac emits an error before the superclass and subclass are generated, but the entire compilation process completes fine correctly generating the source files and compiling all the files together. Java IDEs have varying levels of support for annotation processing; check your IDEs' documentation for details.

The annotation type and processor is only a proof of concept; many possible refinements are left as "exercises for the reader" including:

  • Developing a second annotation type to mark a class separate from the annotation to configure how each field should be treated.

  • Additional structural checks on the annotated code, proper modifiers on fields and constructors, etc.

  • Generation of equals and hashCode methods.

While using an annotation processor to approximate properties is awkward compared to built-in language support, annotation processors can be used today as part of some toolchains and they are configurable by the user. The code provided should be enough of a starting part for others to experiment with using annotation processors in this fashion; have fun.

Friday Aug 07, 2009

Annotation Processing Build Advice: Set source and class file destinations

In addition to following the general build advice of setting source, target, and encoding, when running annotation processors the source and class file destination directories should be set via javac's -s directory and -d directory options, respectively. Using the Filer, annotation processors can generate new source code to be compiled as part of the project and can also even generate new class files directly.

As recommended on the annotation processing forum, a much more hygienic build results when different kinds of generated output files are not intermixed with each other and any kind of output file is not mixed with any input files. (Input files would presumably be tracked under version control where generated files would not be.) In more detail:

  • Set the directory for outputting generated source files using -s

  • Set the directory for outputting generated class file code using -d

  • Do not have the source and class file output locations overlap with each other and do not have either of them overlap with the sourcepath or classpath.

Following these guidelines clearly delineates inputs to the compiler and outputs from the compiler.

Although apt has been deprecated, the same recommendations hold there too.

Monday Jul 27, 2009

An apt replacement

Showing that no rule is so broad as to not admit an exception, the apt tool and its associated API, com.sun.mirror.\*, are being deprecated in JDK 7. The plan is to remove the tool and its API to the next major JDK release after JDK 7.

As a com.sun.\* API, the apt API is not governed by the JCP; however, we don't usually mass-deprecate com.sun.\* APIs either. We introduced apt in JDK 5 to gain experience with annotation processing before standardizing this facility with JSR 269 in JDK 6, which added the javax.annotation.processing and javax.lang.model.\* packages and added annotation processing options to javac.

As the lead engineer for both apt and JSR 269, the JSR 269 API and tool experience should be uniformly better than apt; the newer API is easier to use, more flexible, and should run faster as well. I unconditionally recommend transitioning to the JSR 269 API and javac (or its compiler API) for all your annotation processing needs.

The table below summarizes the apt deprecation information.

Summary of deprecated apt API replacements
apt type Standard Replacement
Package com.sun.mirror.apt
AnnotationProcessor javax.annotation.processing.Processor
AnnotationProcessorEnvironment javax.annotation.processing.ProcessingEnvironment
AnnotationProcessorFactory javax.annotation.processing.Processor
AnnotationProcessorListener No analog.
AnnotationProcessors No analog.
Filer javax.annotation.processing.Filer
Messager javax.annotation.processing.Messager
RoundCompleteEvent No analog.
RoundCompleteListener No analog.
RoundState javax.annotation.processing.RoundEnvironment
Package com.sun.mirror.declaration
AnnotationMirror javax.lang.model.element.AnnotationMirror
AnnotationTypeDeclaration javax.lang.model.element.TypeElement
AnnotationTypeElementDeclaration javax.lang.model.element.ExecutableElement
AnnotationValue javax.lang.model.element.AnnotationValue
ClassDeclaration javax.lang.model.element.TypeElement
ConstructorDeclaration javax.lang.model.element.ExecutableElement
Declaration javax.lang.model.element.Element
EnumConstantDeclaration javax.lang.model.element.VariableElement
EnumDeclaration javax.lang.model.element.TypeElement
ExecutableDeclaration javax.lang.model.element.ExecutableElement
FieldDeclaration javax.lang.model.element.VariableElement
InterfaceDeclaration javax.lang.model.element.TypeElement
MemberDeclaration javax.lang.model.element.Element
MethodDeclaration javax.lang.model.element.ExecutableElement
Modifier javax.lang.model.element.Modifier
PackageDeclaration javax.lang.model.element.PackageElement
ParameterDeclaration javax.lang.model.element.VariableElement
TypeDeclaration javax.lang.model.element.TypeElement
TypeParameterDeclaration javax.lang.model.element.TypeParameterElement
Package com.sun.mirror.type
AnnotationType javax.lang.model.type.DeclaredType
ArrayType javax.lang.model.type.ArrayType
ClassType javax.lang.model.type.DeclaredType
DeclaredType javax.lang.model.type.DeclaredType
EnumType javax.lang.model.type.DeclaredType
InterfaceType javax.lang.model.type.DeclaredType
MirroredTypeException javax.lang.model.type.MirroredTypeException
MirroredTypesException javax.lang.model.type.MirroredTypesException
PrimitiveType javax.lang.model.type.PrimitiveType
PrimitiveType.Kind javax.lang.model.type.TypeKind
ReferenceType javax.lang.model.type.ReferenceType
TypeMirror javax.lang.model.type.TypeMirror
TypeVariable javax.lang.model.type.TypeVariable
VoidType javax.lang.model.type.NoType
WildcardType javax.lang.model.type.WildcardType
Package com.sun.mirror.util
DeclarationFilter javax.lang.model.util.ElementFilter
DeclarationScanner javax.lang.model.util.ElementScanner6
DeclarationVisitor javax.lang.model.element.ElementVisitor
DeclarationVisitors No replacement.
Declarations javax.lang.model.util.Elements
SimpleDeclarationVisitor javax.lang.model.util.SimpleElementVisitor6
SimpleTypeVisitor javax.lang.model.util.SimpleTypeVisitor6
SourceOrderDeclScanner javax.lang.model.util.SimpleElementVisitor6
SourcePosition No replacement.
TypeVisitor javax.lang.model.element.TypeVisitor
Types javax.lang.model.util.Types

Thursday Mar 12, 2009

Language Model Changes as of JDK 7 Build 50

To date, there have been a few API changes to javax.lang.model.\* in JDK 7. Early in the release, SourceVersion.RELEASE_7 was added to correspond to any new source-level changes coming in JDK 7 (6458819). Eventually, there will be changes to support the modulue construct being added by JSR 294; changes may or may not be needed for language changes coming from Project Coin and JSR 308. Once modulues are added, the JSR 269 API elements meant to cope with the expression problem will be tested. In the mean time, some minor API cleanups have occurred, one to make handling exceptions easier (6794071), another to group common functionality in mixin interfaces (6460529), and some minor API clarifications (6501749). Along the way, there has been a little general bug fixing too (6478017, 6583626, 6498938).

Small tweaks and bug fixing will continue to occur in the javax.lang.model.\* and javax.annotation.processing packages throughout JDK 7 in addition to changes needed to support new language features.

Tuesday Nov 07, 2006

JSR 269 Passes Final Approval Ballot

The votes are in and JSR 269 has passed its final approval ballot!

Now that the initial JSR 269 API is defined, I'm looking forward to using the API for various meta-programming tasks, like writing coding convention checkers and incrementally refining the API in JDK 7.

Tuesday Oct 24, 2006

JSR 269 in Final Approval Ballot

I'm very pleased to report that JSR 269 has reached the final approval ballot stage of the JCP process.

The only change since the proposed final draft was to correct the specification of VariableElement.getConstantValue.

JDK 6 Build 102 JSR 269 API Changes

Build 102 included a very small change to the JSR 269 API: the specification of the VariableElement.getConstantValue method was changed so that a value was returned independent of the static-ness of a final field. The implementation already behaved that way; the specification mistakenly excluded the non-static case.

This is the last planned change to the JSR 269 API before its final approval ballot.

Tuesday Oct 10, 2006

JSR 269 Interview on Artima

Artima has an interview with me about JSR 269.

Thursday Oct 05, 2006

JDK 6 Build 101 JSR 269 API Changes

There haven't been any API changes in JSR 269 since build 98 and the proposed final draft; however, build 101 includes a sample annotation processor showing how the API can be used to check program structure.

The sample file is under

from the root JDK installation directory.

As the name suggests, this processor checks that the source being processed follows the naming conventions discussed in the language specification. This name checking is meant to be representative of a class of properties that could be checked for at build time by annotation processors.

A few stylistic points to note in the sample:

  • Extending AbstractProcessor: The AbstractProcessor class is provided to make writing annotation processors more convenient by eliminating repeated boilerplate code. Generally, the results of the getSupportedFoo methods can be specified by applying a @SupportedFoo annotation to the class. The @Override annotations help ensure the methods of the class are behaving as expected. To initialize fields for objects that will persist for life of the processor object, such as the Messager and Filer retrieved from the ProcessingEnvironment, assign the fields in the overridden init method after super.init has been called.
  • Implement functionality with a visitor, but expose the functionality with its own entry point: The NameChecker class provides a checkNames method for clients to pass an element to get its names checked. While this checking is implemented using a visitor, exposing that implementation detail is not friendly to clients.
  • Have a policy for future language versions: The JSR 269 API models the current language, but the language will be changing in some future release and annotation processors written today could be run against new language constructs of the future. Two basic policies are:
    1. Write the processor to only work with the current language version.
    2. Write the processor to cope with unknown future constructs.
    The sample processor does the latter. It does this by returning SourceVersion.latest rather than a particular version constant and by having the visitor's visitUnknown method be non-fatal if called. The visitUnknown method will be called when encountering a construct newer than the platform version the visitor was written to.

For production use, the heuristicallyConstant method probably needs some tweaking; leave comments detailing any changes you find useful.

Monday Sep 18, 2006

JSR 269 in proposed final draft

I'm happy to announce that JSR 269 has progressed to the proposed final draft stage of the JCP process. This draft corresponds to the version of JSR 269 implemented in build 98 of JDK 6. Please send comments to

Changes from the public review draft include:

  • Adding originating elements var-args parameters to the Filer create methods to enable better management of dependencies.
  • Changing the return type of the getFooName methods to a separate Name extends CharSequence interface.
  • Revised Type visitor structure
  • Included a description of the annotation processing discovery process.
  • Various specification clarifications:
    • More explicit null pointer defaults.
    • The modeling API is meant to be used for multiple purposes, including but not limited to annotation processing.
    • More notes on anticipated evolution of the API.



« April 2014

No bookmarks in folder