JSR 199 meets JSR 203

The Compiler API, JSR 199, added in JDK 6, provides the ability to invoke a Java compiler via an API. The API included a simple abstraction, JavaFileManager and related classes, for the way that the compiler and similar tools read and write files. Along with JavaFileManager, the Compiler API provides an interface, StandardJavaFileManager, for a implementation using the standard file system APIs in java.io.\* and java.util.zip.\*. Clients of the Compiler API can either obtain and use a StandardJavaFileManager, or they can provide their own implementation of a JavaFileManager, enabling the compiler to access custom storage, such as the in-memory storage used by an IDE.

Now, in JDK 7, there is a new feature, More New I/O APIs for the Java Platform, JSR 203, which provides a much richer file system abstraction. How do these new APIs relate to the features in the Compiler API?

To find out, this past week I've put together an implementation of the JavaFileManager API, that bridges to the new NIO APIs. This means that anyone who uses or provides a filesystem using the java.nio.file.\* APIs can now use those same APIs in conjunction with the Compiler API. Since the new NIO APIs provide uniform access to a variety of file systems, such as the standard file system, zip files, and user-provided file systems, this is a very attractive way to use the Compiler API as well.

Implementation Details

Just as the main class for accessing the standard file system is java.io.File, its analog in the new NIO world is java.nio.file.Path. So, to start working on the new file manager, I took a copy of StandardFileManager and at least conceptually, replaced all occurrences of File with Path, and called the resulting interface PathFileManager. Eventually, I trimmed the class down some, and added a few more methods: more on that later. Next up, I created a simple new implementation of FileObject that wraps a Path object, and so naturally enough is called PathFileObject. Together, these provide enough to start filling in the "obvious" implementations. For the most part, the implementations either call through to the underlying new NIO API, or use cut-n-paste code from the existing javac file manager. (Eventually, I'll move the cut n paste code into a utility superclass that I can share between the two file managers.)

One of the abstractions in the Compiler API is a Location, which abstractly represents search paths, such as the class path, source path and so on. These are easily modeled as a map of a Location to a list of Paths. The code to initialize default values for these search paths -- most notably the platform class path, or bootclasspath -- uses the existing code in javac.

The biggest non-trivial method to implement in the new file manager was the list method. This takes a Location, and parameters such as a package name, the types of files of interest, and whether to recurse into subpackages. The method iterates over the contents of the Location, searching each entry in turn. Search paths may contain both directories and jar files. Searching a directory is relatively obvious, and with new new NIO API, handling jar files is almost as easy. If the entry on a search path is a file, the file manager attempts to open it as a FileSystem, using FileSystems.newFileSystem This is the new standard way to open files like zip and jar files. If the file manager successfully opens the file as a filesystem, it gets the root directory for the file system, and simply searches in that directory, using the same code as if it were a regular directory.

The other non-trivial method is inferBinaryName. This is a method to guess the binary name for a JavaFileObject without having to open it. The guess is based on the name of the file object. For an entry in a jar file, you can use the entry name as the basis for the binary name. For an entry in a directory, you have to scan the paths for a given Location and see if the file in question is in a directory on the search path, and use the directory-relative name as the basis for the binary name. There may be more information available, depending on how the file object was created. For all these reasons, the new file manager's inferBinaryName method delegates to an abstract method on PathFileObject, and different anonymous subtypes of this class provide different and appropriate implementations of inferBinaryName, depending on the information available at the time the file object is created.

What other methods appear on PathFileManager? The first is a method to get the underlying Path for any file object returned by the PathFileManager. This is useful for any tools that want to use PathFileManager, and which need more operations (such as length(), exists(), etc) than those provided by the basic JavaFileManager API. In addition, there are methods on PathFileManager to get and set a default FileSystem for the file manager to use. Currently, this is used by javac as the file system in which files are created when no specific location has been set, such as with the -d or -s options.

What Next?

The new code cannot trivially be added to javac, because of the standard bootstrap issues: since javac is used to build JDK, it cannot itself use any features that are new to that version of JDK, including any new API. So we need to figure out how to make file manager available. By itself, this is not a big deal, but it would be nice to try and tie it into the existing javax.tools.\* API as well, if possible.

Further out, there have been requests over the years to have javac write class files directly to a jar file. There are also known performance issues with the existing java.util.zip.\* and java.util.jar.\* classes. Currently, the new NIO libraries do not come with "out of the box" support for zip files as FileSystems, although there is a demo library that works well enough for read only access by javac, provided the library is built and placed on the javac classpath. It would be interesting to see if some zip file expert could provide a fast new zip library via the new NIO interfaces. This would not only benefit JDK users in general, but it would also mean that javac would immediately benefit as well, by way of this new file manager.

See Also

Thanks to Alan Bateman for answering all my questions about the new NIO API, and to Joe Darcy for feedback on this note.


There was a long standing feature request against javac to no longer require "listing". In-process compilation typically doesn't want to compile against files but actually against a classloader's set of classes in the current VM (e.g. the JSP compiler needs to compile against the webapp CL).

Typically you are forced to assume that class.getClassLoader() is actually a URLClassLoader and then convert the urls into file system locations. This does not work reliably across J2EE containers.

Just wondering if you stumbled across that/considered changing that aspect of Javac.

Posted by Matthias on September 25, 2009 at 04:54 AM PDT #

@Matthias: The feature request is still open, and under consideration. We cannot solve the general case, because the implicit ability to list the contents of a package is exposed to annotation processors via the javax.lang.model (JSR 269) API. Setting that aside, the work reduces to reimplementing the way the compiler handles imports, and the related scope machinery, which is not a small amount of work.

Posted by Jonathan Gibbons on September 25, 2009 at 07:04 AM PDT #

Post a Comment:
Comments are closed for this entry.

Discussions about technical life in my part of the OpenJDK world.


« April 2014