Friday Jul 20, 2012

Fortress Wrapping Up

After working nearly a decade on the design, development, and implementation of the Fortress programming language, the Oracle Labs Programming Language Research Group is now winding down the Fortress project. Ten years is a remarkably long run for an industrial research project (one to three years is much more typical), but we feel that our extended effort has been worthwhile. Many aspects of the Fortress design were novel, and we learned a great deal from building an interpreter and an initial set of libraries. Nevertheless, over the last few years, as we have focused on implementing a compiler targeted to the Java Virtual Machine, we encountered some severe technical challenges having to do with the mismatch between the (rather ambitious) Fortress type system and a virtual machine not designed to support it (that would be every currently available VM, not just JVM). In addressing these challenges, we learned a lot about the implications of the Fortress type system for the implementation of symmetric multimethod dispatch, and have concluded that we are now unlikely to learn more (in a research sense) from completing the implementation of Fortress for JVM. We also note that, over the last ten years, other languages (Chapel, X10, Clojure, and Scala, among others) have explored some of the same issues that Fortress has addressed, and we have very much enjoyed conversations, collaboration, and friendly competition with those who have explored these ideas in alternative contexts.

The Fortress source code remains open-source, and the code repository will remain available for the foreseeable future to those who wish to work on it. The Programming Language Research Group will continue to respond to inquiries about Fortress (and requests for minor bug fixes as we can). We not do not plan to cease work on Fortress suddenly, but will spend the next few months getting the code and language specification into the best shape that we can. We also plan to write several academic papers about various aspects of the Fortress design and implementation technology. Going forward, we will look for opportunities to apply lessons learned and to transfer Fortress-related technology to other projects.

We thank everyone who has expressed an interest in or support for Fortress, especially DARPA (which funded in part some of the early work as part of its High Productivity Computing Program) and its HPCS review committees; former Programming Language Research Group team members and the many interns who contributed to the design and implementation effort; the management of Sun Microsystems Laboratories and Oracle Labs; and those who became part of the open-source effort, downloading the implementation, trying it out, and asking questions that kept us on our toes.

Here are some of the aspects of Fortress with which we are quite pleased, with commentary:

Generators and reducers
This has turned out to be a powerful way to organize collections classes and their use. It is related to the idea of "map-reduce" but is a bit more general. Moreover, comprehension and "big operator" syntax allows programmers to write code that sometimes focuses on a single instance rather than having to adopt the always-global point of view required by the use of higher-order operators such as map and reduce; this is often a useful stylistic alternative.

Implicit parallelism supported by work-stealing
We have found work-stealing to be an effective implementation mechanism for parallelism on shared-memory processor clusters, and especially for a style of programming in which the compiler can implicitly extract parallelism because function arguments and tuple elements are by definition assumed to be computationally independent.

Nested atomic blocks supported by transactional memory
We believe that this is a powerful and expressive alternative to locks for expressing synchronization among threads in a shared-memory environment.

Parametrically polymorphic types that are not erased
There are advantages (we're talking performance here, not just aesthetics) to being able to inquire, dynamically and cheaply, what is the element type of a collection in order to select an appropriate implementation.

Symmetric multimethod dispatch and parametrically polymorphic methods
Avoiding the need for the "visitor pattern" makes code more flexible and easier to extend; moreover, delegating dynamic dispatch on arguments other than the receiver object to the runtime allows greater opportunity for dynamic code optimization (an area that deserves further exploration). We made excellent headway on elucidating the interactions of this feature with polymorphic types and polymorphic methods.

Multiple inheritance, inheritance symmetry, and type exclusion
Like most modern object-oriented programming languages, Fortress supports multiple inheritance among object types. Unlike most other object-oriented programming languages, in Fortress inheritance is completely symmetric. This led to a need for the programmer sometimes to indicate explicitly when two types are intended to be disjoint (that is, no object can belong to both), and constitutes an interesting and unusual aspect of the Fortress type system.

Mathematical syntax
The idea of "typesetting" the code using LaTeX has received mixed reviews, and many potential users find the issue of keyboarding the extended Unicode character set daunting. Nevertheless, even in its more verbose ASCII form, the syntax of Fortress, especially the notion of treating juxtaposition as a user-definable operation, has proved to be very convenient.

Components and APIs
The Fortress component/API system allows control over namespaces in an unusually fine-grained manner; in particular, we figured out how to allow exporting only part of a set of overloaded functions or methods while correctly maintaining their execution semantics and supporting separate compilation.

Here are some aspects we wish we had been able to explore further (and may yet, in some future project):
Dimensions and units
We designed a way to write such expressions as "2 meters per second" and "15.7 kg" and have them fit smoothly into the rest of the type system and mathematical syntax—still a good idea that has yet to reach fruition.

Explicit descriptions of data distribution and processor assignment
Parallel programming languages still lack a good way to represent complex hierarchical hardware resources so as to allow programs to depend parametrically on these resources in even a moderately portable fashion. We had a paper design for this but did no implementation work.

Conditional inheritance and conditional method definition
We wanted to express such ideas as "a list of elements of type T implements a (lexicographic) total order provided that T implements a total order" and "the binary OPLUS operator applies element-wise to vectors provided that OPLUS is a binary operator on the vector elements, and moreover vector OPLUS is associative (or commutative) provided the OPLUS on the elements is associative (or commutative)". We designed "conditional 'where' clauses" as a notation for expressing such ideas, but never got to implementing them.

There is much work yet to be done in designing and implementing programming languages, and we believe that many of the ideas mentioned above will be important, in some form, in future language designs.

We thank you for YOUR interest in Fortress, and hope that you will be similarly interested in our future efforts in other areas.

—Guy Steele, for the Oracle Labs Programming Language Research Group


Technical discussion of the Fortress programming language and its applications


« July 2012