Closures for Java: The Q&A

At Devoxx last month I said that it’s time to add closures to Java.

That statement, and my follow-up blog entry, elicited reactions ranging from enthusiastic support to skeptical acceptance to cynical indifference. It certainly motivated lots of tweets and lots of questions. In classic Stephen O’Grady style I’ll attempt here to summarize and answer some of the key questions I’ve seen, so far.

Q Before we begin, do you have anything to disclose?

A Yes. At heart I’m an old Lisp hacker; more specifically, I’m a past teacher and implementor of the Scheme dialect. Closures are completely natural to me—along with tail recursion, continuations, and the definition of procedure call in terms of rename plus goto. (I accept that I’m unusual in this regard.)

Having said that, in all my years at Sun I’ve done my level best to respect and preserve the “feel of Java,” as James calls it. Java incorporates some of the great ideas of the Lisp tradition but it is not, and never will be, a Lisp. So I’ve always been a bit nervous at the prospect of adding closures to Java. If we do so then we absolutely must preserve the existing character of Java as a language for “working programmers.”

Q So, why now?

A Two reasons.

First, the JDK 7 schedule has been extended, which in turn means two things: We have more time, and JDK 8 will be even later. If we don’t do closures in 7 then they’ll have to wait for 8, whenever that turns out to be. I don’t think we can wait that long to gain good support for writing scalable parallel programs using bulk-data APIs such as parallel arrays.

Second, some of the key participants in the extremely lively closures debate of a couple of years ago have indicated their willingness to consider a simplified proposal.

Q By the way, why was the JDK 7 schedule extended?

A The schedule was pushed out, to at least September 2010, so that we can finish several critical features as well as have time to submit an official Java SE 7 Platform JSR, form an Expert Group, and allow all the appropriate deliberations to take place. JDK 7 will not ship until the Java SE 7 JSR goes final.

Q Why did you make the decision to include closures in Java SE 7 behind closed doors, without any community input?

A That’s not the decision that I made, contrary to various reports. Making such a unilateral decision is not within my power, nor is it within Sun’s.

Q Okay, so what did you decide?

A I decided to initiate an effort to draft a new proposal within the design space I’ve outlined and along the lines of the straw-man examples I’ve shown, and to prototype an implementation as part of JDK 7 so as to enable broad experimentation. If all goes well then we’ll later submit a language-change JSR which will, in turn, be proposed as a component of the eventual Java SE 7 JSR.

Q Are your design constraints and straw-man examples the effort of months of internal deliberations within Sun?

A No, the constraints and examples are mine alone. They’re a reflection of my sense—which could be wrong!—of what the overall community is most likely to support.

Q Why did you choose FCM as a starting point?

A I didn’t choose FCM; I just happened to choose a notation for the examples in my talk which looks a lot like FCM.

Q Why not start with one of the existing proposals?

A The existing proposals are, in my view, either too simple or too complex. CICE only goes part-way toward making the parallel-array API easy to use, while BGGA and FCM contain more features than are needed to address that key use case.

Q Do you, or does someone else at Sun, have a specific proposal ready for review?

A No.

Q Why did Sun delay the formation of a JSR for closures for so many years?

A There was never a clear consensus in favor of a specific proposal, and in fact key respected members of the Java community supported significantly different proposals. In our view a JSR Expert Group would not have been an effective forum for resolving that disagreement.

Q Why not start a closures JSR now and let its Expert Group do the work of formulating a new proposal?

A For the same reason that there’s not yet a JSR for Project Coin, namely that no new Java SE JSR is likely to be approved until the present dispute in the JCP Executive Committee is resolved. Once that happens, and assuming that consensus forms around the result of this new effort, we fully intend to submit and lead a language-change JSR for closures. In the meantime we’ll proceed with the initial design and prototyping work in the OpenJDK Community, just as we have done with Project Coin.

Q Where will this work be done? How can I participate?

A Anyone with an informed opinion will be welcome to participate in Project Lambda, which I’ve just proposed.

Q Where can I find a written version of the straw-man examples you showed at Devoxx?

A I’ll post an updated and expanded version to the Project Lambda pages, once the Project is approved.

Q Why didn’t you take part in the original closures debate?

A I considered that, but there was an awful lot of heat in those discussions. I thought it better to let the debate run its course, see what I could learn from the proposals and their prototypes, and then try to synthesize something new and simpler.

At that point in time, moreover, we were trying to ship JDK 7 sooner rather than later. Even if broad consensus had formed around a single proposal it likely would not have made it into JDK 7.

Q Why did you use the ‘#’ character in your straw-man syntax?

A The ‘#’ character is traditionally used in Lisp to introduce certain kinds of literals, and in some dialects even closure literals. That it’s also used in the FCM proposal is coincidental, though in retrospect perhaps I should’ve chosen a different character in order to avoid giving the impression that I was backing FCM.

Q Did you really invent your straw-man syntax while flying from San Francisco to Europe?

A Yes. I’d been thinking about a simplified closures proposal for several months, but without a specific syntax. It’s hard to do a presentation about language features without a concrete syntax, however, so while writing my slides for Devoxx, somewhere high above the North Atlantic, I played around with several different possibilities. The one I showed in my talk was the best of them. I mentioned this fact during the talk only to make it clear that this syntax is just a straw-man; apparently some people found that tweet-worthy.

Q Do you expect the final proposal to use your syntax?

A No. My straw-man syntax is meant only to be a starting point, not an actual design—that takes a lot more work! I’m sure there will be a long, and likely bitter, debate over the actual concrete syntax.

Q Isn’t adding closures pretty risky? Won’t it add a lot of risk to JDK 7?

A Every big change carries risk. The key features I’ve outlined, along with many others, have been implemented in the BGGA prototype thanks to Neal Gafter’s fine work. We therefore already know that it’s possible to implement them together in a reasonably harmonious fashion; this greatly reduces the overall risk to both the design and the implementation.

Q Do you have anything else to add?

A No, I think that about covers it. I’m sure there will be many questions, comments, and suggestions going forward. The best place to engage will be the Project Lambda mailing list, once it’s been created.

Comments:

Mark, since you're currently using the synatax AND semantics of FCM, it is pretty hard to claim that what you're proposing isn't FCM (with the rigour from BGGA's investigative work).

(Remember, in language debates, its the semantics that matter, not the syntax. What distinguishes the semantics of CICE/BGGA/FCM is how they handle the 'this' and 'return' keywords in the 'closure' element of each proposal. What you've announced so far follows FCM semantics for these key criteria. As does Neal's new CFJ proposal.)

Posted by Stephen Colebourne on December 02, 2009 at 01:23 AM PST #

Class act. Thanks for the FAQs.

Posted by Patrick on December 02, 2009 at 01:44 AM PST #

+1 for closures/lambdas. Keep up the good work. I hope the debate about project lambda won't be as bitter as the previous.

Posted by Kevin on December 02, 2009 at 02:11 AM PST #

Stephen: I agree that what I've said so far could be construed as a subset of
FCM. I've been careful to say that it's not FCM, however, because I'm not
convinced that FCM, taken whole, is a suitable solution. FCM contains many
good ideas but I'm troubled by, among other things, the subtle distinction
between method literals and method references, the necessity of writing
"m.invoke()" rather than just "m()" to invoke a method, and the complexity of
named inner methods.

Posted by Mark Reinhold on December 02, 2009 at 03:41 AM PST #

I look forward to seeing the draft proposal and participating in its refinement.

Posted by Neal Gafter on December 02, 2009 at 03:50 AM PST #

Mark, now that JDK 7 is being delayed, is there any possibility of multi-catch exceptions being looked at again?

Posted by David Kopp on December 02, 2009 at 03:51 AM PST #

Any chance of parameter name access being considered for JDK 7 ? It was slipped from JDK 6 because Sun ran out of time. It is just a small enhancement to reflection.

Posted by Paul Hammant on December 02, 2009 at 04:07 AM PST #

Mark, thanks for the clarifications. Interestingly, in an unpublished v0.6 of FCM written many moons ago we had removed named inner methods, so I tend to forget they are in v0.5. I do think that Member literals are very useful, especially as annotation values, but they might be clearer with a different syntax. You may find m.invoke() is necessary from the perspective of integration with existing Java.

Posted by Stephen Colebourne on December 02, 2009 at 05:15 AM PST #

About m() vs m.invoke():
in Java, the method scope and the variable scope
are separated (JLS3 6.3 and 15.12),
so m() is ambiguous.

It's possible to modify 15.12 but it will add unnecessary complexity to an already very complex algorithm.

Rémi

Posted by Rémi Forax on December 02, 2009 at 05:22 AM PST #

Mark, I am happy someone finally picks up Closures (or lambdas) as important enhancement for the Java language. May it be inspired or influenced by FCM or not, the choice of semantics (and syntax) shows that it is not that far-fetched for being suited for Java developers. And that's exactly, what Stephen and I had in mind when working on the proposal. And from your post, I read that we have the same goal. :)
Will surely join the party and support working on lambda for Java, and I don't think the goal to be a compromise of existing proposals, but a neat solution that fits well in the language.
Cheers,
Stefan
P.S.: Just as a side-note: the "m.invoke()" syntax is due to the different name spaces of methods and variables. If there is a way to resolve this issue, I'd prefer the method-like invocation too ;)

Posted by Stefan Schulz on December 02, 2009 at 05:44 AM PST #

I think anonymous functions would be a better name for your project. Normally, lambda expressions doesn't change the semantics of subexpressions.

Expressed in Lisp, ((lambda () x)) should be equivalent to x. That is not what you're proposing.

Posted by Peter von der Ahé on December 02, 2009 at 05:51 AM PST #

David: Yes, the improved exception-handling proposal will be reconsidered, as
Joe Darcy already noted in his blog.

Paul: No, parameter-name access is not currently being considered.

Rémi, Stefan: Yes, I understand that methods and variables have different name spaces. I think it's worth investigating how to disambiguate "m()".

Peter: It's true that what I'm proposing does change the semantics of enclosed
statements (though not of enclosed expressions). Naming the effort "Project
Lambda" might confuse a few language theoreticians but I think most Java
developers already understand that Java is not Lisp, nor the lambda calculus.
Besides, "Project Anonymous Functions" doesn't exactly roll off the tongue, or the fingers.

Posted by Mark Reinhold on December 02, 2009 at 08:15 AM PST #

Given that adding this feature's high maintenance complexity to the language will at best save three lines of code, when compared to using already existing language features, what is the business justification for saddling the companies that rely on Java with this additional cost for no reward?

Posted by Mikael Gueck on December 02, 2009 at 10:45 AM PST #

Well said, Mikael. Seems like another "Ruby has it, therefore Java must have it too" feature.
IMO all it would do really is lead to a lot of ways for people to write more obfuscated code, making maintenance work on Java based systems more tedious than it is already (all maintenance is tedious, and the more a language allows for writing obfuscated code the harder it gets).
All of us who've written C and C++ professionally have seen what damage function pointers can do.
They're massively powerful, but also massively easy to abuse and turn code into unreadable masses of rubbish (and yes, I've been guilty of writing those, but at least I documented them).

Retrofitting something like this into an existing language is not to be taken lightly, probably not to be undertaken at all.

Posted by J.T. Wenting on December 02, 2009 at 03:07 PM PST #

I already see a consensus here and a general positive will to help. All the points discussed here don't seem a huge problem for an average developer (I almost wrote lambda coder) like me, except for the extension methods.
That makes me wonder why we couldn't get it sooner :)
The JCP dispute looks like the real cause of road-map shift...so sad

Posted by Olivier Allouch on December 02, 2009 at 06:38 PM PST #

@Mikael
You are saving 3 lines of code at the client-side. Though, what your are saving in terms of API complexity is uncomparable. Currently every API that needs function-interfaces, has to define its own set of them. They are always the same, but have to be defined in each API separately. Adding standard function-interfaces to the JDK is a good thing to do. From there, it is only a small step to add syntactic sugar on top of the function-interfaces. I would even prefer making one further small step and allowing access to non-final local variables.

Posted by Fatih Coskun on December 02, 2009 at 06:58 PM PST #

Yay! FCM :)

Posted by Germán on December 02, 2009 at 08:54 PM PST #

Maybe could we use "m.()" or "m.#()" instead of "m()"?
Shorter than "m.invoke()", unambiguous, and in line with the syntax for declaration of closure/lambda using #
Sure, invoke would be better suited for reflection...

Posted by efl on December 02, 2009 at 11:38 PM PST #

Mikael: The premise of your question is false. You should have phrased it thus: "Given that adding this feature will significantly reduce the maintenance complexity in code where it is used, and save many lines of code in APIs where it is used heavily (like ParallelArray), what is the business justification for saddling Sun with the relatively low implementation cost for such a high reward?

Posted by Neal Gafter on December 03, 2009 at 12:05 AM PST #

I agree with Mikael and Wenting, but I see this as an opportunity to create a new language for large enterprise software, to fill the vacuum left by Java.

Posted by Lawrence Kesteloot on December 03, 2009 at 05:21 AM PST #

Fatih, would it not therefore make more sense to simply ask Stephen Colebourne to develop a few de facto standard interfaces and simple implementations, rather than to change the language to add potential for complexity?

Neal, this proposal adds new potential for complexity, as well as a slew of unintended consequences which themselves increase the potential for complexity, and it doesn't let you do anything that, like you and Fatih said, people aren't already successfully doing with a few lines of code.

Let's use this language's complexity budget to make it able to answer business needs it currently can't easily fulfill, like more rapid development cycles with full object replacement in a running JVM.

Posted by Mikael Gueck on December 03, 2009 at 02:01 PM PST #

This is great! First Collections, then autoboxing, now closures. Java becomes more like Smalltalk every day. ;-)

Posted by Peter Munro on December 03, 2009 at 07:10 PM PST #

I just love that ! People complain counting the lines of code and then other people want more lines of code to keep it unobfuscated.
All I want is my code to be the reflection of my explanations in plain English. When I say "this piece of code gets called when the task is complete", I see a closure and not an implementation of a single-method interface like "taskComplete()" of "TaskCompletionListener".

If you want a language without closures, then yes, I may have to create one yourself, cause almost every one got it, even PHP !
You're so wrong thinking that "large enterprise software" means "good old code like we had in the xx's" (you fill the blanks). They're more experimental than you think.
Trust me, closures make your code much more readable.

Posted by Olivier Allouch on December 03, 2009 at 07:39 PM PST #

Excellently put Neal. :)

From the conversations I have had with developers, I generally find that the ones that are against adding "closures" (method literals/whatever) to Java, are those that have not really used languages that support them well. They also tend to not be developers that try to write immutable code whenever possible. This is not a scientific measure by any means, but dislike of them seems to correlate with less familiarity with a more functional style of development, and all the advantages that brings.

I find the lack of this feature in Java to be frustratingly limiting, and I am extremely pleased they are seriously being considered for Java 7. Great going Mark!

Posted by Robin on December 03, 2009 at 08:02 PM PST #

Post a Comment:
Comments are closed for this entry.
About

This blog has moved to http://mreinhold.org/blog. <script>var p = window.location.pathname.split('/'); var n = p[p.length - 1].replace(/_/g,'-'); if (n != "301") window.location = "http://mreinhold.org/blog/" + n;</script>

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today
News

No bookmarks in folder

Blogroll

No bookmarks in folder

Feeds
RSS Atom