Closures for Java

The free lunch is over.

Multicore processors are not just coming—they’re here.

Leveraging multiple cores requires writing scalable parallel programs, which is incredibly hard.

Tools such as fork/join frameworks based on work-stealing algorithms make the task easier, but it still takes a fair bit of expertise and tuning.

Bulk-data APIs such as parallel arrays allow computations to be expressed in terms of higher-level, SQL-like operations (e.g., filter, map, and reduce) which can be mapped automatically onto the fork-join paradigm.

Working with parallel arrays in Java, unfortunately, requires lots of boilerplate code to solve even simple problems.

Closures can eliminate that boilerplate.

It’s time to add them to Java.

The closures design space In the last few years three serious proposals for adding closures to Java have been put forward: BGGA, CICE, and FCM. These proposals cover a wide range of complexity and expressive power. My view, having studied them all, is that each contains good ideas yet none is entirely appropriate for a “working programmer’s language.”

To support the principal use case of parallel programming we really only need two key features:

  • A literal syntax, for writing closures, and

  • Function types, so that closures are first-class citizens in the type system.

To integrate closures with the rest of the language and the platform we need two additional features:

  • Closure conversion, so that a closure of appropriate type can be used where an object of a single-method interface or abstract class is required, and

  • Extension methods, so that closure-oriented bulk-data methods can be retrofitted onto existing libraries, and in particular the Collections Framework, without breaking compatibility.

A couple of other integration features worth considering are first-class method references and the ability of function types to include exception parameters.

Some of the other features found in the existing proposals carry considerable additional complexity:

  • Capture of non-final variables,

  • Non-local transfer of control, and

  • Library-defined control structures (i.e., control abstraction).

At present I see no need to add any of these to Java.

Let’s be about it It’s time to learn from the past debate and move forward. Sun will initiate the design and implementation of a simple closures feature, as outlined above, and add it to JDK 7 so as to enable broad experimentation. If all goes well then we’ll later submit a language-change JSR which would, in turn, be proposed as a component of the eventual Java SE 7 JSR.

Revising a programming language that’s in active use by millions of developers is no small task. Sun neither can nor should do it alone, so I hereby invite everyone who participated in the earlier closures conversations—as well as anyone else with an informed opinion—to join us.

Comments:

I'm looking forward to closures in Java. Thanks you for leading this effort!

Posted by Lars Vogel on November 24, 2009 at 08:33 AM PST #

I really like the BGGA style and honestly I don't think it's too complicated. IMHO

static <T extends Object & Comparable<? super T>> T min(Collection<? extends T> coll)

is much harder to understand and to explain. The syntax or closures might be new but it not really "hard". As an author and trainer I find Generics challenging to teach but closures are really easy compared to Generics. So: "There’s not a moment to lose!"

Posted by Christian Ullenboom on November 24, 2009 at 12:24 PM PST #

I don't really understand this move... Is it not too late ?
First the decision was taken to only include project coin in Java SE 7 JSR.
And now, several months later, the decision seems to be reconsidered.
What's reason behing this move ? What has changed ?

Posted by shamaz on November 24, 2009 at 06:48 PM PST #

I hope only the restriction on non-final variables can be lifted. This is absolutely essential (Smalltalk etc. use the term "full closures") and it's extremely useful, remarkably as the Java language doesn't support multiple returns so a LambdaExpression cannot produce more than one output (conveniently). And the implementation is very easy (lifting local vars into the closure object); I've seen some people complaining of JMM issues if the closure is executed in another thread, but w/o closures people often do the same implementation writing helper classes that capture the necessary data, so I call BS on that argument.

Posted by Osvaldo Pinali Doederlein on November 24, 2009 at 08:20 PM PST #

I think Neal's new proposal on javac.info is spot on. This includes the access to local variables if you annotate them. That we can handle.

Cheers,
Mikael

ps. Btw, do you know what happend to the MigLayout RFE? It is not accessible anymore. There were hundreds of nice comments so I think it's a shame if it was censured in any way.

Posted by Mikael Grev on November 24, 2009 at 08:23 PM PST #

I'm sure a sensible solution will be reached, but I'd like to put in a word for a new keyword over some sort of magic brackets approach. Either "function" or "fn"

There will be inevitable breakage from people who've called fields and methods "function" but not common, less prevalent than the equivalent "enum" issue and nothing we can't refactor away pretty quickly.

Posted by Dave Minter on November 24, 2009 at 08:31 PM PST #

It is not BGGA-closures that are hard per se. The combination of BGGA with checked exceptions and Generics can become hard to read.

If BGGA would adopt the Groovy "->" arrow notation for closure parameters, I think that this would be easier to read than the current FCM-oriented style.

But I am happy about having closures at all and if FCM-style syntax is the price to pay for that, I am willing to do so.

Posted by Sakuraba on November 24, 2009 at 08:39 PM PST #

Hi Mark,
What about exception transparency and method references ?

Rémi

Posted by Rémi Forax on November 24, 2009 at 10:08 PM PST #

I'd like to help; is there a prototype I can test yet, or should I wait until it's in JDK 7? Is there somewhere in particular to discuss it?

Posted by Ricky Clarkson on November 24, 2009 at 10:13 PM PST #

Great. Finally, with closures, JDK1.7 is actually an interesting upgrade from JDK1.6

Posted by Morten on November 24, 2009 at 10:40 PM PST #

Without capture of non-final variables, you can't properly call these constructs "closures". At least in the classical sense - closures have that name because they 'close' over the free variables of the lexical environment in which they are created, which includes non-finals.

Posted by Yuri Niyazov on November 25, 2009 at 01:51 AM PST #

Great news Mark! Would also like to see the @Shared as Mr. Gafter has proposed. I second what Ricky Clarkson is saying -> JDK 1.7 looks like it will be a worthy upgrade.

Posted by sboulay on November 25, 2009 at 02:27 AM PST #

Really happy to see this! I also agree that the control structure bits of BGGA was probably a miss directed. I guess the problem that was supposed to solve is more static in nature, and would thus probably be better addressed with a good macro system.

WRT syntax I'm quite fond of the Scala approach of lots of optional syntax bits.

(s:String) => println(s)

can be written as

_ => println(_)

or just

println _

or simply

println

yielding easy to read expressions such as

list.foreach(println)

instead of the more verbose

list.foreach((s:String) => println(s))

It's kind of how expressions with operators works ATM.

Consider this statement

int sum = 4 + 5;

I would love if I could write

import static java.lang.Integer.add

int sum = 4 add 5

Instead of

int sum = ((int a, int b) => Integer.add(a,b))(4,5)

or

int sum = 4 (+) 5

to lift + into a lambda taking two arguments.

This short syntax would be used to do things like

int sum = intlist.reduce(add)

or

int sum = intlist.reduce(+)

So to summarize, for the syntax it would be great if argument names, position and types could be inferred and any expression missing an argument could be automatically lifted to a lambda.

int shiftedlist = intlist.map(+1)

Posted by John Nilsson on November 25, 2009 at 04:08 AM PST #

For the three left out I find the 1st (capture of non-final variables) a must have. I know the impact it may have on memory model but it is also huge the boilerplate of having to create a final field each time a variable outside the scope of the closure is needed!

Posted by Marcell Manfrin on November 25, 2009 at 06:02 AM PST #

Can you clarify why you see no need to add control abstraction? That would allow us to really cut down the boilerplate code which often makes Java programming quite painful.

Posted by Nick Radov on November 25, 2009 at 09:13 AM PST #

The feeling I have about all this hype around Closures is that people will dedicate an enourmous amount of time designing and implementing such thing and at the end the Java community will barely use it ... perhaps in a far future when new developers will learn it since the beginning ... may be ...

Perhaps I am just too long in this carrier that I just can't feel any pain working with Java.

\* from time to time I try another languages, for joy or job obligation.. and everytime I start of using Groovy, Scala or other super hype revolutionary language I finish with a set of non-debuggable mess ... and usually I spend much more time trying to adapt my Java culture to the opinion of other people than simply coding that in Java ...

I hope closure add something useful this time :)

Posted by Felipe Gaucho on November 25, 2009 at 07:30 PM PST #

Here is why you'll want to capture non-final variables.

If you don't, programmers will just continue to transform

@Shared int count = 0;

into

final int[] count = int[1].

It's ugly, but it works. EXCEPT when the variable is volatile.

In this case, the compiler can do the right thing, but the programmer can't--at least not in a portable way.

Posted by Cay Horstmann on November 25, 2009 at 10:29 PM PST #

Mark I am no fan of closure at least in Java, but any way if community decided to add the closure to Java my recommendation is to stick with the closure syntax of Groovy. It gives you literal syntax you mentioned earlier, besides this majority of Groovy programmers are Java programmers, so transition to Java closure will be much easier.

Posted by Rashid on November 26, 2009 at 03:57 AM PST #

@Cay what do you expect to happen if a programmer passes a closure which uses
@shared int count
to a method which executes the closure in parallel? Even if you mark it volatile the result is likely to be indeterminate.

Posted by Mark Thornton on November 26, 2009 at 04:04 AM PST #

@Cay, the programmer can use atomic array:
http://download.java.net/jdk7/docs/api/java/util/concurrent/atomic/AtomicIntegerArray.html

Rémi

Posted by Rémi Forax on November 26, 2009 at 09:58 PM PST #

I'd encourage lifting from the Go programming language.

Posted by Noel on November 26, 2009 at 11:15 PM PST #

Forgive me, but I'll copy my comment from Stephen Colebourne blog:

This proposal seems realistic and elegant. But I really can't understand the need to introduce a second notation (expressions) only to avoid a single word (return) as I couldn't understand the need for the "several returns" mess in BGGA !
I thought the point was to keep it simple.

And why not use a new "function" keyword ? This is what it is, isn't it ? Has the enum keyword caused so much trouble in Java/JDK 6 ? Are you afraid of the "Java becomes JavaScript" comments ?

But, first of all: remove expressions.

Posted by Olivier Allouch on December 01, 2009 at 05:26 AM PST #

@Noel

I guess we disagree on this. For me the main reason to support lambdas in Java is to cut down boilerplate code enough to make functional programming viable.

To this end any way to reduce the syntactic overhead of lambdas must be a good thing.

Why do you think this is a risk?

I mean, if syntactic overhead isn't an issue there is no reason to not just continue using anonymous inner classes.

Posted by John Nilsson on December 01, 2009 at 06:38 AM PST #

Sorry, the above was meant @Olivier

Posted by John Nilsson on December 01, 2009 at 07:36 AM PST #

This is a good news but from my point of view is quite incompressible why it will require also the "Extension methods" feature.

Collections framework could not simply extended to support Closures as it has been done for Generics? Why you say it would break compatibility ?

I think "Extension methods" is really against OOP paradigm and promotes really bad coding practices.

Regards,
Paolo

Posted by Paolo Di Tommaso on December 01, 2009 at 03:22 PM PST #

I never said that boilerplate code wasn't an issue. Maybe me wanting to add the "function" keyword may have been caused by my love of JavaScript, or just that it's its name.

But, the main point I raised was the fact that expressions are just wrong for Java. Java isn't JSP or PHP. The goal of those languages is to output a string stream. You throw strings to the output buffer. Java isn't a DSL. There's no such thing as a "current value". The APIs aren't designed to be fluent (except some like StringBuffer/Builder or Joda), and the "return 'this' if it normally returns 'void'" proposal was rejected from Project Coin...and expressions are something you can add later if you really want to.

Posted by Olivier Allouch on December 01, 2009 at 06:36 PM PST #

Add one more vote to "closure over non-final variables is essential".

As was pointed out, without that, you don't really have closures anyway. But beyond that, it's a very useful construct, including for uses of closures in concurrency.

Speaking of which, while not 100% essential (using other synchronization techniques can have the same effect), it would be nice if a local variable could be made volatile.

Posted by pete.d on December 05, 2009 at 09:26 AM PST #

The IBM's listings are not identical. Listing 2 allows the user of withFilter and withMapping to pass ANY filter or mapping, whereas Listing 4 passes a concrete filter and mapping.

Listing 2 should look differently with much less "boilerplate code", when only one concrete filter and a mapping are used.

Posted by Alex Otenko on December 06, 2009 at 09:44 PM PST #

I'd like to vote against adding closures into Java. In my opinion, you folks have failed on two fronts:

1) Using syntax that would be familiar to existing Java developers (focus on readibility instead of brevity)

2) Explaining the need of Closures for the mainstream Java user. I don't care if this uber-cool feature helps all you mathematicians out there. I simply don't see the "itch" that closures are meant to fix.

Roughly half of the Java community seems to agree, so writing them off as "clueless" is disingenuous. If we don't agree it's because you have, thus far, failed to make your case. You might have a good point, but you're not pitching it properly for the target audience.

Posted by Gili on December 07, 2009 at 10:48 AM PST #

@Gili, you may not yet see the need, but how confident are that you will not require concurrent processing in the future or that Java's existing mechanisms will suffice when you do?

Whereas a few years ago all my jobs could be run comfortably on a single core, now an increasing number need to use all the cores I can get to keep the computation time within tolerable limits.

As for brevity, take for example matrix multiplication. This uses two operations '+' and '\*'. Even the proposed syntax adds a significant number of boiler plate characters, while an implementation using inner classes totally loses the significant character (operation).

Posted by Mark Thornton on December 07, 2009 at 11:22 PM PST #

@Mark Thornton

It isn't clear from this post whether closures are going to be a syntactical feature or new JVM instructions are going to be added as well to support new way of concurrent execution.

Can someone clarify?

IBM's ParallelArray quoted above doesn't introduce anything new beyond syntax.

Posted by Alex Otenko on December 08, 2009 at 01:00 AM PST #

@Mark, thus far concurrent processing has made up less than 5% of my application code. I am quite happy with the level of support that Java already provides to that end especially since it is limited to a relatively small part of my program.

I've yet to be convinced that the concurrency "itch" can't be solved without Closures. I am also uncomfortable at the thought that, once introduced, developers will be able to (ab)use Closures everywhere, not only in use-cases used to justify their introduction. People \*will\* abuse this feature and it will lead to less readable code.

A good example is operator overloading. I am in favor of adding the +,-,/,\* operators for BigInteger but against the idea of providing open-ended operator overloading for user classes.

I'd be more ammendable to the idea of Closures if you could limit them to a certain part of the API or otherwise limit the way they may be abused.

Posted by Gili on December 08, 2009 at 03:08 AM PST #

Gili,

How do you imagine they'd be abused? Can you give an example?

Thanks,
Ricky.

Posted by Ricky Clarkson on December 08, 2009 at 05:34 AM PST #

@Ricky Clarkson

to start with, the code in the closure can't be inherited, overridden and even simply reused. That's abuse #1.

Posted by Alex Otenko on December 08, 2009 at 05:55 PM PST #

@Gili, if the itch involves computations like those facilitated by the ParallelArray api, then with current Java, that API is truely horrible.

Closures make API of that kind palatable by hiding the 80 odd interfaces (there are anonymous function types instead, but they aren't so intrusively visible.

That API could also be greatly simplified if generics worked over primitives, but that is considerably harder to implement than closures.

If the JIT could convert arrays of primitive wrappers (Double, etc) into arrays of primitives, then that could also allow a simpler API. Unfortunately this seems to be even harder (unless we can make significant changes to existing Java semantics, especially the 'Object' nature of those wrappers).

Posted by Mark Thornton on December 08, 2009 at 06:04 PM PST #

@Mark,

"That API could also be greatly simplified if generics worked over primitives, but that is considerably harder to implement than closures."

I would much rather see Sun invest effort in cleaning up Generics than in introducing Closures, even if the latter is "easier". There are wide-reaching implications to adding Reification and better primitives support that would do away with most of the learning curve and boilerplate code associated with Generics. That would be a huge step in the right direction.

My feeling is that, for better or for worse, Sun introduced a very painful compromise in 2004. I feel that the situation is different five years later and that Sun should clean up existing features in Java before it proposes to introduce new ones to the language.

I am hoping that project Jigsaw will be the vehicle by which we can begin to remove deprecated code and occasionally even break backwards compatibility for code older than 10 years. A lot has changed in the past decade and occasionally breaking those constraints can yield a more cohesive language.

One approach would be for the JVM to check the classfile version of any classes it loads, and allowing Jigsaw-aware classes to interact with classes that break backwards compatibility while older classes continue running with full backwards compatibility.

Posted by Gili on December 08, 2009 at 10:16 PM PST #

@Alex, closures are just syntax which can be translated to existing constructs, but might also use some of the new features being added to the JVM for other reasons.

@Gili, I also would prefer reification and generics over primitives, but that option is not on the menu for JDK7.

Posted by Mark Thornton on December 08, 2009 at 10:33 PM PST #

Post a Comment:
Comments are closed for this entry.
About

This blog has moved to http://mreinhold.org/blog. <script>var p = window.location.pathname.split('/'); var n = p[p.length - 1].replace(/_/g,'-'); if (n != "301") window.location = "http://mreinhold.org/blog/" + n;</script>

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today
News

No bookmarks in folder

Blogroll

No bookmarks in folder

Feeds
RSS Atom