Tuesday Dec 08, 2009

Simple long polling in Scala with Atmosphere

There are two styles of pushing events from servers to clients: long polling; and HTTP streaming. The former tends to be easier for developers to understand "push", especially when a bit of HTML and JavaScript is utilized, and it does not break the REST request/response constraint. So there are some advantages over HTTP streaming depending on what your requirements are.

I recently took a slight detour into Node.js, impressive stuff. A blog entry written by Simon Willison presented a simple long-polling example.

Hmm... i wonder if i can do something functionality equivalent in Scala and using Atmosphere. I might iron out some bugs in Atmosphere, learn some stuff along the way, and help developers better understand long polling and Atmosphere.

A common pattern for long polling is:

  1. when a client makes a request and the server has data, that data is returned immediately and the connection is not suspended.
  2. when a client makes a request and the server has no data, the connection is suspended, and the client waits for a response. When data is available the server returns that data and the connection is resumed.

In cases 1 or 2 the client make will another request after it has received the data, and the process repeats. Further more, there may be 2 or more clients waiting for data whose connection will be resumed when the same data is available, for example, consider the case of a chat application where a client may send a message to all other clients.

Notice that there is an interval of time when a client processes a response and makes a new request. Within that interval new data may be available. To ensure that the client does not miss out on that data the server must store that data and the client must signal to the server what data it has previously received.

To play with these concepts i wrote a very simple long polling chat server written in Scala using Atmopshere that functions as previously described, you can download it here.

The main class that acts as the chat application is as follows:

01  @Path("/chat")
02  @Singleton
03  @Produces(Array("application/json"))
04  class MessagesResource {
05  
06    var messages = List[Message]();
07  
08    @Suspend { val resumeOnBroadcast = true }
09    @GET
10    def getMessages(@QueryParam("date") lastSeenTime : long) : unit = {
11      val lastSeenDate = new Date(lastSeenTime);
12      
13      messages.filter(_.date.compareTo(lastSeenDate) > 0) match {
14        case Nil => // Suspend and wait for data
15        case x => // Return with data
16          throw new WebApplicationException(
17            Response.ok(x.reverse).build())
18      }
19    }
20  
21    @Broadcast
22    @POST
23    @Consumes(Array("application/x-www-form-urlencoded"))
24    def publishMessage(@FormParam("message") message : String) = {
25      val m = new Message(new Date(), message);
26      messages = m :: messages;
27      List(m);
28    }
29  } 

The root resource class

A JAX-RS root resource class, MessageResource, is served from the "chat" path segment as declared by the @Path annotation at line 1, and there will be one instance of this class created per the application (using the Jersey @Singleton annotation at line 2) because the messages are going to be stored in memory. All resource methods (see later) will produce JSON, as declared by the @Produces annotation at line 3.

The list of chat messages are stored in an immutable list, List[Message], a reference of which can be updated (which is why var is utilized) at line 6.

Sending a chat message

A client will send a chat message by performing a POST request to the chat resource with a representation conforming to the "application/x-www-form-urlencoded" media type, which basically is the default format used by HTML forms of name=value pairs separated by '&' characters, with a name of "message" and the value that is the chat message. Such a contract is defined at lines 22 to 24 with the JAX-RS resource method publishMessage. The JAX-RS @FormParam annotation is utilized to obtain the form parameter value whose name is "message".

Highlighted, in yellow at line 21, is the Atmopshere annotation @Broadcast. This declares that any response entity returned from the publishMessage resource method will be broadcast on all suspended connections. This is how one chat client may send a message to all other clients participating in chatting.

A new message is created, at line 25, that contains the date it was created at and the actual chat message. The date is used later to determine what messages to send back to the client. A new list of messages is then created, at line 26, by concatenating the newly created message and all the previous messages, and that new list is assigned to the messages field. Then the resource method, at list 27, returns a list with one element that contains the newly create message. A list is returned so that a JSON array will be produced with one object that contains the date and message values (more on how the JSON is created later).

Receiving a chat message

A client will receive one or more chat messages by performing a GET request to the chat resource with a query parameter declaring the date of the message it last received. Such a contract is defined at lines 9 and 10 with the JAX-RS resource method getMessages. The JAX-RS @QueryParam is utilized to obtain the query parameter whose name is "date" that corresponds to the date of the last received message.

Highlighted, in yellow at line 8, is the Atmopshere annotation @Suspend. This declares that, if the resource method returns normally, the connection will be suspended and resumed when a broadcast occurs on that connection.

For expediency i found it is easier to pass dates between the client and server as non-negative numbers thus line 11 creates a java.util.Date object (i know i should be using Joda time!) to easily compare dates of messages.

The list of messages is then filtered to obtain a list of those messages that have been broadcast after the date the client last received a message, at line 13. If the list is Nil (or empty), at line 14, the resource method returns and and the connection will be suspended until a chat message is broadcast. If the list is not empty, at line 15, then the list is reversed, so messages are ordered in increasing time from left to right, a Response is built with the list and is thrown with a WebApplicationException. The throwing of the exception tells Atmosphere the connection should not be suspended. So this is the way client may receive any pending data immediately.

The client

It is very easy to test the chat server using curl. For example, if you run the chat server, then in one terminal window type:

curl -v http://localhost:9999/app/chat 

then the connection is suspended and curl will wait for a response. Then in another terminal window type:

curl -v -d message=HELLO http://localhost:9999/app/chat

You will observe that a message similar to the following will be returned by both curl statements:

[{"message":"HELLO","date":1260295792434}] 

Then if you type the following in the first terminal window (substituting the date value as appropriate):

curl -v http://localhost:9999/app/chat?date=1260295792434

the connection will be suspended because no messages have been broadcast after the date declared in the query parameter.

A browser-based client is more interesting and some simple HTML + JavaScript can be easily created as follows (note that i am not so experienced in this area, for example i cannot work out why the heck it works with Firefox but not Safari):

01  <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
02      <head>
03          <script type="text/javascript" src="jquery-1.3.2.js"></script>
04          <script type="text/javascript" src="jquery.form.js"></script>
05          <script type="text/javascript">
06              var last_date = 0;
07              function fetchLatest() {
08                  var ul = $('ul');
09                  $.getJSON('chat?date=' + last_date, function(data){
10  
11                      $.each(data, function() {
12                          ul.prepend($('<li></li>').text(new Date(this.date) + ": " + this.message));
13                          if (last_date < this.date)
14                              last_date = this.date;
15                      });
16                      
17                      setTimeout('fetchLatest()', 1);
18                  });
19              }
20          </script>
21          <script type="text/javascript">
22              $(document).ready(function() {
23                  fetchLatest();
24                  $('#publishForm').ajaxForm();
25              });
26          </script>
27      </head>
28      <body>
29          <p>
30              <form id="publishForm" action="chat" method="post">
31                  Message: <input type="text" name="message" />
32                  <input type="submit" value="Submit Comment" />
33              </form>
34          </p>
35          <p>Waiting for messages...</p>
36          <ul></ul>
37      </body>
38  </html> 

jQuery and the jQuery form plugin make it easy to perform GET and POST requests asynchronously.

Lines 30 to 33 define how the client will send chat messages to itself and other clients utilizing the POST requests. Line 24 uses the jQuery form plugin to perform the POST request asynchronously so the browser does not display the response.

Line 23 initiates long polling calling the function fetchLatest. This function, at lines 7 to 19 will use jQuery to perform a GET request asynchronously that expects a JSON array in response. The message objects in that array are iterated over and the HTML is updated for each message. Finally, at line 17 the function sets a timer to call itself (i am not sure if tail calls are optimized).

If you run the chat server and open two Firefox windows and in each go to the URL http://localhost:9999/app/ it should be possible send and receive chat messages from each window.

How messages are serialized to JSON

Instances of List[Message] are serialized where Message is as follows:

class Message(@BeanProperty val date : Date,
              @BeanProperty val message : String) {
}

The Jackson JAX-RS library is utilized to serialize a list of Message. The @BeanProperty annotation tells the Scala compiler to generate bean getter and setter methods and ths enables Jackson to workout what the bean properties are.

This is where it gets a little hacky (note that i have not investigated any Scala libraries for producing JSON). First, it is necessary to work around a bug in Atmosphere which does not take into account the media type declared in the @Produces when broadcasting (we are gong to fix that!). Second, Jackson does not know how to serialize the instances of the Scala List. So it is necessary to create a message body writer as follows:

01  @Provider
02  @Produces(Array("\*/\*"))
03  class ListProvider extends JacksonJsonProvider {
04  
05    private val arrayListClass = classOf[java.util.ArrayList[_]];
06  
07    override def isWriteable(c: Class[_],
08                             gt: Type,
09                             annotations: Array[Annotation],
10                             mediaType: MediaType) : boolean = {
11      classOf[List[_]].isAssignableFrom(c) &&
12          super.isWriteable(arrayListClass, arrayListClass,
13                            annotations, MediaType.APPLICATION_JSON_TYPE);
14    }
15  
16    override def writeTo(t: Object,
17                         c: Class[_],
18                         gt: Type,
19                         annotations: Array[Annotation],
20                         mediaType: MediaType,
21                         httpHeaders: MultivaluedMap[String, Object],
22                         entityStream: OutputStream) : unit = {
23  
24      val l = t.asInstanceOf[List[_]];
25      val al = new java.util.ArrayList[Any]();
26      for (m <- l) {
27        al.add(m);
28      }
29  
30      super.writeTo(al, arrayListClass, arrayListClass,
31                    annotations, mediaType, httpHeaders, entityStream);
32    }
33  } 

The JacksonJsonProvider is extended and the methods associated with writing entities are overridden. This writer declares that the Scala type List is supported, at line 11, and the List is converted to an instance of java.util.ArrayList that Jackson understands when writing, at lines 24 to 28.

Using the Atmosphere spade server

The application is run using the Atmosphere spade server, which in turn uses embedded Grizzly Comet:

01  object ChatServer {
02    def main(args: Array[String]) {
03      try {
04        AtmosphereSpadeServer.build(args(0), "org.atmosphere.samples.lpchat").start();
05      } catch {
06        case ex : Exception => ex.printStackTrace;
07      }
08    }
09  }

Also since i was having much fun with JAX-RS, Jersey and Scala the HTML and JavaScript is returned from the following root resource class:

01  @Path("/")
02  class FilesResource {
03  
04    @Path("jquery{id}.js")
05    @GET
06    def getJQuery(@PathParam("id") ps : PathSegment) = new File(ps.getPath());
07  
08    @GET
09    def getIndex() = new File("index.html");
10  }

Passing thoughts

While it is obvious that this example is just a toy it does IMHO get across the concept on long polling rather well and example is very concise. I think this could be a good basis for a presentation to a JUG.

Examples such as this could also serve as the basis to develop common patterns and mechanisms in Atmosphere to manage lists of messages. The main problem in this respect is how store and retrieve messages. There are a myriad of ways and Atmosphere should preferably not dictate certain solutions in this respect e.g. perhaps i could convert the chat example to store and query messages using Couch DB?

Friday Jun 06, 2008

JavaOneDemo in Scala

For some light entertainment I converted the JavaOneDemo over to a Scala-based version, see here. It is by no means complete, there is still some Java code (which shows how one can mix the two in the same project) and i know it can be improved by using Scala's native XML support.

I am getting increasingly grumpy with the verbosity of Java and the reduction that Scala offers is good (i wish i could write Jersey completely in Scala, i would be more productive and i think the code would be better, but the downside is it would limit adoption for those wishing to mess around with the code base).

Here is a bit of the Java code:

@ProduceMime("application/xhtml+xml")
public class SystemProperty {
    Properties properties;
    String name;    
    String value;
    SystemProperty(Properties properties, String name) {
        this.properties = properties;
        this.name = name;
        this.value = properties.getProperty(name);
        if (value == null)
            throw new WebApplicationException(Response.Status.NOT_FOUND);
    } 
    @GET
    public Property getProperty() {
        return new Property(name, value);
    }    
    @PUT
    @ConsumeMime("text/plain")
    public void updateProperty(String newValue) {
        properties.setProperty(name, newValue);
    }    
    @DELETE
    public void deleteProperty() {
        properties.remove(name);
    }
}

Now compare that with the equivalent Scala code:

@ProduceMime(Array("application/xhtml+xml"))
class SystemProperty(properties : Properties, name : String) {
    val value = properties.getProperty(name)
    if (value == null)
        throw new WebApplicationException(Response.Status.NOT_FOUND)
    @GET
    def getProperty = new Property(name, value)
    @PUT
    @ConsumeMime(Array("text/plain"))
    def updateProperty(newValue : String) : unit = 
        properties.setProperty(name, newValue)    
    @DELETE
    def deleteProperty() : unit = properties.remove(name)
}

This class uses Scala concise constructors and the Scala compiler is smart enough to know whether the constructor parameters need to be fields on the class. Unfortunately Scala does not perform auto-boxing of arrays, which is why you see Array("application/xhtml+xml") in the @ProduceMime annotation (this gets more annoying when using the JAX-RS builder-related methods). Notice how you can write the equivalent constructor logic directly in the class.

For first @GET method Scala does not require that you declare the return type, it works it out. So this can be written concisely in one line.

I would like to write some "Scala-sugar" for classes like UriInfo and Response to improve EOU in Scala in addition to further investigating use of Scala features like closures as experimented with previously.

Monday Nov 26, 2007

Using Jersey and Scala's features for HTTP preconditions

Jersey has a way to plug in processing of Java methods that are HTTP methods (annotated with a @HttpMethod). A resource method dispatch provider analyzes a method signature (of a resource class) and if the signature is suitable returns a request dispatcher that knows how to transform a HTTP request to instances of the method parameter types and transform an instance of the returned type to a HTTP response.

(Note that this plug-in mechanism is not currently exposed as a documented SPI and it is mostly internal to Jersey. However, the plan is to expose this as documented functionality. Also what is presented below is only possible in the latest builds of Jersey so this is all still very much experimental.)

This can make for interesting ways to define and process HTTP methods especially using features of other JVM-compatible languages, like Scala, that support closures.

A RESTful framework like Jersey has some areas where it would be useful for a developer to return some meta-data and a function where by the runtime analyzes the meta-data and from that data the runtime decides if the function should be called or not. Such areas are:

  • precondition support, an HTTP method returns meta-data consisting of an entity tag and/or a last modified date, and a function to be called if preconditions are met. The runtime checks the meta-data against the associated HTTP request headers and if preconditions are not met a 304 (Not Modified) or a 412 (Precondition Failed) response is returned (and the function is not called).
  • variant selection, an HTTP method that may produce one or more representations consisting of one or more media types, languages, character sets and encodings can return a 'sparse' variant matrix and a function to be called with the variant selected by the runtime according to what the client accepts.

I strongly suspect that 100 (Continue) responses can also fall into area and perhaps more generally Comet-based support, for example, using Scala's actors.

I hacked together a proof of concept resource method dispatch provider written in Scala to support preconditions. So i can write a simple Scala resource class as follows:

import PartialResponse._

@UriTemplate("/")
class MethodService {

    @ProduceMime(Array("text/plain"))
    @HttpMethod
    def get() = partialResponse("1234") {
        "RESPONSE"
    }
} 

(Obviously this example is contrived since the entity tag is hard coded).  

The partialResponse is a method imported on the object PartialResponse and it takes as the first parameter a string that is the entity tag "1234" and as the second parameter a function (essentially a closure) that is the block of code that returns the string "RESPONSE".

I deployed this class using the Light Weight HTTP server:

HttpServerFactory.create("http://localhost:9998/method").start()

(Thanks to a recent feature change Jersey now automatically picks up resource classes by scanning the Java class files, by default, using the java class path.) 

And performed a GET request using curl that returns a 304 (Not Modified) response:

> GET /method/ HTTP/1.1
> Host: localhost:9998
> Accept: \*/\*
> If-None-Match: "1234"
>
< HTTP/1.1 304 Not Modified
< Content-length: 0
< Etag: "1234"

So as you can see in this case preconditions are not being met: essentially the client already has the latest state of the resource "/method". 

The code for PartialResponse is as follows:

object PartialResponse {
    def partialResponse(t: String)(body: => Object):
            PartialResponse = {
        val response = new PartialResponse {
            def entityTag() : String = t
            def respond() : Object = body
        }
        response
    }
}

trait PartialResponse {
    def entityTag() : String
    def respond() : Object
}

The partialResponse method creates a anonymous instance of the trait PartialResponse and sets the methods entityTag and respond accordingly. 

The code for the resource method dispatch provider and request dispatcher is presented at the end of this blog and below i highlight the relevant code that performs the precondition calculation:

01  val pr = method.invoke(resource, null).
02          asInstanceOf[PartialResponse]
03
04  val t = new EntityTag(pr.entityTag)
05  var r = request.evaluate(t)
06  if (r == null) {
07      val ar = pr.respond()
08
09      val mediaType = getAcceptableMediaType(request);
10      r = Response.Builder.representation(ar, mediaType).
11              tag(t).build();
12  }
13  response.setResponse(r);

Line 1 and 2 invokes the Java method that is the HTTP method (here it is assumed the method takes no parameters) and the response is cast to a PartialResponse (the resource method dispatch provider ensures that the request dispatcher will only be created for HTTP method with the appropriate return type).

Line 4 obtains the entity tag declared by the HTTP method.

Line 5 evaluates the entity tag for the HTTP request. If the result of the evaluation is not null then preconditions have not been met and the response is returned. If null then preconditions have been met and the respond method of the application is called at line 7.

Lines 9 and 10 build up the response from the value returned by the respond method, the media type and the entity tag.

Of course i am sure this can be vastly improved but the proof of concept shows that it is possible to integrate Scala language features with Jersey in such a manner that has the potential to make it easier and more 'natural' to develop RESTful Web services. Such things are possible with Java by anonymously extending abstract classes, but i think this is unwieldy for developers over the more imperative approach in application code and maybe more so if preconditions are combined with variant selection and potentially 100 (Continue). Of course Java closures in SE 7 would be most welcome especially if there is good compatibility with Scala (perhaps it might be possible to support SE 7 application code with closures and a Jersey runtime compiled to SE 5 or 6 using the same tricks as i have implemented for Scala?).

The next steps are to refine this to work for variant selection and preconditions. I suspect the variants are best handled by Scala's pattern matching, and it will be interesting to see how variant selection and precondition checking can be combined (since the entity tag or last modified to use may depend on the variant that is selected).


import javax.ws.rs.core._
import com.sun.ws.rest.impl.model.method.dispatch.ResourceMethodDispatchProvider
import com.sun.ws.rest.impl.model.method.dispatch.ResourceJavaMethodDispatcher
import com.sun.ws.rest.api.core.HttpRequestContext;
import com.sun.ws.rest.api.core.HttpResponseContext;
import com.sun.ws.rest.api.model.AbstractResourceMethod
import com.sun.ws.rest.spi.dispatch.RequestDispatcher

class ScalaMethodProvider extends ResourceMethodDispatchProvider {

    class PartialResponseDispatcher(arm: AbstractResourceMethod)
            extends ResourceJavaMethodDispatcher(arm) {

        def _dispatch(resource: Any, request: HttpRequestContext,
                response: HttpResponseContext) : Unit = {
            val pr = method.invoke(resource, null).
                asInstanceOf[PartialResponse]

            val t = new EntityTag(pr.entityTag)
            var r = request.evaluate(t)
            if (r == null) {
                val ar = pr.respond()

                val mediaType = getAcceptableMediaType(request);
                r = Response.Builder.representation(ar, mediaType).
                    tag(t).build();
            }
            response.setResponse(r);
        }
    }

    def create(arm: AbstractResourceMethod) : RequestDispatcher = {
        val rt = arm.getMethod().getReturnType()
       
        if (classOf[PartialResponse].isAssignableFrom(rt)) {
            return new PartialResponseDispatcher(arm);
        } else {
            null
        }
    }
}

Thursday Nov 01, 2007

Concurrency with Scala (was Concurrency with Erlang)

Steve Vinosky has written some really interesting articles on Erlang concurrency and Erlang reliability. I think i am going to buy that book on Erlang he recommends.

After i read his article on concurrency i wondered if the same (contrived but instructive) example he presents could be written in Scala and how it would compare, if anything it is a good exercise in learning.

Here is the simple non-concurrent (tail call) recursive pow function in Scala:

def pow(n: int, m: int): BigInt = {
    def _pow(m: int, acc: BigInt): BigInt  = m match {
        case 0 => acc
        case _ => _pow(m - 1, acc \* n)
    }
     _pow(m, 1)
}

and here is the concurrent cpow function using Scala's event-based actors:

def cpow(n: int, m: int): BigInt = {
    val actors = for (_ <- List.range(0, m)) yield
actor { react { case x: int => sender ! x } }
    actors foreach ( a => a ! n )
    (actors foldLeft BigInt(1)) { (t , a) => self ? match { case x: int => t \* x } }
}    

The value actors, which is a list of m actors, is obtained using list comprehension. As in Steve's example each actor just pings back the value it receives from the sender. Then a message is sent to each actor with the value of n. Finally all the values received from the actors are multipled together (as BigInt types) using the foldLeft function. Rather elegant! and similar in size to the Erlang code.

I measured the time it took for both functions to calculate 502000 on my Acer Ferrari 3400 laptop running Solaris, Java SE 5, and Scala 2.6.0. The pow function took 17.86 ms and the cpow function took 81.28 ms.

Below I re-iterate one of Steve's final points with a slight modification:

..., many developers are comfortable with OO programming. I’d like to advise such developers not to let Erlang’s or Scala's functional nature scare you away, ...

Given that Scala is a hybrid object-oriented and functional programming language it can enable developers to smoothly transition between the imperative and functional programming styles in  small manageable steps i.e. Scala is less scary than it might initially appear.

Wednesday Oct 31, 2007

Tail recursion in the Jersey Mandel example

I was browsing through John Rose's very detailed blog entry on method calls entitled "anatomy of a call site" and it got me browsing through his past entries. When i got to the blog on "tail calls in the VM" it reminded me that i wanted to investigate tail calls in Scala and see how they translate into Java byte code.

The Mandel example in Jersey is written in Scala and generates images of the Mandelbrot set. To determine if a point, C, on the complex plane is a member of the set one calculates Zn using the functions:

Z0 = C

Zi+1 = Zi + C

where n is equal to some limit on the number of iterations or where the modulus of Zn is greater than 2. If n reaches the limit then the point C is considered a member of the set otherwise n represents the rate at which the function escapes to infinity for the point C.

My original code to calculate n was as follows:

def iter(c: Complex, limit: int) = {
    var z = c
    var i = 0;
    while(z.modSquared < 4 && i < limit) {
        z = z \* z + c;
        i = i + 1;
    }

    i
}

Very imperative! How can we make this functional without blowing up the stack? we make tail calls of course! which basically means making sure that the last operation before the return of the value is a function call to the same function (note that if you read some of the comments to John's blog you can see how easy it is to get confused over what constitutes a tail call). So we can re-write the method as follows:

def iter(c: Complex, limit: int) = {
    limit - _iter(c, c, limit)
}

private def _iter(z: Complex, c: Complex, limit: int) : int = {
    if (limit == 0 || z.modSquared >= 4) return limit

    _iter(z \* z + c, c, limit - 1);
}

I used two separate methods so i could decrement the limit in the recursive _iter function whilst retaining the same contract for the iter function as in the previous code. I can verify that scala recognizes this as a tail call by looking at the byte code generated for the _iter function:

 0 iload_3
 1 iconst_0
 2 if_icmpeq 15 (+13)
 5 aload_1
 6 invokevirtual #26 <mandel/Complex.modSquared>
 9 iconst_4
10 i2d
11 dcmpg
12 iflt 17 (+5)
15 iload_3
16 ireturn
17 aload_1
18 aload_1
19 invokevirtual #30 <mandel/Complex.$times>
22 aload_2
23 invokevirtual #33 <mandel/Complex.$plus>
26 aload_2
27 iload_3
28 iconst_1
29 isub
30 istore_3
31 astore_2
32 astore_1
33 goto 0 (-33)

As you can see line 33 jumps all the way back to the start of the function.

To compare with Java compiler i wrote some similar functions in Java:

public int iter(Complex c, int limit) {
    return limit - _iter(c, c, limit);
}
    
private int _iter(Complex z, Complex c, int limit) {
    if (limit == 0 || Complex.modSquared(z) >= 4) return limit;
    
    return _iter(Complex.add(Complex.multiple(z, z), c), c, limit - 1);
}

Here is the byte code for the Java _iter :

 0 iload_3
 1 ifeq 15 (+14)
4 aload_1
 5 invokestatic #3 <mandel/Complex.modSquared>
 8 ldc2_w #4 <4.0>
11 dcmpl
12 iflt 17 (+5)
15 iload_3
16 ireturn
17 aload_0
18 aload_1
19 aload_1
20 invokestatic #6 <mandel/Complex.multiple>
23 aload_2
24 invokestatic #7 <mandel/Complex.add>
27 aload_2
28 iload_3
29 iconst_1
30 isub
31 invokespecial #2 <mandel/Main._iter>
34 ireturn

In the case of Java line 31 invokes the _iter method.

Things would of course be better for Scala and other VM-based languages (or implementations of) if the JVM had support for hard tail calls.

Tuesday Oct 30, 2007

Annotated Scala, Groovy, JRuby, Rhino, Jython ...

Java runtime annotations are stored in Java bytecode using attributes, as specified by the Java virtual machine specification. So one can use toolkits like the ASM toolkit for creating and reading annotations in addition to using the Java reflection API for reading annotations.

Implementations of languages that compile to Java bytecode can also leverage runtime annotations and it can make for really smooth integration between classes/objects written in the language and frameworks, such as Jersey (JAX-RS), Metro (JAX-WS, JAXB), JUnit and Spring etc. written in Java, that process annotations.

So what's the support for runtime annotations like out there for some common JVM-based language implementations?

Scala

Scala has by far the most robust support for annotations. The annotation syntax is very similar to Java. JAX-RS and Jersey specify support for runtime annotations on classes, fields, methods and method parameters. I can annotate Scala classes using JAX-RS annotations and the Jersey runtime correctly processes the thoses classes (as shown here). In fact since the integration between Java and Scala is so smooth and Scala being such a pleasurable language to code in i could not resist including in the Jersey distribution an example written in Scala.

Groovy

Groovy 1.1 supports runtime annotations and like Scala the syntax is very similar to Java. Originally i played around with Jersey and Groovy 1.1 Beta. This version supported annotations on classes, method and field but not on method parameters. Recently downloaded Groovy 1.1 RC 1 and this version supports annotations on method parameters. So i can annotate Groovy classes and support all the Jersey features as i can in Scala (Groovy 1.1 is not yet final where as Scala 2.6 is so Scala's support is considered more robust). Since Groovy requires no compile step i can script Jersey resources (more on this later in another blog).

JRuby (Ruby)

Now things start to get a little fuzzy. My understanding is Ruby does not currently support annotations but that it is very easy to do so because of Ruby's meta-programming capability, as shown here. Charles Nutter (one of the JRuby leads) says here:

"JRuby doesn't support annotations because Ruby doesn't support annotations. So what! We can extend Ruby..."

and further says "I've planted the seed here and on the JRuby dev list...let's see if it germinates.", but i don't know if that seed has taken root...

Rhino (JavaScript)

JavaScript is in a similar position to JRuby, annotations can be added to JavaScript using meta-programming techniques, as shown here.

Rhino has the ability to integrate with Java such that a JavaScript object can implement one or more Java-based interfaces or extend a Java class. However, it does not seem possible to implement a (plain old) JavaScript object and pass it over to the Java side as a (plain old) Java object. Intuitively this would seem possible given the existing support. It looks like the Rhino source code is still being worked on even though the last release (1.6R7) was over a year ago. That's good. So i took a look at the code, specifically the JavaAdaptor and ClassFileWriter classes.

Now i may not know what i am getting into here, i could be pulling on a thread connected to a really big hair ball, it appears "tricky but doable". Once that is done it is just a simple matter of mapping the annotations declared in the JavaScript to equivalent byte code... anyone up for a challenge?

Jython (Python)

Python has function decorators in Python 2.4, function annotations and class decorators implemented in the subversion repository of python 3.0 (or Python 3000, it says here that it will go final in August 2008). So perhaps Python 3000, when it is ready, would be the right place to start any Java-based annotation support.

The latest stable release of Jython is version 2.2.1 supporting Python 2.2. From the roadmap Jython 2.5 is next (no date given) and this release will target Python 2.5. It says the core developers will be:

"concentrating on making a clean 2.5, while improving the java integration"

So maybe something specific to Jython w.r.t. to annotation support could be implemented. Although the Jython roadmap looks on the right track i suspect the 2.5 release won't be happening any time soon and the 3.0 release is way over the horizon. However, the competition between language implementations is in rude health so perhaps things will zip along faster than I expect.

Monday Oct 22, 2007

Mandelbrots as a Web service

I have been mostly silent over the last two weeks. The reason being my wife and I have just had a kid. In between periods of changing, comforting, <insert task here>, and attempting to construct a users guide for a newly introduced mind i have been playing around with something that would be fun, visually appealing and help me learn about Scala and Scala actors.

The result is a simple Web service, utilizing Jersey and written in Scala, that calculates the Mandelbrot set over a specified area (in the complex plane) and renders that area as an image. (See the latest build, which contains this example). I am sure the code shows that i am a Scala novice! (i know for sure the Mandelbrot calculation is not very efficient).

The Mandelbrot Web services is implemented using Jersey as follows:

1 @UriTemplate("({lx},{ly}),({ux},{uy})")
2 class MandelService(
3         @UriParam("lx") lx: double,
4         @UriParam("ly") ly: double,
5         @UriParam("ux") ux: double,
6         @UriParam("uy") uy: double,
7         @DefaultValue("512") @QueryParam("imageSize") imageSize: int,
8         @DefaultValue("512") @QueryParam("limit") limit: int,
9         @DefaultValue("8") @QueryParam("workers") workers: int) {
10
11     val lower = new Complex(lx, ly);
12     val upper = new Complex(ux, uy);
13     val dx = upper.re - lower.re;
14     val dy = upper.im - lower.im;
15
16     val width : int = if (dx >= dy) imageSize else (dx/dy \* imageSize).toInt
17
18     val height : int = if (dy >= dx) imageSize else (dy/dx \* imageSize).toInt
19   
20     @ProduceMime(Array("image/png"))
21     @HttpMethod
22     def getMandelbrot() = {
23         val image = new BufferedImage(width, height, BufferedImage.TYPE_INT_RGB);
24         new ParallelMandelRenderer(workers, lower, upper, limit,

25 image.getRaster()).render();
26         image
27     }
28 }

As you can see the actual Web service is very simple. Read here for general details.

This class will get instantiated every time the request path of the URI matches the URI template declared at line 1.

Lines 2 to 9 utilize Scala's compact syntax for class constructors. The URI parameters, in lines 3 to 6, represent the bottom left and upper right coordinates, as double types, of the area (of the complex plane) to perform calculations over. The query parameters, in lines 7 to 9, have default values declared such that if a query parameter is absent from the URI the default value will be used instead.

Lines 16 and 18 calculate the width and height of the image and ensure that the aspect ratio is the same as the area to perform calculations over.

Lines 20 and 22 declare an HTTP GET method that produces an image in the PNG format. Line 23 creates a new BufferedImage according to the calculated width and height. Lines 23 and 24 calculate the Mandelbrot set, in parallel (or more specifically concurrently), over the specified area and renders the results into the raster of the buffered image. Line 26 returns the image. The example contains an entity provider that serializes types of java.awt.image.RenderedImage, using the javax.imageio.ImageIO class according to media type declared in line 20.

The ParallelMandelRender utilizes event-based Scala actors (see here for a paper on Scala actors, the preliminary results in section 7 are very interesting). The render method is as follows:

1 override def render() : unit = {
2     val c = new JobCoordinator(n)
3     yRanges foreach ( x => c ! (new c.Job { def execute = render(x._1, x._2) }) )
4     c.waitForCompletion
5 }

Line 2 creates a new JobCoordinator where 'n' is the number of workers (actors) to instantiate (more on this later). In line 3 yRanges represents the image height split into ranges according to the number of workers. For each y-range a new job is created to render that y-range and that new job is sent as a message to the coordinating actor encapsulated by the JobCoordinator instance. The coordinating actor then sends that job to a worker that is available to receive work. Line 3 waits until all jobs have completed.

Strictly speaking it should not be necessary to specify the number of workers. Scala's scalability of event-based actors is very good and the underlying runtime should (in theory) utilize as many threads as the system configuration allows. So i could create as many actors as there are horizontal lines in the image and simplify the code (or for every pixel, who wants to port this to a DAP!, but that is a bit extreme). However, the current approach allows me to verify scalability and easily compare thread-based to event-based actors (changing 'react' to 'receive' in the actor code).

Now, all i need is to get my hands one of those T5120 servers and give it a serious workout doing some fractal calculations!

As the Mandelbrot wikipedia entry says the algorithm is embarrassingly parallel. Not only should it be easy to scale up but once the service is exposed as a network-based service it should be easy to scale out. So another way to scale the service is to have a Web client split the area into tiles and send out multiple requests. A client that was capable of browsing the Mandelbrot set a bit like browsing a map using say google maps would be really cool :-) something i would definitely like to include the example.

Monday Sep 24, 2007

Deploying Scala resource classes

In my last blog entry i neglected to mention how i was deploying and testing Scala classes.

I used the light weight HTTP server with the following Scala code: 

 object Server {
import com.sun.ws.rest.api.container.ContainerFactory
import com.sun.net.httpserver.HttpHandler
import com.sun.net.httpserver.HttpServer
import java.net.InetSocketAddress
import java.util.HashSet

def main(args: Array[String]) {
     val resources = new HashSet()
     resources.add(classOf[MyResource])
        val handler =
            ContainerFactory.createContainer(classOf[HttpHandler], resources).
        asInstanceOf[HttpHandler];
        val server = HttpServer.create(
            new InetSocketAddress(9998), 0);
        server.createContext("/service", handler);
        server.setExecutor(null);
        server.start();
        println(server);
        Thread sleep 100000
    }
}

Eduardo wanted to know if it is possible to deploy Scala resource classes in Glassfish. Here is how.

1) Create a new Web application in NetBeans called "ScalaWebApplication".

2) Create a new package called "com.sun.ws.rest.samples.myscala.resources".

3) Create a file ScalaResource.scala in the new package with the following code:

import java.util.List
import javax.ws.rs.{UriTemplate, HttpMethod, ProduceMime, UriParam}
import javax.ws.rs.core.{HttpContext, UriInfo, HttpHeaders}
import scala.collection.mutable.JavaMapAdaptor

package com.sun.ws.rest.samples.myscala.resources {

@UriTemplate("/{id}")
class ScalaResource(@UriParam("id") id: String,
                 @HttpContext var uriInfo: UriInfo,
                 @HttpContext var headers: HttpHeaders) {

    var hmap = new JavaMapAdaptor[String, List](
    headers.getRequestHeaders())

    @ProduceMime(Array("text/html"))
    @HttpMethod
    def getMe() = <html>
    <body>
    <p>ID: { id }</p>
    <p>URI : { uriInfo.getAbsolute() }</p>
    <table>
        <tr>
        <th>Header</th>
        <th>Value</th>
        </tr>
        { for (header <- hmap) yield
        <tr>
        <td>{ header._1 }</td>
        <td>{ header._2 }</td>
        </tr>
        }
    </table>
    </body>
</html>.toString()
}
}

4) Create a file WebResources.scala in the new package with the following code: 

package com.sun.ws.rest.samples.myscala.resources;

import com.sun.ws.rest.api.core.DefaultResourceConfig;

class WebResources extends DefaultResourceConfig {
    getResourceClasses().add(classOf[ScalaResource]);
}

5) Modify the web.xml to be as follows:

<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee">
<servlet>
  <servlet-name>Jersey Web Application</servlet-name>
    <servlet-class>com.sun.ws.rest.impl.container.servlet.ServletAdaptor</servlet-class>
    <init-param>
      <param-name>webresourceclass</param-name>
        <param-value>com.sun.ws.rest.samples.myscala.resources.WebResources</param-value>
      </init-param>
      <load-on-startup>1</load-on-startup>
    </servlet>
    <servlet-mapping>
      <servlet-name>Jersey Web Application</servlet-name>
      <url-pattern>/\*</url-pattern>
    </servlet-mapping>
</web-app>

6) Add the following to the project build.xml

    <property
        name="scala.home"
        value="${scala-2.6.0-final}/share/scala"
    />
  
    <target name="-pre-compile">
        <taskdef resource="scala/tools/ant/antlib.xml">
            <classpath>
                <pathelement location="${scala.home}/lib/scala-compiler.jar"/>
                <pathelement location="${scala.home}/lib/scala-library.jar"/>
            </classpath>
        </taskdef>
              
        <path id="build.classpath">
            <path path="${javac.classpath}"/>
            <pathelement location="${build.classes.dir}"/>
        </path>
      
        <scalac srcdir="${src.dir}"
                destdir="${build.classes.dir}"
                classpathref="build.classpath"
                target="jvm-1.5"
                force="changed">
            <include name="\*\*/\*.scala"/>
        </scalac>
    </target>

7) Add the directory location of the Scala 2.6.0 final distribution to the nbproject/project.properties file using the property name "scala-2.6.0-final". (This is the directory location that contains the directories "bin" and "share".)

8) Add the scala library jar "${scala-2.6.0-final}/share/scala/lib/scala-library.jar" to the list of compile libraries.

9) Add the following jars in the Jersey 0.3 distribution to the list of compile libraries: lib/jersey.jar, lib/persistence-api-1.0.jar, lib/jsr311-api.jar.

10) Deploy.

11) Goto the URL:

http://localhost:8080/ScalaWebApplication/1

... and hopefully you will see some HTML.

Notice that there is no special configuration of Scala and this should work with Glassfish V2 and V3. All that is required is correct configuration of the web.xml with the Servlet Jersey uses to dispatch HTTP requests and the initialization parameter "webresourceclass", which points to the compiled Java class of WebResources.scala. 

Obviously this whole process could be better improved with some built in support for compiling Scala source files in NetBeans.

Latest Jersey with latest Scala

Version 2.6.0 of Scala was recently released so i thought i would try out this version with the new life-cycle changes in 0.3 of Jersey.

The following Scala class works just fine when deployed:
 1 @UriTemplate("/{id}")
2 class MyResource(@UriParam("id") id: String,
3                  @HttpContext var uriInfo: UriInfo,
4                  @HttpContext var headers: HttpHeaders) {
5
6     var hmap = new JavaMapAdaptor[String, java.util.List](
7     headers.getRequestHeaders())
8
9     @ProduceMime(Array("text/html"))
10     @HttpMethod
11     def getMe() = <html>
12     <body>
13     <p>ID: { id }</p>
14     <p>URI : { uriInfo.getAbsolute() }</p>
15    <table>
16        <tr>
17         <th>Header</th>
18         <th>Value</th>
19         </tr>
20        { for (header <- hmap) yield
21        <tr>
22         <td>{ header._1 }</td>
23         <td>{ header._2 }</td>
24        </tr>
25        }
26     </table>
27     </body>
28 </html>.toString()
29 }

The above class supports a HTTP GET method that returns an HTML representation containing the value of the URI parameter, the absolute URI of the request, and a table of the HTTP request headers. 

Scala has a compact syntax for class constructors, see lines 2 to 3. The URI parameter "id" is a field on the class as well as a parameter to the constructor of the class. Jersey correctly passes the URI parameter when creating a new instance of this class.

Line 6 declares a field on the class that adapts the HTTP request headers, that is of the type MultivaluedMap, to a scala.collections.mutable.Map so the headers can be accessed using familiar Scala operations on collections.

Lines 9 to 11 declare an HTTP method that responds to a GET request with a HTML representation generated using Scala's XML support.

Lines 20 to 24 iterate over the HTTP request headers. Each header is a tuple of two values, the header name and a java.util.List of values. This functionality is only possible if the HTTP request headers are adapted (see line 6).

As you can see Scala's close integration with Java makes it easy to use Jersey with Scala. However, i think such integration could be improved by providing appropriate Scala adapters for injected interfaces so that a developer is not required to do such adaption.

I am still very new to Scala and need to get stuck into writing a more "real world application". Perhaps a good exercise would be to write in Scala what Tim is doing in Erlang.

[Update on comments]

Michael, see my other blog entry after this on how get things working with GF. It proved to be reasonably simple given the smooth integration of Java and Scala. What is really missing is NetBeans support for editing and compiling Scala classes. If that was in place deploying to GF would be very simple.

Rafael, thanks for the tip, that is much better. I was wondering how to that and the solution you presented is obvious, of course with hindsight :-)

Friday Jul 06, 2007

Skittr on GlassFish

I finally got around to playing around with Skittr (a Twitter clone written in Scala lift web framework). I built the Skittr example from the main trunk of the lift source and deployed into the latest v2 of GlassFish (v2 beta 2) and it worked:


 

There are HTML rendering issues with Firefox 2.0.0.4 (and Safari 3.0 beta, i have not tried on IE) which makes it hard to enter in messages (Firebug is most useful in this respect!) but otherwise it all seems to function.

Thursday Jul 05, 2007

More scala with Jersey

The following Scala class exercises just about every annotation-based mechanism currently supported by Jersey:

import javax.ws.rs.{UriTemplate, HttpMethod, ProduceMime, UriParam}
import javax.ws.rs.core.{HttpContext, UriInfo, HttpHeaders}

@UriTemplate("/{id}")
class MyResource {

    @HttpContext var uriInfo: UriInfo = _

    @HttpContext var headers: HttpHeaders = _


    @ProduceMime(Array("text/html"))
    @HttpMethod
    def getMe(@UriParam("id") id: String) = <html>
    <body>
    <p>ID: { id }</p>
    <p>URI : { uriInfo.getURI() }</p>
    </body>
</html>.toString()


    @HttpMethod
    def postMe(@UriParam("id") id: String,
        entity: String) : String = <xhtml>
    <body>
    <p>POSTed entity: { entity }</p>
    <p>Content-Type: { headers.getMediaType() }</p>
    </body>
</xhtml>.toString()


    @UriTemplate("sub-method")
    @HttpMethod
    @ProduceMime(Array("text/plain"))
    def getSubMe = "sub-method"

    
    @UriTemplate("sub-class")
    def getSubClassResource = new SubClassResource

    @ProduceMime(Array("text/plain"))
    class SubClassResource {
        @HttpMethod
        def getSubMe = "sub-class resource"
    }


    @UriTemplate("sub-object")
    def getSubObjectResource = SubObjectResource

    @ProduceMime(Array("text/plain"))
    object SubObjectResource {
        @HttpMethod
        def getSubMe = "sub-object resource"
    }
}

The UriInfo and HttpHeaders are injected into fields of the MyResource class. The template parameter value for the template parameter {id}, using the UriParam annotation, is injected as parameter of the getMe and postMe methods. Sub-methods (annotated with UriTemplate and HttpMethod) work fine, as do sub-locator methods (annotated with UriTemplate) getSubClassResource and getSubObjectResource that return a new class or an object, respectively, to continue the URI matching process.

The use of the XML syntax is nice (although i wish there were a way to switch off mixed content i.e. ignoring the white space for indenting). I suspect that it is possible to develop an entity provider for scala.xml.Elem class to convert to a String and to also strip out indenting white space.

Friday Jun 08, 2007

Skittr cloning Twitter

Skittr is a clone of Twitter written by David Pollak in 884 lines of Scala. It uses the lift web framework. What caught my eye was the following:

It can handle 1M+ users on a two Intel Core 2 Duo boxes (one as the web front end and the other as the message broker.) With a weeks of full-time coding, it could be modified to scale linearly with the number of boxes in the system.

Impressive scalability in conjunction with the Java VM and a Servlet engine.

A side question: can functional programming languages help programmers better develop scalable applications for multi-core chips? Cause at the moment there is an issue that is nicely expressed by Sean McGrath:

The processor will stop doubling in speed and halving in cost. Instead, you will find more and more processors shipping in each computer.

This is the future because the hardware people are creating it that way. The software people need to realize that fact and start figuring out how to use all the processors. This future does not just involve just re-compiling your software. It involves turning it on its head in most cases.

Thursday Jun 07, 2007

Resourcescala

 What's the following?

import com.sun.ws.rest.api.UriTemplate
import com.sun.ws.rest.api.HttpMethod

@UriTemplate("/test")
class MyResource {

    @HttpMethod
    def getMe = "HELLO FROM A SCALA RESOURCE"
}

It is a resource i wrote in Scala and deployed to the Java light weight HTTP server using the RESTful API in R2 of SWDP. It is really nice when things just work :-)

Scala appeals as an experiment in scriptingfunctional languages because it supports Java annotations and thus i can use the RESTful runtime. So i can experiment with the design of resources and the producing/consuming of information in ways that are not possible with the Java language. Still, it's been a while since i played around with a functional language...

About

sandoz

Search

Archives
« July 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
  
       
Today