Friday May 16, 2008

Server-side image processing with JRuby-on-Rails and the Java 2D API

One of the many advantages of creating Rails applications with JRuby is the access that JRuby gives you to the rich set of Java libraries available in the Java platform.

This blog entry provides step-by-step instructions for creating a simple Rails application that uses the Java 2DTM API to perform server-side image processing. It also shows you how to install and use the GlassFishTM v3 Gem, which contains only the GlassFish v3 kernel, Grizzly, and other utilities, thereby giving you an application server with a smaller size and a faster start-up time.

Although Rails is intended for developing database-backed web applications, this application does not use a database. Many think that it is better to use the file system rather than a database to store binaries. I'll leave it up to you whether you use a database or not for that. In any case, this blog entry focuses on taking advantage of JRuby to access Java platform libraries from a Rails application.

Installing the Software

To get started, go through these steps to install JRuby, Rails, and the GlassFish v3 Gem if you haven't already.
  1. Download jRuby-bin-1.1.zip from jruby.codehaus.org and unpack the zip file.
  2. Add the path to your jRuby installation to your system path.
  3. Install Rails on your JRuby VM by running this command:
    jruby -S gem install rails
    
  4. Install the GlassFish v3 gem on your JRuby VM by running this command:
    jruby -S gem install glassfish
    

Creating the Rails Application

Now, let's set up the application like you would with any JRuby-on-Rails application.
  1. Go to <JRUBY_INSTALL>/samples.
  2. Generate the application's directory structure:
    jruby -S rails photo
  3. Go to the photo directory you just created.
  4. Generate a controller and a default view:
    jruby script/generate controller home index
  5. Tell Rails that you are not using a database for this application by doing the following:
    1. Open the <JRUBY_INSTALL>/samples/photo/config/environment.rb file in a text editor.
    2. Remove the hash mark (#) from line 21 so that it reads:
      config.frameworks -= [ :active_record, :active_resource, :action_mailer ]

Setting Up the User Interface

This application has a simple UI that displays an image, a combobox, and a button. The user selects an image-filtering effect from the combobox and clicks the button. When the user clicks the button, the application performs an image processing operation on the displayed image to produce the effect and opens a new page that displays the processed image. Here's a screenshot of the first page:

To set up the UI, do the following:

  1. Copy a JPEG image to <JRUBY_INSTALL>/samples/photo/public/images.
  2. Open <JRUBY_INSTALL>/samples/photo/app/views/home/home.html.erb in a text editor.
  3. Replace its contents with the following:
    <html>
    <body>
    <img src="../../images/kids.jpg"/><p>
    <% form_tag :action => 'seeimage' do -%>
    <%= select_tag "operation",
            "<option selected='selected'>Grayscale</option>
            <option>Negative</option>
            <option>Brighten</option>
            <option>Sharpen</option>
    %>
    <<% end -%>
    </body>
    </html>
    
    Here you're using the form tag and select_tag helpers that Rails provides. It's a good idea to use the helpers rather than to write this HTML by hand because the helpers take care of a lot of extra stuff you'd have to add if you wrote the HTML yourself. When you run the application, just view the source of the page and you'll see what HTML the helpers generate for you. Check out Ruby on Rails Manual ActionView::Helpers for more information on the various helpers for views.

    From the combobox on this page, the user can select from four different image effects: Grayscale, Negative, Brighten, and Sharpen. The selected name of the effect is saved into the operation variable and is passed as a request parameter to the seeimage action of the controller. The action will use this request parameter to execute the appropriate image-processing code.

Adding the Image Processing Code to the Controller

This application gives you a good overview of how to access Java libraries from a Rails application using Ruby code. It also includes a nice sampling of the image-processing operations available in the Java 2D API.

While adding the Ruby code that performs the image processing to the controller, you'll learn the following concepts involved in using Java libraries in a Rails application:

  • Giving your controller access to Java libraries
  • Referring to Java classes
  • Performing file input and output using the java.io and javax.imageio packages
  • Assigning Java objects to Ruby objects
  • Calling Java methods and using variables
  • Converting arrays from Java language arrays to Ruby arrays
  • Streaming files to the client
You'll also learn the following concepts associated with performing image processing with Java 2D:
  • Reading an image file into a buffered image so that you can process it.
  • Obtaining a Graphics2D object from the buffered image so that you can draw the processed image.
  • Using the different image operations offered by Java 2D
  • Filtering a buffered image through one of the image operations
  • Writing the processed image to a byte output stream so that you can stream it to the client.
For your convenience, I've included a copy of the controller online here: home_controller.rb.

Giving the Controller Access to Java Libraries

To allow your controller to use Java libraries, perform these steps:
  1. Open <JRUBY_INSTALL>/samples/photo/app/controllers/hello_controller.rb in a text editor
  2. Add the following line right inside the HomeController class declaration:
    include Java
    
This one line is all you need to access Java libraries from your controller.

Referencing Java Classes

The photo example uses constants to reference Java classes it uses frequently.

To add the constants you need for this application, do the following:

  • After the include Java statement, add the following constant declarations in your controller:
      BI = java.awt.image.BufferedImage
      CS = java.awt.color.ColorSpace
      IO = javax.imageio.ImageIO
    
    Now you can use the constant to reference the class later, as shown by this line:
    bi2 = BI.new(w, h, BI::TYPE_INT_RGB)
    
In addition to using constants, you have three ways to reference a Java class from Ruby code:
  • Use the familiar import statement:
    import java.awt.image.BufferedImage
    ...
    bi2 = BufferedImage.new(w, h, BufferedImage::TYPE_INT_RGB)
    
  • Include the class using the include_class statement:
    include_class 'java.awt.image.BufferedImage'
    
  • Reference the fully-qualified name of the class when invoking its methods:
    filename = "#{RAILS_ROOT}/public/images/kids.jpg"
    file = java.io.File.new(filename)
    
  • Creating the Actions Needed in the Controller

    In a typical Rails application, each of your views maps to an action of the same name in your controller. When you access a view in your browser, the corresponding action executes. You have a view named index.html.erb, and so you need an action called index. As I explained in the section on creating the UI, the form submits to the seeimage action in the controller. Normally, you would need a page called seeimage.html.erb to map to this action. But, in this case, the controller will stream the image to the browser, and so you need actions called index and seeimage, but you don't need a view that maps to seeimage.
    1. Inside the HomeController class declaration and after the constants you added in the previous section, add a seeimage action:
      def seeimage
      end
      
    2. After the seeimage action, add an index action:
      def index
      end
      

    Getting Request Parameters

    All request parameters are accessible through the param method, which returns the parameters in a hash.

    The operation request parameter has the value the user selected from the menu on index.html.erb. To get the value of operation, do the following:

    • Inside the seeimage action, read the value of the operation request parameter into a variable called @data:
         @data = params[:operation]
      
    You'll use this variable to select the proper image-processing operation.

    Reading the Image File Into a Buffered Image

    The next step is to input the image file into an in-memory buffered image, represented by a BufferedImage object so that you can perform operations on the image.
    • Inside the seeimage action, right after the assignment of the operation request parameter into the @data variable, add the following code:
      filename = "#{RAILS_ROOT}/public/images/kids.jpg"
      imagefile = java.io.File.new(filename)
      bi = IO.read(imagefile)
      w = bi.getWidth
      h = bi.getHeight
      bi2 = BI.new(w, h, BI::TYPE_INT_RGB)
      big = bi2.getGraphics
      big.drawImage(bi, 0, 0, nil)
      bi = bi2
      
    The preceding code does the following:
    1. Reads the image file into a File object.
    2. Uses the ImageIO class to store the image file into memory as a BufferedImage object so that you can perform operations on it.
    3. Creates a new BufferedImage with the preferred size and bit-depth to facilitate image processing.
    4. Creates a Graphics2D object from the new BufferedImage object so that the graphics context, or drawing surface, has the proper size.
    5. Uses the Graphics2D object to draw the original buffered image to the graphics context.
    6. Saves the new buffered image into the original one.

    As you can see, referencing Java classes and methods from Ruby code is not much different from doing it from Java code. Notable differences are the following:

    • You do not need to declare any types when using Ruby code. Ruby can infer the type based on the return value of the method call or the method's argument list. For example, Ruby can tell that bi is a BufferedImage object because that's what the read method of ImageIO returns.
    • You don't need to add parentheses to a method call when it takes no arguments.
    • You don't add semicolons to the ends of method calls.
    • You use nil instead of null to represent a null value.
    • You use a double colon in between the class name and the field name when referencing static values, such as when referencing the TYPE_INT_RGB field of BufferedImage:
      BI::TYPE_INT_RGB

    Creating a Filter That Can Produce the User's Chosen Effect

    Now that you have the user's chosen image operation saved in @data, you can write a case statement that creates the appropriate filter based on the value of @data. Each condition of the case statement uses a different class from the Java 2D API that can be used to perform a particular image-filtering operation. All of the classes implement BufferedImageOp. For more detail on image filtering in Java 2D, see Using Java 2D's Image Processing Model.

    This section goes into some detail about Java 2D image processing. If you're more interested in using Java libraries with Ruby code rather than the Java 2D API, just look for the Ruby_Info tag delimeters.

    1. After the code to save the image into a buffered image, create a variable to hold the image filter:
      op = nil
      
    2. Add the following case statement:
      case @data
         when "GrayScale"
            colorSpace = CS.getInstance(CS::CS_GRAY)
            op = java.awt.image.ColorConvertOp.new(colorSpace, nil)
         when "Negative"
            lut = Array.new
            for j in 0..255
               lut[j] = 256-j
            end
            jlut = lut.to_java :byte
            blut = java.awt.image.ByteLookupTable.new(0, jlut)
            op = java.awt.image.LookupOp.new(blut, nil)
         when "Brighten"
            op = java.awt.image.RescaleOp.new(1.4, -25, nil)
         when "Sharpen"
            data = [-1, 0, -1, 0, 5, 0, -1, 0, -1]
            dataFloat = data.to_java :float
            sharpen = java.awt.image.Kernel.new(3, 3, dataFloat)
            op = java.awt.image.ConvolveOp.new(sharpen)
      end
      
    <Ruby_Info> The Ruby case statement is similar to the switch statement in the Java programming language, but is more powerful and flexible, partly because it internally tests for multiple conditions at once. For example, one condition of the statement can do a string comparison while another condition can perform regular expression matching, but that's beyond the scope of this blog.</Ruby_Info>

    This case statement has a lot going on. Let's take it one piece at a time.

    Converting the Image to GrayScale

    The first condition of the preceding case statement uses ColorConvertOp to convert the color model of the image to grayscale, essentially making it a black-and-white image instead of a color image:

    The Java 2D API provides a set of color spaces, such as CS_GRAY and CS_CMYK. You just need to create a new ColorConvertOp instance and give it your chosen color space.

    Creating a Negative of the Image

    When the user selects "Negative" from the menu, the application uses the LookupOp class to create a negative of the original image:

    The LookupOp class uses a lookup table to filter the color values of pixels from a source image to a destination image. A pixel's color is made up of three components: red, green, and blue, each of which is represented by a value within the 8-bit range, 0-255.

    To produce the negative of an image, you need to create a lookup table that has the values 0-255 in the reverse order so that each pixel's color will be set to the color's complement, as the following for loop does:

    lut = Array.new
    for j in 0..255
      lut[j] = 256-j
    end
    
    This example uses only one lookup array, which means that it will be used to convert the colors of all three of the color components of each pixel. If you want, you can provide separate arrays for each color component so that each is converted in a different way.

    <Ruby_Info>

    The preceding code uses a Ruby array and a for loop. As with other variables in Ruby, you don't need to declare the type of the array, nor do you need to initialize it to a certain length. Same thing with the for loop: you don't need to initialize the iteration variable. And you don't need to explicitly increment it either. Finally, to indicate the range for the for loop, you just give the starting value and ending value of the iteration variable, separated by two dots.

    After the for loop exits, you have a Ruby array. What you need to do is convert it into a Java array so that you can use the array with Java libraries. To convert the array, you use the to_java function and indicate the type that you want to assign to the array:

    jlut = lut.to_java :byte
    

    </Ruby_Info>

    Now that you have converted the Ruby array to a Java array, you can use it to create a lookup table and pass the lookup table to an instance of LookupOp:

    blut = java.awt.image.ByteLookupTable.new(0, jlut)
    op = java.awt.image.LookupOp.new(blut, nil)
    

    Brightening the Image

    The original image is a little dull and dark. You can use RescaleOp to change the brightness or saturation of the image by applying a multiplier and an offset. The photo example uses RescaleOp to increase the brightness by 40% and shift the color values of each pixel 25 points to the lower part of the range (towards black) to make the image look a little more saturated:
    op = java.awt.image.RescaleOp.new(1.4 -25, nil)
    
    After processing the image with this filter, you'll get the following image:

    Sharpening and Edge-Detection

    The most complicated image filtering operation is convolution. Convolution involves calculating a new color value for a destination pixel by multiplying the color values of the source pixel and its neighboring pixels by a matrix, called a kernel. This operation can produce such effects as sharpening, blurring, or edge-detection, which looks like a line-drawing version of the image.

    The way you perform convolution with the Java 2D API is by creating a Kernel object and then using it to construct a ConvolveOp object. The photo example uses a kernel that causes a sharpening effect:

    To create the filter that will perform this sharpening effect, you would use the following code:

    data = [-1, 0, -1, 0, 5, 0, -1, 0, -1]
    dataFloat = data.to_java :float
    sharpen = java.awt.image.Kernel.new(3, 3, dataFloat)
    op = java.awt.image.ConvolveOp.new(sharpen)
    
    Here again, you need to convert the Ruby array into a Java array before using it to create a Kernel object.

    The following matrix would give you the edge-detection effect:

    [1, 0, 1, 
     0, -4, 0, 
     1, 0, 1]
    
    Here's the result of using this matrix on our example image:

    Filtering the Image

    Once you have your image filter, you can use it to convert your buffered image and draw the filtered image to the graphics context by adding the following code:
    dest = op.filter(bi, nil)
    big.drawImage(dest, 0, 0, nil);
    
    The op variable is the object that represents the image filtering operation from the previous section. The dest variable represents the filtered buffered image.

    Streaming the Image File to the Client

    Just as your used the read method of ImageIO read the image file into a buffered image, you can use the write method to write the filtered buffered image back into a file, or in the case of this example, an output stream, which you can then use to stream the file to the client.

    • Add the following lines to finish up the example by streaming the filtered image to the client:
      os = java.io.ByteArrayOutputStream.new
      IO.write(dest, "jpeg", os)
      string = String.from_java_bytes(os.toByteArray)
      send_data string, :type => "image/jpeg", :disposition => "inline", :filename => "newkids.jpg"
      
    Here, you're writing the data to a byte array output stream. Then, you convert the data to a Ruby string so that you can use send_data to stream it to the browser.

    If you prefer to save the image to a file rather than a stream and save the file to disk, you can use send_file instead of send_data:

    send_file writefilename, :type => 'image/jpg', :disposition => 'inline'
    

    Running the Application

    Running the application is easy using the GlassFish v3 gem:
    1. Deploy the application on the GlassFish v3 GEM:
      jruby -S glassfish_rails photo
      
    2. Run the application by entering the following URL into your browser:
      http://localhost:3000/home/index
      
    3. Select an image filtering operation from the combobox and click Submit.
    4. After the filtered image is displayed in the browser, click the browser's Back button to return to the previous page if you want to filter the image again.

    That's all there is to it. For more information on JRuby, Ruby-on-Rails, and the Java 2D API, visit the following links:

About

jenniferb

Search

Categories
Archives
« May 2008
SunMonTueWedThuFriSat
    
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
       
Today