Friday Jun 14, 2013

Java and the Raspberry Pi Camera (Part 1)

Using the Raspberry Pi Camera with Java I've always liked the idea of computer vision and on the very long list of things I'd like to spend more time exploring is the OpenCV libraries which have a handy set of Java bindings.  In the past I've experimented with, and used some of the other frameworks that are available for image capture in Java, specifically the Java Media Framework (JMF) and the Freedom for Media in Java (FMJ), mostly around the idea of integrating images from a webcam into an application like a security monitoring system.  Sadly, JMF has grown a little dusty over time with the last release being way back in 2002 (you have to be amused when you see that the hardware requirements for this are a 166MHz Pentium processor and 32Mb of RAM).  FMJ is a little more modern, but was last updated in 2007.

The Raspberry Pi Foundation recently announced the launch of a camera that plugs into one of the two ribbon cable connectors on the board (as shown below):

Raspberry Pi Camera

I thought it would be an interesting idea to see how easy it would be to get this working with a Java or JavaFX application.

There are three utilities that are available for testing the camera: raspistill, raspiyuv and raspivid.  These allow you to grab a frame or video from the camera and store it in a file.  This seemed to be a good starting point for figuring out how to use the camera and get the frame data into a Java application, ideally as a BufferedImage (I decided to start with simple image capture and look at video streams later).

I downloaded the code from github and started looking at what it does and how it works.  Initially I thought it would make to sense to use a toolchain to cross compile the code on my quad-core Linux box.  However, having spent a day working on this and failed to get the code to compile cleanly (even using the download of the Raspberry Pi org's toolchain) I decided it might be slower on the Raspberry Pi, but at least it worked.

I also found a useful post from Tasanakorn Phaipool who had created a couple of sample applications that made use of the camera and linked to the OpenCV libraries.  This provided a good starting point as it simplified things compared to the raspistill application and enabled me to figure out a relatively simple build environment (I don't have time right now to climb the learning curve required for cmake).

Getting the code to compile and run was really quite challenging.  I will confess it's been a while since I've done any C coding, but more of the issues I experieced were to do with getting the build process to work correctly.  I used an iterative approach to creating a Makefile, simply resolving issues as I found them, gradually adding header file references and libraries until the code compiled cleanly.  To use the camera we need the multi-media abstraction layer (MMAL) API.  Broadcom have very kindly made this available as source, but documentation-wise you pretty much have to dig through the source code (there is a big comment at the top of the mmal.h file which is the best documentation I've found so far).  Once I'd got the code to compile and link it still would not run, which puzzled me for quite some time until, by comparing the raspistill executable to the one I'd built, I found that I needed to include the in the list of libraries to link.  (This really does confuse me because this library is not required to resolve any function references so the code compiles and links correctly, but without it the necessary camera configuration is not registered and the call to mmal_component_create() will fail).

At this point I have some code that will talk to the camera and display the preview image on the video output (HDMI).  Next I need to modify this so it can be used with JNI and integrate this with a new subclass of ImageInputStream which can then be used to create a BufferedImage in a Java application.

One other thing that is interesting is that when I run the simple test program the preview is displayed and very shortly after the network stops working (all the LEDs on the Pi except the power light go out).  I assume that is a bug somewhere.  Fortunately, I have a serial console connected so can still access the Pi via PuTTY.

I will update my blog as I make more progress on this.

A blog covering aspects of Java SE, JavaFX and embedded Java that I find fun and interesting and want to share with other developers. As part of the Developer Outreach team at Oracle I write a lot of demo code and I use this blog to highlight useful tips and techniques I learn along the way.


« June 2013 »