Sunday May 11, 2008

Java on the NVidia APX 2500

At JavaOne 2008 we demonstrated Java running on the NVidia APX 2500 mobile device. The APX 2500 was unveiled at the 3GSM conference a few months ago, with OpenGL ES 2.0 support in hardware, a fast ARM processor, and features like full HD video decompression in hardware. Our demonstration was shown in our Vice President Jeet Kaul's keynote address, toward the end, 1:00:00 into the video. It was also shown in father of Java James Gosling's keynote address, about 30:30 into the video. A clip also showed up on ZDNet.

We were fortunate to obtain a dev kit from NVidia about a week and a half before JavaOne. At the time we were multitasking on other work that had to be done by the conference. With help from the Java ME team, I successfully compiled the CVM, a highly portable Java SE-class VM, for the device. This took about 24 hours. This stack includes the Connected Device Configuration (CDC) and Foundation Profile (FP), but no graphics stack. At that point we had a headless Java VM with JNI support. We also compiled in the java.nio Buffer classes from the JSR-239 reference implementation, because these are used to hand data down to the OpenGL layer in both the desktop JSR-231 and mobile JSR-239 implementations. However, we were not aiming to run JSR-239 on the device.

Anthony Rogers, lead designer of JavaFX, created a demo design we were aiming for, including 3D graphics, spatialized audio, and touch screen input. In support of his design, I then took out the machete and whacked JOGL down to about 10% of its original size, changing its build process to parse the OpenGL ES 2.0 headers instead of the desktop OpenGL 2.1 headers, and eliminating all utility classes that had dependencies on immediate mode, the fixed-function pipeline, and the AWT or Java 2D. I ported NVidia's basic OpenGL ES 2.0 demo, a spinning triangle with colored vertices, from C to Java to this modified JOGL. After working through several issues, including needing to write an OpenKODE-based launcher for CVM (using kdMain() instead of a normal main()), we had a window on the screen, but no graphics. Fortunately, while whittling down JOGL to the OpenGL ES 2.0 headers, I had left in the composable pipelines, in particular the DebugGL. Plugging in the DebugGL immediately isolated the problem to a missing glUseProgram call. After fixing this, we had a spinning triangle on the screen being driven by a very thin Java stack interfacing directly to the graphics hardware via OpenGL ES 2.0. This was working the Friday night before the conference. Here is a picture of this demo running on the APX 2500, plugged in to the "wingboard" which provides power as well as several breakout connectors.

At that point we focused our efforts on audio. Our first attempts to use the device's hardware support for OpenMAX didn't work, and since it was a Saturday by this point we had to find a fallback. We found that we could send bits out the Windows waveOut device, so I built a very simple pseudo-spatialization engine in pure Java which models the positions of the speakers in 3D, and treats the sound as a point emitter in the 3D space, with 1 / (1 + r\^2) falloff between the source and the speakers. This engine then sends its outputs down to the Windows waveOut device with a very small amount of native code. This took somewhat longer than expected or desired, and we ran into a hiccup where we found that we could only initialize the waveOut device on the APX 2500 in mono mode, not stereo. We contacted NVidia about this and the reason turned out to be a bug in my code where I had incorrectly computed the average bytes per second. Fixing this allowed us to get full stereo out of the APX 2500. This was fixed by the Monday morning before the conference.

At that point we had to get the graphics portion of the demo working. We decided to fall back to using the OpenGL ES 1.1 support on the device because we didn't have the time to write extensive utility libraries to make it easy to write pure shader-based code. While I was working on downgrading the modified JOGL from OpenGL ES 2.0 to OpenGL ES 1.1, Sven Gothel, author of GL4Java, started working on some backup content. That night I ran into extensive problems, including needing to implement the JNI GetDirectBufferAddress entry point in order for JOGL to work properly. I was seeing mysterious JVM crashes while trying to get the simplest red square OpenGL demo (ported to vertex buffer objects for OpenGL ES 1.1) on the screen.

In the meantime, Sven single-handedly ported the San Angeles OpenGL ES 1.1 demo from the Assembly 2004 competition from C to Java in a single night, first getting it to run on the desktop using an unmodified JOGL. He then modified the demo slightly to fit the framework we had running on the APX 2500. (I didn't have time to fully abstract away the EGL, so there were a few places in the code where EGL-specific constructs had to be used.) While I was struggling mightily to get any graphical output to come out of the device, Sven handed me a binary and said, "Come on, give it a try." The first run yielded a NullPointerException. The second run worked. An absolute miracle and awesome team effort. Here are a couple of pictures of the demo running on the NVidia APX 2500. Click for a larger image.

On Sven's suggestion we dropped in a sound sample of the wind blowing, spatialized to oscillate between the right and left speakers, which gave the demo a haunting feel. Probably because we were using the OpenGL ES 1.1 driver, which emulates the fixed function pipeline at the driver level using shaders, we were using more CPU than we should have, so there were some dropouts on the audio where our Java-based mixer thread was starved. We'll fix this in the coming days.

Overall the port of Java, JOGL and 3D spatialized audio to the device, as well as the San Angeles demo, took six days.

We would like to thank Brian Bruning and Keith Galocy from NVidia for getting us the device and helping us quickly with our questions; Nandini Ramani for facilitating the contact with NVidia; Hinkmond Wong, Chris Plummer and Dean Long from the CVM team for getting us a working JVM so quickly; Chris Oliver for extremely helpful discussions and advice; and Anthony Rogers for the original demo idea, although I didn't succeed in executing his vision — yet. :)

Thursday May 10, 2007

NASA World Wind Java at JavaOne

Quick Information Links

Today, May 10, at JavaOne 2007, NASA released the early access SDK of their revolutionary World Wind planetary visualization system for the Java platform.

This is an historic moment. Now leading-edge, high-performance, 3D geospatial visualization is available to developers on all platforms. You can embed it in your Java applications as a component. You can extend it, changing the visual appearance of the globe in any way you can imagine. This technology is deployable inside of a web browser as an applet, or as an application launched with a single mouse click, using the foundations of the Java platform and the OpenGL 3D graphics API. No manual software installation is required (aside from the Java Runtime Environment, which is pre-installed on 9 out of 10 PCs shipped today, and on all Macintosh computers).

Tom Gaskins, the technical lead of World Wind Java, and the team of Dave Collins, Lado Garakanidze, Randy Kim, and alumni Eric Dalgliesh and Chris Maxwell, led by Project Manager Patrick Hogan, have built an elegant, extensible, and embeddable architecture which will resonate throughout the software development community and the world. I extend my heartfelt thanks to them for their perseverence and hard work.

Sun's involvement with NASA World Wind Java dates back to September 2005, when Hans Muller, CTO of Sun's Desktop division, initiated a dialogue between NASA's World Wind development team and Sun. At the time, World Wind was exclusively a .NET application, written in C# and Direct3D, and ran only on Microsoft's Windows platform. Chris Campbell and I met with the World Wind development team at NASA Ames to discuss the feasibility of a Java platform port of World Wind. It turned out that NASA had already been discussing this internally and Tom Gaskins had written an article on EarthFlicks, a stripped-down version of World Wind written in Java using OpenGL via JOGL for its 3D graphics. Chris and I discussed the current state of Java 2D and 3D graphics, and we collectively discussed areas where NASA would need help in bringing World Wind to Java.

The extraordinary nature of NASA's work was immediately apparent, and I initiated discussions throughout Sun to try to help accelerate the development of a Java version of their system by providing funding enabling NASA to hire an additional contractor. My vice-president, Laurie Tolson, agreed to fund half of a collaborative research project, the other half funded by Sun's External Research Office led by Emil Sarpa. Foothill / De Anza College, which has an internship program at NASA Ames, helped facilitate the collaboration, which officially began in April 2006.

In the meantime, NASA had already begun work on a Java version of World Wind. Tom Gaskins had been providing feedback on JOGL for months, and he, Chris Campbell and I had had steady dialog during this time. His need for an easy code path to bring textures in to OpenGL prodded Chris to finish the initial version of JOGL's TextureIO utility classes. Afterward I assumed responsibility for these classes and extended them with compressed texture support which is required for World Wind to use texture memory effectively. Tom and the NASA team improved the quality of JOGL for the entire development community by uncovering several bugs in these utilities along the way which we fixed.

The entire NASA development team made extraordinary strides in a remarkably short period of time, getting quickly to the point where the globe was on the screen and continuing to refine the algorithms used and extend the system's functionality. Areas of development and discussion included the general strategy for producing the geometry and imagery for the planet's surface; support for rendering place names; support for rendering lines and polygons on the planet's surface; the algorithms used to position the camera (or the "view"); integration of new data sets; inclusion of features such as political boundaries and meridian and parallel lines; and selection of features on the surface of the planet. Patrick Hogan retargeted the team's efforts and made World Wind Java the primary focus for ongoing development.

NASA's work on World Wind Java informed bug fixes to JOGL's support for DRI rendering on X11 platforms, among other areas. Based on NASA's experience using JOGL to render text for place names, as well as extensive discussions with Phil Race from the Java 2D team, I added a TextRenderer to JOGL which provides easy, high-quality, and platform-independent text support for OpenGL applications. JOGL's TextRenderer soon after replaced the custom text rendering system built earlier in World Wind Java for rendering place names.

At OOPSLA 2006 I saw an exhibit of a virtual equipment panel out of an airplane (an F-16, in fact). Looking closer I saw the standard Java coffee cup logo in the corner of the window. The sales representative from the company (The DiSTI Corporation) indicated that this was a piece of code produced by their tool, GL Studio, and that they were showing a new Java code generator for their tool which produced Java code using JOGL. I introduced myself as the JOGL technical lead and was soon in touch with Darren Humphrey, the CTO of DiSTI. I put NASA and DiSTI in touch, as DiSTI's F-16 cockpit was a natural fit for flying over NASA World Wind Java's terrain. This resulted in the mind-blowing F-16 flight simulator demo shown at JavaOne 2007, linked below, in which you can fly over the entire world.

In recent months the NASA team has focused on refinements of the view and terrain algorithms, as well as satisfying requirements set by other sponsors of the project such as geometric highlighting of features on the planet's surface. The software has become increasingly ready for consumption by a larger audience.

NASA World Wind Java and The DiSTI Corporation's F-16 flight simulator demonstration were shown in Bob Brewin's technical keynote this past Tuesday to a wonderful audience reception. You can watch a replay of the webcast of the keynote. Part of the segment on NASA World Wind Java even made it to CNET Tom Gaskins presented an excellent technical session today (TS-3489, 3-D Earth Visualization with NASA World Wind) which was extraordinarily well received and which discussed in greater depth World Wind Java's architecture and how it can be utilized, extended, and embedded as a component.

This is just the beginning of a remarkable journey. Sun and NASA eagerly await what the Java developer and user communities will do with these capabilities. The world is at your fingertips.

For More Information

Friday May 04, 2007

Brief Introduction

First post containing a brief introduction and some background. [Read More]



« April 2014