Thursday Apr 01, 2010

How to Write a Memory Leak Unit Test

Unit tests are great for ensuring that your functionality is correct. But how do you make sure you don't have leaks in your code, leaks that
eventually cause your application to crash when it runs out of memory? Unit tests don't typically don't run into memory problems directly because they tend to start up, run a little bit of code, and shut down -- getting a nice fresh memory environment on each test start.

One great way to help protect yourself from uptime problems is to write leak tests. These are unit tests where you first perform some operations, then do normal cleanup, and finally you assert that the objects you were using during the operation have been cleaned up completely. In other words, that there are no references left anywhere on the heap.

Testing for leaks isn't very hard; the basic trick is to create a weak reference to your to-be-cleaned object (by constructing a WeakReference wrapping your object), then you delete all your local references to the object, then you perform garbage collection, and finally you see whether the weak reference still contains your object. This works because the weak reference is handled specially by the garbage collection system.

It turns out things aren't quite as easy as that, because garbage collectors try to be smart, and simply calling System.gc() doesn't mean it's going to actually perform a complete and final garbage collection. This means that your unit test could incorrectly conclude you have a leak, since the reference is still held.

The NetBeans team has built a really good test infrastructure for this. When I wrote unit tests for NetBeans plugins, I could simply call their assertGC method to assert that the given weak reference should be garbage collected, and it would handle the rest. (Internally, they do things like actually allocating a bunch of memory chunks to really force the garbage collection to run.)

Here's an example:

// The following unit test fragment wants to ensure that a graphics object
// which shows the current selection in the authoring tool, is cleared up after
// the user clears the selection.
Node firstHandle = /\* Node showing selection highlight, lookup code here \*/;

// Now clear view selection -- then assert that all the handles have disappeared

WeakReference<Node> ref = new WeakReference<Node>(firstHandle);
firstHandle = null;
assertGC("Selection handle leaked", ref);

When you write these tests you also have to make sure you null out any local variable references you are holding right there in the test.
The above test will pass if and only if the target object, firstHandle, is properly garbage collected.

But wait -- that's not the best part. Let's say you've written a leak test, and it fails. Now what? What do you do -- make the process pause and attach a profiler and try to hunt it down?

This is where their unit test support really shines. When the unit test discovers that the reference is still held somewhere, it uses its own heap walking library to hunt down the offending reference, and dumps it out as part of the test assertion failure message!!!

Here's the output of a real leak test failure:

Testcase: testMemoryLeak(SelectionHandlesTest): FAILED
Selection handle leaked:
public static java.util.List javafx.scene.Scene@dc8a29-$dirtyCSSNodes->

In other words, we see that there is a static list in Scene named dirtyCSSNodes which is holding on to our target Rectangle. The syntax here is that you first see the class name for the object (where a prefix of [L means it's an array-of), then its system id, then the field name - and if it's an array, the array index.

When the unit test fails, it takes a while -- 20 seconds or so for the above test -- to actually produce the above trace. But the important part is that this is only slow when the test fails. It only has to work hard when you have a leak, and you don't want to have leaks!

I've been wanting to write leak tests for the authoring tool (which is written in JavaFX), since uptime matters a great deal in a tool which deals with potentially large objects (such as multimedia assets). And I realized that there is absolutely nothing NetBeans specific about the NetBeans leak unit test support. So I went and pulled out the relevant code into a separate library. The library basically contains two parts: a .jar file which contains the INSANE heap walking library, and a second jar which contains the assertGC() unit test assertion method and supporting infrastructure.

I have extracted this into standalone pieces (outside the NetBeans infrastructure) so you can get the bits easily - download, then add the two jars in there on your test classpath and call assertGc() and assertSize() from your tests as described above. Here's the basic skeleton for all leak tests:

// (1) Create your objects
// (2) Look up the object you want to ensure gets cleaned up later:
Foo foo = // code to get foo

// (3) Call your cleanup code which is supposed to free everything

// (4) Create a weak reference to your object reference, and null out
// your reference
WeakReference<Foo> ref = new WeakReference<Foo>(foo);
foo = null;
assertGC("Foo leaked", ref);

All I've done is extract NetBeans code written by others so I've kept the licenses the same as for NetBeans. All credit goes to the author of INSANE, Petr Nejedly -- and to the authors of the memory assertion stuff in All the source code for INSANE and NbTestCase are in the NetBeans mercurial repository.

In addition to assertGC, there is also assertSize(). This method can be used to ensure that the transitive size of an object graph is below a certain threshold! This can be good for writing tests to not only ensure that you don't have leaks, but that your data structures are of the rough expected size so you don't need excessive amounts of memory. There are more details on this on the INSANE home page.

One special note on JavaFX: The above leak isn't actually a leak; it is a deliberate optimization of the scenegraph, and the reference will be cleared up after the next scene pulse completes. Therefore, for unit leak tests, in addition to actually nulling out the weak references, you also want to run through a scene pulse as well. One really hacky, implementation-dependent and unsupported way to do that is to call scene.$scenePulseListener.pulse();. You probably want to isolate that in a utility method such that you can update it in one place when it needs to change...

Finally, note that I built this on JDK6. If there is interest perhaps we could create a wrapper project for this on Kenai or, where people can also create say a Maven binary for this, a JDK 5 version (there is nothing JDK6 specific so it just needs a recompile, but I don't have JDK5 on this Snow Leopard Mac), etc. Hope you find this all as useful as I have!

Thursday May 14, 2009

Run Tests - Without Focus Loss!

I like unit tests - but running them can be painful. Commit-validation tests
which bring up UI are obviously annoying, but even simple unit tests that
get in the way. Does this look familiar?

The above menubar should look familiar to anyone on a Mac who's run unit tests for client side
Java code. Not necessarily GUI tests, just any test where GUI libraries are loaded.

For every testcase, the test runner fires up a new process, which tells
OSX that "I'm graphical, give me focus!", and this steals the focus from the user.
The test finishes quickly thereafter, the process quits - and then the test runner
goes to the next test and the whole cycle starts over.

This is really painful because I like to have lots of tests. The Python editor support
for Netbeans had 600+ tests;
the counts for the JavaScript support was higher, and
the Ruby support even higher than that. Whenever I run tests, I basically have to
fight with my computer to get focus. Forget trying to write anything - every second
or so my keystrokes get stolen as the next test grabs focus - so I've gotten in
the habit of using the time for browsing, since I'm mostly reading, and I can handle
clicking a link a second time if the first click got lost. But every now and then
somebody will ping me on instant messaging - and it's maddening trying to respond while
this is going on.

If this is sounding painfully familiar to you, I have good news. I've finally figured
out a setup where this is no longer a problem!

The key discovery was that I can run my tests from another account on the system.
With OSX' fast user switching, I can switch to the alternate account, launch the
unit tests, and return back to my regular account where the tests won't interfere
with display focus. In order to let me run my tests from that other account, I
just open a terminal there and su -l tor in the shell to run all the
commands (or NetBeans) as myself.

This was a huge improvement since it removes the 10-30 minutes testrun downtime.
But it had some disadvantages - first, I don't like running tests from a shell, and second,
it's hard to know when things finish - and switching back and forth to check is annoying
since I always have passwords on my accounts so the machine isn't open.

So the second step was to set up Hudson (a
continuous integration server that is trivial to setup, and has a huge number of
plugins which makes
graphing code coverage, unit tests, findbugs results, integrating with version control
systems etc trivial. And it's
not just for Java developers

Instead of logging into the other account
for each test run, I log into the other account once, and start up Glassfish
with Hudson running. From now on I can access, configure and launch builds right
from my own browser in my primary account. The key step here is that Glassfish was
started from the secondary account, so its primary display is associated with the builder account.
When my build in Glassfish gets to the test stage, it's actually doing the display
connection just as before, and if I log into the secondary account, I get the annoying
focus flashing just as before. Look - the tested process is a user visible application in the dock:

Another improvement which really helped is the "File System" version control plugin for Hudson.
I want my Hudson builds to build my current working copy. I don't want to
check my code into Mercurial (even a local repository) just so that Hudson can
grab the code and build it. I want Hudson to build my current changes - my current
edits. After all, I'm trying to test them before checking in! And I discovered that
there is a plugin which will let me do that - it's just
a "file system" version control wrapper - which means you just point Hudson to your local
directory, and off it goes. When the build starts, it makes a quick disk-copy of the
source tree. Even though Mercurial cloning is pretty fast, this is even faster.
The disk copy also lets me specify a filter of files to exclude, so I had it ignore
\*.class files. The diskcopy only takes 10 seconds or so before the build kicks
off, and it's building a snapshot of my current in-progress, edited working copy!
(It can also just update the copy based on changed filestamps - that's even faster,
but it didn't seem to correctly delete removed files, so I let it start from scratch
each time.)

(Note - this plugin isn't in the catalog that you can browse right from within the
Manage Plugins page within Hudson; I downloaded the .hpi file from
and installed it in the Advanced tab.)

The final ingredient is the new Hudson support in NetBeans 6.7. I don't even have
to go to the browser to kick off build jobs. I just point NetBeans to my Hudson
server once, and from then on I have full Hudson integration. When I want to
run my tests I just select Launch job:

I get notified if there's a problem:

I can look at failures and logs:

I can see build logs etc. directly in the output window, and hyperlinks warp to directly to files - to the files as they were in the build, not the current state:

So to recap - with this setup, as I'm editing my code and I want to check the tests,
I just right click on a node and say "Start Job" - and off it goes without bothering
me at all - no more focus interruptions, and no more GUI windows popping up from
interactive tests. It's trivial to check the results. And it's even added one more
level of convenience: I have multiple projects, each with unit tests, and from the
IDE I couldn't have them all run with a single gesture. My build job does that.

I'm really stoked! I was at one point able to do this when I was working on Linux
and Solaris by setting my $DISPLAY variable and doing tricks with VNC. But that
still required my tests to run in a console - which made interpreting the results sucky.

If you haven't played with Hudson, try it - it's unbelievably easy to set up. Just download
the Glassfish appserver and install it, download the Hudson .war file, and drop the hudson.war
into the autodeploy directory of the appserver, and browse to localhost:8080/hudson.
Once you're there you can install
plugins (under Manage hudson), point to your local installations of the JDK, ant,
etc., and configure your build jobs by running scripts, launching maven scripts, writing
ruby scripts, or obviously running ant scripts.

Some final miscellaneous tips:

  1. I don't want Time Machine to back up my builds trees, or Spotlight to
    index data in these directories, so I went to the TimeMachine preferences and had it
    exclude the ~/.hudson/jobs/ directory.

  2. I did the same thing for Spotlight - but unlike
    the Time Machine preferences, there was no checkbox to "Display Invisible Files" (e.g.
    files that start with a dot, such as .hudson) in its file chooser. Here's a tip I didn't
    learn until recently: When a Mac filechooser has focus, you can press slash (/) - and this
    will open a text field where you can directly type the path it should jump to. I typed
    /Users/tor/.hudson and from there I was able to select the jobs directory to exclude.

  3. You might be tempted to skip the "filesystem version control" plugin and just have
    your build symlink to your working copy. Be careful; if Hudson is configured to delete
    older builds you might find yourself without your source code. I'm not saying it will
    follow your symlinks - Java has support for symlinks now - but I haven't tried it, and
    I have been bitten by ant in the past where it decided to follow symlinks in its
    zeal to delete recursively!

  4. I recently discovered that you can reorder the build steps in a Hudson job. The little
    graphic to the left of a build task is a handle you can just drag to reorder!


Tor Norbye


« April 2014