Thursday Apr 01, 2010

How to Write a Memory Leak Unit Test


Unit tests are great for ensuring that your functionality is correct. But how do you make sure you don't have leaks in your code, leaks that
eventually cause your application to crash when it runs out of memory? Unit tests don't typically don't run into memory problems directly because they tend to start up, run a little bit of code, and shut down -- getting a nice fresh memory environment on each test start.



One great way to help protect yourself from uptime problems is to write leak tests. These are unit tests where you first perform some operations, then do normal cleanup, and finally you assert that the objects you were using during the operation have been cleaned up completely. In other words, that there are no references left anywhere on the heap.



Testing for leaks isn't very hard; the basic trick is to create a weak reference to your to-be-cleaned object (by constructing a WeakReference wrapping your object), then you delete all your local references to the object, then you perform garbage collection, and finally you see whether the weak reference still contains your object. This works because the weak reference is handled specially by the garbage collection system.



It turns out things aren't quite as easy as that, because garbage collectors try to be smart, and simply calling System.gc() doesn't mean it's going to actually perform a complete and final garbage collection. This means that your unit test could incorrectly conclude you have a leak, since the reference is still held.



The NetBeans team has built a really good test infrastructure for this. When I wrote unit tests for NetBeans plugins, I could simply call their assertGC method to assert that the given weak reference should be garbage collected, and it would handle the rest. (Internally, they do things like actually allocating a bunch of memory chunks to really force the garbage collection to run.)



Here's an example:


// The following unit test fragment wants to ensure that a graphics object
// which shows the current selection in the authoring tool, is cleared up after
// the user clears the selection.
view.selectAll();
Node firstHandle = /\* Node showing selection highlight, lookup code here \*/;
assertNotNull(firstHandle);

// Now clear view selection -- then assert that all the handles have disappeared
view.selectNone();

WeakReference<Node> ref = new WeakReference<Node>(firstHandle);
firstHandle = null;
assertGC("Selection handle leaked", ref);




When you write these tests you also have to make sure you null out any local variable references you are holding right there in the test.
The above test will pass if and only if the target object, firstHandle, is properly garbage collected.



But wait -- that's not the best part. Let's say you've written a leak test, and it fails. Now what? What do you do -- make the process pause and attach a profiler and try to hunt it down?



This is where their unit test support really shines. When the unit test discovers that the reference is still held somewhere, it uses its own heap walking library to hunt down the offending reference, and dumps it out as part of the test assertion failure message!!!



Here's the output of a real leak test failure:


Testcase: testMemoryLeak(SelectionHandlesTest): FAILED
Selection handle leaked:
public static java.util.List javafx.scene.Scene@dc8a29-$dirtyCSSNodes->
java.util.ArrayList@4ed14172-elementData->
[Ljava.lang.Object;@3c22de9e-[41]->
javafx.scene.shape.Rectangle@612e4cd

In other words, we see that there is a static list in Scene named dirtyCSSNodes which is holding on to our target Rectangle. The syntax here is that you first see the class name for the object (where a prefix of [L means it's an array-of), then its system id, then the field name - and if it's an array, the array index.



When the unit test fails, it takes a while -- 20 seconds or so for the above test -- to actually produce the above trace. But the important part is that this is only slow when the test fails. It only has to work hard when you have a leak, and you don't want to have leaks!



I've been wanting to write leak tests for the authoring tool (which is written in JavaFX), since uptime matters a great deal in a tool which deals with potentially large objects (such as multimedia assets). And I realized that there is absolutely nothing NetBeans specific about the NetBeans leak unit test support. So I went and pulled out the relevant code into a separate library. The library basically contains two parts: a .jar file which contains the INSANE heap walking library, and a second jar which contains the assertGC() unit test assertion method and supporting infrastructure.



I have extracted this into standalone pieces (outside the NetBeans infrastructure) so you can get the bits easily - download leaktests.zip, then add the two jars in there on your test classpath and call assertGc() and assertSize() from your tests as described above. Here's the basic skeleton for all leak tests:


// (1) Create your objects
// (2) Look up the object you want to ensure gets cleaned up later:
Foo foo = // code to get foo
assertNotNull(foo);

// (3) Call your cleanup code which is supposed to free everything

// (4) Create a weak reference to your object reference, and null out
// your reference
WeakReference<Foo> ref = new WeakReference<Foo>(foo);
foo = null;
assertGC("Foo leaked", ref);




All I've done is extract NetBeans code written by others so I've kept the licenses the same as for NetBeans. All credit goes to the author of INSANE, Petr Nejedly -- and to the authors of the memory assertion stuff in NbTestCase.java. All the source code for INSANE and NbTestCase are in the NetBeans mercurial repository.



In addition to assertGC, there is also assertSize(). This method can be used to ensure that the transitive size of an object graph is below a certain threshold! This can be good for writing tests to not only ensure that you don't have leaks, but that your data structures are of the rough expected size so you don't need excessive amounts of memory. There are more details on this on the INSANE home page.



One special note on JavaFX: The above leak isn't actually a leak; it is a deliberate optimization of the scenegraph, and the reference will be cleared up after the next scene pulse completes. Therefore, for unit leak tests, in addition to actually nulling out the weak references, you also want to run through a scene pulse as well. One really hacky, implementation-dependent and unsupported way to do that is to call scene.$scenePulseListener.pulse();. You probably want to isolate that in a utility method such that you can update it in one place when it needs to change...



Finally, note that I built this on JDK6. If there is interest perhaps we could create a wrapper project for this on Kenai or java.net, where people can also create say a Maven binary for this, a JDK 5 version (there is nothing JDK6 specific so it just needs a recompile, but I don't have JDK5 on this Snow Leopard Mac), etc. Hope you find this all as useful as I have!

Monday Jan 05, 2009

NetBeans Screenshot of the Week #40: Python Test Runner UI


In the 7.0 builds, we have a dedicated unit test runner for Python now. Running a file as a test, or running the test project action, will open a test runner with the output docked on its right hand side, instead of just the output window (click for full size screenshot):









Here's the test runner itself:






What you want to see in the output is a single green line which says "All n Tests Passed", where n is hopefully a large number. But when one or more of the tests fail, you see something like the above. Running the Next Error action (Cmd-.) will jump to the next failed test, just like it normally jumps to the next error in the current output window, or the next task in the tasklist.



One thing to notice (and this is new in 7.0) is that we now include the failure message right there in the test summary for each test, so you don't have to drill into an individual test to see the failure message. You can also hover over a test and any output from that test is shown in a tooltip. You can also right click on tests and select "Run Again" to run just one specific test over again.



This is the same window as the one we have for Ruby. In fact, Erno, who wrote the Ruby test runner, has modularized the code now such that it's the same implementation for both languages - we just have language specific plugins to actually hook into the various testing frameworks. Currently, for Python we support the two builtin testing frameworks: unittest and doctest. The above screenshot showed a regular unittest run. Here's a doctest:







One important thing to note is that you don't have to instrument your code with any test running code. As long as you place doctests in your docstrings, or as long as you have any classes extending unittest.TestCase, the test runner will find them. All you have to do is run Test Project:







and when you have done that, both unittests and doctests are found and included in the same test run - here's both failing doctests and unit tests:






Once again, kudos to Erno for writing the excellent test runner support! And as usual, let us know of any problems.



P.S. Jean-Yves Mengant has also done some excellent work recently. In addition to some cool new quickfixes in the Python editor for NetBeans, he has just integrated a multithreaded debugger for Python!




Tuesday Jun 24, 2008

Ruby Screenshot of the Week #29: Test Navigator for RSpec


Let's start with the screenshot:






The thing to notice here is that we have a number of RSpec do-blocks in this file. Until now, only classes, methods, fields and attributes were showing up in the navigator. But in the latest daily builds, rspec do blocks like these are shown in the navigator (bottom left corner), along with the string summary (if any) such that you can quickly navigate between them. Placing the caret in one of these blocks will highlight the corresponding item in the navigator.



Adding a better navigator outline for spec files has been on the TODO list for a while, but apparently in the latest Rails builds regular tests are also switching away from using test methods to using test blocks (according to issue 137925, thanks James). So it seemed high time to do something about this.



The other thing to notice is that these blocks are also folded, so you can use Command-Shift-+ and - to collapse/expand all blocks quickly.



By the way, another old issue was fixed in NetBeans recently, which makes it easy to configure filetypes. In the past, if you wanted say .mab files to get Ruby treatment in the IDE, you'd file an issue and I'd modify the Ruby mimetype resolver to include it. The list is already pretty long -- .rb, .rake, .rxml, .rjs, .rbw, .rbx, .mab, .builder, .gemspec, and so on.



Now you can add any extra types you want yourself, using the Options | Miscellaneous | File Associations. Add any extensions you want, and bind them to the Ruby mime type and all the various Ruby editing machinery should kick in for files of that type.






NetBeans 6.5 Milestone 1 is about to be released and there are a lot of goodies there - both in the Ruby specific featureset as well as generic IDE support (such as the new Quick Search functionality). I'll look into that for another entry. And apologies for the long delay since my last entry - I blame it on a combination of having been sick, having taken some time off and having been busy with work...

About

Tor Norbye

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today