HPROF, HAT, and jhat
By kto on Mar 02, 2006
My Own Little HPROF Nightmare
If you do a google search on "not found for JNI local ref" you will find many hits, this is unfortunately a bug in the new implementation of the hprof agent in JDK 5 (the Tiger release) brought out by using the heap analysis tool HAT or it's improved version in JDK 6 (the upcoming Mustang release) called jhat.
The HAT/jhat tool is one of the few tools that uses the binary hprof format as a binary heap snapshot mechanism. The failure usually looks something like:
java.io.IOException: Thread 200000 not found for JNI local ref at hat.parser.HprofReader.getThreadObjectFromSequence(HprofReader.java:577) at hat.parser.HprofReader.readHeapDump(HprofReader.java:408) at hat.parser.HprofReader.read(HprofReader.java:221) at hat.parser.Reader.readFile(Reader.java:90) at hat.Main.main(Main.java:149)
And pretty much stops HAT and jhat dead in the water.
Unfortunately my attempts to fix this bug in Tiger Update 4 (see 6231936) didn't work out as I'd planned, and hopefully a better fix should be coming with Tiger Update 7 (see 6325354) and the fix is also available in the latest Mustang downloads at http://mustang.dev.java.net/. The basic problem lies in the difference between a java.lang.Thread object existing, the java thread actually being alive and running, and the timing of dumping out objects for a given java thread that are on there way to being garbage collected. My first attempted fix was to try and work around the timing, and the second fix was more along the lines of assuring the heap dump didn't include missing references, mapping any unknown thread references to the catch all 'system thread'. I sure hope this is the last fix to this particular problem. I'm also not kidding myself that it is the last problem with hprof, so if any of you run into issues, please create a forum posting at http://forum.java.sun.com/forum.jspa?forumID=47&start=0, eventually I'll see it.
I could give excuses for this foopaw but I'd rather tell you what has been done and what will be done to prevent this kind of regression in the future. The ability to get heap dump snapshots and view them is a critical need that we recognize and plan on leveraging in the future. [The hprof and HAT/jhat combination is a hard one to test automatically, and if anyone out there reading this has any good ideas, please let me know.]
- We have added HAT to the JDK as "jhat" in Mustang. This was done for many reasons, but a primary one was to enable us to start building up a set of tests with hprof and jhat to verify the accuracy and correctness of the hprof binary heap dumps, and also make sure that hprof and jhat work well together. In addition, by having the jhat tool in the JDK we could make it more readily available and integrate it better with the rest of the JDK. This Mustang jhat can be run on ANY hprof binary format files, even those created by JDK 1.4.2, so even if you don't want to try Mustang JDK 6, you might want to download it just for the sake of having this jhat tool handy.
- We started a project to investigate for Dolphin (JDK 7) a new binary heap dump snapshot file format, with a more formal specification (see https://heap-snapshot.dev.java.net/). The traditional hprof binary format is close to what we want, but it has never been fully specified and we would like to improve upon it and create a more standard format.
- The hprof agent is not the only way to get a hprof binary format file for HAT/jhat. The Mustang release includes a utility called 'jmap' that has an option to dump a heap snapshot file in the hprof binary dump format, one that jhat can read (HAT may be able to read it too, but the stacks for where objects were created will be missing from the data, and sometimes HAT doesn't like that). This jmap & jhat technique is a very fast way to get a look into the heap of a running VM without the overhead of starting the VM with hprof.
- Improve the error recovery of HAT/jhat. The HAT project has been retired but jhat is being improved in the Mustang release (see http://blogs.sun.com/roller/page/sundararajan). The basic HAT program is quite old, but it seems to answer many problems, we would like to see it evolve and improve in the future. By using it more and making it more available, we will make this more stable.
Ultimately we can make hprof and jhat tools industrial strength but it will take some time. And please create forum postings at http://forum.java.sun.com/forum.jspa?forumID=47&start=0 so that we know how things are going.