JavaOne BOF on Memory Leaks

For those of you who came to Frank and myself's BOF session at JavaOne, you can download my "modified" jhat here jhat.jar.  My modification of original jhat include:

  • add exclude functionality on query results to make it easier to filter out hundreds of duplicate reference chains, this way it will be possible to find unique leaks (see Figure 1)
  • add classloader browser to nicely display classloaders' parent-child relationship, as well as number of classes loaded in each classloader (see Figure 2)

To run jhat, simple do "java -Xmx512m -jar jhat.jar <heap_file_name>"

Figure 1.

filter reference chains 

 

Figure 2.

 classloader browser

 

Update: Frank posted the presentation slides and audio recording of this BOF on his blog entry

Comments:

Good software. Thanks. I agree with what Frank suggested at JavaOne BOF to opensource it, I'd go a little bit further, it would be very nice to productizing it and get it bundled in a standardized package. I view it as parallel to jconsole and visualgc.

Posted by Gary Zhu on May 15, 2007 at 01:50 AM PDT #

This is a great addition to jhat - I haven't found any other tools that provide this information nearly so easily.

I do have a question, though - is it possible there's a bug? According to this modification, I've managed to eliminate all references to a WebAppClassLoader that wasn't getting collected on application reload in Tomcat. Unfortunately, it still doesn't get collected. :(

Anyone else run into this?

Thanks!

Posted by Cott Lang on September 24, 2007 at 01:33 AM PDT #

Hi Cott,

I am not aware of any bugs in jhat, the changes I made are on top of JDK 1.6.0_01 source code.

One possibility I can think of is that the WebAppClassLoader is actually "garbage-collectable", but it is not yet reclaimed by JVM. Garbage-collection in JVM is unpredictable and varies depending on the algorithm. One way to make sure that it is leaking in the WebAppClassLoader is to have a automated script that does the deploy/undeploy cycle of your WebApp many many times, and then use JConsole to monitor the PermGen memory space of JVM. If PermGen space usage graph increases continueously over time, then you know you are still having the Classloader leak problem.

Posted by Edward Chou on September 24, 2007 at 01:45 PM PDT #

Edward,

You are, of course, correct. I re-ran my tests and deployed and un-deployed my application a few dozen times before it finally began to garbage collect the WebAppClassLoader. It seems it does not GC until it is stressed for perm gen space.

After that, it allowed 5-15 copies of it to linger steadily through hundreds of reloads.

Thanks again for this wonderful tool - I'd sure love to see this in the JDK distribution.

Posted by Cott Lang on September 28, 2007 at 11:12 AM PDT #

Thanks! I was trying to execute a query to dicover references between objects of different classloaders, but it is taking forever (+1 day and running).

This was the query:
select o1 from instanceof java.lang.Object o1
where classof(o1).loader != heap.findObject('0x046cbf28')
&& contains(referees(o1), "classof(it).loader == heap.findObject('0x046cbf28');")

where 0x046cbf28 is a WebApp classloader of Tomcat.

Your jhat version nicely helped to find out some suspects (sorry if I'm wrong on those):

Thread.contextClassLoader:
Maybe Tomcat does not reset inmediately this member after request processing.

org.apache.tomcat.util.digester.Digester.parser:
It is holding a reference to a Xerces parser in my application lib.

Posted by Imp on January 13, 2008 at 10:38 AM PST #

Imp,

You're not wrong on Xerces - you don't want that in your application lib under Tomcat.

Posted by Cott Lang on January 13, 2008 at 12:54 PM PST #

Thanks Cott.
I think this is the wrong place to post this... :(

Xerces has been deleted from WEB-INF/lib dir. It was there to support java 1.4.2.

The application keeps throwing permgen exceptions after 20 redeploys or so.

A heap dump after the error shows several unreferenced WebappClassLoader (7). Jhat shows no livepaths to those classloaders.

select cl
from instanceof org.apache.catalina.loader.WebappClassLoader cl
where count(heap.livepaths(cl)) == 0
(This query shows 7 classloaders)

The special query of Edward Chou to find Reference Chains from Rootset (Exclude weak refs, filter enabled) also shows the CLs as unreferenced!!

I'm using JDK 6 under Windows 2000.

May somebody shed some light over the issue? Is a reference hidding somewhere? What other reason may prevent the JVM from unloading one CL?

Boy, I needed to tell someone...

Posted by Imp on January 14, 2008 at 01:39 AM PST #

Hi,

Are you going to publish your modifications in source form?

You've very aptly demonstrated that some problems are best approached by customizing the jhat code with extra features - so it would be very nice to be able to build upon your customizations. It would also be nice to be able to update to future jhat versions without losing your additional features.

Max.

Posted by Max Bowsher on September 10, 2008 at 04:43 PM PDT #

Post a Comment:
  • HTML Syntax: NOT allowed
About

wingzero

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today