Classloader leaks: the dreaded "java.lang.OutOfMemoryError: PermGen space" exception

This blog has moved to http://frankkieviet.blogspot.com

Comments:

One point about trying to detect Classloader leak is that it is very difficult to know what exactly is going on inside AppServer. If you are familiar with the source code of AppServer, then it will help you a bit, since you will have a general idea of where to look. Because these leaks likely to be at most unexpected places. I think that is another reason these kind of problems are difficult to debug for application developers.

Posted by edwardchou on October 16, 2006 at 07:19 AM PDT #

Frank, Perfect! This topic came out just in time! We recently came across this issue with dreaded PermGen space exception happening in our Java CAPS integration server. At this point, we have no idea what's causing it and probably it's not reproducible again. I have been thinking about whether it's worth it to run a profiler on our 'logicalhost' (HPJMeter, as our JCAPS runs on HP-UX), but I feel apprehensive about spending time to do it as it could be difficult to point to the exact source of problem.Edward: thanks, your comments confirms my apprehension. or do you guys think it's still worth it to run a profiler?

Posted by Aji on October 26, 2006 at 08:40 PM PDT #

This behaviour of the JDK seems buggy to me. I cannot find any reason for the reference shown with the red arrow in the last figure. I understand that a child class must reference the parent. But the parent class referencing an anonymous child? If the child class disappears, the parent class can continue executing perfectly. Suppose that this reference did not exist. That would allow to free a child anonymous class if there are no instances of this anonymous child. Isn't this the correct behavour?

Posted by Ramon on October 27, 2006 at 07:30 AM PDT #

@Ramon: It's not a JDK issue, its actually an implementation detail of the Level class. Level.class is maintaining a hard, static reference to all instances of Level. Given that Level.class is distributed by the JDK, it will likely have been loaded by the boot classloader. Thus, no instances of Level can be garbage collected meaning their anonymous inner class definitions and the classloaders that point to them cannot be garbage collected.

Posted by Tim on October 27, 2006 at 08:51 AM PDT #

Hi Aji,

Last week we had a customer report of a permgen space problem; it turned to be a problem with an OS that was missing some patches. Although that OS was not HPUX; nevertheless it may be worthwhile to check that your OS has the latest patches installed.

If the problem is easily reproducible and you escalate it to customer support, our engineering department will have a fix in no time for you.

Since HPUX probably doesn't support JDK 6 yet, one way of going about diagnosing this problem would be to run the Integration Server with HPROF enabled, trigger a memory dump (kill -3) and inspect the dump file with the modified jhat from JDK 6.

Frank

Posted by Frank Kieviet on October 28, 2006 at 04:30 PM PDT #

@Tim: Thanks, you are right. However, this seems to me a bug in java.util.logger.Level. It should hold weak references.

Posted by Ramon on October 29, 2006 at 01:00 AM PST #

About whether java.util.logging.Level has a bug: the way that custom Levels are used is confusing and error prone. Why does one have to create a subclass of Level to add a new custom level? The rationale for that completely escapes me. If serialization is the argument, the problem with that is the custom level class would have to be on the unmarshal side, which is likely problematic. Besides, with the existing code there's no issue with marshalling/unmarshalling custom levels.

The best solution would be to add a static factory method to the Level class, e.g.

   Level getLevel(int value, String name)

this method would create a new Level if a level with the same name/value cannot be found in the list of known Levels.

Currently there appears to be no other way to use a custom level in an application in an application server than to use a level that has a number as its name using the

    Level parse(name)

method: the implementation tries to interpret the name as a number. Alternatively one could hack together something serialization to instantiate level objects with arbitrary names and int values.

Frank

Posted by Frank Kieviet on October 29, 2006 at 11:39 AM PST #

I guess that it should be filed as bug (or improvement request) against Level implementation. It seems that changing on known array from storing Level themselves to weak (or soft) references to them should alleviate the problem.

Posted by al0 on November 08, 2006 at 04:43 PM PST #

I guess that it should be filed as bug (or improvement request) against Level implementation. It seems that changing on known array from storing Level themselves to weak (or soft) references to them should alleviate the problem.

Posted by al0 on November 08, 2006 at 04:44 PM PST #

Excellent! Since reading this article the other day, I have found and resolved two memory leaks that have been troubling me for some time.

I resolved one issue by modifying the source code of a package. Once I knew what the memory leak was, I was able to find information about it on the web.

Similarily, for the second issue, I obtained several clues from other online discussions and resolved the leak by moving the offending class to a different classloader.

I never had a way to determine what the leaks were, but your fantastic article has opened the door. Thanks.

Posted by downeyt on November 26, 2006 at 01:34 AM PST #

Hey! so this was the problem we had in here! My boss change de default space for the PermGen to 26 mb, 'cause non of us knows what the default size was ('cause in the app server it wasn't setted), 'till now! Keep on doing this blogs... A little late, but now we know how to deal with this in other installations we make. Bye!

Posted by Gloria Rocío Giménez Gamarra on December 07, 2006 at 05:41 AM PST #

By the way, I have filed a Java bug about this topic: #6543126

Posted by Jesse Glick on April 05, 2007 at 06:12 AM PDT #

Great info Frank. All Framework and AppServer coders should take note. A bad implementation by Level class. You are absolutely right - they should not have forced sub-classing to create new Levels. Even changing Level implementation to use WeakReferences now is going to break existing code.

Posted by Sony Mathew on April 12, 2007 at 06:39 AM PDT #

Since every class has a reference to the classloader that created it, does this mean that every Enum declared in a servlet will cause a memory leak?

It seems that the Enum class is created as a static class and has a static pointer to the classloader.

I have just done a memory test and there is a static reference to the classloader for every value of an Enum. Among these are the values for the Enum for the log4j logger levels.

Posted by Tim Downey on April 20, 2007 at 04:34 AM PDT #

Re Tim Downey:

Hi Tim,

I'm not familiar with the latest versions of Log4J: the one that I used doesn't have Java 5 style enums. In any case, if an enum class is created in classloader X, and a reference to an instance of this enum class is stored in another classloader, the classloader X can indeed not be garbage collected. If this is the case, it looks very much like the problem with the java.util.logging.Level problem described above.

Frank

Posted by Frank Kieviet on April 20, 2007 at 06:15 AM PDT #

Hi Frank, I have looked into the log4j source and the reason for the leaks is the Level class, not enumerations. However, there is a problem with enumerations, too. Even if I create an Enum and only use it in the same class, I get a leak. I am guessing that the implementation of the Enum class is doing something similar to what the Level class is doing.

Posted by Tim Downey on April 20, 2007 at 08:51 AM PDT #

Thank you! I appreciate your taking the time to write it up including great illustrations and also Edward Chou's and others work and discussion. I'll be sure to keep an eye out for what you write about next including the new jdk6 tools.

Posted by Roger on April 26, 2007 at 02:57 AM PDT #

I received the following additional information from Tim Downey:

I was creating an example to demonstrate the leak, when I tried one more test. After reloading the web app, I then ran your servlet that forces a GC collection in the perm gen. After that, the 'leaks' went away.

When a Java 1.5 enum is used, some static references are added to the class that point to the values of the enum. These objects are not collected on a reload, but they are collected when the perm gen is collected.

I guess that this means that such static references will cause more collections in the perm gen, but should not cause an out of memory error.

Tim

Posted by Frank Kieviet on May 01, 2007 at 02:42 AM PDT #

Thanks for this. I found this after attended your BOF.
On the other hand, JVM could do better job to garbage collect CUSTOMLEVEL class and only leave out Level.INFO, Level.SEVERE, etc.. JVM knew they were shared/used by other classes, otherwise, it would not establish that red link(in above diagram) in the first place.

Posted by Gary on May 15, 2007 at 02:33 AM PDT #

Hi Gary,

It's difficult for the JVM to do a better job: the Level.class object can definitely not be GC-ed and hence the static datamembers also cannot be GC-ed.

However, the implementation of the Level class is, eh, shall we say, not the best of all possible designs. For instance, the Level class could do without the list of all instances: it's only used for serialization. Also, why would you need to sub-class the Level class? A factory method would be a better approach.

Frank

Posted by Frank Kieviet on May 15, 2007 at 08:21 AM PDT #

Hi Frank

working on a tapestry application, with log4j, and a heap of other libraries. After running through and searching for leaks as suggested, i see hundreds (if not thousands) of references to static properties, and enums. In terms of fixing our own code, is it just a case of changing all the static properties to regular properties, and all the static methods to regular methods (as well as adding methods to return former static properties), and getting rid of enums? I'm still learning the trade, so forgive my ignorance, my level of understanding is not that great.

thanks
dave

Posted by dave on June 20, 2007 at 01:02 PM PDT #

Hi Dave,

I don't think the problem is in the use of static variables that you see as memory leaks. It's all about identifying the links from the server code to classes in your application. Did you try to use jhat? If you like I can send you the latest version (not available in the JDK yet) that has some more advanced filtering (courtesy of Edward Chou).

Frank

Posted by Frank Kieviet on June 20, 2007 at 05:06 PM PDT #

hmmm... perhaps i didn't select the right links, or maybe i was just interpreting the results incorrectly. i'll read through again, more thoroughly.
yes, i downloaded jdk 6 and played with jmap and jhat, if you could send me the updated version that'd be great.
dtra82 at gmail dot com

thanks

Posted by dave on June 21, 2007 at 09:22 AM PDT #

You can download the updated jhat.jar from Edward Chou's blog.

Posted by Frank Kieviet on June 21, 2007 at 09:31 AM PDT #

Adding a servlet listener to release log4j references on destroy context may help:

public class ApplicationLifecycleListener implements ServletContextListener
{
public void contextDestroyed(final ServletContextEvent sce)
{
LogFactory.release(Thread.currentThread().getContextClassLoader());
}

public void contextInitialized(final ServletContextEvent sce)
{
}
}

Posted by David Mackenzie on September 10, 2007 at 12:29 PM PDT #

+1! garbage collection is my number one feature of java, but...

Posted by karl on September 11, 2007 at 12:47 PM PDT #

In the conclusion the article mentions that the leak is because of reference from outside. From the example given it looks like a case of cyclical reference to me. If Level class not loaded before creating the servlet, then it has to be loaded by the AppClassLoader. Since the class of Level contains reference to CustomLevel, custom level can not be GCed and hence the AppClassLoader can not be GCed and hence any classes loaded by AppClassLoader can not be GCed.

Posted by Shripad Agashe on October 31, 2007 at 02:51 PM PDT #

Re Shripad:

The "outside" is the classloader that loads the Level class. The AppClassloader doesn't have load the Level.class: it will delegate such a load request to the parent classloader. On the other hand, the CustomLevel class is in fact loaded by the AppClassloader, so now there's a reference from the classloader that loaded Level to the classloader that loaded CustomLevel.

Frank

Posted by Frank Kieviet on November 01, 2007 at 04:22 AM PDT #

Hi Frank,
One interesting thought came to me is regarding use of static objects. Lets take a case where a class C1 has object O1 as a static member variable. Now if the O1 is updated after loading, should the class C1 be allowed to be GCed? Since if the class C1 is loaded and unloaded again, it may lose the state of O1. Not sure how this use case would work.

Posted by Shripad Agashe on November 01, 2007 at 05:10 PM PDT #

Can someone answer this question about the bad Level implementation: does it mean that whenever I used the stcLogger that comes with JCAPS JCD, I have the issue of class loader leak? or is the stcLogger not using the Level class that comes with JDK?

Posted by Tony Tung on November 08, 2007 at 08:03 AM PST #

Hai Frank
Excellent article. Currently we have integrated OSGI framework to our appserver. We never considered the points mentioned in this blog. Thinking of reviewing the code of service bundles we are providing and do some testing by doing multiple deploy-undeploy. Can you give some suggestions ?

Posted by Hari on February 04, 2008 at 04:27 PM PST #

Java enums are definitely causing memory leaks when reloading web applications under tomcat.

We are using Java 1.6 and tested with SUN JVM and JRocket. As you mentioned, the enums keep pointers to classloader and the classloader does not unload. This causes memory leaks.

With SUN JVM, you get a perm space out of memory error after just a few reloads, but not with JRocket (since it does not use the permspace).

Does anyone know how to resolve this memory leak caused by Java enums other than not using it?

Thanks,
Frik

Posted by Frik Strecker on March 08, 2008 at 01:57 AM PST #

Re Frik:

I haven't seen this problem myself, perhaps because I haven't been using enums much.

If it's indeed a problem, it's a critical bug that should be fixed asap. Perhaps you can file a ticket for this problem?

Frank

Posted by Frank Kieviet on March 09, 2008 at 05:13 AM PDT #

Based on the comments by Tim Downey on this page, the enums keep a pointer to the classloader which results in a leak on reload.

I am not sure exactly how to proof or disproof what he found in his tests, but I found projects with enums do not get unloaded properly. So there seems to be merit for what he is saying and therefor my previous post. However, Java reflection does not show this pointer, but I do not what the bytecode will show. I will try a decompiler today and see what it shows.

I will be hooking up JProfiler today to see if I can get to the bottom of this, but any ideas are welcome.

As you said, if this is the case, then this is a serious bug that warrants attention.

Posted by Frik Strecker on March 09, 2008 at 08:59 PM PDT #

We also notice this with Inner classes. Even if you manually de-reference the instance, the containing instance seems to retain a reference,

Posted by garys on March 19, 2008 at 08:57 AM PDT #

Re Frik:

If there are any links between a class and enums, it should show up in memory dumps that can be obtained with jmap and analyzed with jhat. I think that would be the easiest way to look into the issue.

Frank

Posted by Frank Kieviet on March 20, 2008 at 03:54 AM PDT #

Re Garys:

I don't believe there are any classloader leaks because of the use of inner classes: these are used all over the place, including in applications of which the classes are in fact GC-ed correctly after undeployment.

Frank

Posted by Frank Kieviet on March 20, 2008 at 03:57 AM PDT #

Hi Everyone,

There is been a problem in my application server trying to deploy an application with multiple web contexts, and the total ear file size is slightly more than a Gig. The JVM throws out the Perm Gen Space error when trying to Unzip the file during deployment.
Application Server being used JBoss 4.2.

Thank you.
Rama

Posted by Rama on March 26, 2008 at 03:53 AM PDT #

Re Rama:

With an EAR of about a Gb, the problem that you're running into may not be a classloader leak, but simply because of the fact that it will take a lot of memory to load all these classes into memory. Try to increase your permspace memory.

Frank

Posted by Frank Kieviet on March 26, 2008 at 06:54 AM PDT #

Hi Frank,

this is hugely informative, thank you so much for the post.

I do have a question though: I am facing the dread PermGen OOM, but the tomcat instances are configured to not allow redeployment of the webapps.
This setting was chosen to address this very specific problem (tomcat+struts suffers from it because of the "static reference to the classloader in a Servlet" pattern).

Now if you take redeployment out of the equation, what else can explain the PermGen OOM? My guess is that it could be the number of jars/classes in my webapps, or String.intern() (through XML parsing for example).

What I am looking for is a way to either 1) inspect the permgen 2) calculate my permgen requirement based on the number of classes / Strings / classloaders

Using YourKit profiler does not help much because the profiler does not tell me what is in what generation.

Any ideas?

- Renaud

Posted by Renaud Bruyeron on April 22, 2008 at 01:57 AM PDT #

Re Renaud,

You can use jconsole to inspect the various memory pools, including permgen space.

To see how much memory you need, you could increase the permgen memory setting, start Tomcat, measure the permgen usage, deploy your application, and measure again.

Frank

Posted by Frank Kieviet on April 25, 2008 at 03:00 AM PDT #

Hi Frank,

thank you for your reply.
The thing is, I am beyond the basics already: the webapp seems to be leaking permgen \*after\* deploy - i.e. permgen grows in spikes after a few days.

See http://www.deuxc.org/issue-permgen-cms.png

This shows a graph of MemoryMXBean.getNonHeapMemoryUsage().getUsed()
over time. Notice the spikes between the 6th and 7th of June, the 9th and 10th and the final one (which caused the famous PermGen OOM) at the end.

When I correlate this to the actual activity of the application, the spikes happen on publication jobs (this is a content management system). This involves a lot of XSL processing and hibernate activity.

What I am really looking for is something to \*inspect\* what is inside the nonheap memory pools so that I can get a sense of 1) why it's growing like it is (i.e. is it a leak?) 2) what the appropriate setting should be avoid OOM if it is not a leak

What do you think?

Posted by Renaud Bruyeron on June 15, 2008 at 06:44 PM PDT #

Thank you, this is very helpful. I am having a permgen problem with eclipse and needed to understand permgen. I added -XX:MaxPermSize=128m
to eclipse.ini which I hope will help in addition to Xms Xmx

Posted by Jol on July 10, 2008 at 10:07 AM PDT #

Thanks for the post, very helpful! I'm using Java 6 + Tomcat 5.5 + Struts 1.2.x + Hibernate and note that the PermGen memory increase each time I deploy/undeploy an application. I tried to use another JVM (BEA JRockit ) but the problem is the same, this JVM has a memory area called "Class Memory" with a behaviour very similar. Maybe the problem is the Struts' classloader.

Posted by Igor on July 14, 2008 at 11:22 PM PDT #

And... Still relevant three years later. Thanks.

Posted by Harry on February 15, 2009 at 03:48 PM PST #

Hi Frank,

It was really a nice article to under stand the concept of classloader and Permgen space.thanks a lot.

I have the following questions:

if such kind of application is deployed in server and the same error comes
what is the quick solution to make the server up?
if we increase the maxpersize in memory arguements does it going to help??

Thx
Souvik

Posted by Souvik Das on August 19, 2009 at 05:29 AM PDT #

Re Souvik:

If there's a true memory leak, increasing the permgen setting of the VM will only postpone the system running out of memory. Depending on how the memory leaks, e.g. only when the application is redeployed, increasing the memory settings may be a useful workaround, similarly to restarting the server when redeploying the leaking application.

HTH,
Frank

Posted by Frank Kieviet on August 24, 2009 at 09:29 AM PDT #

If I understand correctly, the problem resides in the java.util.logging package and the fact that it is used by the JVM that also starts the whole server; what's hapening if one puts rt.jar in WEB-INF/lib? Wouldn't in this case the java.util.logging classes be garbage-collected when the server stops?

Posted by Marian on September 29, 2009 at 07:33 AM PDT #

Re Marian:

First of all, the java.util.logging.Level issue is just an example. Indeed the problem in this example is that the JVM (root classloader = parent classloader) is holding a reference to the class that is loaded in the application classloader.

Moving rt.jar into the web application won't work, because even if you setup the delegation model to self-first, no classloader is allowed to try to self-first delegate classes that start with java.\* or javax.\*.

For other examples, where the classes are not java.\* or javax.\*, moving the jar to the application classloader would indeed likely solve the problem. Instead of moving, one could also copy the jar into a web application and turn on the self-first delegation model.

HTH,
Frank

Posted by Frank Kieviet on October 01, 2009 at 07:27 AM PDT #

<p><a href="http://www.hlcsuperstore.com/mobile-phones_cell-phone-accessories.html">cell phones accessories</a></p>
<p><a href="http://www.hlcsuperstore.com/mobile-phones_cell-phones.html">cell phones</a></p>
<p><a href="http://www.hlcsuperstore.com/mobile-phones_cell-phone-batteries.html">cell phone battery</a></p>

Posted by snio on October 15, 2009 at 10:28 AM PDT #

Thank you for your help. I'm a maven fan and always trying to automate deploy on continuus integration. I don't understand why that OOM is always a fatality on medium/big sized project.

Regards,

Posted by Alexis on October 18, 2009 at 07:03 PM PDT #

Hi Frank,

Great article, thank you. I was wondering if you had any plan to write a similiar article about the code cache in JDK 6? In my situation we have a home grown application server. My code that runs in this server uses Janino to generate code at runtime. All the permGen issues have been resolved i.e. permGen does not increase anymore over time and is correctly garbage collected. However, the code cache increases continually until we get an OutOfMemoryError (swap space). I am finding this more difficult to solve than the permGen issues.

best.

Posted by Paul Nolan on October 26, 2009 at 06:40 PM PDT #

Re Paul:

Hi Paul, I don't have much expertise in native compilation, code caching, or the VM for that matter, but it seems to me that if permgen is constant and if the code cache keeps growing, you've hit a bug in the JVM. Did you try to bring this up with the JDK team?

Frank

Posted by Frank Kieviet on October 29, 2009 at 09:08 AM PDT #

Thanks for the great article...great information shared.

Few pointers here as well,
http://minmaxmim.blogspot.com/2010/01/javalangoutofmemoryerror-java-heap.html

http://minmaxmim.blogspot.com/2010/01/javalangoutofmemoryerror-permgen-space.html

Posted by m M on January 04, 2010 at 02:48 PM PST #

Perfect!! I never knew PermGen is growing exponentially across each deployment. I will never use that setting ever again in my Jrockit as I never use Sun's JDK to avoid memory problems, but unsuccessful till now. I even compile in Jrockit to avoid memory problems, but always got OOM and PermGenSpace

Posted by lava kafle on January 13, 2010 at 06:55 PM PST #

My problem was fixed by putting referenced libraries in the web server's shared library folder rather than the deployed application's library folder.

Posted by Chris on April 21, 2010 at 05:53 PM PDT #

I get this problem quite a bit, cheers for the heads up. Off to your next post to see how I can fix it.

Posted by stamp duty calculator on August 09, 2010 at 10:39 AM PDT #

Hi everybody,
I get the same pb. But it seems not cause by redeployed the applications.
have we anothers reasons ?
thanks

Posted by blueocean on October 19, 2010 at 05:01 PM PDT #

I am facing out of memory permgen exception.I tried to increase my permgen and log class loading and unloading.Unfortunately I am down after two days and when I traced the log I found that loading of each class took place one time and no grabage collector is up until the permgen reached maximum then garbage collector kept on running nt able to unload any class I am wondering what made my permgen increase whileeach class is loaded only once .I forgot to say that I am using tomcat and Sun JVM 6

Posted by Nesma Rashad on October 25, 2010 at 09:11 PM PDT #

Is there a way for the app to monitor perm gen space usage?
e.g. our app writes a WARN to Tomcat log if Runtime.getRuntime().freeMemory() falls below 10 mb, but that doesn't help w/ the perm gen problem.

Posted by Mike Sweeney on December 08, 2010 at 11:56 PM PST #

why? it said i neede to anser a simple math question

Posted by guest on March 14, 2011 at 01:17 AM PDT #

Post a Comment:
Comments are closed for this entry.
About

fkieviet

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today