OutOfMemoryError looks a bit better!

OutOfMemoryError has always been a confusing error. For a long time the HotSpot Virtual Machine threw this error without a detail message or stack trace so typically the thread throwing the error would terminate with this:
  Exception in thread "main" java.lang.OutOfMemoryError

That was confusing. Is my java heap full or does it mean something else? Those familiar with the heap layout that the HotSpot VM uses will know that the "something else" might mean the "permanent generation". This is place where reflective data such as class and method objects are allocated. It is also the place where interned strings are stored. If you've got an application that loads a huge number of classes or interns millions of strings then it's possible that the OutOfMemoryError is because the permanent generation is full rather than the java heap.

In 5.0 the error is less confusing as there is a detail message. This means you will see something like this:

  Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
or:
  Exception in thread "main" java.lang.OutOfMemoryError: PermGen full

In Mustang, there has been further changes to the way that OutOfMemoryError is reported. One obvious one is that the HotSpot VM will attempt to include a stack trace. This means you should see something like this:

  Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at ConsumeHeap$BigObject.(ConsumeHeap.java:22)
        at ConsumeHeap.main(ConsumeHeap.java:47)

In this example we see the stack trace where the allocation failed. If the OutOfMemoryError is because the perm gen it full then you might see String.intern or ClassLoader.defineClass near the top of the stack.

Is a stack trace useful? In some limited cases it can be. For example, suppose you've got a thread looping and in the loop it is allocating objects and putting them into a collection. In that case the stack trace might direct you to a good starting place. More generally, the stack trace is not likely to be useful. If you've got a busy application and the heap is nearly full, then OutOfMemoryError will be likely be thrown in some random place by some random mutator.

One useful improvement is the -XX:+HeapDumpOnOutOfMemoryError option which tells the HotSpot VM to generate a heap dump when an allocation from the java heap or the permanent generation cannot be satisfied. There isn't any overhead to running with this option so it should be useful for production systems where OutOfMemoryError takes weeks (or longer) to surface. The heap dump is in HPROF binary format so it can be analyzed using any tools that can import this format. If you read Sundar's blog then you'll know that Mustang includes a useful tool called jhat which can be used to do rudimentary analysis of the dump. jhat supports Object Query Language so you can easily create your own queries and mine the heap dump.

The next improvement is visible when the system is almost out of swap space and an allocation from the native heap fails in the VM. In that case the VM aborts and you get a one-line error such as the following:

  Exception java.lang.OutOfMemoryError: requested 16 bytes for CHeapObj-new. Out of swap space?

This message has been known to confuse a lot of developers. At first glance it might it might look like that the java heap is full. Of course if you increase the size of the heap with the -mx option it might make the problem worse as the larger java heap means there is less native memory available.

In Mustang this condition will trigger the VM to invoke the fatal error handling mechanism. This means you should get a fatal error log as you would get with a normal (abnormal?) crash. The fatal error log is named hs_err_<pid>.log and contains useful information about the thread, process, and system at the time of the crash. In the case of native heap exhaustion then the heap memory and memory map information in the log can be useful. The exact format is version and platform specific and you will get more information in the J2SE Troubleshooting Guide.

Hopefully, developers will find these improvements useful. It should make OutOfMemoryError a little less confusing and it means the error is no longer a one-line wonder.

Comments:

Alan, is there an easy way to shut down the VM on a "normal" (heap) OutOfMemory? I find that the exception is of no use whatsoever. Au contraire, we often find VMs in the field that encounter partial crashes because one important thread happended to run into OOM. E.g. webapp runs out of heap, tomcat's connection acceptor thread gets OOM and we have a server that runs but does no longer accept connections. If we're lucky, we find the OOM somewhere on stderr. A simple "crash,burn,restart" mechanism would be my favourite. -- Greetings, Matthias

Posted by Matthias Ernst on November 28, 2005 at 02:56 AM PST #

There is another -XX option which can be used to execute actions when the OOME is thrown. The primary motive for that option is to do additional data collection at the point when the first OOME is thrown. The actions can include killing the VM. That said, it might be worth making this simpler with something like +ExitOnOutOfMemoryError. One would need to be careful of course and you wouldn't want to use this in environments where you are loading mobile code.

Posted by alanb on November 28, 2005 at 04:32 AM PST #

This message has been known to confuse a lot of developers

Posted by guest on November 29, 2005 at 03:58 PM PST #

cant you use the uncaught exception handler to detect an OOM error and restart you system should that happen. This is neater that some -XXoutOfMemory handler

Posted by Michael Skells on December 01, 2005 at 06:43 PM PST #

Indeed, it could be useful. Matthias mentioned he was using Tomcat so I don't know if he's got the opportunity to set the default uncaught exception handler. That said, his mileage will vary. If the java heap is completely exhausted then it might not be possible to do very much in the uncaughtException implementation. System.exit might be okay. The script or whatever that started the server will then need to look after restarting it.

Posted by alanb on December 01, 2005 at 08:30 PM PST #

Regarding OOME (and other errors for that matter) and restarts: How about using reflection to start your application (invoke the main() method)? something like this: <code>

import java.lang.reflect.\*;
public class Invoker{
&nbsp public static void main(String[] args){
&nbsp&nbsp int count = 5;
&nbsp&nbsp for(int i=0; i < count; i++){
&nbsp&nbsp try {
&nbsp&nbsp&nbsp Class.forName("MemoryConsumer").getMethod("main", new Class[] {String[].class}).invoke(null, new Object[] {args});
&nbsp&nbsp } catch (InvocationTargetException ite) {
&nbsp&nbsp&nbsp System.err.println("THROWABLE: "+ite+ " " + ite.getTargetException());
&nbsp&nbsp } catch (Exception e){
&nbsp&nbsp&nbsp System.err.println("THROWABLE: "+e);
&nbsp&nbsp }
&nbsp&nbsp System.gc();
&nbsp&nbsp System.gc();
&nbsp&nbsp System.err.println("After loop #"+(i+1)+" Free memory: "+Runtime.getRuntime().freeMemory());
&nbsp&nbsp try{Thread.sleep(5000);}catch(InterruptedException ie){System.err.println(ie);}
&nbsp&nbsp }
&nbsp }
}
</code> Probably not a "best practice", but it works. After the OOME is thrown, there are no references to any class/object in the "MemoryConsumer" application, so the gc is able to free a lot of memory.

Another solution would be to use a native script (batch/sh etc), but that won't be 100% java...
Any opinions?

Posted by guest on December 02, 2005 at 02:17 AM PST #

I would like to add that the Invoker-class should log/alert when an OOME (or other uncaught exception) is thrown of course, so appropriate action can be taken. But UNTIL I have fixed the bug, or increased heap space, it would be nice if the downtime for my application is as low as possible. (eg automcatic restart)

Posted by guest on December 02, 2005 at 02:20 AM PST #

Invoker only catches exceptions thrown by the main thread. When the java heap is full then OOME could be thrown by a different thread or by many threads at around the same time. Even if you catch the OOME then there isn't a general way to recover, and any attempting recovery could also fail with OOME.

Posted by alan on December 02, 2005 at 05:46 AM PST #

There is actually a possibility to automaticly restart arbitrary java application eg. Matthias Tomcat Server. The tanukisoftware opensource service wrapper does an almost perfect job on an occurence of a OOM Error! See the example on: http://wrapper.tanukisoftware.org/doc/english/prop-filter-x-n.html

Posted by simon on January 10, 2006 at 09:55 PM PST #

There needs to be a specific OutOfMemoryError detail for when the Graphics card runs out of accelerated graphics memory when creating VolatileImages. I experienced this problem and took a week to figure out what it was. Keith

Posted by guest on April 11, 2006 at 05:22 PM PDT #

If there is option to ask the JVM print all the collections variable name and corresponding size, it will be nice, if possible and not slow the application when OOME not throw

Posted by Carfield Yim on August 31, 2006 at 03:32 AM PDT #

On personal opinion, I find this very helpful. Guys, I have also posted some more relevant info further on this, not sure if you find it useful: http://www.bidmaxhost.com/forum/

Posted by guanhua on March 23, 2007 at 02:38 PM PDT #

i'm experiencing "java.lang.OutOfMemoryError: Java heap space" errors so i added the -XX:+HeapDumpOnOutOfMemoryError option, but when the first error is thrown, no dump file is created. Where is supposed to be created the dump? Any help would be appreciated.

Posted by daad on April 29, 2007 at 07:13 PM PDT #

hi every body If you are getting outofmemory exception <no stact trace available> then here are some tips to avoid it...... 1: avoid the creation of same type of objects again and again in a loop because whens it created use the object name rather recreating it. this avoid the memory heap to rise and also helps the garbagecollector to keep proper track and free it........ 2:avoid the use of recursions if any........ because return address may fill your return memory stack...... if you still have a doubth try to use recursion in a loop and check what error you get.... love java......... bye

Posted by ganesh modgekar on July 17, 2007 at 09:46 PM PDT #

You said: "There is another -XX option which can be used to execute actions when the OOME is thrown."

What is that option? I've seen rumours of a -XXOnError, but I haven't been able to find more details about how to use this option, if it exists.

I want to be able to tell the JVM to die on an OOM, instead of having it hang around in a crippled state. Then something like the Java Service Wrapper can restart it for another try.

Posted by Albert Strasheim on September 27, 2007 at 01:31 AM PDT #

Here is a good list of jvm options http://blogs.sun.com/watt/resource/jvm-options-list.html

Posted by Jack on March 04, 2008 at 05:12 AM PST #

Post a Comment:
Comments are closed for this entry.
About

user12820862

Search

Top Tags
Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today
News
Blogroll

No bookmarks in folder