X

Poonam Bajaj's Blog

  • October 10, 2014

Running on a 64bit platform and still running out of memory

Poonam Parhar
Consulting Member of Technical Staff

It sounds strange that one is running with 64bit JVM but still running out of native memory or Java Heap space. On a 64bit machine, with 64bit JVM we get almost unlimited memory then how is it possible to run out of memory? Well, it may happen in certain situations.

HotSpot JVM has a feature called CompressedOops with which it uses 32-bit (compressed) pointers on 64-bit platforms to have smaller footprints and better performance on 64-bit platforms. 64-bit address values are stored into 32-bit pointers using an encoding and are retrieved back using the corresponding decoding.

CompressedOops implementation tries to allocate Java Heap using different strategies based on the heap size and the platform it runs on. If the heap size is less than 4Gb then it tries to allocate it into low virtual address space (below 4Gb) so that the compressed oops can be used without encoding/decoding. If it fails or heap size > 4Gb then it tries to allocate the heap below 32Gb to use zero based compressed oops. If this also fails then it switches to regular compressed oops with narrow oop base.

The heap base where the Java Heap would be based at is computed by the JVM function Universe::preferred_heap_base():
http://hg.openjdk.java.net/jdk8/jdk8/hotspot/file/87ee5ee27509/src/share/vm/memory/universe.cpp#l672

Heap base computation depends on the heap size and the HeapBaseMinAddress value for the platform we are running on. If the heap_size+HeapBaseMinAddress can fit in the first 4G address space, then heap is allocated below 4G. Then it checks if it can fit under 32G boundary otherwise java heap is allocated in the higher addresses.

In doing so, it puts a cap on the size that the native memory can grow up to, and also limits the growth of the Java Heap, and this may lead to out of memory errors for memory intensive applications. The following process map shows that the java heap is allocated just next to the native heap.

0000000100000000          8K r-x--  /sw/.es-base/sparc/pkg/jdk-1.7.0_60/bin/sparcv9/java
0000000100100000          8K rwx--  /sw/.es-base/sparc/pkg/jdk-1.7.0_60/bin/sparcv9/java
0000000100102000         56K rwx--    [ heap ]
0000000100110000       2624K rwx--    [ heap ]   <--- native Heap
00000001FB000000      24576K rw---    [ anon ]   <--- Java Heap starts here
0000000200000000    1396736K rw---    [ anon ]
0000000600000000     700416K rw---    [ anon ]

Workarounds for this situation:
1. Run with -XX:-UseCompressedOops (keeping in mind that you would lose the performance benefit that comes with enabling CompressedOops). This will instruct the JVM to run without the Compressed Oops feature and it would not try to fit the Java Heap in the first 4GB of heap.
2. Keep the CompressedOops and set the base of the Java Heap with the JVM option -XX:HeapBaseMinAddress=n to specify the address where the Java Heap should start from. If you are running out of native memory, set it to a higher address so that there is enough room available for the native mallocs, and if Java Heap is getting exhausted then set it to a lower value. And if you need java heap to be allocated beyond 4GB or 32GB then HeapBaseMinAddress should be set to a value so that HeapBaseMinAddress + heap size > 4GB or 32GB.
The default value of HeapBaseMinAddress on most of the platforms is 2GB.

Let's run a simple test program on solaris-sparc 10 and solaris-x64 11 and see where the Java Heap is placed on these platforms when using CompressedOops.

On Solaris-sparc 10 with JDK 1.7.0_67

% uname -a
SunOS svr 5.10 Generic_148888-03 sun4v sparc SUNW,Sun-Fire-T200

% java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b32)
Java HotSpot(TM) Server VM (build 24.65-b04, mixed mode)

% java -d64 -XX:+PrintCommandLineFlags -XX:+PrintFlagsFinal TestClass &
[3] 22638
% -XX:InitialHeapSize=266338304 -XX:MaxHeapSize=4261412864 -XX:+PrintCommandLineFlags -XX:+PrintFlagsFinal -XX:+UseCompressedOops -XX:+UseParallelGC
[Global flags]
    uintx AdaptivePermSizeWeight                    = 20              {product}
    ...<snip>..
    uintx GCTimeRatio                               = 99              {product}
    uintx HeapBaseMinAddress                        = 4294967296      {pd product}
    bool HeapDumpAfterFullGC                        = false           {manageable}
    ....<snip>...
    Heap Parameters:
ParallelScavengeHeap [ PSYoungGen [ eden =  [0xffffffff1e800000,0xffffffff1ebf5d88,0xffffffff22a00000] , from =  [0xffffffff235000
00,0xffffffff23500000,0xffffffff24000000] , to =  [0xffffffff22a00000,0xffffffff22a00000,0xffffffff23500000]  ] PSOldGen [  [0xfffffffe75400000,0xfffffffe75400000,0xfffffffe7fc00000]  ] PSPermGen [  [0xfffffffe70400000,0xfffffffe706843a0,0xfffffffe71c00000]  ]

The default value of HeapBaseMinAddress is 4GB here and the Java Heap is allocated in the higher virtual address space.

On Solaris-x64 11 with JDK 1.7.0_67

bash-4.1$ uname -a
SunOS hsdev-15 5.11 11.0 i86pc i386 i86pc

bash-4.1$ java -d64 -XX:+PrintCommandLineFlags -XX:+PrintFlagsFinal TestClass &
[1] 6049
bash-4.1$ -XX:InitialHeapSize=805015296 -XX:MaxHeapSize=12880244736 -XX:+PrintCommandLineFlags -XX:+PrintFlagsFinal -XX:+UseCompressedOops -XX:+UseParallelGC
[Global flags]
    ....
    uintx GCTimeRatio                               = 99              {product}
    uintx HeapBaseMinAddress                        = 268435456       {pd product}
    bool HeapDumpAfterFullGC                       = false           {manageable}
    ....
bash-4.1$ java -d64 -classpath $JAVA_HOME/lib/sa-jdi.jar sun.jvm.hotspot.CLHSDB 6049
Attaching to process 6049, please wait...
hsdb> universe
Heap Parameters:
ParallelScavengeHeap [ PSYoungGen [ eden =  [0x0000000700000000,0x00000007007ae238,0x000000070c000000] , from =  [0x000000070e0000
00,0x000000070e000000,0x0000000710000000] , to =  [0x000000070c000000,0x000000070c000000,0x000000070e000000]  ] PSOldGen [  [0x0000000500400000,0x0000000500400000,0x0000000520200000]  ] PSPermGen [  [0x00000004fb200000,0x00000004fb483380,0x00000004fc800000]  ]

bash-4.1$ pmap 6049
0000000000400000          4K r-x--  /java/bin/amd64/java
0000000000410000          4K rw---  /java/bin/amd64/java
0000000000411000       2288K rw---    [ heap ]
00000004FB200000      22528K rw---    [ anon ]
0000000500400000     522240K rw---    [ anon ]
0000000700000000     262144K rw---    [ anon ]

Here, the default value of HeapBaseMinAddress is 256MB and the Java Heap is allocated in the 32G address space.

On Solaris-x64 11 with JDK 1.8.0_20

bash-4.1$ java -version
java version "1.8.0_20"
Java(TM) SE Runtime Environment (build 1.8.0_20-b26)
Java HotSpot(TM) 64-Bit Server VM (build 25.20-b23, mixed mode)
bash-4.1$ java -d64 -XX:+PrintCommandLineFlags -XX:+PrintFlagsFinal TestClass &
[1] 19069
bash-4.1$ -XX:InitialHeapSize=805015296 -XX:MaxHeapSize=12880244736 -XX:+PrintCommandLineFlags -XX:+PrintFlagsFinal -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseParallelGC
[Global flags]
    uintx AdaptiveSizeDecrementScaleFactor          = 4                                   {product}
    ....
    uintx GCTimeRatio                               = 99                                  {product}
    uintx HeapBaseMinAddress                        = 2147483648                          {pd product}
    bool HeapDumpAfterFullGC                        = false                               {manageable}
    .....
bash-4.1$ java -d64 -classpath $JAVA_HOME/lib/sa-jdi.jar sun.jvm.hotspot.CLHSDB 19069
Attaching to process 19069, please wait...
hsdb> universe
Heap Parameters:
ParallelScavengeHeap [ PSYoungGen [ eden =  [0x00000006c0200000,0x00000006c190a4d8,0x00000006cc200000] , from =  [0x00000006ce2000
00,0x00000006ce200000,0x00000006d0200000] , to =  [0x00000006cc200000,0x00000006cc200000,0x00000006ce200000]  ] PSOldGen [  [0x00000004c0400000,0x00000004c0400000,0x00000004e0400000]  ]  ]

bash-4.1$ pmap 19069
19069:  java -d64 -XX:+PrintCommandLineFlags -XX:+PrintFlagsFinal TestClass
0000000000411000       3696K rw---    [ heap ]
00000004C0400000     524288K rw---    [ anon ]
00000006C0200000     262144K rw---    [ anon ]
00000007C0000000        512K rw---    [ anon ]

Here, HeapBaseMinAddress is 2GB by default and heap is allocated in the first 32GB address space.

The default value of HeapBaseMinAddress for Solaris-x64 systems in jdk1.7.0_xx is 256M. It has been changed to 2G in jdk8 with the following bug fix: https://bugs.openjdk.java.net/browse/JDK-8013938. 

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha