Case study: How to leverage +LargePages option of J2SE1.5 on Linux hugemem kernel for 11i Oracle Applications java servlet based modules (Experiments from the Field..Based on True Stories)

Preface

This case study deals with Oracle configurator product (shortcode CZ) of 11i Oracle Applications and how to make it more stable and scalable by leveraging the LargePages option provided in J2SE1.5. This paper is in continuation to earlier article Common Incorrect beliefs about process address limits on 32bit platforms
It is assumed that the maintenance release of 11i Oracle Applications is 11.5.10 as per certification specified in Note 304099.1 - Using J2SE Version 5.0 with Oracle E-Business Suite 11i. To find the latest version of Java certified with the E-Business Suite application tier, please see OracleMetaLink Note 300482.1, titled "Overview of Using Java with Oracle E-Business Suite Release 11i".
This study assumes RHAS3 as the running platform OS.

The general approach and the case for this study

To increase stability and response time of a java servlet based application (say Oracle configurator), at least these two conditions must be satistifed:

Garbage collection must be minimized - The more the overall heapsize, the lesser will be the contention for space across different sessions, assuming the user load remains the same. The lesser the demand for object or session allocation in the JVM heap, the lesser will be the garbage collection.
Response time must be reasonably high - This, again is dependent upon garbage collection. During full garbage collection (GC) cycles, the application is not able to respond to the user activity because its doing background maintenance. Therefore, the lesser the garbage collection, the higher is the response time.

There might also be a requirement for doing a considerable caching of configurator models in the JVM heap, which adds to the memory overhead. Using LargePage option can be useful in such a situation.

Installing the hugemem kernel

Based on the article Redhat Linux Kernels and process space limits using Java.. Some eye-openers , it is already established that the hugemem kernel allows to address upto 3.4G of process address space.

Install the hugemem kernel rpm for the same linux kernel level on the configurator tier. As an example For 2.4.21-40 level, the kernel-hugemem-2.4.21-40.EL.i686.rpm is the corresponding rpm.

As root:

# rpm -Uvh kernel-hugemem-2.4.21-40.EL.i686.rpm

Since you would be persisting with the hugemem kernel across boots, make Hugemem the default kernel by specifying the default=0 line in the Boot loader configuration file. Assuming the boot Loader is GRUB, the file would be /etc/grub.conf. default= specifies the order of the kernel listed in loader configuration file, starting with 0 as the first. It might be set to different value(1,2,3) depending on however many kernels installed on the machine.

Note: Installing the hugemem kernel from the rpm does put the hugemem kernel as the first entry in the boot loader configuration file (/etc/grub.conf), therefore default=0 will be needed to be specified.

#more /etc/grub.conf

# grub.conf generated by anaconda

# Note that you do not have to rerun grub after making changes

# to this file

# NOTICE: You have a /boot partition. This means that

# all kernel and initrd paths are relative to /boot/, eg.

# root (hd0,0)

# kernel /vmlinuz-version ro root=/dev/rootvg/lv1

# initrd /initrd-version.img

#boot=/dev/sda

default=0

timeout=10

splashimage=(hd0,0)/grub/splash.xpm.gz

title Red Hat Enterprise Linux ES (2.4.21-40.ELhugemem)

root (hd0,0)

kernel /vmlinuz-2.4.21-40.ELhugemem ro root=/dev/rootvg/lv1 hda=ide-scsi

initrd /initrd-2.4.21-40.ELhugemem.img

.....

Create a symbolic link for System.map in /boot to point to System.map file specific to hugemem kernel.

#pwd
/boot
# ls -l System.map
lrwxrwxrwx 1 root root 30 May 24 20:55 System.map -> System.map-2.4.21-40.ELhugemem

This is a also good time to install the jdk-1_5_0_06-linux-i586.rpm (J2SE(TM) Development Kit 5.0 Update 6 from http://java.sun.com).

As root:

# rpm -Uvh jdk-1_5_0_06-linux-i586.rpm

Warn the Operating System that the limits will be challenged:

Add the following in /etc/security/limits.conf

applmgr soft nofile 65536
applmgr hard nofile 65536

applmgr soft nproc 16384

applmgr hard nproc 16384

applmgr soft stack unlimited

applmgr hard stack unlimited

applmgr soft memlock unlimited

applmgr hard memlock unlimited

Set the ulimit -a for the unix user running the JVM like this:

# ulimit -a
time(cpu-seconds) unlimited

file(blocks) unlimited

coredump(blocks) 0

data(kbytes) unlimited

stack(kbytes) unlimited

lockedmem(kbytes) unlimited

memory(kbytes) unlimited

nofiles(descriptors) 65536

processes 16384

Implement hugepage & supporting features in OS

Make kernel configuration changes for supporting Hugepages and shared memory segments. The example assumes that you would be locking 9GB of real memory on the machine with 12G of RAM.

Add this the /etc/sysctl.conf :

##############################################################
# Added for huge page access

# This is worth of 9GB locked in Real Memory

# This limit needs to be changed as per the available memory

# on the Box with 12g RAM. Change this as per your

# requirement
#

##############################################################

vm.hugetlb_pool=9216

##############################################################

# Added to keep maxsize of shared memory segment at 3.5g

# This is required to support LargePages feature of JDK1.5

# and to avoid the following message at JVM startup:
# Java HotSpot(TM) Server VM warning: Failed to reserve

# shared memory (errno = 22).
##############################################################

kernel.shmmax=3758096384

kernel.shmall=3758096384

##############################################################

Mount hugepage filesystem as root:

# mkdir -p /mnt/hugepages

The following entries should be there in /etc/fstab:

# vi /etc/fstab
..

##############################################################

# These entries are needed to make hugemem work

##############################################################

none /dev/shm tmpfs defaults,size=9G 0 0

##############################################################

# In the above setting, size should be modified as real

# memory being locked in sysctl.conf for vm.hugetlb_pool

##############################################################

nodev /mnt/hugepages hugetlbfs rw 0 0

##############################################################

Reboot the machine to boot into Hugemem Kernel and for allocating hugepages to reflect in /proc/meminfo. After the reboot, it should look like this:

$ grep -i huge /proc/meminfo
HugePages_Total: 4608

HugePages_Free: 4608

Hugepagesize: 2048 kB

Note: For dual core CPU Dell machines, it was found that even though Hugepagesize was 2mb, the number pages to be specified in /etc/sysctl.conf had to be doubled to make it worth a particular memory limit, say 9G. You are advised to experiment & verify with the output of previous command on your machine to check if this holds true for you too.

Check that the hugepagefs and tmpfs filesystems have been mounted with the required options:

# mount | grep huge
nodev on /mnt/hugepages type hugetlbfs (rw)

# mount | grep tmpfs

none on /dev/shm type tmpfs (rw,size=9G)

Prepare the Java servlet application for leveraging the LargePage JVM option

As applmgr (or application code tree owner unix user), with environment variable $IAS_ORACLE_HOME set on dedicated configurator tier (Note 343677.1 - Setting Up a Dedicated JServ for Running Only Oracle Configurator),

$ whoami

applmgr

$ vi $IAS_ORACLE_HOME/Apache/Apache/bin/java.sh

# ###############################################################

# This file is automatically generated by AutoConfig. It will be

# read and overwritten. If you were instructed to edit this file,

# or if you are not able to use the settings created by

# AutoConfig, refer to Metalink document 165195.1 for assistance.

# ###############################################################

# Added ulimit -s 8192 to make jdk1.5 work with modified native

# thread stack size ONLY for this servlet. This will keep the

# effect local.

#################################################################

ulimit -s 8192

JSERVJAVA=/usr/java/jdk1.5.0_06/jre/bin/java

#################################################################

ARGV="$*"

JVMLOGDIR=/ORACLE/prod2/9iAS/Apache/Jserv/logs/jvm

STDOUTLOG=$JVMLOGDIR/jserv.stdout

STDERRLOG=$JVMLOGDIR/jserv.stderr

...

#################################################################

In $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.properties, add the JVM switches directive line in the # Begin customizations & # End customizations block at the very end of the file. Remember to put that all these JVM switches in the same line:

# vi $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.properties

...

# Begin Customizations

################################################################

# Added to make JVM1.5 work with Oracle Configurator

################################################################

wrapper.bin=/ORACLE/prod3/9iAS/Apache/Apache/bin/java1.5.sh

################################################################

# the max heapsize specified by -Xmx will have to be changed as

# per the max available RAM On the config machine e.g. if the

# RAM=12g and we are reserving 9g for JVM heaps, then each JVM

# can have a max Heapsize of 2.5g or so:

# (2.7 g + 300g -> for perm generation overhead ) x 3 JVMs ~ 9g

# minus spare

################################################################

wrapper.bin.parameters=-verbose:gc -Xmx2750M -Xms2750m -Xss128k -XX:PermSize=128m -XX:MaxPermSize=256M -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:ParallelGCThreads=4

-XX:+PrintGCTimeStamps -XX:+UseTLAB -XX:+UseLargePages -server

################################################################

# End Customizations

Note: It is important to keep these directives in # Begin & End Customization section so as to preserve these settings across AutoConfig sessions.

You will need to comment out the following 2 lines in the starting section of $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.properties :

#######################

# Execution parameters

#######################

# The Java Virtual Machine interpreter.

# Syntax: wrapper.bin=[filename] (String)

# Note: specify a full path if the interpreter is not visible in

# your path.

# wrapper.bin=/ORACLE/prod2/9iAS/Apache/Apache/bin/java.sh

# Arguments passed to Java interpreter (optional)

# Syntax: wrapper.bin.parameters=[parameters] (String)

# Default: NONE

################################################################

# comment out this line so that the java servlet will at least

# startup

################################################################

# wrapper.bin.parameters=-verbose:gc -Xmx1024M -Xms128M -XX:MaxPermSize=128M -XX:NewRatio=2 -XX:+PrintGCTimeStamps -

XX:+UseTLAB

################################################################

Note: It should be remembered that the default wrapper.bin directive for java.sh and default JVM options will get uncommented after running Autoconfig. So you would have to remember to comment them again after running Autoconfig, or else the Apache servlet will not start.

Why dont we do it this way?

One good question would be: why dont we put the JVM switches in the autoconfig token jvm_options in $CONTEXT_FILE? This would alleviate the need for manually updating the $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.properties each time after AutoConfig. For example.

<jvm_options oa_var="s_jvm_options" osd="Linux">-verbose:gc -Xmx1024M -Xms128M -XX:MaxPermSize=128M -XX:NewRatio=2 -XX:+PrintGCTimeStamps -XX:+UseTLAB</jvm_options>

The answer is simple. Setting the JVM options here will cause EACH servlet setup in the configurator tier to use the same switches. Assume that there are 2 more servlets configured to be running on the same machine (iPayment and OXTA servlets), the total memory taken up the servlets will be 3 X 3g ~ 9G. If you were planning on running 3 confugurator JVMs with LargePage option, all your real memory is gone already.

Hence the manual change step. At the same time, if your IAS_ORACLE_HOME is setup in such a way through the *.properties file that ONLY the configurator servlet is supposed to startup, then this method would work as well. So, this is also a viable option & probably a better one in this situation.

Reap the benefits

Now when you start the confgurator servlet using $SCRIPTS/adapcctl.sh, you can easily verify that the servlet JVM process has 3G of heapsize allocated and that the Garbage collection time spent is considerably reduced after using the parallel GC JVM options used above.

........

45990.174: [ParNew 1640767K->1624634K(2457536K), 0.0106000 secs]

46718.218: [ParNew 1640888K->1624758K(2457536K), 0.0133860 secs]

46913.196: [ParNew 1641014K->1628789K(2457536K), 0.0291830 secs]

........

.........

The advantage of using largepages is that the heap memory will be locked and is not eligible for paging out on swap. The access of objects in heap in real memory is obviously faster.

Hence, using Largepages option is a good option for achieving performance goals, especially when the caching demands in the servlet are considerable and there is a section of RAM on the machine running servlet, which is not being put to any use.

Experiments from the Field..Based on True Stories