Enabling large page support on supported operating environments can give a significant boost to performance. This is especially true for applications with large datasets or running with large heap sizes. Below is a summary of how to enable large pages on Solaris, Windows, and Linux. The text is largely from the "HotSpot VM Options Page ":http://java.sun.com/docs/hotspot/VMOptions.html, but I've had a lot of questions about this and thought it merited highlighting the information here. Stay tuned for a revamped "HotSpot VM Options Page ":http://java.sun.com/docs/hotspot/VMOptions.html coming your way in the next few weeks.
Beginning with Java SE 5.0 there is now a cross-platform flag for requesting large memory pages: -XX:+UseLargePages
(on by default for Solaris, off by default for Windows and Linux). The goal of large page support is to optimize processor Translation-Lookaside Buffers.
A Translation-Lookaside Buffer (TLB) is a page translation cache that holds the most-recently used virtual-to-physical address translations. TLB is a scarce system resource. A TLB miss can be costly as the processor must then read from the hierarchical page table, which may require multiple memory accesses. By using bigger page size, a single TLB entry can represent larger memory range. There will be less pressure on TLB and memory-intensive applications may have better performance.
However please note sometimes using large page memory can negatively affect system performance. For example, when a large mount of memory is pinned by an application, it may create a shortage of regular memory and cause excessive paging in other applications and slow down the entire system. Also please note for a system that has been up for a long time, excessive fragmentation can make it impossible to reserve enough large page memory. When it happens, either the OS or JVM will revert to using regular pages.
Operating system configuration changes to enable large pages:
As of Solaris 9, which includes Multiple Page Size Support (MPSS), no additional configuration is necessary. If you're running 32-bit J2SE versions prior to J2SE 5.0 Update 5 on AMD Opteron hardware additional tuning is necessary. Due to a bug in HotSpot Large page code, the default large page size running the 32-bit x86 binary is 4mb. Since 4mb pages is not supported on Opteron, the large page request fails and the page size defaults to 8k. To get around this, explicitly set the large page size to 2mb with the following flag:
Large page support is included in 2.6 kernel. Some vendors have backported the code to their 2.4 based releases. To check if your system can support large page memory, try the following:
# cat /proc/meminfo | grep Huge
Hugepagesize: 2048 kB
If the output shows the three "Huge" variables then your system can support large page memory, but it needs to be configured. If the command doesn't print out anything, then large page support is not available. To configure the system to use large page memory, one must log in as root, then:
# Increase SHMMAX value. It must be larger than the Java heap size. On a system with 4 GB of physical RAM (or less) the following will make all the memory sharable:
# echo 4294967295 > /proc/sys/kernel/shmmax
# Specify the number of large pages. In the following example 3 GB of a 4 GB system are reserved for large pages (assuming a large page size of 2048k, then 3g = 3 x 1024m = 3072m = 3072 \* 1024k = 3145728k, and 3145728k / 2048k = 1536):
# echo 1536 > /proc/sys/vm/nr_hugepages
Note the /proc values will reset after reboot so you may want to set them in an init script (e.g. rc.local or sysctl.conf). Also, internal testing has shown that root permissions may be necessary to get large page support on various flavors of Linux, most notably Suse SLES 9.
Only Windows Server 2003 supports large page memory. In order to use it, the administrator must first assign additional privilege to the user who will be running the application:
# select Control Panel -> Administrative Tools -> Local Security Policy
# select Local Policies -> User Rights Assignment
# double click "Lock pages in memory", add users and/or groups
# reboot the machine
As always, every application is different and true performance is always defined by each individual running their own application. If you run into problems or have questions about Java performance visit the java.net performance forum
or feel free to send me a comment.