We covered huge pages and their benefits in part 1 of the blog series – “Huge Pages: In the context of Exadata.” This blog will discuss static huge pages and transparent huge pages, a technique to allocate huge pages dynamically. Databases, especially Oracle, when run on Linux x86-64-based systems, provide excellent performance when data is resident in memory and not fetched from the underlying storage. Huge pages are pre-allocated to configure large memory for the Oracle database before a database instance is started. Huge Pages have played a vital performance optimization role when running databases. Frequently used data is kept in memory, especially for mission-critical transactional databases, to provide the lowest read latency, a critical function for optimal database performance. A large chunk of memory known as the System Global Area (SGA) – which includes the database buffer cache – is allocated to database workloads. This blog will explain the scenarios of huge pages or transparent huge pages being beneficial in delivering extreme performance.
What are Huge Pages?
System memory is organized in blocks known as pages. A page in Linux x86-based systems is 4K in size (4096 bytes), with total memory stretching into terabytes. A system with 1 gigabyte of memory has 256,000 pages using a 4K page size. The CPU’s memory management unit (MMU) manages all these pages. Consider, for example, a system with 1 TB (terabyte) of memory using the standard 4K (4096 bytes) page size will have to manage more than 268 million pages! The overhead for the MMU to manage that many pages is the potential for degraded performance.
Huge Pages is a feature of the Oracle Linux kernel that allows the operating system to use memory pages that are larger than the default (4K) page size. To optimize memory allocation, memory-intensive applications like Oracle database can use Huge Pages before starting a database instance. Oracle recommends using a Huge Page size of 2MB (yes there is a 1G large page size as well) for allocating memory to the Oracle System Global Area. For optimal performance, the aggregate allocation of all Oracle database SGAs (the SGA size for all database instances on the server) must reside inside the huge pages. We talked about configuring huge pages in the prior blog on the topic.
What are Transparent Huge Pages?
Transparent Huge Pages, or THP, is a layer within the Linux kernel that, when enabled, can allocate huge pages dynamically to applications that request them using a special system call. Transparent Huge Pages memory differs from standard Huge Pages memory because the khugepaged kernel thread allocates huge pages dynamically. Standard or static huge pages memory is pre-allocated at instance startup and does not change during runtime. Database instances, when shut down, do not release this block of memory to the available memory pool.
Some application binaries (also known as executables) – like the Oracle executable – can benefit from being cached using huge pages and can request the allocation of additional huge pages during run time. The allocation of transparent huge pages during run time provides the necessary huge pages for such applications. In the context of large applications, like Oracle executables running on the server, there may be a scenario where these executables can benefit from large page sizes rather than managing thousands of 4K pages. THP, again with a 2 MB page size, dynamically allocates huge pages at run time, which improves the execution of the application.
We allocated standard huge pages, so why configure transparent huge pages?
There are cases when larger applications, other than the database instances, can benefit from huge pages. Pre-allocating or reserving memory for many huge pages (for both applications and databases) will reduce the available memory for other processes running on the system. When a large application needs a chunk of memory, we can allocate huge pages during run time and these huge pages are referred to as transparent huge pages.
Databases make use of static huge pages to ensure that there is minimal swapping and that the most frequently accessed data is resident in memory. We do not want to allocate more than the required number of huge pages as the configuration of static huge pages pre-allocates memory and is not released (remains allocated as huge pages) when the database instance is shut down. Some applications can also make use of larger page sizes to eliminate swapping by dynamically requesting huge pages. These pages are released when the application does not need them. Oracle, with its large .text region in the executable that contains the compiled instructions of the Oracle database engine, is such an application. This region is read-only and heavily accessed during execution as it holds the core logic for query processing, memory management, and I/O operations. With a large .text region, the operating system must allocate thousands of standard 4K pages and must be managed by the operating system. Large .text regions are accessed frequently as Oracle executes SQL statements, manages transactions, and handles concurrency. Using transparent huge pages ensures that allocations span fewer page boundaries, reducing page table lookups and improving memory utilization.
How does application request transparent huge pages?
What if some applications require huge pages at run time? This is where ‘madvise’ parameter of transparent_hugepage helps us. When “madvise” is configured for transparent huge pages, and applications make a madvise system call to request huge pages, then the huge pages are allocated. Applications requiring huge pages at run time must explicitly use a madvise system call to inform the kernel about their memory needs and the respective allocations of huge pages to optimize the application performance. Other applications not configured to make madvise call will not be able to use transparent huge pages.
Beginning with the 23ai Oracle, Oracle recommends turning on transparent huge pages so that large applications like Oracle executable can benefit from transparent huge pages. You must still allocate static huge pages so that aggregate SGA size configured for all database instances can 100% fit their memory requirements inside of huge pages. We recommend Oracle database SGA to be allocated via static huge pages that are pre-allocated for that purpose.
A tale of two recommendations
For databases prior to Oracle Database 23ai, Oracle recommends configuring and using static huge pages. Beginning with Oracle Database 23ai, Oracle recommends enabling transparent huge pages along with static huge pages to allow databases to take advantage of the improvements in the latest UEK. The good news is that on Exadata X8M and later, with Exadata System Software 25.1 and newer, UEK7 and transparent huge pages are enabled out-of-the-box during deployment! If you’re upgrading from an earlier release and want to check if transparent huge pages are setup, see below for an easy way to check. But first, let’s discuss what to do when both Oracle Database 19c and 23ai are present on the same database server.
What if I run the 19c database and the 23ai database on the same server
Transparent huge page memory is enabled by default on Oracle Linux and Exadata. The ‘madvise’ parameter used in the kernel command enables transparent huge pages and assigns huge pages to memory regions explicitly requested by applications using the madvise system call. In the case that Oracle Database 19c and 23ai are running on the same server, Oracle recommends setting transparent huge pages to ‘madvise’.
Checking your system is configured with transparent huge pages
To check if your Exadata database server is configured with transparent huge pages, you need to check the grub bootloader options and the current memory management settings.
Checking grub is as simple as using ‘grep -i hugepage /boot/grub2/grub.cfg’ and checking the output is as follows:
GRUB_CMDLINE_LINUX=”crashkernel=auto rhgb quiet numa=off transparent_hugepage=madvise“
The current memory management settings can be determined using :
‘cat /sys/kernel/mm/transparent_hugepage/enabled’
The output should look like this – the square braces around madvise indicate that this is the active setting.
always [madvise] never
Conclusion
Huge pages in an Exadata environment utilize the memory of the system efficiently. When running Oracle databases, for optimal performance, make sure that the entire SGA (the sum of SGA of all database instances on the server) resides on static huge pages. This will ensure that the Oracle database is not starving for memory and fragmentation does not occur under heavy load. When large applications like the Oracle executable are run, the large Oracle .text region will explicitly use madvise parameter of transparent huge pages and allocate large pages during run time. It will ensure the application is run in the most efficient manner. We continue to allocate static huge pages for optimal database memory utilization and transparent huge pages to optimize the running of large applications like the Oracle executable.