Oracle Database 21c - Persistent Memory Database – Eliminate Latency with Directly Mapped Buffer Cache

Oracle Database 21c Persistent Memory Database maps the database buffer cache directly onto Persistent Memory, which means the buffer cache and storage become one in the same. The majority of input/output operations simply disappear because the database directly uses data that resides in storage and operates on that data in the same manner it uses data in memory in the database buffer cache. The Directly Mapped Buffer Cache is the defining feature of Oracle Database 21c Persistent Memory Database and embodies the true promise of Persistent Memory.

Oracle Database 19c also supports Persistent Memory as outlined in our previous blog (Oracle Database 19c Running @Memory Speed) but in a quite different manner. Oracle Database 19c allows database files to reside on Persistent Memory, which gives the benefit of dramatically improved I/O performance. Oracle Database 21c eliminates I/O by mapping the buffer cache into the Persistent Memory contents, allowing the database to bypass the I/O layer entirely.  Of course, these features are also quite different from Exadata, which uses Persistent Memory in the storage-side. Both Oracle Database 19c (using OMS) and Oracle Database 21c Persistent Memory Database use server-side Persistent Memory.

In this blog, we will go through the process of setting up PMEM devices and creating an Oracle database on Persistent Memory. We will explore how Oracle databases can be created from scratch (using the CREATE DATABASE command) or using RMAN ACTIVE DUPLICATE command to bring an active database from another server.

What is Persistent Memory

Persistent Memory is a solid-state, high-performance, byte-addressable technology (like DRAM) that resides on the memory bus, giving PMEM the access and speed characteristics of DRAM. Memory (or DRAM) is ephemeral, and storage (such as Disk or Flash) has traditionally been required for persistent copies of data. For any computer application to work, the application data stored in HDD/SSD must be loaded in memory, introducing latency into the process.

Persistent Memory, on the other hand serves a dual purpose. Persistent Memory is a storage device that sits on the memory bus and can be used for memory expansion or adding storage to a server.

With the advancements in infrastructure technology (compute, storage, memory, networking etc.), and fast running database systems, there has always been a struggle to optimize database workloads and improve performance. Increasing data volumes have taken a big toll on bringing the data (IO system) to the compute for processing, and disks or flash (persistent storage) performance have been a bottleneck. Although we have seen huge advancement in storage technologies in the hardware itself (mechanical to flash to NVME) and clever technologies on top of them (cache layer built on top of storage devices), IO has always been a bigger bottleneck.

In the later part of the blog, we will show how to set up an existing Oracle database on a Persistent Memory store. Let’s first look at the memory storage hierarchy.

Memory Storage Hierarchy

In the below diagram, you can see where Persistent Memory sits in the memory and storage hierarchy and the latency to bring data to the compute.

How does Oracle utilize Persistent Memory?

Persistent Memory with Oracle databases can be utilized in three different ways.

1. Persistent Memory Database using Oracle Memory Speed (OMS). This feature is available starting Oracle Database 19c. This feature was covered in the blog here.

2. Persistent Memory Database with Directly Mapped Buffer Cache (available from Oracle Database 21c onwards) – Accelerates DBMS operations, I/O is done via memory copy and/or direct access. Data is accessed directly from PMEM as an integral part of the database buffer cache. Persistent Memory File-Store is tightly coupled with the database. (we will talk about the capability in this blog)

3. Storage side (Exadata) – Combined with RDMA (Remote Direct Memory Access), Oracle database access data cached in PMEM on the storage side of Exadata. Read more about Persistent Memory magic in Exadata.

In this blog, we will focus on the Directly Mapped Buffer Cache feature which fully utilizes the potential of persistent memory. So, what is the Directly Mapped Buffer Cache? Let us find out.

Introduction to Directly Mapped Buffer Cache

Typically, when data is read storage, it is necessary to copy data blocks into DRAM (Dynamic Random-Access Memory) regardless of whether the underlying storage device is a Hard Disk or Flash Storage or even Persistent Memory (as we have seen with Oracle Database 19c using OMS). Using the Directly Mapped Buffer Cache feature of Oracle Database 21c, Oracle Database accessed data directly from the PMEM device, rather than copying the data into buffer cache (DRAM), eliminating the majority of I/O operations entirely. With Directly Mapped Buffer Cache, database files on PMEM become part of the database buffer cache as we will see in the set up.

Avoiding Fractured Blocks (aka torn writes) and Data loss with Persistent Memory

Persistent Memory, like DRAM, is byte addressable rather than block addressable as with conventional disk or Flash storage. With PMEM, data is persisted 8-bytes at a time rather than in increments of 4,096 bytes with the typical O/S pages or storage sector sizes. If an Oracle Database writes a block of data (8192 bytes = 8K), the data would be persisted into PMEM in 1024 chunks (1024 chunks of 8 bytes each). Any disruption (power failure for example) during the time of this write, can leave data “fractured” or “torn” with inconsistent data in some of those 1024 chunks (old and new data). Both the 19c (OMS) and Directly Mapped Buffer Cache (21c) solutions eliminates this “fractured” block problem. Oracle accomplishes this internally by doing an out of place copy with metadata update to reference the new block only after the entire block is written successfully written.

While database blocks are accessed directly in PMEM as part of the Directly Mapped Buffer Cache, Oracle copies blocks into DRAM when they are modified. All changed blocks are written by the Database Writer (DBWR) process, including retaining copies of prior block versions in rollback segments for read consistency purposes. In any well-tuned system, writing of blocks is done asynchronously from the transaction and is not latency sensitive. Of course, writing of REDO blocks is latency sensitive, so REDO is written directly to PMEM in Oracle Database 21c Persistent Memory Database. Oracle Database 21c includes logic to prevent fracturing (or tearing) of REDO blocks to guard against corruption of REDO as well.

Let us get into details of how to configure Oracle Database with Persistent Memory using Directly Mapped Buffer Cache. We will start with basic requirements including, software versions and then move on to create PMEM File store followed by creating an Oracle database.

Requirements and guidelines for Directly Mapped Buffer Cache

Requirements:

• • An Intel Cascade Lake or Ice Lake based server or later.
  • Persistent Memory devices (PMEM) in each socket capable of memory interleaving.
  • Oracle Linux with Huge-Page support.
  • A DAX capable (DAX) file system such as XFS.
  • Oracle version 21.3 with patch – 33516412 (DBRU 21.5.0.0.0)

bash-4.2$ $ORACLE_HOME/OPatch/opatch lspatches

33907983;MERGE ON DATABASE RU 21.5.0.0.0 OF 33339444 33486067

33516412;Database Release Update: 21.5.0.0.220118 (33516412)

Persistent Memory Filestore:

Oracle Database 21c Persistent Memory Database includes the integrated Persistent Memory Filestore to store all files.

Guidelines:

                Directly Mapped Buffer Cache feature is only supported for a single instance Oracle database deployment, not with Oracle Real Application Clusters. In addition, each PMEM Filestore can only contain Oracle datafiles from one Oracle Database. Multiple PMEM Filestores are required to deploy multiple databases on a single server.

Let us start setting up:

1. Setup environment variables (tst21pm is our target instance on PMEM Filestore)

export ORACLE_HOME=/u01/app/oracle/product/21.5

export ORACLE_BASE=/u01/app/oracle

export PATH=$ORACLE_HOME/bin:$PATH

export PATH=$ORACLE_HOME/perl/bin: $PATH

export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH

export TWO_TASK=

export ORACLE_SID=tst21pm

2. The following command lists some of the XFS file systems created using PMEM devices

df -hT

/dev/pmem0p1                 xfs       1.5T  1.4T   57G  97% /mnt/pmem12

/dev/pmem1p1                 xfs       1.5T  1.4T   57G  97% /mnt/pmem13

3. Create sample init.ora (initialization file) to start up the instance

*.audit_file_dest='/u01/app/oracle/admin/tst21pm/adump'

*.audit_trail='db'

*.compatible='21.0.0'

*.control_files='/u01/app/oracle/oradata/TST21PM/control01.ctl','/u01/app/oracle/oradata/TST21PM/control02.ctl'

*.db_block_size=8192

*.db_file_name_convert='/u01/app/oracle/oradata/TSTLOCAL’, ‘/u01/app/oracle/oradata/TST21PM'

*.log_file_name_convert=’ /u01/app/oracle/oradata/TSTLOCAL’, '/u01/app/oracle/oradata/onlinelog/TST21PM', ‘/u01/app/oracle/oradata/onlinelog/TSTLOCAL’, ‘/u01/app/oracle/oradata/onlinelog/TST21PM’

*.db_name='tst21pm'

*.db_recovery_file_dest='/u01/app/oracle/oradata/fast_recovery_area'

*.db_recovery_file_dest_size=10g

*.diagnostic_dest='/u01/app/oracle'

*.dispatchers='(PROTOCOL=TCP) (SERVICE=tst1smplXDB)'

*.enable_pluggable_database=true

*.local_listener='(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=<Host_Name>)(PORT=<Port_Number>)))'

*.log_archive_format='%t_%s_%r.dbf'

*.nls_language='AMERICAN'

*.nls_territory='AMERICA'

*.open_cursors=300

*.pga_aggregate_target=1g

*.processes=8960

*.remote_login_passwordfile='EXCLUSIVE'

*.sga_target=8g

*.undo_tablespace='UNDOTBS1'

4. Make sure /pmemfs/TST21PM directory exists, and user oracle has proper permissions to it.
5. Edit /etc/fuser.conf file to add (or uncomment if one already exists)

‘user_allow_other’ option

$ cat /etc/fuser.conf

user_allow_other

6. Linux userid that owns Oracle database server binary (located at $ORACLE_HOME/bin/oracle) must have execute permissions for /bin/fusermount program. This can be accomplished in different ways, such as by changing file permissions, adding this user to the Access Control List associated with /bin/fusermount, etc.

Please note that /bin/fusermount program has setuid bit set, and it is typically owned by root.

$ ls -l /bin/fusermount

-rwsr-x---. 1 root root 32584 Sep  6  2016 /bin/fusermount

$ chmod o+rx /bin/fusermount

$ ls -l /bin/fusermount

-rwsr-xr-x. 1 root root 32584 Sep  6  2016 /bin/fusermount

7. Create SPFILE

Create a spfile (from the initial init file we created in step 3). The reason for that is when "CREATE PMEM FILESTORE" command is run, spfile will record the mountpoint

for the PMEM device for subsequent startup of the instance. In case you chose to use init file instead of spfile, you will have to manually specify the parameter pmem_filestore and its mapping.

SQL> CREATE spfile=’/u01/app/oracle/dbs/spfiletst21pm.ora’ from pfile=’/u01/app/oracle/dbs/inittst21pm.ora’;

8. SQL > startup nomount

SQL > create pmem filestore TST21PM MOUNTPOINT '/u01/app/oracle/oradata/TST21PM' BACKINGFILE '/pmemfs/TST21PM/tst21pm.bf' SIZE 80G;

PMEM filestore created.

In alert log you will see following messages:

PDBID-1 OFS:TST21PM of type fsdirect CREATED

PDBID-1 OFS:TST21PM MOUNTED at /u01/app/oracle/oradata/TST21PM/ w/type fsdirect

2022-03-04T19:30:16.545076-08:00

PDBID-1 OFS:/u01/app/oracle/oradata/TST21PM/ UNMOUNTED

PDBID-1 OFS:TST21PM of type fsdirect DESTROYED

2022-03-04T19:30:16.624065-08:00

Creating PMEM Filestore with the following parameters:

  Backing Storage    :/pmemfs/TST21PM/tst21pm.bf

  Filestore Name     :TST21PM

  Block Size         :8192

  Filestore Size     :85899345920

  Auto Extend Size   :0

  Max Size           :85899345920

  InternalVsize      :167503724544

  InternalRootSz     :10485760

2022-03-04T19:35:28.440379-08:00

PMEM Filestore is mounted with the following parameters:

  Backing Storage    :/pmemfs/TST21PM/tst21pm.bf

  Filestore Name     :TST21PM

  Block Size         :8192

  Filestore Size     :86769664000

  Auto Extend Size   :0

  Max Size           :86769664000

  InternalVsize      :167503724544

  InternalRootSz     :10485760

2022-03-04T19:35:28.442010-08:00

PDBID-1 OFS:TST21PM of type fsdirect CREATED

PDBID-1 OFS:TST21PM MOUNTED at /u01/app/oracle/oradata/TST21PM/ w/type fsdirect

2022-03-04T19:35:28.593751-08:00

Completed: create pmem filestore TST21PM MOUNTPOINT '/u01/app/oracle/oradata/TST21PM' BACKINGFILE '/pmemfs/TST21PM/tst21pm.bf' SIZE 80G

spfile is updated with the following parameter

SQL> show parameter pfile

NAME                             TYPE        VALUE

------------------------------------ ----------- -------------------------------------------------

spfile                               string      /u01/app/oracle/dbs/spfiletst21pm.ora

SQL> show parameter pmem

NAME                        TYPE        VALUE

------------------------------------ ----------- --------------------------------------------------

pmem_filestore      string      /u01/app/oracle/oradata/TST21PM, /pmemfs/TST21PM/tst21pm.bf

9. Make sure you can connect to source database, and to the PMEM DB (that we are going to create) using the same SYS password. You may want to create static listener for the PMEM DB so that you can connect to it using SYS in nomount mode.
10. I will use RMAN ACTIVE DUPLICATE command to clone the source database to the PMEM DB.

I have a script that I will use to run for the DUPLICATE command. tstlocal is the source database running on a different host. tst21pm is the target database that will be a PMEM database.

connect target sys/<password>@tstlocal;

connect auxiliary sys/<password>@tst21pm;

RUN {

ALLOCATE CHANNEL d1 TYPE DISK;

ALLOCATE CHANNEL d2 TYPE DISK;

ALLOCATE CHANNEL d3 TYPE DISK;

ALLOCATE CHANNEL d4 TYPE DISK;

ALLOCATE AUXILIARY CHANNEL aux1 TYPE DISK;

ALLOCATE AUXILIARY CHANNEL aux2 TYPE DISK;

ALLOCATE AUXILIARY CHANNEL aux3 TYPE DISK;

ALLOCATE AUXILIARY CHANNEL aux4 TYPE DISK;

DUPLICATE TARGET DATABASE TO TST21PM  FROM ACTIVE DATABASE                NOFILENAMECHECK;

RELEASE CHANNEL d1;

RELEASE CHANNEL d2;

RELEASE CHANNEL d3;

RELEASE CHANNEL d4;

RELEASE CHANNEL aux1;

RELEASE CHANNEL aux2;

RELEASE CHANNEL aux3;

RELEASE CHANNEL aux4;

}

11. Once the database is cloned, let us validate the data files and their destination file system.

To validate datafiles are located on the right mount point, query the name and size of the datafiles.

Resources and Further Reading

Oracle Database 19c Running @Memory Speed

Persistent Memory Primer – By Chris Craft

Conclusion:

                In this blog, we have seen how we can clone an Oracle database running on a different server to PMEM Filestore. From the memory-storage hierarchy diagram earlier in the blog, we can see that PMEM devices are about ~3000x faster than SSDs. We have significantly reduced the IO latency when our database run on an PMEM device. With Directly Mapped Buffer Cache, the data is read directly from the PMEM Filestore, rather than buffer cache as we see in traditional stores or even in OMS solution of PMEM. As we see in Memory-Storage hierarchy, DRAM is still more efficient compared to PMEM, this solution also uses intelligent caching policies where, frequently accessed blocks are still cached in DRAM. Both the OMS and Directly Mapped Buffer Cache solution of PMEM database ensures that updates to Oracle data blocks are always atomic and free from fractured block issues. You may wonder about the capability of key features of Oracle database - Scalability and Availability. PMEM databases can scale horizontally using Oracle database Sharding capability. Availability of databases can be achieved by using local Data Guard or remote site data guard set up.