Introduction

Oracle Exadata Database Machine is a widely deployed engineered system focused entirely on being the ideal platform to run Oracle Database. Organizations often deploy multiple Exadata to support a variety of workloads, including AI, mission-critical OLTP, and analytics. Exadata deployments can be enterprise-wide and support different processing and lifecycle stages, such as Dev/Test, UAT, and Production. Customers increasingly use Exadata’s built-in virtualization capabilities to increase database consolidation density and return on investment.

By design, Exadata is a highly available and secure platform built using industry-standard hardware components with redundancy and security at every layer. Exadata System Software – a core component of Exadata – is engineered alongside the hardware. It provides intelligent database-aware capabilities that leverage features such as Smart Scan (SQL Offload) and Exadata RDMA (Remote Direct Memory Access) Data Accelerator to extract and deliver maximum performance. Robust out-of-the-box deployment methodologies including the Oracle Maximum Availability Architecture (MAA) best practices ensure both performance and availability.

As is needed with any hardware or software platform, Oracle makes regular maintenance updates available to ensure that Exadata customers have access to the latest functionality and, just as importantly, the latest security fixes. These updates are necessary to ensure the updated and secure operation of the platform and the Oracle Database.

Oracle Exadata System Software 24ai (24.1) introduces Exadata Live Update, a new capability for performing interruption-free software updates on Exadata database servers and increase Exadata’s availability profile. Using industry-standard utilities such as Ksplice and RPM, Exadata Live Update enables customers to update their Exadata database servers – bare metal, VM Hosts, and VM Guests and, more specifically, kernel/system packages, firmware, user space packages and Exadata packages  – online! The need to apply security updates rapidly is also an increasingly common requirement. Exadata Live Update implements key differentiating functionality to address the selective and online application of updates. Exadata Live Update allows only security updates for customers who need to prioritize security over the latest Exadata innovations. The addition of online database server updates joins the rich availability and security capabilities of Exadata and increases the operational flexibility and agility of Oracle customers. 

Addressing Security Vulnerabilities

Security has always been at the core of everything Oracle does. As Exadata is an engineered system, security requires a holistic approach. This ensures that security updates are applied in a timely manner and that the updates are as unobtrusive as possible to the application – Oracle Database in this case. To ensure that customers have the flexibility and choice to comprehensively apply all updates, including all security updates or just the most critical security fixes, Exadata Live Update uses the well-known and understood Common Vulnerabilities and Exposures (CVE) to group security-related packages into tiers corresponding to criticality. CVEs, as the vulnerability issues are collectively referred to, are publicly disclosed computer security flaws, each with a CVSS score that ranges from 1(Low impact) to 10 (High impact). 

Many organizations have policies to apply critical (CVSS 7 and higher) security fixes and/or all security fixes (CVSS 1 and higher) within a specific period of the available CVE fix. Exadata Live Update enables customers to meet these critical security requirements while meeting or exceeding their availability requirements. 

Exadata Live Update builds on the existing Exadata update process, that uses patchmgr, by introducing several update tiers – all of which can be applied online. This enables the application of packages and other updates (e.g. firmware) aligned to the following tiers:

The diagram below helps visualize the functional scope and relative size (shown at a representative, not exact, scale) of each update tier.  

Importantly, when using Exadata Live Update, you do not need to choose a tier and be restricted to only applying that tier on an ongoing basis. You can easily switch between tiers from one release to the next, allowing you to apply security fixes every month and full updates with less frequency aligned, for example, with regular patching windows and/or DR switchover tests. 

Exadata Live Update is capable of updating database servers (physical hosts of VM guests) running the same major Linux release (e.g., OL7 ->OL7 or OL8 -> OL8). Customers running  Exadata System Software 23.1 (which updated Linux to OL8) or higher can be updated to Exadata System Software 24ai (also on OL8, with a minimum target release of 24.1.x) using Exadata Live Update without server downtime. The Exadata Live Update feature works on all Exadata System Software 24ai supported platforms (X6 and higher and running 23.1 or higher) as long as the update is within the same major OS release. 

Exadata Live Update is prechecked and executed using the Exadata patchmgr utility, and the command takes the following syntax.

Outstanding Work

Some content cannot be applied live, or requires a server reboot to apply. For example, server firmware cannot be applied while the server is running. Exadata Live Update is intelligent and classifies such packages as ‘outstanding’ work, which is then staged for application at a later time. By default, staged outstanding work is applied during the next graceful server reboot. 

While outstanding work is applied at the next reboot as a default behavior,  it is also possible to specify a reboot window. This enables graceful reboots to be performed as needed without any staged outstanding work being applied. When you are ready to apply the outstanding work, adjust the reboot window or reset to the default behavior. The customized scheduling is enabled using the live-update-schedule-outstanding-work option with the patchmgr command. Outstanding work may also be deferred by choosing the ‘never‘ option. 

The Oracle MAA best practice recommendation is to schedule outstanding work at least once every three months.

Finally, use the ‘reset‘ option to restore the default behavior.

Application examples

Exadata Live Update transparently uses Ksplice uptrack to apply updates to the live Linux kernel and related critical packages. At the same time, Exadata Live Update stages the related updates to the on-disk kernel to ensure that when the server is rebooted and the related outstanding work is applied, the latest kernel and system packages are used, ensuring critical security fixes are always available and active.

Finally, Exadata System Software includes the Java JDK. Over the years, significant improvements have been made to the way Java is updated, and this continues to be a focus area for the Java team. Today, updates to the included JDK are staged as outstanding work by Exadata Live Update.

Exadata Live Update Runtime

Exadata Live Update is performed in a rolling manner where updates are applied one server at a time. The diagrams below show a representative progression sequence of a rolling update in a two server cluster.

Application to Exascale Direct Volumes (EDV)

With the advent of Exascale with Exadata System Software 24ai, a series of new components was introduced. Of these, and as part of the decoupling of storage management from the database server to the storage server, a new system-level component was introduced – Exascale Direct Volumes (EDV).  Analogous to ASM Dynamic Volumes Manager (ADVM), EDV allows Exascale Volumes to be used as a generic POSIX block device.

This storage component is, in turn, enabled by a new kernel-level component which allows the Linux OS block layer, file system layer, and application layer to take advantage of the smart storage of Exadata storage servers.  A representation of the functional flow between the block device and Exadata Storage Servers is shown in the diagram. Because a kernel-level component enables this functionality, this component has been designed with high availability and supportability, ensuring that updates can be done to this component while maintaining system uptime.

With the current integration, EDV is patched in both the Online and the Outstanding Work phases of Exadata Live Update.  The 2 phases implement different aspects of EDV patching with different end goals.

Types of EDV Patches

In a single EDV patch set for Exadata, there are 2 sets of updates. The command “edvutil lsinitiator” can be used to view the details. The screen capture below shows the Base and Online version information for a recent release. In the screen capture:

• Base EDV Version – this version of EDV is installed when outstanding work is completed upon a reboot of a node.  This Base EDV version is similar to the base Linux kernel version running on the system.

• Online Patch EDV Version – this version is the Base EDV Version + the provided Online EDV patches.  This is equivalent to the Linux effective kernel version for Ksplice.

Update Type I : Online Patching of EDV

The Online patching version of Exascale Live Update will patch EDV when used with the ‘full’ option.  Other options, such as highcvss or allcvss do not patch EDV.  Online patching of EDV utilizes Oracle Ksplice technology, similar to Oracle Linux Kernel patches in online mode.  This method of patching allows EDV volumes to continue to be used, while delivering critical fixes for EDV volumes.  This method can be utilized across monthly updates.

When utilizing only online patching with EDV, the Base version of EDV will remain at the original install version.  Only the Online EDV version is increased. This is pictorially shown in the diagram.

Viewing the applied EDV Patches

Once the EDV online updates are installed, a new RPM is placed in the system RPM registry.

To view what online updates have been installed, a query of the installed RPM can be used.

A sample query output is captured and is shown.

Update Type II: Outstanding-Work patching of EDV

To install the full set of EDV fixes, including new functionality and other non-critical updates, the EDV install is updated when the Exadata Live Update ‘full’ option is invoked and subsequently ensures that all outstanding work is completed, via graceful reboot. When this happens, new kernels and EDV modules are installed.  The reboot installs a new Base EDV version and updates the exadata-oracleedv RPM installed in the system RPM registry. 

At this point, the EDV Base Version will match the new Exadata Release Version. 

Server backup before using Exadata Live Update

The patchmgr utility automatically takes a backup of the active system partition of a Database server or VM on to an inactive logical volume. If a rollback is required in the most unlikely of scenarios, patchmgr –rollback provides the ability to rollback to the previous backup reducing the time needed to return the server to a known and working state.

Use of Oracle Unbreakable Linux Network (ULN) 

The Exadata team provides regular releases focused on new functionality and innovations, as well as maintenance releases that include ongoing fixes and security updates. As part of our release strategy, we publish these releases on the Oracle Unbreakable Linux Network (ULN) as channels that customers can use directly or, more commonly, via ULN mirrors.

For those unfamiliar with ULN and its capabilities, an excellent blog on ULN and Exadata Database Servers that addresses channels, access, usage etc. makes good reading.

As Exadata is an engineered system, the packages included in the channels and installed on the database servers are curated, tested, and certified with both the Exadata hardware and Oracle Database. Exadata Live Update can use these channels—directly from ULN or using a ULN mirror—to simplify access to these regular updates and reduce (or potentially eliminate) the need to copy updates across multiple database servers. This not only reduces network utilization but also reduces the time taken to apply updates. 

Conclusion

Exadata Live Update, an excellent new capability of Exadata System Software 24ai (24.1), is a powerful utility that is designed to simplify Oracle Exadata database server maintenance and update efficiency. It ensures that kernel/system packages, user space package updates and Exadata packages can be applied online with zero impact on the availability and performance of critical applications. Exadata Live Update enables organizations to remain secure with the capability to apply critical security updates and fixes quickly. When CVEs are published with Exadata Live Update, it becomes possible to apply the security patches to the database servers while keeping mission-critical workloads available. Exadata Live Update significantly improves the operational experience for Exadata administrators. It enables mission-critical databases to run continuously while updates are applied online, helping secure business operations and continuity.