Introduction to Exadata Architecture

Oracle Exadata Database Machine is an engineered system that runs Oracle Database in a high performance, cost effective, and scalable manner than can be installed on-premise or in a cloud like Oracle OCI. Exadata deployments are comprised of two physical server types, database servers and storage servers, that are interconnected by a network supporting Remote Direct Memory Access (RDMA). In the latest Exadata systems, the network implementation is RDMA over Converged Ethernet (RoCE), and the network adapters are Nvidia ConnectX-7s. Virtual machines (VMs) running on the database servers constantly communicate with one another and with the storage servers via RDMA. This is great from a performance and availability point of view, but it has made it difficult to implement a frequently requested feature — the ability to live migrate a VM from one database server to another, which is helpful from load balancing and maintenance points of view. The latest Exadata release, 26.1, adds support for live migrating database VMs, which required significant development work on QEMU, the virtual machine manager used by Exadata. This blog post discusses the challenges associated with live migrating Exadata VMs, along with a high level view of the QEMU enhancements required to enable the feature.

QEMU Live Migration

Live migration is a mechanism that moves a VM from one physical server to another while it’s still running, ideally without the guest detecting that migration has occurred. The state information that’s transferred from the source to the destination includes both persistent and runtime state. Persistent state includes things like the CPU type and count, the system bus layout and the types of devices connected to it, the contents of NVRAM, and the connected block storage devices. Runtime state includes things like the contents of memory, the contents of CPU registers, the internal state of connected devices, and the state of the networking stack. VMs that use passthrough devices like a network adapter’s SR-IOV virtual function (VF) will also have runtime state associated with those devices; Exadata’s database VMs rely on ConnectX-7 VFs passed through to implement RDMA functionality.

QEMU’s live migration process can be split into five phases: 1) initialization, 2) pre-copy, 3) stop-and-copy, 4) resume, and 5) cleanup. During the initialization phase, the destination QEMU instance is started and persistent state information is transferred from the source to configure it. During pre-copy, the source QEMU instance iteratively copies memory to the destination while the source VM is running. All of the source’s memory pages are transferred to the destination at least once, and the source QEMU instance tracks the pages dirtied by the VM and copies them to the destination again as necessary; it’s important that the mechanisms the source QEMU instance uses to detect dirty pages are accurate and efficient. During the stop-and-copy phase, the source VM is stopped and all remaining runtime state information is copied to the destination, including any remaining dirty pages. During the resume phase, the destination VM is started, and during the cleanup phase the source VM is shut down. The downtime observed by the VM coincides with the stop-and-copy phase, and QEMU uses the iterative pre-copy phase to minimize that downtime — the pre-copy phase continues until QEMU believes it can hit a target downtime given the amount of runtime state information that has to be transferred to the destination during the stop-and-copy phase.

multifd Option

multifd is a QEMU live migration option that uses multiple threads to scan memory pages and transfer dirty pages from the source to the destination. When this option is enabled, QEMU’s main live migration thread drives the process, scanning for dirty pages and handing off work to multifd threads. The number of multifd threads used is configurable, but picking a value is not straightforward — the target downtime, the amount of VM memory, the dirty page rate, and the number of CPUs available to run multifd threads are all factors. It’s also possible for the main live migration thread to be so busy that it cannot parcel work out to all of the configured multifd threads. QEMU’s multifd threads include some performance optimizations, such as detecting that a page has been written completely with zeros, in which case the thread writes a message to the destination saying the page is zero instead of writing a page full of actual zeros. Exadata systems have an additional optimization that takes advantage of 2MB huge pages, handing them off to multifd threads instead of 4KB pages; this reduces the number of synchronization operations between the main thread and the multifd threads and exploits memory locality.

Challenges Associated with Downtime

Live migration downtime is a particularly challenging problem for Exadata database VMs because they’re constantly communicating among themselves via RDMA, and the entire system is managed by Real Applications Clusters (RAC) software. If during live migration a VM is unresponsive for too long, a communication protocol timeout can occur and/or RAC may decide the VM has died and initiate cluster recovery procedures. This is particularly difficult to manage with respect to RDMA VFs, which can have 20 megabytes or more of runtime state information that needs to be transferred during the stop-and-copy phase of QEMU’s live migration process. QEMU has very little control over how long it takes to read the runtime state information from the source VM’s VF, nor how long it takes to write that state information into the destination VM’s VF.

The shortest timeouts in the system are associated with RDMA operations, and we handle that case by taking advantage of the fact that the Exadata system knows which VMs are communicating with one another. When a VM that’s being live migrated enters a state where it can no longer process RDMA operations, we use Linux’s FWCTL interface to communicate with the Nvidia firmware on all of the network adapters affected by the migration, repeatedly issuing the MLX5_CMD_OP_POSTPONE_CONNECTED_QP_TIMEOUT command to postpone queue pair timeouts that would otherwise signal a communication failure. In the worst case, the live migration operation will be cancelled and the source VM will be resumed, together with its communication channels.

Parallel VFIO State Transfer

One thing QEMU can do about the large amount of runtime state associated with RDMA VFs is to transfer it from the source to the destination in parallel, a feature that was added to QEMU 10.0 by Oracle engineer Maciej Szmigiero. Prior to this change, QEMU transferred VFIO device state serially on its main migration thread, one device at a time, which is a slow process for Exadata VMs with four RDMA VFs. After this change, the source QEMU process creates a thread for each VFIO device at stop-and-copy time. That thread reads device state from the kernel, splits it into chunks, wraps the chunks in indexed packets, and queues the packets for transmission via multifd threads; the main migration thread only carries packet ordering information. The destination QEMU process reassembles the packets in order and a thread dedicated to each VFIO device writes the result to the adapter. The last packet contains adapter configuration information that requires special handling because the process of loading it into a destination adapter requires calling QEMU functions that are not fully asynchronous, but otherwise the transfer process is highly parallelized. We tested VMs with four ConnectX-7 RDMA VFs and roughly 20 megabytes of runtime state information per VF, and using parallel state transfer reduced the VF portion of stop-and-copy time from 1783 ms to 1068 ms.

Dirty Page Tracking

The performance of dirty page tracking during QEMU’s live migration pre-copy phase is extremely important, and it falls into two categories: tracking writes initiated by software running on CPUs, and writes initiated via RDMA.

Tracking CPU Writes

KVM’s traditional method of tracking memory writes relies on write protecting secondary page table entries — when dirty logging is enabled, the memory pages associated with the VM are write protected. When CPUs write that memory, it triggers a page fault. KVM marks the page as dirty, removes the write protection, allows the write to proceed, and re-enables write protection. This mechanism provides accurate information about dirty 4KB pages, but the page faults and frequent page table walks have a significant negative impact on performance. The kernel uses a reverse mapping (rmap) data structure to help mitigate this — it maps a physical page to every page table entry that maps it, so when a page fault is taken it’s straightforward to find the physical page, mark it as dirty in a bitmap, and update all of the page table entries referencing it as needed. Later, QEMU can retrieve the dirty bitmap from KVM and use it to transfer the dirty pages during live migration.

Later improvements to the Linux kernel, especially the two-dimensional paging (TDP) MMU, eased performance issues in this area by enabling multiple CPUs to process page faults concurrently. In our testing, combining the TDP MMU with features like eager page splitting improved performance significantly, but when we ran VMs with workloads that generated heavy write traffic, we topped out at a per-CPU dirty page tracking rate of 11GB/s, which didn’t meet our requirements.

Modern AMD processors provide hardware Accessed (A) and Dirty (D) bits in Nested Page Table (NPT) entries to improve dirty page tracking performance. On these processors, instead of trapping writes, KVM periodically scans page table entries, collects dirty-bit state, clears the bits, and updates its dirty page bitmaps. This changes dirty tracking from a fault-driven mechanism into an exercise in dirty-bit collection. The VM’s memory write path is unimpeded, live migration overhead is reduced, and scalability is improved on systems with large amounts of memory.

Another performance improvement derives from tracking 2MB pages instead of 4KB pages. A single 2MB NPT entry in an AMD processor represents 512 individual 4KB pages, so there are far fewer page table entries for KVM to scan. There’s a cost, however — writing a single word to a 2MB page marks the entire page as dirty, and if you take no other action, QEMU is stuck sending an entire 2MB page from the source to the destination as 512 4KB pages (QEMU’s live migration bitmaps operate at 4KB granularity). The solution developed for Exadata 26.1 uses hashing to identify the dirty 4KB pages within a 2MB page, which will be detailed below.

Tracking RDMA Writes

Some DMA capable hardware supports reporting the pages it has written via the kernel’s VFIO dirty page logging interface, and that’s the case for the Nvidia ConnectX-7 adapters used in Exadata systems capable of live migration. QEMU uses the kernel interface to start VFIO dirty page logging, then periodically retrieves a bitmap representing dirty pages within a range of memory available to a VF; the bitmap describes pages that match the VM’s page size. The ConnectX-7 implementation of this mechanism has limitations, however. One limitation is the number of VFs that can have dirty page logging enabled simultaneously: four. A second, more important, limitation, manifests as degraded DMA performance while dirty page tracking is enabled. This impeded the performance of VMs that were not being live migrated, so we dropped this approach.

Oracle engineers contributed to an upstream Linux kernel API, IOMMUFD, that improves this process. An IOMMU (Input-Output Memory Management Unit) is a hardware abstraction layer that sits between a device that generates DMA transactions and system memory. Under the kernel’s control, an IOMMU translates a logical address associated with a DMA operation (also known as I/O virtual addresses, or IOVAs) into a host physical memory address. In the context of VMs and RDMA adapters passed through to them, an IOMMU confines the RDMA adapter’s memory accesses to regions associated with the VM, as opposed to a different VM or the hypervisor. The IOMMU also handles mapping interrupts generated by the RDMA adapter to interrupt handlers associated with the VM.

IOMMUFD is a user space interface to kernel functionality that enables sharing IOMMU acceleration features in newer CPUs between a VM and the hypervisor, including enabling the operating system running in the VM to map logical DMA addresses to guest physical addresses; a second mapping stage controlled by the hypervisor translates those guest physical addresses to host physical addresses. QEMU uses the IOMMUFD API to coordinate VM and hypervisor actions, which in turn enables it to use hardware IOMMU acceleration to implement operations that were previously implemented in (much slower) emulation code. Importantly for live migration, that hardware acceleration includes IOMMU dirty page tracking. On AMD processors, this IOMMU dirty page tracking mirrors the mechanism described above that’s used for tracking pages written by the CPU.

Hash-based Dirty Page Tracking

At the end of the dirty page tracking process, QEMU’s multifd threads have data structures describing 2MB pages that have been written somewhere by a CPU or a RDMA VF. If the multifd threads attempt to transfer these 2MB pages from the source to the destination, live migration typically does not complete for database VMs — too many 2MB pages are dirtied too fast for QEMU and the network interfaces to keep up, which means QEMU can’t hit its target downtime, which means the live migration process is vulnerable to RDMA and RAC timeouts.

We decided to solve this problem by throwing CPU cycles and memory at it, specifically by computing and storing hashes of the 4KB pages within the 2MB pages. If a 2MB page is reported as dirty, a QEMU multifd thread computes hashes of the 4KB pages within it and compares them to previously computed hashes. If the hashes match, the destination QEMU process already has that page. If the hashes differ, the page is transmitted again. We’ve carefully selected a hash algorithm with a very low risk of collisions and a very fast implementation on AMD processors, but on large database VMs the hashing operations are expensive in terms of CPU cycles, cache traffic, and memory bandwidth. This approach is working, and live migration of database VMs is a reality in Exadata 26.1. We don’t expect to keep this solution forever, as processor and network adapter vendors are telling us we’ll see more efficient dirty page tracking in future products; that, combined with switching to newer kernels, should give us a better solution in time.