How to enable AMD AVIC and speed up your VMs

March 28, 2024 | 21 minute read
Text Size 100%:

Introduction

In an ideal scenario, a virtual machine would match the performance of the bare metal host on which it runs. But given the extreme complexity and variability of hardware, purely software-based virtualization suffers from unavoidable (and significant) overhead. To help with this challenge, hardware vendors have come to the rescue by providing hardware-assisted virtualization capabilities to help mitigate the biggest bottlenecks. If you are reading this blog, you are likely familiar with many of them: AMD-V/Intel VT-x processor extensions, NPT/EPT for memory virtualization, AMD-Vi/Intel VT-d for I/O virtualization, and no less importantly AMD AVIC/Intel APICv for interrupt virtualization. This blog entry will focus specifically on the AMD AVIC capability.

Why do we care about interrupt virtualization? It turns out that interrupts happen a lot, and are therefore pretty important for system performance. Unsurprisingly, interrupt handling is one of the main performance bottlenecks in virtualized environments.

Picture a network packet destined for an application running on a virtual machine: When an interrupt arrives from the NIC to notify the guest OS that a packet is ready for processing, it is likely that a VM-Exit is required for the hypervisor to inject the interrupt, and the vCPU will only process the interrupt after the next VM-Entry. This overhead negatively impacts end-to-end network latency for applications running on the guest.

Core counts for virtual machines continue to grow, and the Linux kernel scheduler uses IPIs (inter-processor interrupts) for task placement and management. Guest vCPUs attempting to communicate with each other issue IPIs, and in the typical interrupt virtualization mechanism implemented in KVM, a VM-Exit/VM-Entry cycle is required to handle these events.

Enabling AVIC can help to remove the hypervisor intervention in these performance sensitive operations, and close the gap between virtualized and bare metal performance. This blog post describes how to configure OCI BM.Standard.E5.192 Bare Metal instances to enable AVIC so that guests running on them can take advantage of hardware-accelerated interrupt delivery.

What is the Advanced Virtual Interrupt Controller (AVIC)?

A specialized component called the Advanced Programmable Interrupt Controller (APIC) is essential for interrupt handling in modern multiprocessor systems. On x86, the most recent generation is the x2APIC architecture, and it is typically integrated in the processor/chipset. In virtualization, a guest operating system also needs access to an interrupt controller to handle device and interprocessor interrupts, and must rely on the VMM to emulate the interrupt controller, intercept guest access to the interrupt controller registers, and deliver interrupts to the target guest virtual processor. These operations cause numerous VM-Exits, which are a known cause of virtualization overhead. Considering the high rate of inter-processor and device interrupts in scenarios such as cloud workloads, hardware acceleration of these operations can result in significant performance gains, in the form of reduced VM-Exit counts.

The AVIC feature provides hardware support to deliver frequently triggered interrupts (IPIs and device) directly to running guests, avoiding the overhead associated with using the legacy interrupt injection model. The specific mechanisms by which this is accomplished are documented on the AMD64 Architecture Programmer’s Manual Volume 2 System Programming (Section 15.29) and AMD I/O Virtualization Technology (IOMMU) Specification, and are better discussed in a separate post. Fortunately, enabling this acceleration on KVM/QEMU guests requires adjusting only a few simple settings, and this blog entry will show how to do it.

x2avic support in OCI BM.Standard.E5.192 instances

Zen 4 (Genoa) AMD EPYC processors support up to 96 cores (192 threads) per socket, so 2 socket systems have up to 384 threads and we can easily create guests with more than 256 vCPUs, which requires enabling x2apic mode. Earlier AVIC implementations only supported xapic mode (up to 255 vCPUs), but E5 Standard Bare Metal instances now add support for x2AVIC, meaning that x2apic virtualization is supported and we can enable hardware-accelerated interrupt virtualization for up to 512 vCPUs.

In x2apic mode, accesses to APIC registers are done via the Model Specific Register (MSR) interface, so x2AVIC capability means that the hardware is now able to recognize when the guest issues certain read/write MSRs accessing the x2APIC Register Address Space, and emulate the side effects of a subset of these operations without the need for a VM-Exit.

Guests with more than 255 vCPUs require specifying a vIOMMU device with x2apic support, to be used by the guest for interrupt remapping so that it can route interrupts to vCPUs with APIC IDs larger than 255. With newer QEMU and guest kernel we can make use of a paravirtualized feature (kvm-msi-ext-dest-id). This blog will use the latter for simplicity.

How to verify hypervisor support

OCI E5 Standard bare metal shapes already enable the required capabilities by default in BIOS settings. We will briefly mention the essential settings and provide pointers to the official documentation for reference, so this section is still worth a look if you like to tinker with your host firmware or enjoy reading technical documentation.

SVM AVIC and x2AVIC support

AVIC support on specific hardware implementation is indicated by CPUID Fn8000_000A_EDX SVM Feature Identification.

CPUID Fn8000_000A_EDX[AVIC] (bit 13) = 1 indicates support for the AVIC architecture on the hardware. Additionally, CPUID Fn8000_000A_EDX[x2AVIC] (bit 18) = 1 indicates support for the AVIC architecture when the guest is using x2APIC MSR interface.

We can use the cpuid utility to determine if support is present:

$ cpuid -1 -l 0x8000000A | grep -i avic
    AVIC: AMD virtual interrupt controller  = true
    X2AVIC: virtualized X2APIC              = true

These capabilities are also reported in the Flags section of the lscpu command output:

$ lscpu | grep -o 'x2avic\|avic'
avic
x2avic

IOMMU AVIC support

The AMD IOMMU Extended Feature register (documented in MMIO Offset 0030h IOMMU Extended Feature Register) can be queried via the sysfs interface to confirm that the Guest Virtual APIC feature is present:

$ cat /sys/class/iommu/ivhd*/amd-iommu/features | uniq
25bf732fa2295afe

The relevant fields are named GASup and GAMSup. Using Table 21: Interrupt Virtualization Controls for Upstream Interrupts, we find that both GASup (bit 7) and GAMSup (bit 23:21) controls must be set to 1, which is the case for the value read earlier.

The following sample script checks for the relevant fields:

#!/bin/bash

function check_iommu_avic_support() {

    local gasup=7
    local gamsup=21
    local iommu_caps=$(( (1 << $gasup) | (1 << $gamsup) ))

    if [ -f /sys/class/iommu/ivhd0/amd-iommu/features ]
    then
        local bitmap=$(cat /sys/class/iommu/ivhd0/amd-iommu/features)
        bitmap=${bitmap##*:}
    else
        echo "Unable to find AMD IOMMU Extended Feature Register. Is this an AMD EPYC host with IOMMU enabled?"
        echo "Exiting..."
        exit 2
    fi

    if [[ $(( (16#$bitmap & $iommu_caps) )) -eq $iommu_caps ]]; then
        echo "IOMMU AVIC capability is SUPPORTED"
    else
        echo "IOMMU AVIC capability is NOT SUPPORTED"
    fi
}

check_iommu_avic_support

Host parameters to enable AVIC/x2AVIC

In order to support all of the options mentioned in this document, the UEK minimum version is:

UEK7-U2: V5.15.0-204.146.1

In short, when running a recent hypervisor kernel, AVIC can be enabled by loading the kvm_amd module with avic=1. All KVM guests will then enable both SVM and IOMMU AVIC functionality, unless restricted via kernel parameters or inhibited due to specific guest configurations that are not supported by AVIC hardware. For a detailed description of those specific cases and settings, see the following sections.

KVM module parameters

The avic parameter of the kvm_amd module controls both AVIC and x2AVIC state. The specific feature enabled depends on the hardware support. When kvm_amd.avic=1, both SVM AVIC/x2AVIC and IOMMU AVIC will be enabled in guests that support it.

The avic parameter is disabled by default. Edit /etc/modprobe.d/kvm.conf to enable avic parameter in kvm_amd module. An example:

$ cat /etc/modprobe.d/kvm.conf
options kvm_amd avic=1

Alternatively, for a non-persistent option just reload the module with the above parameters using modprobe:

$ sudo modprobe -r kvm_amd
$ sudo modprobe kvm_amd avic=1

IOMMU AVIC

Device (I/O) interrupts acceleration is provided by AVIC and x2AVIC. IOMMU AVIC will be enabled by default when kvm_amd.avic=1.

A recent update to the AMD I/O Virtualization Technology (IOMMU) Specification(2.2.5.2 Interrupt Virtualization Tables with Guest Virtual APIC Enabled) states that when IOMMU AVC is enabled, certain commonly accessed data fields are not cached by IOMMU and therefore do not require invalidating the Interrupt Remapping Table whenever they are modified. This change is available since Linux v6.8, meaning that in newer kernels, no additional configuration is required in order to fully utilize IOMMU AVIC.

IOMMU AVIC independent controls

While still enabling IPI acceleration (SVM AVIC/x2AVIC), the IOMMU AVIC functionality can be disabled independently by using:

amd_iommu_intr=legacy

in the host kernel parameters.

QEMU parameters

In order to enable AVIC, certain conditions need to be met, otherwise AVIC is inhibited (sometimes only temporarily), and interrupt delivery reverts to using the standard mechanism. A list of the various reasons that cause AVIC inhibition can be found in the kernel source code.

During guest launch, we can use the ftrace utility to display the inhibit reasons using the kvm_apicv_inhibit_changed tracepoint:

qemu-kvm-1661305 [141] ..... 429704.679440: kvm_apicv_inhibit_changed: set reason=2, inhibits=0x4
qemu-kvm-1661305 [141] ..... 429704.680307: kvm_apicv_inhibit_changed: cleared reason=2, inhibits=0x0
qemu-kvm-1661310 [160] ..... 429708.378870: kvm_apicv_inhibit_changed: set reason=7, inhibits=0x80
qemu-kvm-1661310 [160] ..... 429708.378879: kvm_apicv_inhibit_changed: set reason=7, inhibits=0x80
qemu-kvm-1661310 [160] ..... 429708.378881: kvm_apicv_inhibit_changed: cleared reason=7, inhibits=0x0

Reason 2 corresponds to APICV_INHIBIT_REASON_ABSENT, which is set during early VM initialization when the in-kernel emulation of the local APIC has not been enabled yet via KVM ioctls KVM_CREATE_IRQCHIP or KVM_CAP_SPLIT_IRQCHIP.

Reason 7 corresponds to APICV_INHIBIT_REASON_IRQWIN, which means that an IRQ window was requested, and AVIC was temporarily inhibited.

Fortunately, the majority of the settings necessary are already applied by default when using a recent QEMU and guest kernel version. Some are explicitly added in the following example QEMU command line, to highlight their relevance.

/usr/libexec/qemu-kvm \
-machine q35,accel=kvm,kernel-irqchip=split \
-cpu host,topoext=on,x2apic=on \
-smp 380 \
-m 64G \
-blockdev node-name=drive0,driver=qcow2,file.driver=file,file.filename=./OracleLinux-8.6-uefi-x86_64.qcow2 \
-device virtio-blk-pci,drive=drive0,id=virtio-disk0 \
-drive if=pflash,format=raw,unit=0,file=/usr/share/OVMF/OVMF_CODE.pure-efi.fd,readonly=on \
-drive if=pflash,format=raw,unit=1,file=./OVMF_VARS.pure-efi.fd \
-device vfio-pci,host=a1:00.1 \
-nodefaults \
-no-user-config \
-serial mon:stdio -nographic

Important options

x2apic=on

This is required to support guests with more than 255 vCPUs. As discussed earlier, the x2AVIC feature means that guests can function in x2apic mode and accesses to their local APIC (using MSR interface) are hardware-accelerated and avoid VM-Exits.

kernel-irqchip=split

This property controls whether the interrupt controller(s) emulation will be done by the kernel (KVM), or user space (QEMU/other VMM). The implementation is dependent on architecture and machine model, but for x86 and pc-q35 (as is the case for this blog), using split means that the local APICs for the vCPUs will be in the kernel, while IOAPIC/PIC/PIT controllers are emulated by QEMU. Using a split irqchip allows us to use the paravirtualized feature kvm-msi-ext-dest-id, which is necessary for supporting more than 255 vCPUs without interrupt remapping. This is a paravirtualized feature that requires both QEMU and guest kernel support.

An alternative is to use a vIOMMU with support for x2apic and interrupt remapping (e.g. -device intel-iommu,intremap=on,eim=on,caching-mode=on), which also requires a split irqchip to work. This option can be useful if we wish to use AVIC to accelerate guests running on kernels older than Linux v5.11, which cannot make use of kvm-msi-ext-dest-id.

A particularly observant reader might wonder why use an Intel vIOMMU model when running on an AMD host. The reason is that we need a vIOMMU that supports both VFIO to allow for passthrough devices and x2apic mode, and that functionality is currently not available in the AMD vIOMMU model that is implemented in QEMU (although there is ongoing work in this area).

Using split irqchip also avoids the need for specifying the option -global kvm-pit.lost_tick_policy=discard, since when the PIT is emulated in kernel, the default mode (reinject) causes AVIC to be inhibited.

As we can see from the example QEMU command line, the are no changes needed for passthrough devices. When the guest is using SR-IOV devices and IOMMU AVIC is properly enabled as discussed above, device interrupts will be accelerated and delivered to the target vCPU without incurring a VM-Exit, unless the vCPU is not currently running, in which case a mechanism called Guest Virtual APIC (GA) Logging ensures that the interrupt is delivered once the vCPU is active again. We’ll use tracepoints associated with this mechanism to verify that IOMMU AVIC is active.

Equivalent Libvirt parameters

In case you manage your guests using libvirt, these are the equivalent changes that need to be applied to the guest XML:

<features>
  <acpi/>
  <apic eoi='off'/>
  <kvm>
    <mem-lock state='on'/>
    <cpu-pm state='off'/>
  </kvm>
  <ioapic driver='qemu'/>                                       ---> split irqchip mode
</features>
<cpu mode='host-passthrough' check='none'>
  <feature policy='force' name='invtsc'/>
  <feature policy='force' name='topoext'/>
  <feature policy='force' name='x2apic'/>                       ---> enable x2apic
  <feature policy='require' name='arch-capabilities'/>
</cpu>

# If a vIOMMU is necessary
<iommu model='intel'>                                           ---> Presents an Intel IOMMU to guest
  <driver intremap='on' caching_mode='on' eim='on'/>
</iommu>

Verification

If you have carefully followed the advice above, AVIC will be active when your guest(s) runs, so IPIs and I/O device interrupts will be accelerated, but the guest has no awareness of it. From the hypervisor side, in order to verify that AVIC is working as expected, we can detect certain events such as VM-Exit types that are associated with AVIC.

SVM AVIC verification

Under typical guest operations, the vCPUs regularly exit guest mode due to idling, causing a VM-Exit. In such cases, the vCPU is not running on a physical processor anymore, and when an IPI targeting this vCPU is sent by another vCPU it cannot be delivered and will cause a VM-Exit on the sending vCPU with an exit code of AVIC_INCOMPLETE_IPI. See Section 15.29.9.1 AVIC IPI Delivery Not Completed for details.

We can detect this fairly common event (IPIs are very frequent after all, which is why we are interested in AVIC in the first place) using the perf kvm tool on the host. Running perf kvm stat live will show regular updates with top VM-Exit reasons. When AVIC is active we expect to see avic_incomplete_ipi events, and no vintr during regular guest operation, while the opposite will happen if AVIC is disabled or inhibited for one or more of the possible reasons mentioned earlier.

When guests are running with AVIC enabled, we expect to see similar output to the following:

Analyze events for all VMs, all VCPUs:

             VM-EXIT    Samples  Samples%     Time%    Min Time    Max Time         Avg time

                 msr        167    71.06%     2.21%      0.52us  50069.52us   1268.37us ( +-  45.24% )
                 hlt         52    22.13%    97.79%    548.00us 1100494.39us 180463.13us ( +-  23.08% )
           interrupt         14     5.96%     0.00%      0.27us      0.55us      0.39us ( +-   6.72% )
 avic_incomplete_ipi          2     0.85%     0.00%      3.64us      3.77us      3.70us ( +-   1.76% )

Total Samples:235, Total events handled time:9595913.03us.

IOMMU AVIC verification

In a similar scenario as described for SVM AVIC earlier, when an I/O device interrupt targets a vCPU that is not currently running, the AMD IOMMU has a mechanism called Guest Virtual APIC (GA) Logging, which ensures that the interrupt is delivered once the target vCPU is active again. We can use tracepoints on this code path to verify that IOMMU AVIC is enabled. The instructions below assume of course that a device such as a VF from an SR-IOV capable NIC (hint there are two Mellanox ConnectX-6 Dx available in an E5-2c BM shape) are passed to the guest, and will be a source of I/O device interrupts. The following shows a simple example of how to enable the relevant event and the expected output:

cd /sys/kernel/debug/tracing/
echo nop > current_tracer
echo 1 > tracing_on

# Enable only GA Log events to avoid noisy output
grep ga_log available_events > set_event

# Verify correct event is set. Expect to see "kvm:kvm_avic_ga_log"
cat set_event

With a guest running, there will be periodic events received:

head -n 20 trace
# tracer: nop
#
# entries-in-buffer/entries-written: 5893/5893   #P:384
#
#                                _-----=> irqs-off
#                               / _----=> need-resched
#                              | / _---=> hardirq/softirq
#                              || / _--=> preempt-depth
#                              ||| /     delay
#           TASK-PID     CPU#  ||||   TIMESTAMP  FUNCTION
#              | |         |   ||||      |         |
   irq/29-AMD-Vi-2740    [005] .... 253283.601523: kvm_avic_ga_log: vmid=33, vcpuid=8
   irq/29-AMD-Vi-2740    [005] .... 253283.601918: kvm_avic_ga_log: vmid=33, vcpuid=8
   irq/29-AMD-Vi-2740    [005] .... 253283.604476: kvm_avic_ga_log: vmid=33, vcpuid=8
   irq/29-AMD-Vi-2740    [005] .... 253283.607779: kvm_avic_ga_log: vmid=33, vcpuid=8
   irq/29-AMD-Vi-2740    [005] .... 253283.613228: kvm_avic_ga_log: vmid=33, vcpuid=8
   irq/29-AMD-Vi-2740    [005] .... 253283.618425: kvm_avic_ga_log: vmid=33, vcpuid=8
   irq/29-AMD-Vi-2740    [005] .... 253283.622187: kvm_avic_ga_log: vmid=33, vcpuid=8
   irq/29-AMD-Vi-2740    [005] .... 253283.628061: kvm_avic_ga_log: vmid=33, vcpuid=8
   irq/29-AMD-Vi-2740    [005] .... 253283.633674: kvm_avic_ga_log: vmid=33, vcpuid=8

VM-Exit reduction with AVIC enabled

Enabling AVIC can drastically reduce VM-Exit counts for certain workloads that generate heavy IPI traffic and device interrupts. The effect of AVIC will vary with different workloads and configurations, so it is worth testing with your specific use case.

For a simple example, we use a smaller guest with 16 vCPUs which has the advantage of avoiding large data files and long processing times when using the perf tools. An example QEMU command line:

/usr/libexec/qemu-kvm \
-nodefaults \
-no-user-config \
-serial mon:stdio -nographic \
-machine q35,accel=kvm,kernel-irqchip=split \
-cpu host,topoext=on,x2apic=on,svm=off \
-smp 16 \
-m 64G \
-blockdev node-name=drive0,driver=qcow2,file.driver=file,file.filename=./OracleLinux-8.6-uefi-x86_64.qcow2 \
-device virtio-blk-pci,drive=drive0,id=virtio-disk0 \
-drive if=pflash,format=raw,unit=0,file=/usr/share/OVMF/OVMF_CODE.pure-efi.fd,readonly=on \
-drive if=pflash,format=raw,unit=1,file=./OVMF_VARS.pure-efi.fd \
-device vfio-pci,host=a1:00.1

In the guest, we can launch a workload using the sysbench benchmark tool, which is available from the EPEL packages for test and development repository.

There are multiple test modes available in sysbench, but we will use the threads workload with a simple invocation:

sysbench threads --threads=16 --thread-locks=8 --time=20 run

After launching the workload on the guest, we collect VM-Exit statistics for 20 secs on the host using the perf tool:

$ sudo perf kvm stat record -p $(pgrep qemu) -- sleep 20

The data captured can be processed using:

$ sudo perf kvm stat report

Note: Both the capture and processing of the data will take significantly longer than the 20 sec interval allocated for the capture, since a large number of events needs to be processed even for a relatively small guest.

Comparison of VM-Exit count

Using the default KVM interrupt virtualization mechanism, we find the following VM-Exit profile:

Analyze events for all VMs, all VCPUs:

             VM-EXIT    Samples  Samples%     Time%    Min Time    Max Time         Avg time

           interrupt   10980413    46.29%    52.97%      0.17us  36735.18us      0.82us ( +-   0.43% )
                 msr   10748435    45.32%    35.02%      0.18us    908.53us      0.56us ( +-   0.02% )
               vintr    1987115     8.38%     8.61%      0.30us     51.03us      0.74us ( +-   0.03% )
                 npf       1961     0.01%     0.01%      0.28us     20.11us      1.10us ( +-   3.48% )
                 hlt        740     0.00%     3.31%      0.45us  13752.90us    764.07us ( +-   8.06% )
               pause        126     0.00%     0.07%      0.55us  11529.83us     92.88us ( +-  98.51% )
               cpuid        113     0.00%     0.00%      0.20us      1.43us      0.39us ( +-   5.19% )
                 nmi         39     0.00%     0.00%      0.40us      1.69us      0.93us ( +-   6.27% )
                  io         15     0.00%     0.00%      2.34us     29.40us      7.53us ( +-  27.77% )
           hypercall          5     0.00%     0.00%      1.66us    141.75us     32.03us ( +-  85.82% )

Total Samples:23718962, Total events handled time:17061220.16us.

Where the same experiment with AVIC enabled results in the following distribution:

Analyze events for all VMs, all VCPUs:

             VM-EXIT    Samples  Samples%     Time%    Min Time    Max Time         Avg time

           interrupt     720091    68.61%    31.56%      0.17us  17136.59us      0.67us ( +-   5.67% )
                 msr     325980    31.06%    21.87%      0.42us     61.70us      1.02us ( +-   0.08% )
                 npf       1293     0.12%     0.19%      0.28us    328.47us      2.27us ( +-  19.95% )
                 hlt        976     0.09%    46.06%      0.39us  13840.11us    717.36us ( +-   6.72% )
 avic_incomplete_ipi        834     0.08%     0.13%      0.38us     17.32us      2.46us ( +-   1.74% )
               pause        141     0.01%     0.01%      0.43us      5.70us      1.26us ( +-   4.17% )
               cpuid        113     0.01%     0.00%      0.20us      1.61us      0.37us ( +-   5.52% )
                 nmi         47     0.00%     0.00%      0.49us      1.86us      0.73us ( +-   5.80% )
                  io         29     0.00%     0.03%      1.84us    148.78us     13.82us ( +-  48.57% )
           hypercall          2     0.00%     0.14%     14.22us   2080.35us   1047.29us ( +-  98.64% )

Total Samples:1049506, Total events handled time:1520217.47us.

Comparing the total number of samples:

AVIC disabled:  23718962

AVIC enabled:   1049506

we can see that enabling AVIC results in approximately a 96% reduction in the number of VM-Exits generated for the same workload.

It is worth noting that this drastic reduction in VM-Exits will not translate to a 96% improvement in performance for the benchmark, which would only be the case for a workload that is fully bottlenecked due to VM-Exit and interrupt injection latency. However, enabling AVIC will certainly show improvements to a degree, since as shown above it significantly reduces one of the overheads in virtualization setups.

AVIC effect on IPI speed

Another simple experiment to quantify the advantages of enabling AVIC is to use the kvm-unit-tests IPI test, which reports the number of cycles (measured using RDTSC) that it takes to deliver an IPI.

Running the standard invocation of the ipi test will cause AVIC to be inhibited since QEMU by default uses the in-kernel irqchip mode, which as we have discussed earlier means that a programmable interval timer (PIT) is emulated in kernel, and the default mode (reinject) causes AVIC to be inhibited. To avoid this, we will use split irqchip mode, and a single run looks like the following (with AVIC enabled):

$ QEMU=/home/opc/Git/qemu/build/qemu-system-x86_64 ./x86-run ./x86/vmexit.flat -smp 2 -append 'ipi' -machine kernel-irqchip=split
/home/opc/Git/qemu/build/qemu-system-x86_64 --no-reboot -nodefaults -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio -device pci-testdev -machine accel=kvm -kernel ./x86/vmexit.flat -smp 2 -append ipi -machine kernel-irqchip=split # -initrd /tmp/tmp.iW3UXG2Dq6
enabling apic
smp: waiting for 1 APs
enabling apic
setup: CPU 1 online
paging enabled
cr0 = 80010011
cr3 = 1007000
cr4 = 20
pci-testdev at 0x10 membar febff000 iobar c000
ipi 3061
  ipi ipi 3036

Collecting 20 runs for each scenario and taking the average shows:

AVIC disabled:  5708 cycles

AVIC enabled:   3027 cycles

a significant 47% reduction in cycles taken to deliver an IPI when AVIC is enabled.

Conclusion

This article presents a brief introduction to the AVIC capability available in Zen4 AMD EPYC processors, and provides instructions on how to enable the feature using an Oracle Cloud Infrastructure Bare Metal instance to demonstrate the process. Support for AVIC, and more specifically for x2AVIC, is available on recent mainline and UEK7 kernels, allowing for hardware-accelerated interrupt virtualization in guests with up to 512 vCPUs.

Alejandro Jimenez


Previous Post

Maximize database performance with Oracle Exadata and Oracle Linux

Gursewak Sokhi | 8 min read

Next Post


Improve your Linux file system know-how with the Oracle Linux Storage Management Video Playlist

Nicolas Pares | 3 min read