Introduction

A new debug feature recently introduced in Oracle’s Open Virtual Machine Firmware (OVMF) makes debugging much easier by supporting retrieval of OVMF debug messages directly from a QEMU process or a QEMU core file. This blog provides an overview and usage details for the new feature.

Background

The UEFI boot firmware used with virtual machines, OVMF, is traditionally difficult to debug. Should an OVMF issue occur, the current method to debug the issue is to enable debug messages, reproduce the problem to capture the debug messages, and inspect the OVMF debug messages for clues. This method suffers from two significant drawbacks:

  1. To enable OVMF debug messages, changes to the QEMU configuration must be made to specify a “debug” build version of OVMF and to capture the OVMF debug messages to a host file. Changing the QEMU configuration may not be straightforward, particularly in cloud environments.

  2. After the above configuration changes are made, the issue must be reproduced. This can be difficult if the issue happens intermittently or infrequently.

Cloud customers who experience a VM boot issue are often asked to capture OVMF debug messages using the above method. Customers generally do not like using their VMs to debug issues. It would be preferable if OVMF debug messages could be obtained upon the first occurrence of an issue, and without the need for configuration changes – and now we can!

New Feature Details

Oracle Linux addresses the above drawbacks with a new OVMF feature named “OVMF Memory Debug Logging.” Oracle’s OVMF (version 20250102+) maintains a circular memory buffer of debug messages which are constantly logged under normal operation. A new utility, ovmf-debug-view, extracts the messages directly from the VM’s associated running QEMU process or from a QEMU core file. If a suspected OVMF issue occurs, the utility can be used to extract the OVMF debug messages for inspection or to send them to a support team for analysis — there’s no longer a need to make QEMU configuration changes, and to then reproduce the issue.

Example Usage

Let’s take a look at the utility in action. For demonstration, we instrumented the OVMF code to hit an ASSERT when there are no viable boot options. This ASSERT will cause OVMF to enter a CPU spin loop and the VM will appear to “hang” on boot. Below, we use the new utility to inspect the OVMF debug messages to determine the source of the hang.

From the host, find the PID of the “hung” VM’s QEMU process (note that VM’s guest name is “Guest1”):

$ ps -ef | grep qemu | grep Guest1
root   382119  382112 63 14:23 pts/2    00:00:17 /usr/libexec/qemu-kvm -name guest=Guest1...

Using the QEMU PID above (382119), extract and validate the OVMF debug message buffer header information:

$ ovmf-debug-view --pid 382119 | head -7

Scanning....................................................................................................
...........................................................DONE

OVMF Debug Info:
OVMF Firmware Version:   20250102
OVMF Debug Log Size:     0x3fba (bytes)
OVMF Debug Log:

As you can see above, the utility successfully scanned the QEMU process memory to find the OVMF debug memory buffer and displayed the header information including the OVMF version (20250102) and memory debug log buffer size (0x3fba bytes).

Run the utility again to see the tail of the log, which indicates the reason for the VM hang:

$ ovmf-debug-view --pid 382119 | tail -2
[Bds] Unable to boot!
ASSERT /home/ajyoung/Src/EDK2/edk2-1.8.0-3/rpmbuild/BUILD/edk2-20250102/OvmfPkg/Library/PlatformBootManagerLib/BdsPlatform.c(2127): 0

As you can see above, an ASSERT was hit in BdsPlatform.c line 2127, which is the ASSERT we instrumented into OVMF for our demonstration.

You can capture all the OVMF memory debug log messages to a file (for later analysis), like so:

$ ovmf-debug-view --pid 382119 > ovmf-debug-log.txt

The utility can also be used on a QEMU core file to extract the OVMF debug log messages:

$ gcore -a -o gcore1 382119     # create a QEMU process core file including all memory mappings
$ ovmf-debug-view --core-file gcore1.382119 | tail -2
[Bds] Unable to boot!
ASSERT /home/ajyoung/Src/EDK2/edk2-1.8.0-3/rpmbuild/BUILD/edk2-20250102/OvmfPkg/Library/PlatformBootManagerLib/BdsPlatform.c(2127): 0

Summary

Oracle’s new “OVMF Memory Debug Logging” feature allows for easy and immediate extraction of OVMF debug messages to troubleshoot boot issues upon first occurrence. This obsoletes previous OVMF debug methods which typically require QEMU configuration changes and reproduction of the issue. We hope this new feature allows users to more easily troubleshoot OVMF related problems.

Notes

  1. The ovmf-debug-view utility can be found in the edk2-tools RPM (version 20250102+).
  2. As of this writing, the feature does not work with Confidential Computing (CC) VMs as the guest memory is encrypted and thus not readable by the host. Support for CC VMs my be added in the future.
  3. Since the most recent debug messages are typically the most relevant to root cause an issue and to reduce the memory footprint, the feature is configured to capture only the latest (most recent) 16KB worth of messages by default. Thus the message buffer will eventually overflow and older OVMF debug messages will be lost/truncated. If desired, to capture more (or all) of the OVMF debug messages, the size of the memory buffer can by dynamically increased using the -fw_cfg name=opt/ovmf/MemDebugLogPages,string=N QEMU option, where N denotes the message buffer size in 4K pages. A value of 0 disables the memory debug logging feature.
  4. To work with the OVMF Memory Debug Logging feature, QEMU core files must be an ELF 64-bit LSB type core file containing all memory mappings. See gcore(1) -a for more details.
  5. The feature is also supported with Arm-based VMs with AAVMF version 20250102+. However, there are a couple of limitations:
    • The fw_cfg name=opt/ovmf/MemDebugLogPages,string=N QEMU option is not supported with AAVMF as fw_cfg is not supported early enough in the boot process.
    • Some very early (SEC/PEI phase) AAVMF debug messages are not captured to the memory debug log due to limitations with the AAVMF code. These messages are typically not needed to debug issues.