Introduction

Windows guests in KVM/QEMU virtualized environments may encounter runtime problems, crashes, and hangs. This article describes practical tools for debugging those issues under different circumstances. If a Windows guest runs into I/O failures or features that do not work correctly, the best next step is to collect Windows system and driver logs. If the guest crashes and leaves a dump file behind, the next step is to analyze the crash dump. If the guest hangs, or crashes without generating a Windows dump, the best fallback is host-side memory capture.

Debugging a Running Windows Guest

If the Windows guest is still alive, start with live trace collection. The following sections introduce several ways to collect Windows system and VirtIO driver logs: the Troubleshooting Script Toolset (TSS), Windows VirtIO scripts for collecting Windows system logs, and Windows VirtIO driver log collection.

Collect TSS Logs From a Windows Guest

The TSS tool can be downloaded from Microsoft. It must be run with Administrator privileges. In PowerShell, you can enable it in the current session like this:

Set-ExecutionPolicy -Scope Process -ExecutionPolicy RemoteSigned -Force

You can then start log collection with a PowerShell command such as:

.\TSS.ps1 -Collectlog DND_SetupReport

The logs will be compressed and saved in a folder named “MSDATA”. TSS can also work with other tools, such as PerfMon, Wireshark, ProcMon, and PktMon to collect more specialized logs. For more details, see Introduction to Troubleshooting Script Toolset (TSS).

Collect Windows System Logs with a VirtIO Script

If you want to collect a simplified set of system logs, the Windows VirtIO driver PowerShell script CollectSystemInfo.ps1 is an option. It’s available in the Tools/debug directory of the Windows VirtIO driver repository, and it mainly collects system-level data, such as system configuration, Windows updates, and event logs.

Run the following PowerShell command first:

Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope Process -Force

Then execute the PowerShell script to collect logs:

.\CollectSystemInfo.ps1 -IncludeSensitiveData

A “SystemInfo” directory is generated with log files.

Collect VirtIO Driver Logs

The logs collected by the two methods above are system logs, not VirtIO driver-specific logs. If you suspect VirtIO drivers are causing an issue, it is better to collect logs from the VirtIO drivers directly. The vioscsi and NetKVM drivers are used as examples below.

Collect Logs for vioscsi and NetKVM

Users can either run TraceView or a batch command to collect logs from Windows VirtIO drivers. For vioscsi, the default ETW trace log level is TRACE_ALL. For NetKVM, users have to enable logging and set the appropriate logging level through the NIC’s Advanced Properties in Device Manager. To locate the NetKVM logging settings, open Device Manager, expand Network adapters, right-click on Oracle VirtIO Ethernet Adapter, select Properties, and then look for the following options on the Advanced tab:

  • Logging.Enable
  • Logging.Level

Collect Logs Through Batch Commands

In the Tools/trace directory of the the Windows VirtIO driver repository, you’ll find batch files for tracing the VirtIO drivers.

  1. Run vioscsi.bat for the vioscsi driver or netkvm.bat for the NetKVM driver; these need to be run as an Administrator.
  2. Reproduce the problem.
  3. Stop tracing and save vioscsi.etl or netkvm.etl.

If you need readable output instead of a raw ETL file, decode the ETL using the matching vioscsi.pdb or netkvm.pdb file. These PDB files are generated alongside VirtIO driver binaries but are not included in official end-user releases, so you may need to obtain them from the build team. If that is not possible, build the PDBs from source and ensure they exactly match the driver binaries installed in your Windows guest. If the guest was installed with drivers from this packaging flow, the matching PDBs may also be available there. To decode the ETL file, use TraceView or ParseTrace.bat together with tracefmt.exe and tracepdb.exe.

Debugging a Crashed Guest From a Memory Dump File

If the Windows guest crashes and successfully generates a memory dump, we need to switch from live tracing to dump analysis. Note: Any crash dump or memory dump file collected during debugging may contain sensitive information, so think before sharing them.

Windows Memory Dump Files

Depending on the advanced system settings in Windows, the crash dump may be a complete memory dump, a kernel memory dump, or a small memory dump. The first two are located in %SystemRoot%\Memory.dmp, while the small memory dump is stored in %SystemRoot%\Minidump\*.dmp.

Minidumps are small and easy to collect, so they are good for triage. But they are often not adequate for deep system debugging because they do not contain detailed debug information. In such cases, collecting a kernel memory dump is recommended.

Debugging with MEMORY.DMP or a Minidump File

  1. Copy %SystemRoot%\Memory.dmp or %SystemRoot%\Minidump\*.dmp from the guest.
  2. Open it in WinDbg or WinDbg Preview on Windows.
  3. Set up symbols for Windows and for the Windows VirtIO driver build’s PDB file. You may be able to obtain the PDB file from the development team that built the driver. If the Windows guest was installed with VirtIO drivers from this packaging flow, the matching PDB may also be available there.
  4. Start with:
!analyze -v
lm
kv

Then continue with driver-specific inspection.

Debugging a Hung or Crashed Guest Without a Windows Dump File

Sometimes, a Windows guest hangs or crashes without generating a crash dump. In that case, host-side capture becomes important.

Trigger NMI to Crash the Hung Windows Guest

If the Windows guest hangs but still responds to an NMI interrupt, one option is to trigger an NMI to force a crash and collect a dump for further analysis. Windows 8 and later usually create crash dumps by default when they receive an NMI. Older versions, such as Windows 7 and Windows Server 2008 R2, may ignore NMI unless NMICrashDump is enabled in the registry:

Path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CrashControl
Value name: NMICrashDump
Type: REG_DWORD
Value: 1

This QEMU HMP command will trigger a NMI from the host:

nmi

The equivalent QEMU QMP command is:

{"execute": "inject-nmi"}

If the Windows guest responds to NMI properly, it crashes with code 0x00000080 (NMI_HARDWARE_FAILURE). If a crash dump file is generated successfully, the user can reboot the guest and copy it out for further debugging.If the Windows guest does not respond to NMI, or if no crash dump file is generated, we have to switch to host-side memory dump collection. Note that the file size of host-side memory dump is usually the same as the guest memory size, so make sure the host has enough free storage to hold it.

Dump Memory in Windows Dump Format on the Host

QEMU supports dumping Windows guest memory in Windows crash dump format. For this to work, the Windows guest has to be booted with -device vmcoreinfo and it must have a valid fw_cfg driver installed in the guest. The following QEMU HMP command will dump guest memory:

dump-guest-memory -w memory.dmp

The memory.dmp file will be generated on the host for further WinDbg analysis.

Use elf2dmp to Convert a Host-side ELF Memory Dump File

If the Windows guest does not have a valid fw_cfg driver installed, or if this driver is not working properly, another option is to dump the Windows guest memory in ELF format. You can then use elf2dmp to convert the host-side ELF dump file into a Windows dump file that can be opened in WinDbg. elf2dmp is normally stored as /usr/bin/elf2dmp on the host.

  1. Issue this QEMU HMP command to dump guest memory in ELF format: dump-guest-memory vmcore.elf
  2. Convert the ELF dump file to a Windows dump file: elf2dmp vmcore.elf memory.dmp

Then open the memory.dmp file in WinDbg for further debugging.

Summary

For a running Windows guest, start with live logging. Collect traces first because they are the least disruptive and often good enough to explore the problem. If the guest crashes and produces MEMORY.DMP or a minidump, move to WinDbg; prefer MEMORY.DMP whenever you need full debugging capability. If the guest hangs or Windows never writes a dump file, switch to host-side memory capture. Convert ELF dump files with elf2dmp, or trigger a controlled crash if the guest is still reachable and your environment supports it.