Using systemtap For Tracing libvirt

January 25, 2024 | 13 minute read
Text Size 100%:

Introduction

systemtap is a scripting language and a tool for dynamically tracing/probing activities in Linux kernel/userspace.

In context of its usage with libvirt, we will use systemtap to dynamically trace libvirt (userspace) in this article.

We will use libvirt-7.10.0-3 in this article but you may also try with other libvirt versions.

Setting up the playground

Step 1 : Create an OCI BM

Go to https://www.oracle.com/cloud/ and spin up a BM (Bare Metal) instance with the Oracle Linux 8 Operating System image.

You can also start using the command line utility yo (https://github.com/oracle/yo) to easily create and manage your OCI instances.

Step 2 : Install libvirt from ULN

# Enable kvm appstream repo
$ sudo dnf config-manager --enable ol8_kvm_appstream

# Enable the virt:kvm_utils2 module and stream
$ sudo dnf module enable virt:kvm_utils2 -y

# Install base packages from the virt:kvm_utils2 stream
$ sudo dnf module install virt:kvm_utils2 -y

# Install all libvirt rpms
$ sudo dnf install libvirt\*7.10.0-3\* -y

Step 3 : UEK and Syncing with ULN

In this blog, we will use UEK6 (Unbreakable Enterprise Kernel 6), but you can also follow along with other UEK versions. Read more about UEK here https://docs.oracle.com/en/operating-systems/uek/.

# To see all installed UEK versions
$ sudo rpm -qa | grep kernel-uek

# To see the running kernel version
$ uname -r

# To see the effective running kernel version (if you have ksplice)
$ uptrack-uname -r

For reading more about ksplice and uptrack, check out https://ksplice.oracle.com.

Now, let's sync our Oracle Linux distro with the default ULN (Unbreakable Linux Network) repositories in order to get the latest updates:

# sync all repos and perform all required upgrades/downgrades
$ sudo dnf distro-sync --allowerasing -y

# Reboot to apply changes (especially if a newer kernel version is available)
$ sudo reboot

Step 4 : Check dtrace support (if following with non-oracle libvirt)

By default, Oracle Linux libvirt rpms are built with dtrace enabled, so if you are using the official libvirt rpms from ULN repo, you are already good to go.

If you have custom libvirt rpms or trying with some other distro, check that libvirt is built with dtrace to support tracing in systemtap. We can enable dtrace support during the build as follows :

For libvirt < 6.7.0 :

$ mkdir build && cd build # to keep build separate from source code repo
$ ../autogen.sh --with-dtrace
$ make

For libvirt >= 6.7.0 :

$ meson build -Ddtrace=enabled
$ ninja -C build

If you do not know about the options configured while building your libvirt rpms and wish to know if dtrace support was enabled, you can install the rpms and check whether libvirt probes are available as shown in next step.

Step 5 : Check that libvirt probes are now available

libvirt probes are the events/functions available for us to trace in libvirt. You can think of them as functions in libvirt source code that are available for us to trace.

The userspace markers/probes are installed by the libvirt-client rpm when systemtap-client is available.

If /usr/share/systemtap/tapset/libvirt_* are present, then libvirt probes are available.

You can use the following command to see the probe files :

$ ls /usr/share/systemtap/tapset/ | grep libvirt
libvirt_functions.stp
libvirt_probes-64.stp
libvirt_qemu_probes-64.stp

To list all libvirt probes :

$ grep 'probe libvirt.[a-z_0-9.]*' /usr/share/systemtap/tapset/libvirt_* -o|cut -f 2 -d :
probe libvirt.event_glib.add_handle
probe libvirt.event_glib.update_handle
probe libvirt.event_glib.remove_handle
probe libvirt.event_glib.remove_handle_idle
probe libvirt.event_glib.dispatch_handle
probe libvirt.event_glib.add_timeout
probe libvirt.event_glib.update_timeout
probe libvirt.event_glib.remove_timeout
probe libvirt.event_glib.remove_timeout_idle
probe libvirt.event_glib.dispatch_timeout
probe libvirt.object.new
probe libvirt.object.ref
probe libvirt.object.unref
probe libvirt.object.dispose
probe libvirt.rpc.socket_new
probe libvirt.rpc.socket_dispose
probe libvirt.rpc.socket_send_fd
probe libvirt.rpc.socket_recv_fd
probe libvirt.rpc.server_client_new
probe libvirt.rpc.server_client_dispose
probe libvirt.rpc.server_client_msg_tx_queue
probe libvirt.rpc.server_client_msg_rx
probe libvirt.rpc.client_new
probe libvirt.rpc.client_dispose
probe libvirt.rpc.client_msg_tx_queue
probe libvirt.rpc.client_msg_rx
probe libvirt.rpc.server_client_auth_allow
probe libvirt.rpc.server_client_auth_deny
probe libvirt.rpc.server_client_auth_fail
probe libvirt.rpc.tls_context_new
probe libvirt.rpc.tls_context_dispose
probe libvirt.rpc.tls_context_session_allow
probe libvirt.rpc.tls_context_session_deny
probe libvirt.rpc.tls_context_session_fail
probe libvirt.rpc.tls_session_new
probe libvirt.rpc.tls_session_dispose
probe libvirt.rpc.tls_session_handshake_pass
probe libvirt.rpc.tls_session_handshake_fail
probe libvirt.rpc.keepalive_new
probe libvirt.rpc.keepalive_dispose
probe libvirt.rpc.keepalive_start
probe libvirt.rpc.keepalive_stop
probe libvirt.rpc.keepalive_send
probe libvirt.rpc.keepalive_received
probe libvirt.rpc.keepalive_timeout
probe libvirt.qemu.monitor_new
probe libvirt.qemu.monitor_ref
probe libvirt.qemu.monitor_unref
probe libvirt.qemu.monitor_close
probe libvirt.qemu.monitor_send_msg
probe libvirt.qemu.monitor_recv_reply
probe libvirt.qemu.monitor_recv_event
probe libvirt.qemu.monitor_io_process
probe libvirt.qemu.monitor_io_read
probe libvirt.qemu.monitor_io_write
probe libvirt.qemu.monitor_io_send_fd

Step 6 : Install debuginfo rpms

  • Create/Enable debuginfo yum repo

    $ sudo bash -c 'cat > /etc/yum.repos.d/ol8_debuginfo.repo << EOF
    [ol8_debuginfo]
    name=Debuginfo for Oracle Linux $releasever ($basearch)
    baseurl=https://oss.oracle.com/ol8/debuginfo/
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
    gpgcheck=1
    enabled=1
    EOF'
  • Install kernel debuginfo rpms (optional) : skip this if you do not intend to trace kernel probes/functions

    $ sudo dnf install kernel-uek-devel-$(uname -r) kernel-uek-debuginfo-$(uname -r) kernel-uek-debuginfo-common-$(uname -r) -y
  • Confirm that kernel-uek-* rpms of same version as $(uname -r) are installed

    $ rpm -qa | grep kernel-uek
  • Install libvirt debuginfo rpms to enable more probes

    $ sudo dnf install libvirt\*debug\*7.10.0-3\* -y
    
    # OR (if the above command failed with metadata errors)
    $ sudo dnf download libvirt\*debug\*7.10.0-3\* --downloaddir /tmp/libvirt-rpms
    $ sudo rpm -ivh /tmp/libvirt-rpms/libvirt*debug*7.10.0-3*.rpm

    If debuginfo rpms of your specific libvirt version are not available in the public repo, file a SR (support request) with oracle support (https://support.oracle.com) to get debuginfo rpms for your specific version.

    If you cannot wait that long, then you can do a libvirt build and/or make the probes visible to systemtap based on the build method.

Step 7 : Install systemtap

$ sudo dnf install systemtap systemtap-client systemtap-runtime systemtap-devel -y

Step 8 : Verify that system is prepared for systemtap tracing

$ stap-prep
Configuring for kernel release 5.4.17-2136.325.5.el8uek.x86_64
Debuginfo automatic downloading is not configured via $DEBUGINFOD_URLS

We can also pass the kernel version as an argument to the stap-prep script for downloading a different kernel version

There were no errors in above command, so we are good to proceed further. We could have avoided the above warning by setting this environment variable :

$ export DEBUGINFOD_URLS="https://oss.oracle.com/ol8/debuginfo/"

Step 9 : Verify that libvirtd is running

$ sudo systemctl start libvirtd
# OR
$ sudo systemctl enable libvirtd --now

Step 10 : List all libvirt probes available to systemtap

$ stap -L 'libvirt.**'
<snip>
libvirt.qemu.monitor_new mon:long fd:long $arg1:long $arg2:long
libvirt.qemu.monitor_recv_event mon:long event:string $arg1:long $arg2:long
libvirt.qemu.monitor_recv_reply mon:long reply:string $arg1:long $arg2:long
libvirt.qemu.monitor_send_msg mon:long msg:string fd:long $arg1:long $arg2:long $arg3:long
<snip>

The above list can be smaller than the list of probes available in /usr/share/systemtap/tapset/libvirt_*

This can happen due to missing libvirt-debuginfo rpms or a version mismatch with the installed debuginfo rpms.

Now verify the version of the installed libvirt debuginfo rpms is the same as the version of the installed libvirt rpms.

$ rpm -qa | grep libvirt
$ rpm -qa | grep libvirt | grep debuginfo

Step 11 : Check that systemtap is working properly

$ sudo stap -e 'probe oneshot{ printf("hello world\n")}'

A simple systemtap script

Let’s see a very simple “hello world” type systemtap script for tracing libvirtd. (systemtap scripts have “.stp” extension in their filename)

# qmp.stp
probe begin {
  printf("Start tracing\n")
}

probe libvirt.qemu.monitor_send_msg {
  printf("QMPs: %s", msg);
}

Now, let’s break down what you see in the script above which has 2 probes(/events) :

  • begin : this is simply the beginning of execution of our script, on which “Start tracing” is printed using printf to the console’s standard output.

  • libvirt.qemu.monitor_send_msg : When libvirt sends a message via qemu monitor, this probe/event is triggered and we can print the ‘msg’ QMP parameter from the output of this probe to standard console to see what QMP command was sent from libvirt to qemu.

You must be wondering how the ‘msg’ parameter came into the picture all of a sudden and how I came to know about it.

The contents of the ‘msg’ data is actually obtained as an output of the probe and you can check all the outputs from libvirt.qemu.monitor_send_msg probe by looking into the following file :

# /usr/share/systemtap/tapset/libvirt_qemu_probes-64.stp
...
probe libvirt.qemu.monitor_send_msg = process("/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so").mark("qemu_monitor_send_msg") {
  mon = $arg1;
  msg = user_string($arg2);
  fd = $arg3;
}
...

To read more about the structure of event probes, see : https://sourceware.org/systemtap/SystemTap_Beginners_Guide/userspace-probing.html

Usage

Let’s create a guest virtual machine and attach a disk to it while we parallely trace libvirt (libvirtd process) using systemtap.

Follow these steps to create VMs via virt-install : https://docs.oracle.com/en/operating-systems/oracle-linux/kvm-user/kvm-KVMUsage.html#basic-usage

Example :

# create ol7.qcow2 boot disk for a new guest VM
$ sudo qemu-img create -f qcow2 /var/lib/libvirt/images/ol7.qcow2 10G

# download OL7.9 ISO from https://yum.oracle.com/oracle-linux-isos.html
$ sudo wget https://yum.oracle.com/ISOS/OracleLinux/OL7/u9/x86_64/OracleLinux-R7-U9-Server-x86_64-dvd.iso -P /var/lib/libvirt/

# install virt-install
$ sudo dnf install virt-install -y

# create virtual machine named ol7
$ sudo virt-install --virt-type=kvm --name=ol7 --vcpus=2,maxvcpus=20 --memory=4096 --location /var/lib/libvirt/OracleLinux-R7-U9-Server-x86_64-dvd.iso --console pty,target_type=serial --disk path=/var/lib/libvirt/images/ol7.qcow2,format=qcow2,size=10 --network default --graphics none --noautoconsole

# install OS (OL7.9) on ol7 guest by following the text instructions (or optionally postpone it)
$ sudo virsh console ol7

# press 'Ctrl+]' to exit from guest's console back to your host

# check that VM is running
$ sudo virsh list --all

Now, let’s open two sessions/terminals on our host machine.

In the first session/terminal, we will run a virsh command (command line utility of libvirt) to do some operation on our guest ol7.

In the second session/terminal, we will run our systemtap script for tracing the libvirtd process.

Syntax for tracing using stap :

$ sudo stap $script_name.stp -x $pid_of_process_to_trace

If you prefer running stap without sudo, add your user to these group(s) : “stapusr”, “stapsys”, “stapdev”.

We need a virtual disk for one of the examples, so let’s create a virtual disk by using the qemu-img command :

$ sudo qemu-img create -f qcow2 /var/lib/libvirt/images/scsi-disk.qcow2 1G
  • $GUEST : name/ID/UUID of your virtual machine
  • $CURRENT_VCPU : current vCPUs (live) of your virtual machine

You can check the current vCPUs of your guest virtual machine by using this command :

$ sudo virsh vcpucount $GUEST
maximum      config        20
maximum      live          20
current      config        2
current      live          2

Now, let’s see some usage examples for tracing libvirtd process

1. Disk Hotplug

Attach a disk to a guest VM :

$ sudo virsh attach-device $GUEST scsi-disk.xml
# scsi-disk.xml
<disk type='file' device='disk'>
  <driver name='qemu' type='raw'/>
  <source file='/var/lib/libvirt/images/scsi-disk.qcow2'/>
  <backingStore/>
  <target dev='sdm' bus='scsi'/>
  <controller type='scsi' model='virtio-scsi'/>
</disk>
$ sudo stap qmp.stp -x $(pgrep libvirtd)
Start tracing
QMPs: {"execute":"blockdev-add","arguments":{"driver":"file","filename":"/var/lib/libvirt/images/scsi-disk.qcow2","node-name":"libvirt-23-storage","auto-read-only":true,"discard":"unmap"},"id":"libvirt-549"}
QMPs: {"execute":"blockdev-add","arguments":{"node-name":"libvirt-23-format","read-only":false,"driver":"raw","file":"libvirt-23-storage"},"id":"libvirt-550"}
QMPs: {"execute":"device_add","arguments":{"driver":"scsi-hd","bus":"scsi1.0","scsi-id":5,"device_id":"drive-scsi1-0-5","drive":"libvirt-23-format","id":"scsi1-0-5"},"id":"libvirt-551"}
QMPs: {"execute":"qom-list","arguments":{"path":"/machine/peripheral"},"id":"libvirt-552"}

2. Disk Hotunplug

Detach the disk from the guest VM :

$ sudo virsh detach-device $GUEST scsi-disk.xml
$ sudo stap qmp.stp -x $(pgrep libvirtd)
Start tracing
QMPs: {"execute":"device_del","arguments":{"id":"scsi1-0-5"},"id":"libvirt-414"}
QMPs: {"execute":"blockdev-del","arguments":{"node-name":"libvirt-5-format"},"id":"libvirt-415"}
QMPs: {"execute":"blockdev-del","arguments":{"node-name":"libvirt-5-storage"},"id":"libvirt-416"}
QMPs: {"execute":"qom-list","arguments":{"path":"/machine/peripheral"},"id":"libvirt-417"}

3. vCPU Hotplug

Attach a vCPU by increasing vCPU count by 1 :

$ sudo virsh setvcpus $GUEST (( $CURRENT_VCPU+1 ))
$ sudo stap qmp.stp -x $(pgrep libvirtd)
Start tracing
QMPs: {"execute":"device_add","arguments":{"driver":"Haswell-noTSX-IBRS-x86_64-cpu","id":"vcpu9","core-id":0,"thread-id":0,"socket-id":9},"id":"libvirt-399"}
QMPs: {"execute":"query-hotpluggable-cpus","id":"libvirt-400"}
QMPs: {"execute":"query-cpus-fast","id":"libvirt-401"}

4. vCPU Hotunplug

Detach a vCPU by decreasing vCPU count by 1 :

$ sudo virsh setvcpus $GUEST (( $CURRENT_VCPU-1 ))
$ sudo stap qmp.stp -x $(pgrep libvirtd)
Start tracing
QMPs: {"execute":"device_del","arguments":{"id":"vcpu9"},"id":"libvirt-402"}
QMPs: {"execute":"query-hotpluggable-cpus","id":"libvirt-403"}
QMPs: {"execute":"query-cpus-fast","id":"libvirt-404"}

5. Device block stats

Display block statistics of our running domain :

$ sudo virsh domblkstat $GUEST
 rd_req 70549
 rd_bytes 154394112
 wr_req 2665
 wr_bytes 7231488
 flush_operations 254
 rd_total_times 2065345419
 wr_total_times 195290762
 flush_total_times 55365761
$ sudo stap qmp.stp -x $(pgrep libvirtd)
QMPs: {"execute":"query-blockstats","id":"libvirt-405"}
QMPs: {"execute":"query-blockstats","arguments":{"query-nodes":true},"id":"libvirt-406"}
QMPs: {"execute":"query-blockstats","id":"libvirt-407"}
QMPs: {"execute":"query-blockstats","arguments":{"query-nodes":true},"id":"libvirt-408"}

Using scripts installed by libvirt and systemtap

There are some systemtap scripts pre-packaged with the libvirt-docs rpm that are ready-to-use for tracing.

There are also scripts available with the systemtap-client rpm that are ready-to-use for tracing.

If you have installed the above two rpms, then you already have these scripts installed on your system.

To verify that you have installed these rpms :

$ sudo dnf install libvirt-docs systemtap-client -y

You can see the example scripts available with the installed rpm like this :

$ rpm -ql libvirt-docs | grep examples/systemtap/
/usr/share/doc/libvirt-docs/examples/systemtap/events.stp
/usr/share/doc/libvirt-docs/examples/systemtap/lock-debug.stp
/usr/share/doc/libvirt-docs/examples/systemtap/qemu-monitor.stp
/usr/share/doc/libvirt-docs/examples/systemtap/rpc-monitor.stp
$ rpm -ql systemtap-client | grep examples/virtualization/ | grep stp
/usr/share/systemtap/examples/virtualization/kvm_service_time.stp
/usr/share/systemtap/examples/virtualization/qemu_count.stp
/usr/share/systemtap/examples/virtualization/qemu_io.stp

Let’s take one of these above scripts: qemu-monitor.stp as an example. For instance look at /usr/share/doc/libvirt-docs/examples/systemtap/qemu-monitor.stp This script monitors the messages received from the qemu monitor as well as the messages sent that we were tracing in our simple script.

$ sudo virsh attach-device $GUEST scsi-disk.xml
$ sudo stap /usr/share/doc/libvirt-docs/examples/systemtap/qemu-monitor.stp -x $(pgrep libvirtd)
  0.000 begin
 20.238 > 0x7fcddc1948b0 {"execute":"device_add","arguments":{"driver":"Haswell-noTSX-IBRS-x86_64-cpu","id":"vcpu9","core-id":0,"thread-id":0,"socket-id":9},"id":"libvirt-409"}
 20.247 ! 0x7fcddc1948b0 {"timestamp": {"seconds": 1700830017, "microseconds": 324418}, "event": "ACPI_DEVICE_OST", "data": {"info": {"device": "vcpu9", "source": 1, "status": 0, "slot": "9", "slot-type": "CPU"}}}
 20.291 < 0x7fcddc1948b0 {"return": {}, "id": "libvirt-409"}
 20.292 > 0x7fcddc1948b0 {"execute":"query-hotpluggable-cpus","id":"libvirt-410"}
 20.293 < 0x7fcddc1948b0 {"return": [{"props": {"core-id": 0, "thread-id": 0, "socket-id": 9}, "vcpus-count": 1, "qom-path": "/machine/peripheral/vcpu9", "type": "Haswell-noTSX-IBRS-x86_64-cpu"}, {"props": {"core-id": 0, "thread-id": 0, "socket-id": 8}, "vcpus-count": 1, "qom-path": "/machine/unattached/device[9]", "type": "Haswell-noTSX-IBRS-x86_64-cpu"}, {"props": {"core-id": 0, "thread-id": 0, "socket-id": 7}, "vcpus-count": 1, "qom-path": "/machine/unattached/device[8]", "type": "Haswell-noTSX-IBRS-x86_64-cpu"
 20.293 > 0x7fcddc1948b0 {"execute":"query-cpus-fast","id":"libvirt-411"}
 20.294 < 0x7fcddc1948b0 {"return": [{"thread-id": 167114, "props": {"core-id": 0, "thread-id": 0, "socket-id": 0}, "qom-path": "/machine/unattached/device[0]", "cpu-index": 0, "target": "x86_64"}, {"thread-id": 167117, "props": {"core-id": 0, "thread-id": 0, "socket-id": 1}, "qom-path": "/machine/unattached/device[2]", "cpu-index": 1, "target": "x86_64"}, {"thread-id": 167120, "props": {"core-id": 0, "thread-id": 0, "socket-id": 2}, "qom-path": "/machine/unattached/device[3]", "cpu-index": 2, "target": "x86_64"},

You can trace other operations in libvirt using sample scripts from the above list as a fun activity.

Printing the function stack trace using systemtap

Now, let’s see how we can use more advanced features while tracing libvirtd like printing userspace function stack trace.

$ sudo virsh list

Using the demo.stp script below, we can trace rpc calls from clients/users to libvirtd and print the userspace function trace when the probe is hit.

# demo.stp
probe libvirt.rpc.client_msg_rx {
  printf("client=%p len=%d program=%d version=%d procedure=%d type=%d status=%d serial=%d\n",
         client, len, prog, vers, proc, type, status, serial);
  print_ustack(ubacktrace())
}
$ sudo stap -DTASK_FINDER_VMA_ENTRY_ITEMS=7680 demo.stp
...
client=0x55ff7b8b2010 len=36 program=536903814 version=1 procedure=66 type=1 status=0 serial=0
 0x7fd42239b0b8 : virNetClientIOHandleInput+0x5a8/0x8e0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239be88 : virNetClientIOEventLoop+0x328/0x600 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239c309 : virNetClientSendInternal+0x1a9/0x4e0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239dc27 : virNetClientSendWithReply+0x27/0x40 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239868f : virNetClientProgramCall+0x2cf/0x830 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4223e5c8b : callFull.isra.3+0x7b/0xc0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd422401b5e : doRemoteOpen+0xc7e/0x2630 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd422403600 : remoteConnectOpen+0xf0/0x240 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4224955bc : virConnectOpenInternal+0x77c/0xe50 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4224960c4 : virConnectOpenAuth+0x74/0x90 [/usr/lib64/libvirt.so.0.7010.0]
 0x55ff7a2a3fc4 : 0x55ff7a2a3fc4 [/usr/bin/virsh+0x3dfc4/0xbe000]
client=0x55ff7b8b2010 len=32 program=536903814 version=1 procedure=60 type=1 status=0 serial=1
 0x7fd42239b0b8 : virNetClientIOHandleInput+0x5a8/0x8e0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239be88 : virNetClientIOEventLoop+0x328/0x600 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239c309 : virNetClientSendInternal+0x1a9/0x4e0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239dc27 : virNetClientSendWithReply+0x27/0x40 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239868f : virNetClientProgramCall+0x2cf/0x830 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4223e5c8b : callFull.isra.3+0x7b/0xc0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4223ed85b : remoteConnectSupportsFeatureUnlocked.isra.5+0x4b/0x80 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42240215d : doRemoteOpen+0x127d/0x2630 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd422403600 : remoteConnectOpen+0xf0/0x240 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4224955bc : virConnectOpenInternal+0x77c/0xe50 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4224960c4 : virConnectOpenAuth+0x74/0x90 [/usr/lib64/libvirt.so.0.7010.0]
 0x55ff7a2a3fc4 : 0x55ff7a2a3fc4 [/usr/bin/virsh+0x3dfc4/0xbe000]
client=0x55ff7b8b2010 len=28 program=536903814 version=1 procedure=1 type=1 status=0 serial=2
 0x7fd42239b0b8 : virNetClientIOHandleInput+0x5a8/0x8e0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239be88 : virNetClientIOEventLoop+0x328/0x600 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239c309 : virNetClientSendInternal+0x1a9/0x4e0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239dc27 : virNetClientSendWithReply+0x27/0x40 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239868f : virNetClientProgramCall+0x2cf/0x830 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4223e5c8b : callFull.isra.3+0x7b/0xc0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4224021f0 : doRemoteOpen+0x1310/0x2630 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd422403600 : remoteConnectOpen+0xf0/0x240 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4224955bc : virConnectOpenInternal+0x77c/0xe50 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4224960c4 : virConnectOpenAuth+0x74/0x90 [/usr/lib64/libvirt.so.0.7010.0]
 0x55ff7a2a3fc4 : 0x55ff7a2a3fc4 [/usr/bin/virsh+0x3dfc4/0xbe000]
client=0x55ff7b8b2010 len=48 program=536903814 version=1 procedure=110 type=1 status=0 serial=3
 0x7fd42239b0b8 : virNetClientIOHandleInput+0x5a8/0x8e0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239be88 : virNetClientIOEventLoop+0x328/0x600 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239c309 : virNetClientSendInternal+0x1a9/0x4e0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239dc27 : virNetClientSendWithReply+0x27/0x40 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239868f : virNetClientProgramCall+0x2cf/0x830 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4223e5c8b : callFull.isra.3+0x7b/0xc0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42240232d : doRemoteOpen+0x144d/0x2630 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd422403600 : remoteConnectOpen+0xf0/0x240 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4224955bc : virConnectOpenInternal+0x77c/0xe50 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4224960c4 : virConnectOpenAuth+0x74/0x90 [/usr/lib64/libvirt.so.0.7010.0]
 0x55ff7a2a3fc4 : 0x55ff7a2a3fc4 [/usr/bin/virsh+0x3dfc4/0xbe000]
client=0x55ff7b8b2010 len=32 program=536903814 version=1 procedure=60 type=1 status=0 serial=4
 0x7fd42239b0b8 : virNetClientIOHandleInput+0x5a8/0x8e0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239be88 : virNetClientIOEventLoop+0x328/0x600 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239c309 : virNetClientSendInternal+0x1a9/0x4e0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239dc27 : virNetClientSendWithReply+0x27/0x40 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239868f : virNetClientProgramCall+0x2cf/0x830 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4223e5c8b : callFull.isra.3+0x7b/0xc0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4223ed85b : remoteConnectSupportsFeatureUnlocked.isra.5+0x4b/0x80 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42240222b : doRemoteOpen+0x134b/0x2630 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd422403600 : remoteConnectOpen+0xf0/0x240 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4224955bc : virConnectOpenInternal+0x77c/0xe50 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4224960c4 : virConnectOpenAuth+0x74/0x90 [/usr/lib64/libvirt.so.0.7010.0]
 0x55ff7a2a3fc4 : 0x55ff7a2a3fc4 [/usr/bin/virsh+0x3dfc4/0xbe000]
client=0x55ff7b8b2010 len=32 program=536903814 version=1 procedure=60 type=1 status=0 serial=5
 0x7fd42239b0b8 : virNetClientIOHandleInput+0x5a8/0x8e0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239be88 : virNetClientIOEventLoop+0x328/0x600 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239c309 : virNetClientSendInternal+0x1a9/0x4e0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239dc27 : virNetClientSendWithReply+0x27/0x40 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239868f : virNetClientProgramCall+0x2cf/0x830 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4223e5c8b : callFull.isra.3+0x7b/0xc0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4223ed85b : remoteConnectSupportsFeatureUnlocked.isra.5+0x4b/0x80 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd422402277 : doRemoteOpen+0x1397/0x2630 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd422403600 : remoteConnectOpen+0xf0/0x240 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4224955bc : virConnectOpenInternal+0x77c/0xe50 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4224960c4 : virConnectOpenAuth+0x74/0x90 [/usr/lib64/libvirt.so.0.7010.0]
 0x55ff7a2a3fc4 : 0x55ff7a2a3fc4 [/usr/bin/virsh+0x3dfc4/0xbe000]
WARNING: Missing unwind data for a module, rerun with 'stap -d /usr/bin/virsh'
client=0x55ff7b8b2010 len=28 program=536903814 version=1 procedure=360 type=1 status=0 serial=6
 0x7fd42239b0b8 : virNetClientIOHandleInput+0x5a8/0x8e0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239be88 : virNetClientIOEventLoop+0x328/0x600 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239c309 : virNetClientSendInternal+0x1a9/0x4e0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239dc27 : virNetClientSendWithReply+0x27/0x40 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239868f : virNetClientProgramCall+0x2cf/0x830 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4223e5c8b : callFull.isra.3+0x7b/0xc0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4223ed47a : remoteConnectRegisterCloseCallback+0xaa/0x120 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4224c07c0 : virConnectRegisterCloseCallback+0x90/0x140 [/usr/lib64/libvirt.so.0.7010.0]
 0x55ff7a2a4201 : 0x55ff7a2a4201 [/usr/bin/virsh+0x3e201/0xbe000]
client=0x55ff7b8b2010 len=36 program=536903814 version=1 procedure=273 type=1 status=0 serial=7
 0x7fd42239b0b8 : virNetClientIOHandleInput+0x5a8/0x8e0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239be88 : virNetClientIOEventLoop+0x328/0x600 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239c309 : virNetClientSendInternal+0x1a9/0x4e0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239dc27 : virNetClientSendWithReply+0x27/0x40 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239868f : virNetClientProgramCall+0x2cf/0x830 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4223e5c8b : callFull.isra.3+0x7b/0xc0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4223fe3d5 : remoteConnectListAllDomains+0x95/0x210 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4224a98bc : virConnectListAllDomains+0x8c/0x100 [/usr/lib64/libvirt.so.0.7010.0]
 0x55ff7a2cfd11 : 0x55ff7a2cfd11 [/usr/bin/virsh+0x69d11/0xbe000]
client=0x55ff7b8b2010 len=28 program=536903814 version=1 procedure=361 type=1 status=0 serial=8
 0x7fd42239b0b8 : virNetClientIOHandleInput+0x5a8/0x8e0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239be88 : virNetClientIOEventLoop+0x328/0x600 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239c309 : virNetClientSendInternal+0x1a9/0x4e0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239dc27 : virNetClientSendWithReply+0x27/0x40 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239868f : virNetClientProgramCall+0x2cf/0x830 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4223e5c8b : callFull.isra.3+0x7b/0xc0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4223ed362 : remoteConnectUnregisterCloseCallback+0x82/0xf0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4224c08f0 : virConnectUnregisterCloseCallback+0x80/0x130 [/usr/lib64/libvirt.so.0.7010.0]
 0x55ff7a2a3e2e : 0x55ff7a2a3e2e [/usr/bin/virsh+0x3de2e/0xbe000]
client=0x55ff7b8b2010 len=28 program=536903814 version=1 procedure=2 type=1 status=0 serial=9
 0x7fd42239b0b8 : virNetClientIOHandleInput+0x5a8/0x8e0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239be88 : virNetClientIOEventLoop+0x328/0x600 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239c309 : virNetClientSendInternal+0x1a9/0x4e0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239dc27 : virNetClientSendWithReply+0x27/0x40 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd42239868f : virNetClientProgramCall+0x2cf/0x830 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4223e5c8b : callFull.isra.3+0x7b/0xc0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd422400881 : remoteConnectClose+0x61/0x170 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4224d52da : virConnectDispose+0x2a/0x40 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd4222c17be : vir_object_finalize+0x7e/0xb0 [/usr/lib64/libvirt.so.0.7010.0]
 0x7fd420c768a9 : 0x7fd420c768a9 [/usr/lib64/libgobject-2.0.so.0.5600.4+0x158a9/0x253000]
WARNING: Missing unwind data for a module, rerun with 'stap -d /usr/lib64/libgobject-2.0.so.0.5600.4'

We need to use the “-DTASK_FINDER_VMA_ENTRY_ITEMS=7680” option with stap while printing userspace backtrace because of a bug in systemtap runtime which is fixed in later versions.

If you are wondering where print_ustack() came from and where you can check other functions like that, feel free to browse under /usr/share/systemtap/tapset/ locally OR https://sourceware.org/systemtap/tapsets/index.html

print_ustack() specifically is defined in : /usr/share/systemtap/tapset/linux/ucontext-symbols.stp

Conclusion

On an ending note, try writing your own systemtap script using different probes available in libvirt for any use case that you may be interested in tracing.

If you face any troubles with tracing, feel free to file a SR (Support Request) with oracle support (https://support.oracle.com) and we will help you out.

I hope that you liked this article and if you did, don’t forget to share it with others who may find it useful as well.

References

Shaleen Bathla


Previous Post

Guarding beyond the perimeter—Zero Trust security with Oracle Linux

Gursewak Sokhi | 2 min read

Next Post


Building the Linux Kernel with LLVM on Oracle Linux 8

Dongli Zhang | 2 min read