Introduction
systemtap is a scripting language and a tool for dynamically tracing/probing activities in Linux kernel/userspace.
In context of its usage with libvirt, we will use systemtap to dynamically trace libvirt (userspace) in this article.
We will use libvirt-7.10.0-3 in this article but you may also try with other libvirt versions.
Setting up the playground
Step 1 : Create an OCI BM
Go to https://www.oracle.com/cloud/ and spin up a BM (Bare Metal) instance with the Oracle Linux 8 Operating System image.
You can also start using the command line utility yo (https://github.com/oracle/yo) to easily create and manage your OCI instances.
Step 2 : Install libvirt from ULN
# Enable kvm appstream repo $ sudo dnf config-manager --enable ol8_kvm_appstream # Enable the virt:kvm_utils2 module and stream $ sudo dnf module enable virt:kvm_utils2 -y # Install base packages from the virt:kvm_utils2 stream $ sudo dnf module install virt:kvm_utils2 -y # Install all libvirt rpms $ sudo dnf install libvirt\*7.10.0-3\* -y
Step 3 : UEK and Syncing with ULN
In this blog, we will use UEK6 (Unbreakable Enterprise Kernel 6), but you can also follow along with other UEK versions. Read more about UEK here https://docs.oracle.com/en/operating-systems/uek/.
# To see all installed UEK versions $ sudo rpm -qa | grep kernel-uek # To see the running kernel version $ uname -r # To see the effective running kernel version (if you have ksplice) $ uptrack-uname -r
For reading more about ksplice and uptrack, check out https://ksplice.oracle.com.
Now, let’s sync our Oracle Linux distro with the default ULN (Unbreakable Linux Network) repositories in order to get the latest updates:
# sync all repos and perform all required upgrades/downgrades $ sudo dnf distro-sync --allowerasing -y # Reboot to apply changes (especially if a newer kernel version is available) $ sudo reboot
Step 4 : Check dtrace support (if following with non-oracle libvirt)
By default, Oracle Linux libvirt rpms are built with dtrace enabled, so if you are using the official libvirt rpms from ULN repo, you are already good to go.
If you have custom libvirt rpms or trying with some other distro, check that libvirt is built with dtrace to support tracing in systemtap. We can enable dtrace support during the build as follows :
For libvirt < 6.7.0 :
$ mkdir build && cd build # to keep build separate from source code repo $ ../autogen.sh --with-dtrace $ make
For libvirt >= 6.7.0 :
$ meson build -Ddtrace=enabled $ ninja -C build
If you do not know about the options configured while building your libvirt rpms and wish to know if dtrace support was enabled, you can install the rpms and check whether libvirt probes are available as shown in next step.
Step 5 : Check that libvirt probes are now available
libvirt probes are the events/functions available for us to trace in libvirt. You can think of them as functions in libvirt source code that are available for us to trace.
The userspace markers/probes are installed by the libvirt-client rpm when systemtap-client is available.
If /usr/share/systemtap/tapset/libvirt_* are present, then libvirt probes are available.
You can use the following command to see the probe files :
$ ls /usr/share/systemtap/tapset/ | grep libvirt libvirt_functions.stp libvirt_probes-64.stp libvirt_qemu_probes-64.stp
To list all libvirt probes :
$ grep 'probe libvirt.[a-z_0-9.]*' /usr/share/systemtap/tapset/libvirt_* -o|cut -f 2 -d : probe libvirt.event_glib.add_handle probe libvirt.event_glib.update_handle probe libvirt.event_glib.remove_handle probe libvirt.event_glib.remove_handle_idle probe libvirt.event_glib.dispatch_handle probe libvirt.event_glib.add_timeout probe libvirt.event_glib.update_timeout probe libvirt.event_glib.remove_timeout probe libvirt.event_glib.remove_timeout_idle probe libvirt.event_glib.dispatch_timeout probe libvirt.object.new probe libvirt.object.ref probe libvirt.object.unref probe libvirt.object.dispose probe libvirt.rpc.socket_new probe libvirt.rpc.socket_dispose probe libvirt.rpc.socket_send_fd probe libvirt.rpc.socket_recv_fd probe libvirt.rpc.server_client_new probe libvirt.rpc.server_client_dispose probe libvirt.rpc.server_client_msg_tx_queue probe libvirt.rpc.server_client_msg_rx probe libvirt.rpc.client_new probe libvirt.rpc.client_dispose probe libvirt.rpc.client_msg_tx_queue probe libvirt.rpc.client_msg_rx probe libvirt.rpc.server_client_auth_allow probe libvirt.rpc.server_client_auth_deny probe libvirt.rpc.server_client_auth_fail probe libvirt.rpc.tls_context_new probe libvirt.rpc.tls_context_dispose probe libvirt.rpc.tls_context_session_allow probe libvirt.rpc.tls_context_session_deny probe libvirt.rpc.tls_context_session_fail probe libvirt.rpc.tls_session_new probe libvirt.rpc.tls_session_dispose probe libvirt.rpc.tls_session_handshake_pass probe libvirt.rpc.tls_session_handshake_fail probe libvirt.rpc.keepalive_new probe libvirt.rpc.keepalive_dispose probe libvirt.rpc.keepalive_start probe libvirt.rpc.keepalive_stop probe libvirt.rpc.keepalive_send probe libvirt.rpc.keepalive_received probe libvirt.rpc.keepalive_timeout probe libvirt.qemu.monitor_new probe libvirt.qemu.monitor_ref probe libvirt.qemu.monitor_unref probe libvirt.qemu.monitor_close probe libvirt.qemu.monitor_send_msg probe libvirt.qemu.monitor_recv_reply probe libvirt.qemu.monitor_recv_event probe libvirt.qemu.monitor_io_process probe libvirt.qemu.monitor_io_read probe libvirt.qemu.monitor_io_write probe libvirt.qemu.monitor_io_send_fd
Step 6 : Install debuginfo rpms
-
Create/Enable debuginfo yum repo
$ sudo bash -c 'cat > /etc/yum.repos.d/ol8_debuginfo.repo << EOF [ol8_debuginfo] name=Debuginfo for Oracle Linux $releasever ($basearch) baseurl=https://oss.oracle.com/ol8/debuginfo/ gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle gpgcheck=1 enabled=1 EOF'
-
Install kernel debuginfo rpms (optional) : skip this if you do not intend to trace kernel probes/functions
$ sudo dnf install kernel-uek-devel-$(uname -r) kernel-uek-debuginfo-$(uname -r) kernel-uek-debuginfo-common-$(uname -r) -y
-
Confirm that kernel-uek-* rpms of same version as $(uname -r) are installed
$ rpm -qa | grep kernel-uek
-
Install libvirt debuginfo rpms to enable more probes
$ sudo dnf install libvirt\*debug\*7.10.0-3\* -y # OR (if the above command failed with metadata errors) $ sudo dnf download libvirt\*debug\*7.10.0-3\* --downloaddir /tmp/libvirt-rpms $ sudo rpm -ivh /tmp/libvirt-rpms/libvirt*debug*7.10.0-3*.rpm
If debuginfo rpms of your specific libvirt version are not available in the public repo, file a SR (support request) with oracle support (https://support.oracle.com) to get debuginfo rpms for your specific version.
If you cannot wait that long, then you can do a libvirt build and/or make the probes visible to systemtap based on the build method.
Step 7 : Install systemtap
$ sudo dnf install systemtap systemtap-client systemtap-runtime systemtap-devel -y
Step 8 : Verify that system is prepared for systemtap tracing
$ stap-prep Configuring for kernel release 5.4.17-2136.325.5.el8uek.x86_64 Debuginfo automatic downloading is not configured via $DEBUGINFOD_URLS
We can also pass the kernel version as an argument to the stap-prep script for downloading a different kernel version
There were no errors in above command, so we are good to proceed further. We could have avoided the above warning by setting this environment variable :
$ export DEBUGINFOD_URLS="https://oss.oracle.com/ol8/debuginfo/"
Step 9 : Verify that libvirtd is running
$ sudo systemctl start libvirtd # OR $ sudo systemctl enable libvirtd --now
Step 10 : List all libvirt probes available to systemtap
$ stap -L 'libvirt.**' <snip> libvirt.qemu.monitor_new mon:long fd:long $arg1:long $arg2:long libvirt.qemu.monitor_recv_event mon:long event:string $arg1:long $arg2:long libvirt.qemu.monitor_recv_reply mon:long reply:string $arg1:long $arg2:long libvirt.qemu.monitor_send_msg mon:long msg:string fd:long $arg1:long $arg2:long $arg3:long <snip>
The above list can be smaller than the list of probes available in /usr/share/systemtap/tapset/libvirt_*
This can happen due to missing libvirt-debuginfo rpms or a version mismatch with the installed debuginfo rpms.
Now verify the version of the installed libvirt debuginfo rpms is the same as the version of the installed libvirt rpms.
$ rpm -qa | grep libvirt $ rpm -qa | grep libvirt | grep debuginfo
Step 11 : Check that systemtap is working properly
$ sudo stap -e 'probe oneshot{ printf("hello world\n")}'
A simple systemtap script
Let’s see a very simple “hello world” type systemtap script for tracing libvirtd. (systemtap scripts have “.stp” extension in their filename)
# qmp.stp
probe begin {
printf("Start tracing\n")
}
probe libvirt.qemu.monitor_send_msg {
printf("QMPs: %s", msg);
}
Now, let’s break down what you see in the script above which has 2 probes(/events) :
-
begin : this is simply the beginning of execution of our script, on which “Start tracing” is printed using printf to the console’s standard output.
-
libvirt.qemu.monitor_send_msg : When libvirt sends a message via qemu monitor, this probe/event is triggered and we can print the ‘msg’ QMP parameter from the output of this probe to standard console to see what QMP command was sent from libvirt to qemu.
You must be wondering how the ‘msg’ parameter came into the picture all of a sudden and how I came to know about it.
The contents of the ‘msg’ data is actually obtained as an output of the probe and you can check all the outputs from libvirt.qemu.monitor_send_msg probe by looking into the following file :
# /usr/share/systemtap/tapset/libvirt_qemu_probes-64.stp
...
probe libvirt.qemu.monitor_send_msg = process("/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so").mark("qemu_monitor_send_msg") {
mon = $arg1;
msg = user_string($arg2);
fd = $arg3;
}
...
To read more about the structure of event probes, see : https://sourceware.org/systemtap/SystemTap_Beginners_Guide/userspace-probing.html
Usage
Let’s create a guest virtual machine and attach a disk to it while we parallely trace libvirt (libvirtd process) using systemtap.
Follow these steps to create VMs via virt-install : https://docs.oracle.com/en/operating-systems/oracle-linux/kvm-user/kvm-KVMUsage.html#basic-usage
Example :
# create ol7.qcow2 boot disk for a new guest VM $ sudo qemu-img create -f qcow2 /var/lib/libvirt/images/ol7.qcow2 10G # download OL7.9 ISO from https://yum.oracle.com/oracle-linux-isos.html $ sudo wget https://yum.oracle.com/ISOS/OracleLinux/OL7/u9/x86_64/OracleLinux-R7-U9-Server-x86_64-dvd.iso -P /var/lib/libvirt/ # install virt-install $ sudo dnf install virt-install -y # create virtual machine named ol7 $ sudo virt-install --virt-type=kvm --name=ol7 --vcpus=2,maxvcpus=20 --memory=4096 --location /var/lib/libvirt/OracleLinux-R7-U9-Server-x86_64-dvd.iso --console pty,target_type=serial --disk path=/var/lib/libvirt/images/ol7.qcow2,format=qcow2,size=10 --network default --graphics none --noautoconsole # install OS (OL7.9) on ol7 guest by following the text instructions (or optionally postpone it) $ sudo virsh console ol7 # press 'Ctrl+]' to exit from guest's console back to your host # check that VM is running $ sudo virsh list --all
Now, let’s open two sessions/terminals on our host machine.
In the first session/terminal, we will run a virsh command (command line utility of libvirt) to do some operation on our guest ol7.
In the second session/terminal, we will run our systemtap script for tracing the libvirtd process.
Syntax for tracing using stap :
$ sudo stap $script_name.stp -x $pid_of_process_to_trace
If you prefer running stap without sudo, add your user to these group(s) : “stapusr”, “stapsys”, “stapdev”.
We need a virtual disk for one of the examples, so let’s create a virtual disk by using the qemu-img command :
$ sudo qemu-img create -f qcow2 /var/lib/libvirt/images/scsi-disk.qcow2 1G
- $GUEST : name/ID/UUID of your virtual machine
- $CURRENT_VCPU : current vCPUs (live) of your virtual machine
You can check the current vCPUs of your guest virtual machine by using this command :
$ sudo virsh vcpucount $GUEST maximum config 20 maximum live 20 current config 2 current live 2
Now, let’s see some usage examples for tracing libvirtd process
1. Disk Hotplug
Attach a disk to a guest VM :
$ sudo virsh attach-device $GUEST scsi-disk.xml
# scsi-disk.xml <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/scsi-disk.qcow2'/> <backingStore/> <target dev='sdm' bus='scsi'/> <controller type='scsi' model='virtio-scsi'/> </disk>
$ sudo stap qmp.stp -x $(pgrep libvirtd)
Start tracing
QMPs: {"execute":"blockdev-add","arguments":{"driver":"file","filename":"/var/lib/libvirt/images/scsi-disk.qcow2","node-name":"libvirt-23-storage","auto-read-only":true,"discard":"unmap"},"id":"libvirt-549"}
QMPs: {"execute":"blockdev-add","arguments":{"node-name":"libvirt-23-format","read-only":false,"driver":"raw","file":"libvirt-23-storage"},"id":"libvirt-550"}
QMPs: {"execute":"device_add","arguments":{"driver":"scsi-hd","bus":"scsi1.0","scsi-id":5,"device_id":"drive-scsi1-0-5","drive":"libvirt-23-format","id":"scsi1-0-5"},"id":"libvirt-551"}
QMPs: {"execute":"qom-list","arguments":{"path":"/machine/peripheral"},"id":"libvirt-552"}
2. Disk Hotunplug
Detach the disk from the guest VM :
$ sudo virsh detach-device $GUEST scsi-disk.xml
$ sudo stap qmp.stp -x $(pgrep libvirtd)
Start tracing
QMPs: {"execute":"device_del","arguments":{"id":"scsi1-0-5"},"id":"libvirt-414"}
QMPs: {"execute":"blockdev-del","arguments":{"node-name":"libvirt-5-format"},"id":"libvirt-415"}
QMPs: {"execute":"blockdev-del","arguments":{"node-name":"libvirt-5-storage"},"id":"libvirt-416"}
QMPs: {"execute":"qom-list","arguments":{"path":"/machine/peripheral"},"id":"libvirt-417"}
3. vCPU Hotplug
Attach a vCPU by increasing vCPU count by 1 :
$ sudo virsh setvcpus $GUEST (( $CURRENT_VCPU+1 ))
$ sudo stap qmp.stp -x $(pgrep libvirtd)
Start tracing
QMPs: {"execute":"device_add","arguments":{"driver":"Haswell-noTSX-IBRS-x86_64-cpu","id":"vcpu9","core-id":0,"thread-id":0,"socket-id":9},"id":"libvirt-399"}
QMPs: {"execute":"query-hotpluggable-cpus","id":"libvirt-400"}
QMPs: {"execute":"query-cpus-fast","id":"libvirt-401"}
4. vCPU Hotunplug
Detach a vCPU by decreasing vCPU count by 1 :
$ sudo virsh setvcpus $GUEST (( $CURRENT_VCPU-1 ))
$ sudo stap qmp.stp -x $(pgrep libvirtd)
Start tracing
QMPs: {"execute":"device_del","arguments":{"id":"vcpu9"},"id":"libvirt-402"}
QMPs: {"execute":"query-hotpluggable-cpus","id":"libvirt-403"}
QMPs: {"execute":"query-cpus-fast","id":"libvirt-404"}
5. Device block stats
Display block statistics of our running domain :
$ sudo virsh domblkstat $GUEST rd_req 70549 rd_bytes 154394112 wr_req 2665 wr_bytes 7231488 flush_operations 254 rd_total_times 2065345419 wr_total_times 195290762 flush_total_times 55365761
$ sudo stap qmp.stp -x $(pgrep libvirtd)
QMPs: {"execute":"query-blockstats","id":"libvirt-405"}
QMPs: {"execute":"query-blockstats","arguments":{"query-nodes":true},"id":"libvirt-406"}
QMPs: {"execute":"query-blockstats","id":"libvirt-407"}
QMPs: {"execute":"query-blockstats","arguments":{"query-nodes":true},"id":"libvirt-408"}
Using scripts installed by libvirt and systemtap
There are some systemtap scripts pre-packaged with the libvirt-docs rpm that are ready-to-use for tracing.
There are also scripts available with the systemtap-client rpm that are ready-to-use for tracing.
If you have installed the above two rpms, then you already have these scripts installed on your system.
To verify that you have installed these rpms :
$ sudo dnf install libvirt-docs systemtap-client -y
You can see the example scripts available with the installed rpm like this :
$ rpm -ql libvirt-docs | grep examples/systemtap/ /usr/share/doc/libvirt-docs/examples/systemtap/events.stp /usr/share/doc/libvirt-docs/examples/systemtap/lock-debug.stp /usr/share/doc/libvirt-docs/examples/systemtap/qemu-monitor.stp /usr/share/doc/libvirt-docs/examples/systemtap/rpc-monitor.stp
$ rpm -ql systemtap-client | grep examples/virtualization/ | grep stp /usr/share/systemtap/examples/virtualization/kvm_service_time.stp /usr/share/systemtap/examples/virtualization/qemu_count.stp /usr/share/systemtap/examples/virtualization/qemu_io.stp
Let’s take one of these above scripts: qemu-monitor.stp as an example. For instance look at /usr/share/doc/libvirt-docs/examples/systemtap/qemu-monitor.stp This script monitors the messages received from the qemu monitor as well as the messages sent that we were tracing in our simple script.
$ sudo virsh attach-device $GUEST scsi-disk.xml
$ sudo stap /usr/share/doc/libvirt-docs/examples/systemtap/qemu-monitor.stp -x $(pgrep libvirtd)
0.000 begin
20.238 > 0x7fcddc1948b0 {"execute":"device_add","arguments":{"driver":"Haswell-noTSX-IBRS-x86_64-cpu","id":"vcpu9","core-id":0,"thread-id":0,"socket-id":9},"id":"libvirt-409"}
20.247 ! 0x7fcddc1948b0 {"timestamp": {"seconds": 1700830017, "microseconds": 324418}, "event": "ACPI_DEVICE_OST", "data": {"info": {"device": "vcpu9", "source": 1, "status": 0, "slot": "9", "slot-type": "CPU"}}}
20.291 < 0x7fcddc1948b0 {"return": {}, "id": "libvirt-409"}
20.292 > 0x7fcddc1948b0 {"execute":"query-hotpluggable-cpus","id":"libvirt-410"}
20.293 < 0x7fcddc1948b0 {"return": [{"props": {"core-id": 0, "thread-id": 0, "socket-id": 9}, "vcpus-count": 1, "qom-path": "/machine/peripheral/vcpu9", "type": "Haswell-noTSX-IBRS-x86_64-cpu"}, {"props": {"core-id": 0, "thread-id": 0, "socket-id": 8}, "vcpus-count": 1, "qom-path": "/machine/unattached/device[9]", "type": "Haswell-noTSX-IBRS-x86_64-cpu"}, {"props": {"core-id": 0, "thread-id": 0, "socket-id": 7}, "vcpus-count": 1, "qom-path": "/machine/unattached/device[8]", "type": "Haswell-noTSX-IBRS-x86_64-cpu"
20.293 > 0x7fcddc1948b0 {"execute":"query-cpus-fast","id":"libvirt-411"}
20.294 < 0x7fcddc1948b0 {"return": [{"thread-id": 167114, "props": {"core-id": 0, "thread-id": 0, "socket-id": 0}, "qom-path": "/machine/unattached/device[0]", "cpu-index": 0, "target": "x86_64"}, {"thread-id": 167117, "props": {"core-id": 0, "thread-id": 0, "socket-id": 1}, "qom-path": "/machine/unattached/device[2]", "cpu-index": 1, "target": "x86_64"}, {"thread-id": 167120, "props": {"core-id": 0, "thread-id": 0, "socket-id": 2}, "qom-path": "/machine/unattached/device[3]", "cpu-index": 2, "target": "x86_64"},
You can trace other operations in libvirt using sample scripts from the above list as a fun activity.
Printing the function stack trace using systemtap
Now, let’s see how we can use more advanced features while tracing libvirtd like printing userspace function stack trace.
$ sudo virsh list
Using the demo.stp script below, we can trace rpc calls from clients/users to libvirtd and print the userspace function trace when the probe is hit.
# demo.stp
probe libvirt.rpc.client_msg_rx {
printf("client=%p len=%d program=%d version=%d procedure=%d type=%d status=%d serial=%d\n",
client, len, prog, vers, proc, type, status, serial);
print_ustack(ubacktrace())
}
$ sudo stap -DTASK_FINDER_VMA_ENTRY_ITEMS=7680 demo.stp ... client=0x55ff7b8b2010 len=36 program=536903814 version=1 procedure=66 type=1 status=0 serial=0 0x7fd42239b0b8 : virNetClientIOHandleInput+0x5a8/0x8e0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239be88 : virNetClientIOEventLoop+0x328/0x600 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239c309 : virNetClientSendInternal+0x1a9/0x4e0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239dc27 : virNetClientSendWithReply+0x27/0x40 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239868f : virNetClientProgramCall+0x2cf/0x830 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4223e5c8b : callFull.isra.3+0x7b/0xc0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd422401b5e : doRemoteOpen+0xc7e/0x2630 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd422403600 : remoteConnectOpen+0xf0/0x240 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4224955bc : virConnectOpenInternal+0x77c/0xe50 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4224960c4 : virConnectOpenAuth+0x74/0x90 [/usr/lib64/libvirt.so.0.7010.0] 0x55ff7a2a3fc4 : 0x55ff7a2a3fc4 [/usr/bin/virsh+0x3dfc4/0xbe000] client=0x55ff7b8b2010 len=32 program=536903814 version=1 procedure=60 type=1 status=0 serial=1 0x7fd42239b0b8 : virNetClientIOHandleInput+0x5a8/0x8e0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239be88 : virNetClientIOEventLoop+0x328/0x600 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239c309 : virNetClientSendInternal+0x1a9/0x4e0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239dc27 : virNetClientSendWithReply+0x27/0x40 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239868f : virNetClientProgramCall+0x2cf/0x830 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4223e5c8b : callFull.isra.3+0x7b/0xc0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4223ed85b : remoteConnectSupportsFeatureUnlocked.isra.5+0x4b/0x80 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42240215d : doRemoteOpen+0x127d/0x2630 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd422403600 : remoteConnectOpen+0xf0/0x240 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4224955bc : virConnectOpenInternal+0x77c/0xe50 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4224960c4 : virConnectOpenAuth+0x74/0x90 [/usr/lib64/libvirt.so.0.7010.0] 0x55ff7a2a3fc4 : 0x55ff7a2a3fc4 [/usr/bin/virsh+0x3dfc4/0xbe000] client=0x55ff7b8b2010 len=28 program=536903814 version=1 procedure=1 type=1 status=0 serial=2 0x7fd42239b0b8 : virNetClientIOHandleInput+0x5a8/0x8e0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239be88 : virNetClientIOEventLoop+0x328/0x600 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239c309 : virNetClientSendInternal+0x1a9/0x4e0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239dc27 : virNetClientSendWithReply+0x27/0x40 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239868f : virNetClientProgramCall+0x2cf/0x830 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4223e5c8b : callFull.isra.3+0x7b/0xc0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4224021f0 : doRemoteOpen+0x1310/0x2630 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd422403600 : remoteConnectOpen+0xf0/0x240 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4224955bc : virConnectOpenInternal+0x77c/0xe50 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4224960c4 : virConnectOpenAuth+0x74/0x90 [/usr/lib64/libvirt.so.0.7010.0] 0x55ff7a2a3fc4 : 0x55ff7a2a3fc4 [/usr/bin/virsh+0x3dfc4/0xbe000] client=0x55ff7b8b2010 len=48 program=536903814 version=1 procedure=110 type=1 status=0 serial=3 0x7fd42239b0b8 : virNetClientIOHandleInput+0x5a8/0x8e0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239be88 : virNetClientIOEventLoop+0x328/0x600 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239c309 : virNetClientSendInternal+0x1a9/0x4e0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239dc27 : virNetClientSendWithReply+0x27/0x40 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239868f : virNetClientProgramCall+0x2cf/0x830 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4223e5c8b : callFull.isra.3+0x7b/0xc0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42240232d : doRemoteOpen+0x144d/0x2630 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd422403600 : remoteConnectOpen+0xf0/0x240 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4224955bc : virConnectOpenInternal+0x77c/0xe50 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4224960c4 : virConnectOpenAuth+0x74/0x90 [/usr/lib64/libvirt.so.0.7010.0] 0x55ff7a2a3fc4 : 0x55ff7a2a3fc4 [/usr/bin/virsh+0x3dfc4/0xbe000] client=0x55ff7b8b2010 len=32 program=536903814 version=1 procedure=60 type=1 status=0 serial=4 0x7fd42239b0b8 : virNetClientIOHandleInput+0x5a8/0x8e0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239be88 : virNetClientIOEventLoop+0x328/0x600 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239c309 : virNetClientSendInternal+0x1a9/0x4e0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239dc27 : virNetClientSendWithReply+0x27/0x40 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239868f : virNetClientProgramCall+0x2cf/0x830 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4223e5c8b : callFull.isra.3+0x7b/0xc0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4223ed85b : remoteConnectSupportsFeatureUnlocked.isra.5+0x4b/0x80 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42240222b : doRemoteOpen+0x134b/0x2630 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd422403600 : remoteConnectOpen+0xf0/0x240 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4224955bc : virConnectOpenInternal+0x77c/0xe50 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4224960c4 : virConnectOpenAuth+0x74/0x90 [/usr/lib64/libvirt.so.0.7010.0] 0x55ff7a2a3fc4 : 0x55ff7a2a3fc4 [/usr/bin/virsh+0x3dfc4/0xbe000] client=0x55ff7b8b2010 len=32 program=536903814 version=1 procedure=60 type=1 status=0 serial=5 0x7fd42239b0b8 : virNetClientIOHandleInput+0x5a8/0x8e0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239be88 : virNetClientIOEventLoop+0x328/0x600 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239c309 : virNetClientSendInternal+0x1a9/0x4e0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239dc27 : virNetClientSendWithReply+0x27/0x40 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239868f : virNetClientProgramCall+0x2cf/0x830 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4223e5c8b : callFull.isra.3+0x7b/0xc0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4223ed85b : remoteConnectSupportsFeatureUnlocked.isra.5+0x4b/0x80 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd422402277 : doRemoteOpen+0x1397/0x2630 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd422403600 : remoteConnectOpen+0xf0/0x240 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4224955bc : virConnectOpenInternal+0x77c/0xe50 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4224960c4 : virConnectOpenAuth+0x74/0x90 [/usr/lib64/libvirt.so.0.7010.0] 0x55ff7a2a3fc4 : 0x55ff7a2a3fc4 [/usr/bin/virsh+0x3dfc4/0xbe000] WARNING: Missing unwind data for a module, rerun with 'stap -d /usr/bin/virsh' client=0x55ff7b8b2010 len=28 program=536903814 version=1 procedure=360 type=1 status=0 serial=6 0x7fd42239b0b8 : virNetClientIOHandleInput+0x5a8/0x8e0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239be88 : virNetClientIOEventLoop+0x328/0x600 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239c309 : virNetClientSendInternal+0x1a9/0x4e0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239dc27 : virNetClientSendWithReply+0x27/0x40 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239868f : virNetClientProgramCall+0x2cf/0x830 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4223e5c8b : callFull.isra.3+0x7b/0xc0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4223ed47a : remoteConnectRegisterCloseCallback+0xaa/0x120 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4224c07c0 : virConnectRegisterCloseCallback+0x90/0x140 [/usr/lib64/libvirt.so.0.7010.0] 0x55ff7a2a4201 : 0x55ff7a2a4201 [/usr/bin/virsh+0x3e201/0xbe000] client=0x55ff7b8b2010 len=36 program=536903814 version=1 procedure=273 type=1 status=0 serial=7 0x7fd42239b0b8 : virNetClientIOHandleInput+0x5a8/0x8e0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239be88 : virNetClientIOEventLoop+0x328/0x600 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239c309 : virNetClientSendInternal+0x1a9/0x4e0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239dc27 : virNetClientSendWithReply+0x27/0x40 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239868f : virNetClientProgramCall+0x2cf/0x830 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4223e5c8b : callFull.isra.3+0x7b/0xc0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4223fe3d5 : remoteConnectListAllDomains+0x95/0x210 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4224a98bc : virConnectListAllDomains+0x8c/0x100 [/usr/lib64/libvirt.so.0.7010.0] 0x55ff7a2cfd11 : 0x55ff7a2cfd11 [/usr/bin/virsh+0x69d11/0xbe000] client=0x55ff7b8b2010 len=28 program=536903814 version=1 procedure=361 type=1 status=0 serial=8 0x7fd42239b0b8 : virNetClientIOHandleInput+0x5a8/0x8e0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239be88 : virNetClientIOEventLoop+0x328/0x600 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239c309 : virNetClientSendInternal+0x1a9/0x4e0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239dc27 : virNetClientSendWithReply+0x27/0x40 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239868f : virNetClientProgramCall+0x2cf/0x830 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4223e5c8b : callFull.isra.3+0x7b/0xc0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4223ed362 : remoteConnectUnregisterCloseCallback+0x82/0xf0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4224c08f0 : virConnectUnregisterCloseCallback+0x80/0x130 [/usr/lib64/libvirt.so.0.7010.0] 0x55ff7a2a3e2e : 0x55ff7a2a3e2e [/usr/bin/virsh+0x3de2e/0xbe000] client=0x55ff7b8b2010 len=28 program=536903814 version=1 procedure=2 type=1 status=0 serial=9 0x7fd42239b0b8 : virNetClientIOHandleInput+0x5a8/0x8e0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239be88 : virNetClientIOEventLoop+0x328/0x600 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239c309 : virNetClientSendInternal+0x1a9/0x4e0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239dc27 : virNetClientSendWithReply+0x27/0x40 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd42239868f : virNetClientProgramCall+0x2cf/0x830 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4223e5c8b : callFull.isra.3+0x7b/0xc0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd422400881 : remoteConnectClose+0x61/0x170 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4224d52da : virConnectDispose+0x2a/0x40 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd4222c17be : vir_object_finalize+0x7e/0xb0 [/usr/lib64/libvirt.so.0.7010.0] 0x7fd420c768a9 : 0x7fd420c768a9 [/usr/lib64/libgobject-2.0.so.0.5600.4+0x158a9/0x253000] WARNING: Missing unwind data for a module, rerun with 'stap -d /usr/lib64/libgobject-2.0.so.0.5600.4'
We need to use the “-DTASK_FINDER_VMA_ENTRY_ITEMS=7680” option with stap while printing userspace backtrace because of a bug in systemtap runtime which is fixed in later versions.
If you are wondering where print_ustack() came from and where you can check other functions like that, feel free to browse under /usr/share/systemtap/tapset/ locally OR https://sourceware.org/systemtap/tapsets/index.html
print_ustack() specifically is defined in : /usr/share/systemtap/tapset/linux/ucontext-symbols.stp
Conclusion
On an ending note, try writing your own systemtap script using different probes available in libvirt for any use case that you may be interested in tracing.
If you face any troubles with tracing, feel free to file a SR (Support Request) with oracle support (https://support.oracle.com) and we will help you out.
I hope that you liked this article and if you did, don’t forget to share it with others who may find it useful as well.