The stapsdt provider, added to DTrace in version 2.0.4, means that DTrace users on Linux can now trace execution of Python programs with built-in probes in the language itself. Here, we describe how, as well as how to add custom USDT probes to Python programs.

Using DTrace

If you are just getting started with DTrace, there are great training resources here.

Here is a bit of background necessary for the following if you are new to DTrace.

In these examples we are going to use process-specific probes; we pass -p <pid> to dtrace to tell it to trace a specific process, and reference that process throughout with $target. Each probe will have an optional predicate, which dictates whether the body of the clause fires. For example the following

python$target:::function-entry
/ stringof(arg1) == "foo" /
{
	printf("%s for %s\n", probename, stringof(arg1));
}

will restrict execution of the printf() to when the function name associated with the function-entry probe for the pid $target is foo. The function name is the second argument of the function-entry probe, arguments are ordered, arg0, arg1, arg2 etc.

We will also use thread-specific variables; these are referenced as self->something. Thread-specific variables are useful for tracing things specific to a thread of execution like depth of calling, increasing self->depth every time we enter a function and decreasing it on return.

With those concepts in place, we can start tracing Python code!

USDT probes in the Python language itself

We talked about USDT probes before but as a reminder, they are essentially markers you can add to your code to denote events of interest and provide information about them via parameters to those probes.

As far back as Python 3.6, DTrace/systemtap support for probes in Python was added. If Python is built with –with-dtrace, probes are added for specific events

audit
function-entry
function-return
line
gc-start
gc-done
import-find-load-start
import-find-load-done

See here for a full description of probes and arguments.

These are added as is-enabled probes. What that means is we can do extra work required to prepare probe data only for cases where the probe is enabled. When defining the probe, we also provide information about a count value – a semaphore – which the kernel will increase when tracing is enabled for the probe, and decrease when it is disabled. Code for a probe “myprobe” associated with provider “myprov” looks like this:

#include <sys/sdt.h>

unsigned short myprov_myprobe_semaphore = 0;

...


	if (myprov_myprobe_semaphore) {
		/* setup argdata here... */
		STAP_PROBE1(myprov, myprobe, argdata);
	}

With this sort of arrangement, we only incur the cost of setting up argdata if someone has enabled the probe (myprov_myprobe_semaphore > 0). This is very important in the case of Python, since the overhead of preparing arguments to the probes for firing is non-zero, and we would not want to incur such overhead every time a function is called if no one is watching!

It is this dynamic, low-overhead approach that makes DTrace powerful; rather than having to enable some logging parameter at startup in case something goes wrong, we can switch on analysis when needed. Imagine trying to debug a large Python program like an LLM which spends hours churning during training; we would not want the cost of logging to be incurred on the off-chance something bad might happen. Rather we want to be able to switch on – and off ! – analysis if we see something wrong, like perhaps an errant system not completing a task in a timely manner.

Under the hood, probe declarations like those above are added as ELF notes in a .note.stapsdt section. These declarations include the provider name, probe name, where the probe is located in the binary and how to find the probe arguments.

For example, we can see the metadata for the python USDT probes as follows. In this case the probes are in libpython; this may vary depending on your Python verison. However, the version of Python you are using must have been built with probes enabled (–with-dtrace) for this to work. We can determine this by running :

$ python3 -m sysconfig |grep with-dtrace
	CONFIG_ARGS = "'--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--with-platlibdir=lib64' '--enable-ipv6' '--enable-shared' '--with-computed-gotos=yes' '--with-dbmliborder=gdbm:ndbm:bdb' '--with-system-expat' '--with-system-ffi' '--enable-loadable-sqlite-extensions' '--with-dtrace' '--with-lto' '--with-ssl-default-suites=openssl' '--with-builtin-hashlib-hashes=blake2' '--with-valgrind' '--without-ensurepip' '--enable-optimizations' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CC=gcc' 'CFLAGS= -O2  -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS  -fstack-protector-strong   -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv ' 'LDFLAGS= -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now    -g ' 'CPPFLAGS=' 'PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig'"

Now that we know it was built with USDT probes, we will dig a bit further. First we find libpython3, then we will read the ELF notes sections (including .note.stapsdt with the probes):

$ ldd /usr/bin/python3
	linux-vdso.so.1 (0x00007fff8a3fa000)
	libpython3.9.so.1.0 => /usr/lib64/libpython3.9.so.1.0 (0x00007fad0944b000)
	libc.so.6 => /usr/lib64/libc.so.6 (0x00007fae09640000)
	libm.so.6 => /usr/lib64/libm.so.6 (0x00007fae09564000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fad09818000)
	libksplice_helper.so => /usr/lib64/libksplice_helper.so (0x00007fae0955e000)
$ readelf -n /usr/lib64/libpython3.9.so.1.0

Displaying notes found in: .note.gnu.property
  Owner                Data size 	Description
  GNU                  0x00000010	NT_GNU_PROPERTY_TYPE_0
      Properties: x86 feature: IBT, SHSTK

Displaying notes found in: .note.gnu.build-id
  Owner                Data size 	Description
  GNU                  0x00000014	NT_GNU_BUILD_ID (unique build ID bitstring)
    Build ID: 14bce601f87ecc5e5354162b898ab9ba261336e2

Displaying notes found in: .note.stapsdt
  Owner                Data size 	Description
  stapsdt              0x00000033	NT_STAPSDT (SystemTap probe descriptors)
    Provider: python
    Name: audit
    Location: 0x0000000000060878, Base: 0x00000000002fcee8, Semaphore: 0x0000000000397dd8
    Arguments: 8@%rbp 8@%r14
  stapsdt              0x00000030	NT_STAPSDT (SystemTap probe descriptors)
    Provider: python
    Name: gc__done
    Location: 0x00000000000641a5, Base: 0x00000000002fcee8, Semaphore: 0x0000000000397dde
    Arguments: -8@%rbp
  stapsdt              0x00000036	NT_STAPSDT (SystemTap probe descriptors)
...

Using these NT_STAPSDT probe descriptions, we can trace function calls, garbage collection, auditing etc. Note that the double-underscore, e.g. “gc__done” is converted to a “-” by DTrace – “gc-done”.

As an example, we will trace function entry and return when help() is called within python3. If we do the following

$ python3
Python 3.9.21 (main, Aug 19 2025, 00:00:00) 
[GCC 11.5.0 20240719 (Red Hat 11.5.0-5.0.1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> help()

while in another window tracing function entry and return in this python3 instance. The tracing should be started before calling help().

$ pgrep python3
1813959
$ $ sudo /sbin/dtrace -p 1813959 -n 'python$target:::function-entry,python$target:::function-return { printf("%s %s()\n", stringof(arg0), stringof(arg1)); 
}'
dtrace: description 'python$target:::function-entry,python$target:::function-return ' matched 2 probes
CPU     ID                    FUNCTION:NAME
  0   4194                  :function-entry <stdin> <module>()

  0   4194                  :function-entry /usr/lib64/python3.9/_sitebuiltins.py __call__()

  0   4194                  :function-entry <frozen importlib._bootstrap> _find_and_load()

  0   4194                  :function-entry <frozen importlib._bootstrap> __init__()

  0   4196                 :function-return <frozen importlib._bootstrap> __init__()

  0   4194                  :function-entry <frozen importlib._bootstrap> __enter__()

  0   4194                  :function-entry <frozen importlib._bootstrap> _get_module_lock()

  0   4194                  :function-entry <frozen importlib._bootstrap> __init__()
...

The output above was heavily snipped. As we can see, a simple call to help() requires a fair bit of work! Most of it relates to importing libraries; we will see below how to get rid of this extra info if it is unwanted.

Note that the script in the Python docs – Instrumenting CPython with DTrace and SystemTap – provides a great example of Python tracing, but it does have one small issue; specifically we need to use stringof() rather than copyinstr().

The script maintains a per-thread (self) variable to count indentation level in function call, increasing it on entry and decreasing it on return.

Below we have added an additional tweak; we wait until all importing is done to avoid a lot of noise in the output. We use import-find-load-start/done to count imports, increasing the count on -start and reducing it on -done.

Then we use that count to check whether we should emit function entry or return, only emitting it when not in the context of an import.

This is a different approach to the script, which looks for the “start” function entry to start tracing and “start” function return to end tracing. The below will also work on interactive python3 sessions while omitting noise from imports:

#!/usr/sbin/dtrace -qs

self int indent;
self int importing;

python$target:::import-find-load-start
{
	self->importing++;
}

python$target:::import-find-load-done
{
	self->importing--;
}

python$target:::function-entry
/ self->importing == 0/
{
        printf("%d\t%*s: ", timestamp, 15, probename);
        printf("%*s", self->indent, "");
        printf("%s:%s:%d\n", stringof(arg0), stringof(arg1), arg2);
        self->indent++;
}

python$target:::function-return
/ self->importing == 0/
{
        self->indent--;
        printf("%d\t%*s: ", timestamp, 15, probename);
        printf("%*s", self->indent, "");
        printf("%s:%s:%d\n", stringof(arg0), stringof(arg1), arg2);
}

The output is quite nice – we run it with -q (quiet mode):

$ sudo /sbin/dtrace -p 1960543 -qs call_stack.d 
138630745237065	function-return:  /usr/lib64/python3.9/pydoc.py:getline:2041
138630745321064	 function-entry:  /usr/lib64/python3.9/pydoc.py:help:2047
138630745344027	 function-entry: /usr/lib64/python3.9/pydoc.py:doc:1779
138630745353785	 function-entry:  /usr/lib64/python3.9/pydoc.py:render_doc:1752
138630745363283	 function-entry:   /usr/lib64/python3.9/pydoc.py:resolve:1738
138630745373502	 function-entry:    /usr/lib64/python3.9/pydoc.py:locate:1713
138630745384754	 function-entry:     /usr/lib64/python3.9/pydoc.py:<listcomp>:1715
138630745394131	function-return:     /usr/lib64/python3.9/pydoc.py:<listcomp>:1715
138630745408238	 function-entry:     /usr/lib64/python3.9/pydoc.py:safeimport:414
138630746355517	function-return:     /usr/lib64/python3.9/pydoc.py:safeimport:451
138630746374613	function-return:    /usr/lib64/python3.9/pydoc.py:locate:1729
138630746385022	function-return:   /usr/lib64/python3.9/pydoc.py:resolve:1743
138630746391896	function-return:  /usr/lib64/python3.9/pydoc.py:render_doc:1757
138630746438022	function-return: /usr/lib64/python3.9/pydoc.py:doc:1788
138630746447731	 function-entry: /usr/lib64/python3.9/pydoc.py:output:1992
138630746455555	function-return: /usr/lib64/python3.9/pydoc.py:output:1994
138630746475543	function-return:  /usr/lib64/python3.9/pydoc.py:help:2066
138630746484190	 function-entry:  /usr/lib64/python3.9/pydoc.py:getline:2038
138630746497345	 function-entry: /usr/lib64/python3.9/pydoc.py:input:1988
138630746504799	function-return: /usr/lib64/python3.9/pydoc.py:input:1990
^C

Adding USDT probes to Python programs

As we saw in a previous post we can add our own USDT probes to a Python program via stapsdt/libstapsdt. These are then added dynamically to a created-on-the-fly ELF notes section that supports tracing of probe firing. DTrace can now read such probe definitions and trace them. As per the example in that post, build libstapsdt:

$ git clone https://github.com/linux-usdt/libstapsdt
$ cd libstapsdt
$ make && sudo make install

Then using stapsdt from https://pypi.org/project/stapsdt/ we can write a program containing a probe:

#!/usr/bin/python3

from time import sleep

import stapsdt

provider = stapsdt.Provider("pythonapp")
probe = provider.add_probe(
"firstProbe", stapsdt.ArgTypes.uint64, stapsdt.ArgTypes.int32)
provider.load()


while True:
    print("Firing probe...")
    if probe.fire("My little probe", 42):
        print("Probe fired!")
    sleep(1)

Now run it:

$ python3 example.py
Firing probe...
Firing probe...
Firing probe...
Firing probe...

Finally let us see that firing using DTrace:

$ pgrep example.py

933309
$ sudo dtrace -p 933309 -n 'pythonapp$target::: { printf("%s %d\n", stringof(arg0), arg1); }'
 sudo /sbin/dtrace -p 1814184  -n 'pythonapp$target::: { printf("%s %d\n", stringof(arg0), arg1); }'
dtrace: description 'pythonapp$target::: ' matched 1 probe
CPU     ID                    FUNCTION:NAME
  1   4194                      :firstProbe My little probe 42

  1   4194                      :firstProbe My little probe 42

^C

Summary

With facilities like the above, we can use DTrace to trace Python program execution and indeed add our own probes to them!

Tracing Python with DTrace

Using DTrace

USDT probes in the Python language itself

Adding USDT probes to Python programs

Summary

References

Alan Maguire

Setup QEMU vhost-user-scsi and SPDK on Oracle Linux 8

Tracing Python with DTrace

Using DTrace

USDT probes in the Python language itself

Adding USDT probes to Python programs

Summary

References

Authors

Alan Maguire

Setup QEMU vhost-user-scsi and SPDK on Oracle Linux 8