The oled tool’s trace subcommand dynamically generates tracing scripts for a list of functions that represent a workload and executes the tracing script. The gathered data includes the entry and return traces of each function in the workload list. The oled trace tool is essential for optimizing performance and identifying bottlenecks in workloads. By dynamically generating tracing scripts, it provides detailed insights into the behavior of each function in a given workload.

The tool helps developers and system administrators monitor entry and return traces, making it easier to detect inefficiencies, performance issues, or areas needing improvement. Moreover, if certain functions cannot be traced, the tool will seamlessly skip them and continue tracing other functions without interrupting the overall analysis process.

DTrace is the fundamental tool used within oled trace. It is a powerful dynamic tracing framework that allows the collection of detailed performance data at runtime. DTrace is leveraged to trace the execution of functions, monitor system events, and collect performance statistics that are crucial for understanding the behavior of a workload. This tracer provides a flexible and efficient way to gather insights into system performance, making it an essential component of the oled trace tool.

Workload

A workload refers to a set of tasks, processes, or functions that a system, application, or service is expected to handle or execute. It represents the computational demands placed on a system, often reflecting the actual activities a user or program carries out. In this context, a workload might consist of a list of functions or operations that are being tested or measured to assess how well the system handles them.

For instance, a CPU hot plug operation involves several processes, including libvirt, QEMU, KVM and kernel, which execute specific tasks across both the host (hypervisor) domain and the guest domain. The host domain handles certain management tasks, while the guest domain executes kernel and KVM-related operations. During this workload, various functions or methods are called across both domains and associated software layers, ensuring proper communication and task execution.

DTrace over Others

When designing the tracing capabilities, DTrace was selected as the foundational technology. DTrace allows for detailed system and application tracing, performance analysis, and debugging. It offers a consistent set of providers and probes, which makes it easier to trace different system events. On Linux, DTrace uses kernel mechanisms to collect observability data and is well supported by the Oracle Linux community.

Ease of Use

  • For system administrators and developers familiar with the DTrace framework, it can be remarkably straightforward to write simple, yet powerful, tracing scripts to diagnose performance issues or investigate system behavior.

Mature and Battle-Tested

  • DTrace has a long history and is a mature technology, having been used extensively in production environments for many years. This maturity translates to a robust and well-understood framework with a wealth of existing scripts and knowledge available.

Using oled trace

trace is part of the oled-tools rpm. For more information about the OLED project and installation instructions, please see Oracle Linux Enhanced Diagnostics.

After installation, trace can be invoked as follows:

# oled trace
Workload need to be specified for trace
usage: trace [-h] [-l WORKLOAD | -f WORKLOADFILE] [-p PID] [-d] [-v] [-L]
               [-R] [-T PID] [-e] [-o OUTFILE | -P]

Trace a workload events.

optional arguments:
  -h, --help            show this help message and exit
  -l WORKLOAD, --workload WORKLOAD
                        Name of workload
  -f WORKLOADFILE, --workloadfile WORKLOADFILE
                        Path to the file containing workload/function list
  -p PID, --pid PID     pid to trace. Must be used with -l or -f
  -d, --debug           Enable debug logging
  -v, --version         Display oled trace version number
  -L, --list            Display available workloads
  -R, --runs            Display currently running traces.
  -T PID, --terminate PID
                        Terminates the specified tracer.
  -e, --expand          Display functions associated with a workload. Must be
                        used with -l
  -o OUTFILE, --outfile OUTFILE
                        Output will be redirected to the file specified by
                        OUTFILE.
  -P, --print           Display output on console

trace requires a workload in order to create a tracer script. The available workloads can be viewed using the -L option.

# oled trace -L
Available workloads are:

Kernel workloads:
kern_cpuhp

User workloads:
libvirt_cpuhp
qemu_cpuhp

When running trace with the kern_cpuhp workload, the tool retrieves the list of functions from the workload and checks them for validity. It also verifies whether each function can be traced using DTrace. After validation, the tool automatically generates a DTrace entry for each valid function. Once the DTrace script is ready, it is executed and the output is redirected to a log file by default. You can override this behavior and specify a custom output file using the -o flag. Additionally, if you prefer to view the logs on the console, you can use the -P option.

# oled trace -l kern_cpuhp
Starting workload: kern_cpuhp

Started tracing, output logs are being continuously redirected to /var/oled/trace/dtlog/2025-10-10_08-30-11_kern_cpuhp.d.log
Please monitor the file for real-time updates. Press Ctrl + C to stop the tool

# oled trace -l kern_cpuhp -P
Starting workload: kern_cpuhp

Started tracing, output is printed to console.
Press Ctrl + C to stop the tool

[2025 Oct 10 09:05:59] [Time: 7607099774856008] [pid: 3245207] [comm: bash] [cpu: 1] _cpu_down Entry.
[2025 Oct 10 09:05:59] [Time: 7607099780023594] [pid: 3245207] [comm: bash] [cpu: 1] cpuhp_kick_ap_work Entry.
[2025 Oct 10 09:05:59] [Time: 7607099780076464] [pid: 3245207] [comm: bash] [cpu: 1] cpuhp_kick_ap Entry.
[2025 Oct 10 09:05:59] [Time: 7607099780192624] [pid: 24] [comm: cpuhp/2] [cpu: 2] cpuhp_should_run Entry.
[2025 Oct 10 09:05:59] [Time: 7607099780242929] [pid: 24] [comm: cpuhp/2] [cpu: 2] cpuhp_should_run Return.
[2025 Oct 10 09:05:59] [Time: 7607099780251615] [pid: 24] [comm: cpuhp/2] [cpu: 2] cpuhp_thread_fun Entry.
[2025 Oct 10 09:05:59] [Time: 7607099780290509] [pid: 24] [comm: cpuhp/2] [cpu: 2] cpuhp_invoke_callback Entry.
...

Tracing a Specific PID

If you want to trace a specific PID, this tool has you covered, the -p option allows you to specify a particular PID. For example, suppose you want to run the libvirt_cpuhp workload. First, obtain the PID of the libvirtd process and provide it to the tracing tool. The tool will then use this PID to create a DTrace probe that tracks the specified process.

# ps aux | grep libvirtd
root      49885  0.0  0.0 2030632 54648 ?       Ssl  Oct01   7:37 /usr/sbin/libvirtd --timeout 120

# oled trace -l libvirt_cpuhp -p 49885
Starting workload: libvirt_cpuhp

Started tracing, output logs are being continuously redirected to /var/oled/trace/dtlog/2025-10-13_00-55-29_trace_libvirt_cpuhp_49885.d.log
Please monitor the file for real-time updates. Press Ctrl + C to stop the tool

Tracing a Custom Workload

The tracer also allows users to provide a custom workload file using the -f option, creating a tracer based on the functions or system calls listed in that file. This approach lets you consolidate all the functions or syscalls you want to track into a single file. For example, in the snippet below, I demonstrate how to track all XFS file and directory operations.

This feature is particularly useful for the following reasons:

  • Not all workloads may be defined with the default oled trace workloads. For a new workload or to address specific issues, you can simply create a new workload file tailored to your needs and run it using this option.
  • Before officially adding a new workload to the main set, you can test and validate its workload files using the -f option to ensure it captures the required data.
# cat xfs_file.fnlist
xfs_file_llseek
xfs_file_read_iter
xfs_file_write_iter
xfs_file_ioctl
xfs_file_compat_ioctl
xfs_file_mmap
xfs_file_open
xfs_file_release
xfs_file_fsync
xfs_file_fallocate
xfs_file_fadvise
xfs_file_remap_range
xfs_dir_open
xfs_file_readdir
xfs_dir_fsync
xfs_file_ioctl
xfs_file_compat_ioctl

# oled trace -f xfs_file.fnlist -P
Starting workload: user_workload

Started tracing, output is printed to console.
Press Ctrl + C to stop the tool
[2025 Oct 13 05:13:44] [Time: 7852364426892203] [pid: 4154645] [comm: pgrep] [cpu: 3] xfs_file_mmap Return.
[2025 Oct 13 05:13:44] [Time: 7852364426928512] [pid: 4154645] [comm: pgrep] [cpu: 3] xfs_file_open Entry.
[2025 Oct 13 05:13:44] [Time: 7852364426936116] [pid: 4154645] [comm: pgrep] [cpu: 3] xfs_file_open Return.
[2025 Oct 13 05:13:44] [Time: 7852364426948068] [pid: 4154645] [comm: pgrep] [cpu: 3] xfs_file_mmap Entry.
[2025 Oct 13 05:13:44] [Time: 7852364426954050] [pid: 4154645] [comm: pgrep] [cpu: 3] xfs_file_mmap Return.

Debug Options

trace also supports a debug option -d, which dumps debug data such as the functions used to create the DTrace script and additional logs in the workflow to aid with debugging into a separate debug file. This debug file also includes a copy of the DTrace script. Additionally, trace keeps track of all running traces in the file /var/oled/trace/trace.run, which can be displayed using the -R option.

# oled trace -R
Running tracers are:
PID: 3245050, Time 2025-10-10_09-05-25, Command: oled trace -l  kern_cpuhp  -P

PID: 3277687, Time 2025-10-10_11-30-23, Command: oled trace -l  kern_cpuhp

These tracers can be terminated using the -T option of trace. The -T option takes the PID displayed in the -R output as its argument.

# oled trace -T 3245050
Workload PID: 3245050 terminated
# oled trace -T 3277687
Workload PID: 3277687 terminated

# oled trace -R
No active tracers

Adding new workloads

Currently, the tool supports the following predefined workloads: kern_cpuhp, libvirt_cpuhp, and qemu_cpuhp. To introduce a new workload, please follow these steps:

  1. Identify Functions to Trace
    Begin by determining which functions you want to trace for your workload. Make a list of these functions.
  2. Test with the -f Option
    Use the -f option to test your workload by providing your new workload file. Iterate and refine your function list until you are satisfied with the tracing results.
  3. Integrate the Workload File
    • Clone the source repository: git clone https://github.com/oracle/oled-tools
    • Add your new workload file to the following directory: oled-tools/tools/trace
    • Name your file following the convention of existing workloads, for example:
      kern_cpuhp.fnlist, libvirt_cpuhp.fnlist, qemu_cpuhp.fnlist
  4. Register the Workload
    Add the name of your new workload to either kern_workload.list or proc_workload.list, depending on its type:
    • For Kernel workloads: $ cat kern_workload.list kern_cpuhp <your_new_workload>
    • For Process workloads: $ cat proc_workload.list libvirt_cpuhp qemu_cpuhp <your_new_workload>

Once you complete these steps, your new workload will be ready for integration. However, if you just want to run the workload on your machine, list all the functions and system calls you want to track in a file (for example, custom_workload), and then run them using the -f option: oled trace -f custom_workload.

Improvements

  • Expanding OLED trace to support additional well-defined workloads, beyond the current three predefined ones.
  • Enhancing trace with advanced filters to reduce data noise when tracing aggressive drivers.
  • Extending support to kernel loadable modules.

Conclusion

The oled trace tool offers a powerful and flexible solution for tracing workloads, enabling developers and system administrators to gain insights into system performance. By dynamically generating tracing scripts and utilizing DTrace as its core tracing engine, the tool provides detailed visibility into function calls, entry and exit points, and potential performance bottlenecks. Whether you’re optimizing system performance, diagnosing inefficiencies, or gaining a better understanding of system behavior, oled trace simplifies the process of tracing and performance monitoring.

Reference