Kernel engineers often need to inspect kernel function return values while developing features or debugging live systems. With trace-cmd support for function_graph return-value decoding, correlating each function exit with the value returned to its caller is now straightforward in day-to-day development and live-system triage.
This article walks through the end-to-end usage, practical use cases, and a tracefs fallback in case you’re more familiar with using ftrace for tracing.
Quick start (trace-cmd)
If you are on the latest version of Oracle Linux 8, 9, or 10, you can start with two commands:
First, record with trace-cmd record [1]. By default, it writes trace.dat in the current directory. Press Ctrl-C to stop.
sudo trace-cmd record -p function_graph -P <target_pid>Second, report with trace-cmd report [2] (it reads trace.dat by default), then look for (ret=...) annotations.
sudo trace-cmd report -O fgraph:tailprintExpected result: trace-cmd report lines include (ret=...), for example:
... funcgraph_entry: 0.451 us | _raw_spin_unlock_irqrestore(); (ret=0)
... funcgraph_exit: 4.359 us | } /* __traceiter_sched_switch */ (ret=0)If you do not see (ret=...), check the kernel support and fallback sections below.
Kernel support for function return values
Availability
The return-value feature was merged upstream under commit a1be9ccc57f0 [3] (see the LKML thread [4]) and is available in mainline kernels on or after v6.5. Before this commit, function_graph provided call-graph timing/depth data but not per-function return values.
Oracle Linux UEK kernels also have this feature on or after UEK-5-U5 (v4.14.35-2047.538.1), UEK-6-U3 (v5.4.17-2136.333.1), UEK-7-U2 (v5.15.0-207.156.4), and UEK-8-GA (v6.12.0-0.7.5).
You can verify whether the kernel supports it in either of these two ways:
- Inspect the boot-time configuration:
grep CONFIG_FUNCTION_GRAPH_RETVAL /boot/config-$(uname -r)- Look for the runtime knobs under tracefs:
ls /sys/kernel/tracing/options/funcgraph-retval*If you see CONFIG_FUNCTION_GRAPH_RETVAL=y or the funcgraph-retval option files, the tracer can record each function exit value alongside the normal call/return timing data.
How the tracer captures return values
function_graph captures return values on the function-exit path by routing control through its return handler before returning to the original caller. In that handler path, the tracer records return-time metadata and computes function duration.
With CONFIG_FUNCTION_GRAPH_RETVAL=y, the handler also saves the function return register(s) so the trace record can include retval. At runtime, the tracefs option /sys/kernel/tracing/options/funcgraph-retval controls whether return-value capture is enabled.
trace-cmd: reporting return values
trace-cmd is the primary user-space front-end for this workflow. It consumes ftrace data directly [5], records efficiently, and produces consistent function_graph reports. I developed the trace-cmd return-value decoding for function_graph, and the feature was accepted upstream and backported to Oracle Linux 8, 9, and 10 (see the upstream commit series starting at [6]).
For Oracle Linux users, update trace-cmd to the latest package available for your release. If your trace-cmd build still does not print return values, build the latest upstream trace-cmd from source [7] or use the tracefs fallback later in this article.
Reading report output
Use the quick-start reporting command and look for (ret=...). If you also want call depth in the output, add -O fgraph:depth.
The following excerpt comes from a single-PID function_graph report on a failing write(2) path. In this example, proc_reg_write() returns -5, and that error propagates through vfs_write, ksys_write, and __x64_sys_write. The diagnosis use case below describes how engineers use this pattern during triage.
Sample output:
<...>-1136215 [006] ..... 1724848.377495: funcgraph_entry: | __cond_resched() { (3)
<...>-1136215 [006] ..... 1724848.377495: funcgraph_entry: 0.200 us | rcu_all_qs(); (4) (ret=0x0)
<...>-1136215 [006] ..... 1724848.377496: funcgraph_exit: 0.601 us | } /* __cond_resched */ (3) (ret=0x0)
<...>-1136215 [006] ..... 1724848.377496: funcgraph_entry: 0.581 us | proc_reg_write(); (3) (ret=-5) <-- first `-5` appears here
<...>-1136215 [006] ..... 1724848.377497: funcgraph_exit: 5.059 us | } /* vfs_write */ (2) (ret=-5) <-- propagated to vfs_write
<...>-1136215 [006] ..... 1724848.377497: funcgraph_exit: 6.332 us | } /* ksys_write */ (1) (ret=-5) <-- propagated to ksys_write
<...>-1136215 [006] ..... 1724848.377497: funcgraph_exit: + 18.414 us | } /* __x64_sys_write */ (0) (ret=-5) <-- reaches syscall boundaryBy default, the formatter prints errno-style negatives in decimal and other return values in hexadecimal, so failures are easy to spot.
Formatting controls
The fgraph plugin includes options to control return-value formatting:
-O fgraph:retval-skiphides return values entirely when you only care about execution time.-O fgraph:retval-decforces decimal output everywhere (useful when chasing reference counts or boolean returns).-O fgraph:retval-hexforces hexadecimal output.
Combine them with existing helpers such as fgraph:tailprint and fgraph:depth to tailor the report for long-running traces.
Use cases worth sharing
- Diagnosis (runtime failures). Engineers often see a syscall failure first (for example
-EIOor-EPERM) but cannot tell where it originated inside the kernel. The workflow is: match the error code to(ret=...)intrace-cmd report, identify the first function that returns it (proc_reg_write()in this example), then follow propagation to the syscall boundary. This narrows triage from “somewhere in the stack” to a specific function and context, while keeping timing and call structure in the same trace. - Development (post-change validation). After implementing or backporting a kernel change, engineers need fast evidence that error-handling behavior did not regress before broad QA coverage. This feature lets them run the same workload on baseline and modified builds, then verify that key functions return expected values directly in
trace-cmd report(ret=...) instead of inferring behavior indirectly from final syscall outcomes. In practice, that catches unintended return-code changes early, speeds code review discussions, and provides concrete trace artifacts that are easier to share across kernel, QE, and support teams.
Fallback: tracefs (no trace-cmd support)
If your trace-cmd build does not decode retval yet, you can still use kernel-side function_graph output directly from tracefs or debugfs.
Run these commands as root.
If your system uses tracefs at /sys/kernel/tracing, run:
ls /sys/kernel/tracing/options/funcgraph-retval /sys/kernel/tracing/options/funcgraph-retval-hexIf /sys/kernel/tracing is missing, mount tracefs first:
sudo mount -t tracefs nodev /sys/kernel/tracingStart a capture
cd /sys/kernel/tracing
echo 0 > tracing_on
echo > trace
echo function_graph > current_tracer
echo my_function > set_graph_function # optional filter
echo $$ > set_ftrace_pid # or pick a PID manually
echo 1 > options/funcgraph-retval
echo 0 > options/funcgraph-retval-hex # keep errno values in decimal
echo 1 > tracing_onStop the capture
echo 0 > tracing_on
cat trace > /tmp/trace-fgraph-retval.log
echo > set_ftrace_pid
echo > set_graph_function
echo nop > current_tracer
echo > traceExpected result: /tmp/trace-fgraph-retval.log includes function_graph lines with trailing return-value annotations such as /* = 0x0 */.
For example:
36) | finish_task_switch.isra.0() {
36) 0.571 us | _raw_spin_unlock(); /* = 0x0 */
36) 4.166 us | } /* finish_task_switch.isra.0 = 0x0 */
36) 0.411 us | module_put(); /* = 0x0 */
36) 0.361 us | module_put(); /* = 0x0 */
36) 0.261 us | module_put(); /* = 0x0 */
36) 2.214 us | module_put(); /* = 0x0 */
36) 0.560 us | module_put(); /* = 0x0 */If your system uses debugfs at /sys/kernel/debug/tracing, run the same commands with that path instead of /sys/kernel/tracing.
Conclusion
trace-cmd report can now decode and print function_graph return values as (ret=...) alongside the existing call graph and timing data. This makes it easier to move from a syscall error code to the first kernel function that returns it, which speeds both diagnosis and post-change validation.