Basic boot diagnostics usually start with kernel verbosity, early console output, initcall timing, Dracut debugging, and systemd analysis. Those tools are covered in Cracking the Linux Boot Code: Linux Boot-Time Diagnostics Foundations.

This second part goes deeper into kernel-side instrumentation that can be enabled during boot: dynamic debug, boot-time tracing, ftrace, bootconfig, and kprobes. These techniques are useful when ordinary logs show that something is slow or failing, but do not expose enough detail about the driver, subsystem, function path, or trace event involved.

Dynamic Debug During Boot

Dynamic debug is useful when you want targeted debug messages from specific kernel source files or subsystems without enabling broad tracing. It is especially helpful for driver-heavy areas such as PCI, IRQ, storage, and networking, where normal loglevel output may not include enough detail.

Enable dynamic debug from the kernel command line with dyndbg=. For example, the following command enables verbose logging and turns on dynamic debug messages for PCI and IRQ-related kernel source files:

grubby --update-kernel=DEFAULT --args="loglevel=8 dyndbg='file drivers/pci/* +p; file kernel/irq/* +p'"

After boot, compare logs from a normal boot and a dynamic-debug boot to confirm that additional messages are being captured:

# dmesg | grep "save config" | wc
    992    7936   63616
# dmesg | grep "irq: Added domain " | wc
     29     174    1586

Dynamic debug can also be enabled after boot through debugfs:

sysctl -w kernel.printk="8 4 1 7"

echo "file drivers/pci/* +p" > /sys/kernel/debug/dynamic_debug/control
echo "file kernel/irq/* +p" > /sys/kernel/debug/dynamic_debug/control

Use dynamic debug selectively to avoid excessive log data.

Use Boot-Time Tracing

Boot-time tracing allows kernel tracing features such as trace events, ftrace, dynamic debug, and kprobes to be configured during boot. This is useful when the issue happens before normal user-space tracing tools are available, or when the system cannot reach a login prompt. It helps measure kernel boot performance, inspect driver and device initialization, capture scheduler or process activity, and debug early failures that would otherwise disappear before logs can be collected.

Boot-time tracing is especially useful for identifying slow initialization paths, driver probe delays, timeout-heavy code paths, early crashes, and boot-time dependency behavior. Use it selectively, because broad tracing can generate very large logs, increase boot overhead, and change timing-sensitive behavior.

Common boot-time tracing options include:

  • tp_printk: Writes trace events into the printk buffer so trace data can appear in kernel logs.
  • trace_event / trace-events: Enables trace events during boot, such as initcall:*, scheduler events, block events, or driver tracepoints.
  • ftrace: Enables a selected tracer, such as function or function_graph.
  • ftrace_filter: Limits function tracing to selected functions or wildcard patterns.
  • ftrace_graph_filter: Limits function-graph tracing to selected functions.
  • ftrace_notrace: Excludes selected functions from tracing.
  • ftrace_pid: Restricts tracing to a specific PID, such as PID 1 for the init process.
  • trace_buf_size: Sets the trace buffer size to reduce data loss during boot.
  • ftrace_boot_snapshot: Preserves a boot-time trace snapshot for later inspection.
  • kprobe_events: Adds dynamic trace points on selected kernel functions without rebuilding the kernel.
  • trace_options: Controls output details such as stacktrace, sym-addr, and timestamps.
  • ftrace_dump_on_oops: Dumps the trace buffer to the console when the kernel hits an oops or panic.

Note: This blog is not a hands-on guide to ftrace. The intent is to help users enable and apply ftrace at boot time.

Ftrace Boot-Time Example

Use ftrace when you want to trace kernel functions, function call flow, scheduler activity, or boot-time execution paths. Function graph tracing is especially useful because it shows nested function calls and timing information.

Enable scheduler trace events during boot:

grubby --update-kernel=DEFAULT --args="trace_event=sched:sched_process_fork,sched:sched_process_exit"

After boot, inspect the trace buffer:

cat /sys/kernel/debug/tracing/trace | more

kthreadd-2 [000] ..... 0.124640: sched_process_fork: comm=kthreadd pid=2 child_comm=rcu_tasks_kthre child_pid=15
kthreadd-2 [000] ..... 0.124657: sched_process_fork: comm=kthreadd pid=2 child_comm=rcu_tasks_rude_ child_pid=16
kthreadd-2 [000] ..... 0.124669: sched_process_fork: comm=kthreadd pid=2 child_comm=rcu_tasks_trace child_pid=17
kthreadd-2 [000] ..... 0.124801: sched_process_fork: comm=kthreadd pid=2 child_comm=ksoftirqd/0 child_pid=18

To trace function call flow, enable function graph tracing. The following example restricts tracing to PID 1, which is usually the init process, and also enables scheduler process fork and exit events:

grubby --update-kernel=DEFAULT --args="ftrace=function_graph ftrace_pid=1 trace_event=sched:sched_process_fork,sched:sched_process_exit"

Example function graph output:

# tracer: function_graph
#
# CPU  DURATION                  FUNCTION CALLS
# |     |   |                     |   |   |   |
 1)               |        hrtimer_cancel() {
 1)   0.231 us    |          hrtimer_active();
 1)               |          hrtimer_try_to_cancel.part.0() {
 1)   0.241 us    |            _raw_spin_lock_irqsave();
 1)   0.270 us    |            __remove_hrtimer();

If the trace buffer is too small, increase it with trace_buf_size. Verify the configured trace buffer size after boot:

cat /sys/kernel/debug/tracing/buffer_size_kb

For more structured boot-time tracing, use bootconfig. First, confirm that the kernel supports bootconfig and boot-time tracing:

grep -E "CONFIG_BOOT_CONFIG|CONFIG_BOOTTIME_TRACING" /boot/config-$(uname -r)

Expected configuration:

CONFIG_BOOT_CONFIG=y
CONFIG_BOOTTIME_TRACING=y

Create a bootconfig file:

ftrace.instance.boot_trace {
    tracer = "function"
    ftrace.filters = "vfs_*"
    buffer_size = 8KB
    enable
}

Back up the initramfs, apply the bootconfig, and enable the bootconfig kernel parameter:

cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak

bootconfig -a ./ftrace_conftest.conf /boot/initramfs-$(uname -r).img
bootconfig /boot/initramfs-$(uname -r).img

grubby --update-kernel=DEFAULT --args="bootconfig"
reboot

After reboot, inspect the boot-time tracing instance:

cd /sys/kernel/tracing/instances/
ls -ld boot_trace
head boot_trace/trace -n 40

# tracer: function
#
# entries-in-buffer/entries-written: 2851/1742353   #P:8
#
#           TASK-PID        CPU#   TIMESTAMP   FUNCTION
wlp-agent.ext-2412 [005] ..... 108.112448: vfs_unlink <-do_unlinkat
wlp-agent.ext-2412 [005] ..... 108.215174: vfs_open <-do_open
wlp-agent.ext-2412 [005] ..... 108.215185: vfs_fstat <-__do_sys_newfstat

Kprobe Boot-Time Example

Use kprobes when you need targeted instrumentation of a specific kernel function during boot without rebuilding the kernel. This is useful when a suspected function, driver path, or subsystem needs deeper inspection.

Create a bootconfig file for a boot-time kprobe:

ftrace.event.kprobes.my_boot_probe {
    probes = "do_sys_openat2 dfd=%ax filename=+0(+0(%si)):string"
    actions = "stacktrace"
}
ftrace.tuning.bufsize = 8

In this example, the buffer size is in KB. Apply the configuration to the initramfs and enable bootconfig:

bootconfig -a ./kprobe_boot.conf /boot/initramfs-$(uname -r).img
bootconfig /boot/initramfs-$(uname -r).img

grubby --update-kernel=DEFAULT --args="bootconfig"
reboot

After boot, verify that the kprobe is registered:

cat /sys/kernel/debug/kprobes/list
ffffffff909ae530  k  do_sys_openat2+0x0    [FTRACE]

Inspect the trace buffer:

cat /sys/kernel/tracing/trace | head -n 40

# tracer: nop
#
# entries-in-buffer/entries-written: 795/749794   #P:8
#
wlp-agent.ext-2982 [005] ..... 135.819135: my_boot_probe: (do_sys_openat2+0x0/0xe0) dfd=0xb0000 filename=(fault)
wlp-agent.ext-2982 [005] ..... 135.819140: <stack trace>
 => kprobe_trace_func
 => kprobe_dispatcher
 => kprobe_ftrace_handler
 => do_sys_openat2
 => __x64_sys_openat
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe

Probe only the function or path you need.

Cleanup

After collecting the required data, remove temporary tracing parameters from the boot entry:

grubby --update-kernel=DEFAULT --remove-args="ftrace ftrace_pid trace_event trace_buf_size ftrace_boot_snapshot bootconfig"

If you embedded bootconfig into the initramfs, remove it when it is no longer needed:

bootconfig -d /boot/initramfs-$(uname -r).img

Then verify the boot entry and reboot:

grubby --info=DEFAULT
reboot

Prefer focused trace events, ftrace_filter, ftrace_graph_filter, dynamic debug rules, or targeted kprobes for the subsystem, driver, or function under investigation.

Conclusion

Dynamic debug, boot-time tracing, ftrace, and kprobes are the tools to use when foundational boot logs are not enough. Keep the scope narrow, size trace buffers intentionally, preserve the collected output, and remove temporary tracing parameters after the investigation. Used carefully, these techniques turn early kernel startup from a black box into a traceable execution path.

For the foundational workflow, see Cracking the Linux Boot Code: Linux Boot-Time Diagnostics Foundations.

References