When a Linux system crawls, freezes, or vanishes into a blank screen during startup, the real story is often hidden in the earliest seconds of boot. Linux boot-time debugging helps uncover that story by identifying whether the delay or failure comes from a problematic service, driver, mount, firmware handoff, or kernel initialization path.

During startup, the Linux kernel initializes hardware, mounts the root file system, prepares the initramfs flow, and hands control to user space, often before familiar debugging tools are available. This first part focuses on the foundation: kernel verbosity, early console output, initcall timing, initramfs debugging with Dracut, and systemd startup analysis.

For deeper kernel instrumentation with dynamic debug, boot-time tracing, ftrace, and kprobes, continue with Cracking the Linux Boot Code: Deep-Dive Linux Boot Diagnostics.

Sometimes, the delay is long enough to deserve a little humor: Why did the sysadmin bring a sleeping bag to work? Because systemd-analyze blame said the network timeout had a long story to tell.

Why Boot-Time Diagnostics Are Important

Boot-time diagnostics are important because many startup problems occur before the system is fully operational. They provide early visibility into the boot path and help identify whether a system is slow, stuck, or failing because of a service, driver, mount, firmware handoff, kernel issue, or resource constraint.

  • Fixing boot failures: Early-stage bugs can freeze or crash the system before traditional logging utilities such as syslog, journald, or systemd are available.
  • Optimizing boot performance: Fast boot times matter for cloud micro-instances, embedded devices, automotive dashboards, and production systems where startup delay directly affects availability or user experience.
  • Identifying driver bottlenecks: Hardware initialization can stall the boot process when drivers wait too long for device responses, firmware handshakes, or timeout events.
  • Security auditing: Early tracing helps verify that kernel security modules, secure boot flows, and related initialization steps execute correctly before user space takes control.
  • Finding resource constraints: Early diagnostics can reveal memory allocation issues, CPU stalls, and other bottlenecks that occur before normal user-space monitoring tools are available.

Boot-Time Debugging Toolkit

This guide covers the foundational techniques used to debug and measure the Linux boot path: kernel verbosity, early console output, initcall timing, Dracut debugging, and systemd startup analysis. The companion deep-dive article covers dynamic debug, boot-time tracing, ftrace, and kprobes.

Linux Boot Flow at a Glance

Linux boot debugging becomes easier when the issue is mapped to the correct boot phase. Each phase has different responsibilities, different failure patterns, and different levels of logging or tool availability.

PhasePurposeTypical Debug Focus
BIOS / firmwareInitializes platform hardware, performs firmware checks, selects the boot device, and transfers control to the boot loader.Firmware settings, boot device selection, secure boot policy, hardware discovery, platform initialization failures.
Boot loaderLoads the Linux kernel and required boot artifacts into memory, passes kernel command-line parameters, and transfers control to the kernel.Kernel image selection, initramfs selection, boot entry configuration, kernel command-line correctness.
Kernel initializationDiscovers CPUs, memory, buses, and devices; initializes built-in subsystems and drivers; processes kernel boot parameters; mounts the root file system; prepares user space.Kernel command-line parameters, driver initialization, device discovery, root file system discovery, early crashes, hangs, or timeout paths.
Initramfs / early user spaceRuns a temporary early user-space environment to discover storage, load required modules, assemble volumes, unlock disks, and mount the real root file system.Dracut issues, missing drivers, storage discovery failures, LVM/RAID/multipath problems, root mount failures, emergency shell analysis.
Service startupStarts the main user-space service manager, launches services, mounts file systems, manages dependencies, and brings the system to the requested target state.Slow services, failed units, dependency ordering, network or storage waits, mount delays, timeout-heavy startup paths.

This article focuses mainly on post-boot-loader diagnostics: kernel initialization, initramfs / early user space, and service startup.

Kernel Console Log Level

The kernel console log level is one of the simplest and most useful controls for early boot debugging. It determines which kernel messages are printed to the console while the system is starting, especially before full user-space logging is available.

Use the loglevel= kernel command-line parameter to control console verbosity:

0  KERN_EMERG    System is unusable
1  KERN_ALERT    Immediate action is required
2  KERN_CRIT     Critical condition
3  KERN_ERR      Error condition
4  KERN_WARNING  Warning condition
5  KERN_NOTICE   Normal but significant condition
6  KERN_INFO     Informational message
7  KERN_DEBUG    Debug-level message

Kernel messages use severity levels from 0 to 7, where lower numbers represent more critical conditions and higher numbers include more detailed output.

For boot debugging, increasing the console log level can help expose driver initialization failures, device discovery issues, root file system problems, mount delays, and other early kernel warnings that may otherwise remain hidden.

  • Append the parameter through the boot loader, such as GRUB or U-Boot.
  • After the system boots, verify the active kernel command line with cat /proc/cmdline.
  • To force the kernel to print all messages regardless of the configured console log level, use ignore_loglevel.
  • Use high verbosity carefully on production systems because debug-level output can be noisy, may slow down console logging, and can make important messages harder to identify when the boot path generates a large amount of output.

Capture Very Early Messages with earlyprintk

earlyprintk helps print kernel messages before the normal console driver is initialized.

  • This is useful when the system crashes, hangs, or goes silent very early in the boot process, before standard console output or user-space logging is available.
  • A common serial-console example is earlyprintk=serial,ttyS0,115200. Use this option when debugging early platform initialization, CPU or memory setup, firmware handoff issues, or early driver bring-up problems where regular kernel logs may not yet be visible.

Time Built-In Initialization with initcall_debug

Built-in kernel drivers and subsystems are initialized through a sequence of functions known as initcalls. Because these functions run early in the boot process, a slow or failing initcall can delay startup, trigger timeouts, or stop the system before normal debugging tools are available. To trace this stage, add the initcall_debug parameter to the kernel command line. It prints timing and return information for each built-in initialization function, making it easier to identify slow, failed, or suspicious initialization paths.

After boot, inspect the kernel log with: dmesg | grep "_init"

Ensure the following kernel configuration options are enabled:

# grep -E 'CONFIG_EARLY_PRINTK=|CONFIG_SERIAL_8250=|CONFIG_SERIAL_8250_CONSOLE=' /boot/config-$(uname -r)
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_EARLY_PRINTK=y

Set earlyprintk and initcall_debug:

# grubby --update-kernel="/boot/vmlinuz-$(uname -r)" \
  --args="earlyprintk=serial,ttyS0,115200 console=ttyS0,115200 loglevel=8 ignore_loglevel initcall_debug"
# reboot

Check console logs:
[    0.243767] calling  rcu_sysrq_init+0x0/0x27 @ 10 usecs
[    0.243768] initcall rcu_sysrq_init+0x0/0x27 returned 0

This output shows when an initcall started, when it returned, and whether it completed successfully. It is especially useful for debugging built-in driver initialization, subsystem bring-up delays, and early boot paths that increase kernel startup time.

Debug Initramfs Issues with Dracut

Dracut is used by many Linux distributions to generate and manage the initramfs image used during early boot. The initramfs contains the tools, drivers, scripts, and configuration needed to discover storage, load required modules, assemble devices, and mount the real root file system before control is handed over to the main system.

When boot fails in this stage, Dracut may drop the system into an emergency shell. In many failure cases, it also generates the diagnostic report file /run/initramfs/rdsosreport.txt. This file contains boot logs, Dracut output, kernel messages, and command output collected from the initramfs environment.

Save /run/initramfs/rdsosreport.txt to a mounted partition such as /boot, or copy it to removable storage if available. Include this file in incident reports or support requests, because it often contains the most useful evidence for diagnosing root file system discovery failures, missing drivers, storage configuration issues, LVM or RAID problems, and early boot timeouts.

Common Dracut Kernel Command-Line Options

Dracut provides several rd.* kernel command-line options that help debug initramfs failures, root file system discovery problems, storage delays, and early boot hangs. These options are added through the boot loader, along with the normal kernel parameters.

  • rd.info: Prints informational Dracut output, even when quiet is present on the kernel command line.
  • rd.shell: Drops to an emergency shell if Dracut cannot mount the root file system.
  • rd.debug: Enables verbose shell tracing for Dracut scripts. If systemd is active in the initramfs, output can be inspected with journalctl -ab; otherwise, logs are written to dmesg and /run/initramfs/init.log.
  • rd.memdebug=0-5: Prints memory usage information at increasing verbosity levels, where higher values produce more detailed output.
  • rd.break=<stage>: Drops to a shell at a specific Dracut stage, such as cmdline, pre-udev, pre-trigger, initqueue, pre-mount, mount, pre-pivot, or cleanup.
  • rd.udev.info: Sets the udev log level to info during initramfs execution.
  • rd.udev.debug: Sets the udev log level to debug during initramfs execution.

Boolean rd.* parameters can usually be enabled by specifying the parameter alone or by setting it to =1, and disabled by setting it to =0. If the same parameter appears multiple times on the kernel command line, the last value is the one Dracut uses.

Dracut Debugging Examples

The following examples show common Dracut kernel command-line options used when debugging initramfs or root file system discovery issues.

Enable verbose Dracut debug output:

This is the least disruptive test. It should boot normally but produce more Dracut/initramfs logging.

sudo grubby \
  --update-kernel=/boot/vmlinuz-6.12.0-202.76.4.1.el9uek.x86_64 \
  --remove-args="rhgb quiet" \
  --args="rd.debug log_buf_len=1M"

If Dracut used systemd in the initramfs, rd.debug output can be inspected with journalctl -ab.

Drop to a shell before switching root:

Use rd.break=pre-pivot to stop Dracut after root file system discovery and mount, but before switching to the real OS root.

# sudo grubby \
  --update-kernel=/boot/vmlinuz-6.12.0-202.76.4.1.el9uek.x86_64 \
  --remove-args="rhgb quiet" \
  --args="rd.debug rd.shell rd.break=pre-pivot log_buf_len=1M"

# reboot

On the console:
pre-pivot:/#

It helps confirm:

  • Did Dracut find the root device?
  • Is the real root mounted at /sysroot?
  • Are LVM, RAID, multipath, or encrypted devices assembled?
  • Are the expected drivers and devices present?
  • Is the failure before or after initramfs handoff?

Key checks from the shell:

  • If /sysroot is mounted correctly, Dracut found the root file system.
  • If /sysroot is missing or empty, the issue is still in initramfs storage/root discovery.

Emergency shell

rd.shell enables the Dracut emergency shell on failure. The emergency shell is the troubleshooting prompt you enter after Dracut cannot continue booting.

With rd.debug rd.shell:

  • If boot succeeds, no shell appears.
  • If Dracut cannot find or mount the root file system, it may drop into an emergency shell.

Analyze Service Startup with systemd

Most modern Linux distributions use systemd as the service manager during user-space startup. After the kernel and initramfs stages complete, systemd starts services, tracks processes, manages mount and automount points, applies dependency ordering, and brings the system to the requested target state. When the system boots slowly after user space begins, systemd tools are often the best place to start.

To increase systemd logging during boot, add the following kernel command-line parameter:

systemd.log_level=debug

Additional logging controls, such as systemd.log_target=, systemd.log_location=, and systemd.log_time=, may also be useful depending on the environment and systemd version.

Measure Overall Boot Time

Use systemd-analyze without arguments to get a high-level breakdown of boot time:

$ systemd-analyze
Startup finished in 1.083s (kernel) + 4.116s (initrd) + 12.996s (userspace) = 18.197s
multi-user.target reached after 12.643s in userspace.

This separates the time spent in the kernel, initrd, and user-space phases.

Inspect the systemd Manager State

Use systemd-analyze dump to print a detailed, human-readable view of the current systemd manager state:

$ systemd-analyze dump

Manager: systemd 252 (252-55.0.3.el9_7.9)
Features: +PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSE>
Timestamp kernel: Thu 2026-06-18 04:54:16 GMT
...
...
Timestamp initrd-units-load-start: Thu 2026-06-18 04:54:17 GMT
Timestamp initrd-units-load-finish: Thu 2026-06-18 04:54:17 GMT
Subscribed: :1.2
Subscribed: :1.1
-> Unit udisks2.service:
        Description: Disk Manager
        Instance: n/a
        Unit Load State: loaded
...
...
        SystemCallErrorNumber: kill
        -> ExecStart:
                Command Line: /usr/bin/cloud-init init
                        PID: 2175
                        Start Timestamp: Thu 2026-06-18 04:54:27 GMT
                        Exit Timestamp: Thu 2026-06-18 04:54:28 GMT
                        Exit Code: exited
                        Exit Status: 0
        CPUAccounting: yes
        IOAccounting: no
...
...

The output is usually long and is intended for debugging. It should not be treated as a stable machine-readable interface.

Identify Slow Units

Use systemd-analyze blame to list units ordered by the time they took to initialize:

$ systemd-analyze blame
5.962s unified-monitoring-agent_config_downloader.service
4.353s dev-rfkill.device
4.353s sys-devices-virtual-misc-rfkill.device
4.281s sys-devices-pnp0-00:00-00:00:0-00:00:0.0-tty-ttyS0.device
4.280s dev-ttyS0.device
4.274s sys-devices-platform-serial8250-serial8250:0-serial8250:0.3-tty-ttyS3.device
4.274s dev-ttyS3.device
4.269s sys-devices-platform-serial8250-serial8250:0-serial8250:0.2-tty-ttyS2.device
4.268s dev-ttyS2.device
...
...
   6ms iscsi.service
   4ms pmie_check.service
   2ms sys-fs-fuse-connections.mount
   1ms pmlogger_farm_check.service
   1ms pmie_farm_check.service

This helps identify units with long startup times. However, a unit with a high duration is not always responsible for delaying the final boot target; it may have started in parallel and may not be on the critical path.

Show the Critical Chain

Use systemd-analyze critical-chain to display the dependency path that directly affected when the target was reached:

$ systemd-analyze critical-chain
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

multi-user.target @12.643s
└─pmie.service @7.928s +1.713s
  └─pmcd.service @6.656s +1.250s
    └─network-online.target @6.616s
      └─cloud-init.service @6.197s +416ms
...
...
                                      └─systemd-journald.socket
                                        └─system.slice
                                          └─-.slice

In this output, the value after @ shows when the unit became active or started, and the value after + shows how long the unit took to start.

Generate Boot Charts

Generate an SVG boot timeline:

systemd-analyze plot > sysd.svg

Generate a dependency graph using Graphviz:

# sudo systemd-analyze dot | dot -Tsvg > systemd.svg
   Color legend: black     = Requires
                 dark blue = Requisite
                 dark grey = Wants
                 red       = Conflicts
                 green     = After

To view it, open the image in a web page or directly with a Linux desktop GUI. Click here to view the full-size image.

Practical Debugging Workflow

A structured workflow helps narrow down whether a boot issue belongs to the kernel, initramfs, driver initialization, storage discovery, or user-space service startup path.

  • Check the active kernel command line: Verify the parameters used for the current boot with cat /proc/cmdline.
  • Increase kernel verbosity: Use loglevel=7 for debug-level console output, or ignore_loglevel to print all kernel messages regardless of the configured console log level.
  • Capture very early failures: If the system fails before the regular console is initialized, enable earlyprintk or the appropriate early console option for the platform.
  • Debug initramfs and root mount failures: If the system reaches initramfs but cannot mount the root file system, use Dracut options such as rd.debug, rd.shell, or a targeted breakpoint like rd.break=<stage>.
  • Save Dracut diagnostics: From an emergency shell, save /run/initramfs/rdsosreport.txt before rebooting. This report often contains the most useful evidence for initramfs and storage discovery problems.
  • Trace driver and subsystem initialization: Use initcall_debug to identify slow, failing, or suspicious built-in driver and subsystem initialization paths.
  • Analyze user-space startup: After the system reaches user space, use systemd-analyze, systemd-analyze blame, and systemd-analyze critical-chain to isolate slow services, dependency delays, and startup bottlenecks.
  • Escalate to deeper tracing when needed: If foundational logs do not explain the issue, use the companion deep-dive article for dynamic debug, boot-time tracing, ftrace, and kprobe-based instrumentation.

Best Practices

  • Enable one debug option at a time: Add kernel and Dracut options incrementally so the impact of each parameter is easy to understand.
  • Record the exact kernel command line: Save the full output of cat /proc/cmdline for every test boot. This makes results reproducible and helps compare different debugging attempts.
  • Preserve early boot logs before rebooting: If Dracut drops to an emergency shell, copy /run/initramfs/rdsosreport.txt to a mounted partition or removable storage before restarting the system.
  • Keep production-like systems lean: On production or performance-sensitive systems, avoid high-volume tracing unless the issue cannot be reproduced elsewhere. Use narrow trace filters and collect only the data needed.
  • Compare against a known-good boot: When possible, compare logs and traces from a failing boot with a successful boot using similar hardware, kernel, boot loader, initramfs, and service configuration.
  • Watch for timing side effects: Heavy logging and verbose debug settings can slow the boot path and affect the issue being investigated. Treat diagnostic results with that overhead in mind.
  • Clean up after debugging: Remove verbose boot parameters, tracing options, temporary breakpoints, and debug-only kernel command-line settings after the investigation is complete.

Conclusion

Linux boot diagnostics turn an otherwise silent startup path into something visible, measurable, and debuggable. Start with basic kernel verbosity, move to early console logging or Dracut diagnostics when needed, use initcall_debug for built-in initialization timing, and use systemd-analyze once the system reaches user space. If the problem needs deeper kernel instrumentation, continue with Cracking the Linux Boot Code: Deep-Dive Linux Boot Diagnostics.

References