Unexpected reboots are among the harder problems to investigate after a system comes back online. In many cases the reboot or crash may go unnoticed until much later. Even when it is noticed, important details are often collected long after the issue occurred, by which time logs might be lost and the original failure context may be harder to understand.

The oled lastboot tool bridges this gap by automatically detecting abnormal reboots when the system comes back online. It installs a small systemd service that runs after each boot. The service determines whether the previous reboot was graceful or unexpected. It also checks whether the event looks like a kernel crash and saves a structured report for later review. For abnormal reboots, the tool can collect data through a simple hook mechanism. It can also upload the resulting artifacts to a configured endpoint when upload is enabled, including vmcore data when present.

Using oled lastboot

lastboot is part of the oled-tools package. For more information about the OLED project and installation instructions, please see Oracle Linux Enhanced Diagnostics.

The user-facing command is:

# oled lastboot
usage: lastboot [-h] {enable,disable,status,report} ...

Enable or disable OLED lastboot report systemd services

positional arguments:
  {enable,disable,status,report}
    enable              Enable lastboot services
    disable             Disable lastboot services
    status              Show lastboot service status
    report              Show the most recent saved lastboot report

options:
  -h, --help            show this help message and exit

The command manages the oled_lastboot_report.service systemd service. When enabled, this service runs once during boot after local filesystems are available. It is not a background process that keeps running. Its job is to inspect the previous reboot, save the report, run any abnormal-reboot hooks, and then exit.

Each service run writes one JSON report under:

/var/oled/lastboot-report/boot_report_YYYY-mm-dd_HH:MM:SS.json

After installation, enable the boot-time service:

# oled lastboot enable
Enabled services:
 - oled_lastboot_report.service

Check whether the service is enabled:

# oled lastboot status
oled_lastboot_report.service: enabled

After the next boot, use journalctl to verify that the service ran and generated a report:

# journalctl -u oled_lastboot_report.service -b
Feb 20 10:48:05 ol-demo lastboot-report[725]: Last boot report generated: /var/oled/lastboot-report/boot_report_2026-02-20_10:47:55.json

The report includes the host name, boot time, local timezone, reboot type, kernel crash status, detected vmcore path, and recommended follow-up actions.

Detecting graceful and unexpected reboots

The first job of the service is to determine what happened before the current boot. It uses recent reboot and shutdown records from wtmp. wtmp is the system login accounting file that records events such as boots, shutdowns, logins, and logouts.

If a shutdown record is found before the reboot record, the previous reboot is treated as graceful. If a reboot record appears without a matching shutdown, the event is treated as unexpected.

For graceful reboots, the tool saves the report and does not run any heavy collection workflow. A normal report looks like this:

# oled lastboot report
OLED Last Boot Report
=====================

Host        : ol-demo
Boot Time   : 2026-02-20 10:45:30
Time Zone   : UTC
Status      : Graceful reboot

No action required.

For unexpected reboots, lastboot continues with crash detection, recommended actions, optional hooks, and optional artifact upload.

Detecting a kernel crash

An unexpected reboot can happen for many reasons, such as power loss, host reset, kernel panic, or another failure path. To identify possible kernel crashes, lastboot searches for vmcore files in kdump locations.

It reads the path directive from:

/etc/kdump.conf

It also searches the default locations:

/var/crash
/var/oled/crash

If a vmcore file is found and its modification time is within the crash-detection window before the current boot, the event is marked as a kernel crash. lastboot uses a 20-minute window.

An abnormal reboot report with a vmcore looks like this:

# oled lastboot report
OLED Last Boot Report
=====================

Host        : ol-demo
Boot Time   : 2026-02-20 10:47:55
Time Zone   : UTC
Status      : Kernel crash
Crash Dump  : /var/crash/ol-demo-2026-02-20-10:45:38/vmcore

Recommended Actions
-------------------
- Collect vmcore
- Collect sosreport using: sos report --batch --all-logs
- Collect PCP archive for the reboot window from /var/oled/pcp/pmlogger/ol-demo/
- Collect console or serial logs if available
- File a JIRA ticket for further investigation

Collecting data with hooks

For many incidents, the most useful data is specific to the environment. One system may need RDMA data, another may need storage controller state, and another may need application-specific logs. lastboot supports this by running executable hooks after abnormal reboots.

A hook is a small executable script that lastboot runs after an abnormal reboot. Each hook can collect whatever data is useful on that system, such as logs, command output, sosreports, vmcore metadata, or application-specific files.

Hooks are installed under:

/usr/libexec/oled-tools/lastboot-hooks.d/

lastboot follows a deterministic hook model:

  • Hooks run only after an abnormal reboot is detected.
  • Hook entries are considered in lexical filename order.
  • Only regular executable files are runnable.
  • Non-executable files and non-regular entries are skipped and recorded in the saved state.
  • Each runnable hook gets a default timeout of 300 seconds.
  • A failed or timed-out hook does not prevent later hooks from running.

Each abnormal reboot event gets a deterministic event ID based on the host name and boot time. Artifacts are written under:

/var/oled/lastboot-artifacts/<event_id>/

The per-event state file, including hook execution results and related workflow details, is saved under:

/var/lib/oled/lastboot/state/<event_id>.json

Each hook receives environment variables that describe the event and where it should write output:

LASTBOOT_EVENT_ID
LASTBOOT_OUTPUT_DIR
LASTBOOT_BOOT_TIME
LASTBOOT_HOSTNAME
LASTBOOT_HOOK_NAME
LASTBOOT_HOOK_OUTPUT_DIR

A hook can be simple. To test the mechanism on a disposable system, install a small hook that writes the event metadata passed by lastboot:

# install -d -m 755 /usr/libexec/oled-tools/lastboot-hooks.d
# cat > /usr/libexec/oled-tools/lastboot-hooks.d/00-lastboot-env.sh <<'EOF'
#!/bin/bash
set -euo pipefail

out_dir="${LASTBOOT_HOOK_OUTPUT_DIR}"
mkdir -p "${out_dir}"

{
    echo "event=${LASTBOOT_EVENT_ID}"
    echo "boot_time=${LASTBOOT_BOOT_TIME}"
    echo "host=${LASTBOOT_HOSTNAME}"
} > "${out_dir}/event.txt"
EOF
# chmod 755 /usr/libexec/oled-tools/lastboot-hooks.d/00-lastboot-env.sh

After the next abnormal reboot, lastboot creates an event workspace and the hook output appears under that workspace:

# find /var/oled/lastboot-artifacts -maxdepth 3 -type f | sort
/var/oled/lastboot-artifacts/ol-demo-20260220T104755/report.json
/var/oled/lastboot-artifacts/ol-demo-20260220T104755/00-lastboot-env.sh/event.txt

The state file records the hook result. This is useful because it tells you whether a hook ran, failed, timed out, or was skipped:

# python3 -m json.tool /var/lib/oled/lastboot/state/ol-demo-20260220T104755.json
{
  "event_id": "ol-demo-20260220T104755",
  "hooks": [
    {
      "name": "00-lastboot-env.sh",
      "status": "success",
      "exit_code": 0,
      "timeout": false,
      "timeout_seconds": 300
    }
  ]
}

Per-hook timeout overrides are configured by filename in /etc/oled/lastboot.json. For example, a sosreport hook may need more than the default 300 seconds:

{
  "hooks": {
    "scripts": {
      "50-sosreport.sh": {
        "timeout_seconds": 1200
      }
    }
  }
}

Reading service and report results

The report generator starts under systemd, but it is not a long-running daemon. It is a Type=oneshot service: it runs at boot, writes the report, runs any abnormal-reboot workflow, and exits. As a result, a successful service may show as inactive after it has completed. The important part is the exit status and the journal output.

# systemctl status oled_lastboot_report.service --no-pager
○ oled_lastboot_report.service - OLED last boot analysis report
     Loaded: loaded (/usr/lib/systemd/system/oled_lastboot_report.service; enabled; preset: disabled)
     Active: inactive (dead) since Fri 2026-02-20 10:48:05 UTC
   Main PID: 725 (code=exited, status=0/SUCCESS)

The service journal shows what happened during boot:

# journalctl -u oled_lastboot_report.service -b
Feb 20 10:48:04 ol-demo lastboot-report[725]: Hook 00-lastboot-env.sh completed successfully
Feb 20 10:48:05 ol-demo lastboot-report[725]: Last boot report generated: /var/oled/lastboot-report/boot_report_2026-02-20_10:47:55.json

The easiest summary is still the oled lastboot report command. The Status field tells you how to read the event:

  • Graceful reboot means a shutdown record was found before the reboot record. lastboot saves the report and does not run abnormal-reboot hooks.
  • Unexpected reboot means the previous boot did not have a matching graceful shutdown record. Hooks can run, but no matching vmcore was found.
  • Kernel crash means the reboot was unexpected and a recent vmcore was found in a kdump search directory.

For abnormal reboots, the JSON state file provides the detailed workflow result. The most useful fields to check are:

  • report – the saved reboot classification and vmcore information.
  • hooks – one entry per hook candidate, including status, exit_code, timeout, and stderr_tail.
  • reboot_loop – whether hook execution was skipped because repeated abnormal reboots were detected.

Uploading artifacts

After hooks run, lastboot can upload the collected abnormal reboot data to an HTTP endpoint. This is configured in:

/etc/oled/lastboot.json

The default configuration keeps upload disabled:

{
  "hooks": {
    "scripts": {}
  },
  "upload": {
    "enabled": false,
    "vmcore_enabled": true,
    "reboot_loop": {
      "enabled": true,
      "threshold_minutes": 20,
      "state_file_upload_limit": 3
    },
    "endpoint": "",
    "method": "PUT",
    "idempotency_header": "",
    "timeout_seconds": 1200,
    "retries": 3,
    "initial_backoff_seconds": 10,
    "headers": {},
    "auth": {
      "token_file": "",
      "token_header": "Authorization",
      "token_prefix": "Bearer ",
      "header_name": "",
      "header_value": ""
    }
  }
}

When upload is enabled, lastboot creates a single tar.gz archive containing the event workspace. If upload.vmcore_enabled is true and a vmcore was detected, regular files from the detected kdump directory are added to the same archive.

The state file records the upload result as well. The most useful upload fields to check are:

  • upload.status – whether upload succeeded, failed, or was skipped.
  • upload.skipped_reason or upload.files[].error – why upload did not complete.
  • upload.files[].http_status – the HTTP response code returned by the upload endpoint.

The archive name is:

<event_id>.tar.gz

The upload endpoint can be a fixed URL, or it can include {filename} or <filename> as a placeholder. The placeholder is replaced with the generated archive name. This is useful when the endpoint is an OCI Object Storage PAR URL or another HTTP destination where each event should land as a distinct object.

The following example enables upload. It includes kdump files in the archive when a vmcore is detected and sends the archive with an HTTP PUT request:

{
  "upload": {
    "enabled": true,
    "vmcore_enabled": true,
    "endpoint": "<upload-url>/{filename}",
    "method": "PUT",
    "idempotency_header": "X-Idempotency-Key",
    "timeout_seconds": 1200,
    "retries": 3,
    "initial_backoff_seconds": 10,
    "headers": {},
    "auth": {
      "token_file": "/etc/oled/lastboot-upload.token",
      "token_header": "Authorization",
      "token_prefix": "Bearer ",
      "header_name": "",
      "header_value": ""
    }
  }
}

lastboot retries transient failures such as HTTP 429 and 5xx responses. If an idempotency header is configured, lastboot sends an idempotency key in the form:

<event_id>:<archive_sha256>

This helps the receiver identify repeated uploads of the same event archive.

Reboot-loop suppression

During a reboot loop, repeatedly running heavy collection hooks can make the situation worse. lastboot includes optional reboot-loop suppression to reduce that risk.

When this setting is enabled, lastboot compares the current abnormal boot with the previous saved abnormal boot. If the time between them is within the configured threshold, the event is treated as part of a reboot loop. In that case, lastboot skips hook execution and other heavy collection work. It can upload only the state file for the first few loop events. The state_file_upload_limit setting controls how many of those state-file-only uploads are allowed.

The default configuration enables this protection with a 20-minute threshold and a state-file upload limit of 3:

{
  "upload": {
    "reboot_loop": {
      "enabled": true,
      "threshold_minutes": 20,
      "state_file_upload_limit": 3
    }
  }
}

Viewing reports and runtime data

To stop generating reports on future boots, disable the service:

# oled lastboot disable
Disabled services:
 - oled_lastboot_report.service

Display the latest saved report:

# oled lastboot report
OLED Last Boot Report
=====================

Host        : ol-demo
Boot Time   : 2026-02-20 10:45:30
Time Zone   : UTC
Status      : Graceful reboot

No action required.

Useful runtime locations are:

# Latest boot reports
/var/oled/lastboot-report/

# Per-event artifact workspaces
/var/oled/lastboot-artifacts/

# Persisted abnormal reboot state
/var/lib/oled/lastboot/state/

# Service logs for the current boot
journalctl -u oled_lastboot_report.service -b

End-to-end validation

For end-to-end validation on a disposable VM, enable the service and trigger a test kernel crash:

# oled lastboot enable
# echo 1 > /proc/sys/kernel/sysrq
# echo c > /proc/sysrq-trigger

After the system reboots, inspect the report, service logs, artifacts, and state:

# oled lastboot report --json
# journalctl -u oled_lastboot_report.service -b
# find /var/oled/lastboot-artifacts -maxdepth 3 -type f | sort
# find /var/lib/oled/lastboot/state -maxdepth 1 -type f | sort

Conclusion

The oled lastboot tool gives administrators a simple workflow for reviewing what happened during the previous reboot. After you enable the service, it generates a report after each boot. Use oled lastboot report to view the latest result. Use journalctl and the saved state file when more detail is needed.

This is especially useful for abnormal reboots because the tool detects the event and starts data collection as soon as the system comes back online.

For abnormal reboots, lastboot can do more than report the event. It can run hooks, collect site-specific data, and upload artifacts when configured. This makes it useful on systems where post-reboot evidence must be collected quickly and consistently.