As Ksplice has evolved, we’ve had to adapt it for new technologies including altinstructions, jump labels, static calls, retpolines. Recently, we’ve had a few reports of incompatibility between Ksplice and eBPF programs tracing the kernel. That issue involved a known incompatibility between ksplice and ftrace which we’ve now resolved. This adaptation for ftrace and eBPF is the latest evolution as we adapt Ksplice to new features in the kernel. As the Linux kernel continues to evolve, you can count on Ksplice to deliver live patching so you can continue to update your kernels without downtime.

Previously, Ksplice intentionally disabled ftrace for functions that we patched in order to prevent conflicts. This wasn’t a problem because most users were not tracing the same functions we were patching. Recently, we’ve found that anti-virus software may want to trace the same functions that we have patched, but would be prevented because ftrace was disabled. With our latest updates, ftrace, eBPF and Ksplice can now patch and trace the same functions. Read on for details!

About tracing and eBPF

eBPF is the most popular way for loading and executing programs within the kernel space, often related to tracing in the kernel. eBPF provides a flexible and efficient way to filter, monitor, and manipulate network packets, system calls, and other kernel events. It enables custom networking, security, and performance monitoring solutions, and is widely used in cloud, container, and serverless environments.

eBPF programs often leverage ftrace to attach eBPF programs to kernel events.

Ftrace stack

When ftrace is enabled, the -pg flag is passed to GCC, which adds a call to the mcount symbol to the prologue of each function. A section containing all these locations is created, either by GCC or the Linux kernel build system. During boot, the Linux kernel ftrace infrastructure uses this section to replace the calls with nops if not already done at compile time. When a user enables ftrace for a function, ftrace replaces the nops with a call to a ftrace handler, which can perform different actions depending on the tracer. For example, the ‘function’ tracer prints information about function calls, including the function name, the call stack, and the time spent in the function.

A Conflict over 5 bytes

Historically, Ksplice has used a trampoline in the prologue of a function to redirect its execution. However, this conflicts with ftrace, which also uses the first 5 bytes of the function. To resolve this issue, Ksplice had to disable ftrace for the functions being patched and abort patching if ftrace was enabled during the patching process. This meant that ftrace users had to attach to the patched function for tracing instead of the original one. Customers using anti-virus software had to disable that software prior to applying Ksplice updates – this isn’t what we wanted, we want Ksplices to apply without requiring the operator to make other changes.

Shifting by 5 bytes

The solution is, of course, to move our trampoline over by 5 bytes and remove the code that disables ftrace. But this is more challenging than it appears. In addition to shifting the trampoline over, we also had to make sure a number of other fixes could be applied, including:

  • Handle kernel text segment modifications: The kernel has multiple ways to modify its text segment at runtime, such as jump labels used by pr_debug() calls. Ksplice had to make sure that its trampoline was not overwritten by these modifications.
  • Resolve symbol duplication: Ksplice was emitting symbols for the function being patched with the same name as the original symbol. This caused issues with eBPF program attachment, which relies on kallsyms to find matching addresses. To resolve this, Ksplice now mangles the name of the function replacing the function being patched.
  • Provide consistent function calls: Ksplice had to ensure that all calls to a function always go through the original function, even if it’s been patched. This required changes to the Ksplice relocation resolution mechanism.

Ksplice has come a long way since its inception patching v2.6.32 kernels, adapting to the changing Linux kernel landscape and addressing the challenges posed by new technologies, as we’ve had to do for many other technologies like altinstructions, jump labels, static calls, retpolines. This adaptation for ftrace and eBPF is the latest evolution as we adapt Ksplice to new features in the kernel. As the Linux kernel continues to evolve, you can count on Ksplice to deliver live patching so you can continue to update your kernels without downtime.

Getting the Fix

Ksplice has been enhanced when used with some newer kernels, reducing the risk of hitting this error when third-party tools are installed. Updating and rebooting into one of the following boot kernels (uname -r) may reduce or eliminate this issue:

  • UEK Relese 7 on Oracle Linux 8 or 9: 5.15.0-308.179.6, or newer for x86, 5.15.0-312.187.5 or new for aarch64
  • UEK Release 8 on Oracle Linux 9: 6.12.0-0.20.20, or newer for x86, 6.12.0-102.36.5.2 or newer for aarch64
  • Ubuntu 22.04 (Jammy) within Oracle Cloud: 5.15.0-142.152, or newer

Oracle recommends updating boot kernels to the latest versions available, which will still contain these fixes, rather than installing these specific versions.