Introduction

In the first part of this three part blog series, I explained how to set up an Oracle Linux virtual machine where you can experiment with the debugging techniques that will be discussed in the blog series. This post will cover “basic” debugging techniques which utilize gdb, and the final post will discuss some more advanced techniques which use tools specifically designed for kernel debugging.

Getting Started

If you followed the setup instructions in the previous post, you should have an Oracle Linux virtual machine set up on your system. Before we begin, you’ll want to start your VM (if it’s not already running), using a command like this:

sudo virsh start ol8.4

Once the VM has booted, log into it with ssh, as described in the previous post. After logging in, we’re ready to begin experimenting with some basic debugging techniques.

gdb on /proc/kcore

This is probably the most basic method of debugging a running kernel. It allows you to get a “read-only” view of what’s currently going on in kernel-space. This can be very useful for looking at things that are in a “steady state” or that can be controlled in some manner, but it won’t be useful for anything that is rapidly changing.

Before trying out this technique, we need to ensure that gdb is installed in our testing VM. Log into the VM as root and run:

dnf install -y gdb

Now that gdb is installed, we can look at a few examples of things that we can do with it.

Finding the Value of the linux_banner String Using its Address

A good example of something valid to examine with gdb on /proc/kcore is the linux_banner string. This is a constant string that is built into the kernel binary, so it will never change while the kernel is running.

We’ll first use /proc/kallsyms to find the address of the string, and then dump it out using gdb. This technique can be useful when a debug info kernel is not readily available, as the /proc/kallsyms file is available on any kernel that was built with CONFIG_KALLSYMS=y.

Get the Address of the linux_banner Symbol

We first need to determine the address of the linux_banner string in the running kernel. The simplest way to do this is to grep for linux_banner in /proc/kallsyms in our test VM, like this:

# grep linux_banner /proc/kallsyms
ffffffff81e001c0 R linux_banner

This technique can be used for finding the address of any global/exported symbol in the kernel.

Use gdb to Read the String at the Address

First, point gdb at /proc/kcore:

gdb -q /proc/kcore

Now use the x/s command to examine the string at the address we just grabbed:

(gdb) x/s 0xffffffff81e001c0
0xffffffff81e001c0:     "Linux version 5.4.17-2102.201.3.el8uek.x86_64 (mockbuild@host-100-100-224-44) (gcc version 8.3.1 20190507 (Red Hat 8.3.1-4.5.0.8) (GCC)) #2 SMP Fri Apr 23 09:05:57 PDT 2021\n"

Finding the Value of the linux_banner Symbolically

Knowing how to look up symbol addresses in /proc/kallsyms is a valuable skill, but we’ve gone to the trouble of acquiring the debug info kernel, so we can look up this string symbolically as well.

Copy the Debug Info Kernel Into the VM

We need to copy the vmlinux binary built with debug info into the VM in order to use it with gdb. Running a command like this on the host system will copy it into root‘s home directory on the VM:

scp /usr/lib/debug/usr/lib/modules/5.4.17-2102.201.3.el8uek.x86_64/vmlinux root@192.168.122.202:~

Use gdb in Combination With the Debug Kernel to Dump the linux_banner String

We can now provide the copied-in vmlinux binary as the executable file for gdb, which gives us access to all the debugging information that has been built into that kernel:

gdb -q /root/vmlinux /proc/kcore

Similar to before, we can use x/s to dump out the desired string. However, this time we don’t need to know the address in advance, we can just provide the linux_banner symbol name:

(gdb) x/s linux_banner
0xffffffff81e001c0 <linux_banner>:      "Linux version 5.4.17-2102.201.3.el8uek.x86_64 (mockbuild@host-100-100-224-44) (gcc version 8.3.1 20190507 (Red Hat 8.3.1-4.5.0.8) (GCC)) #2 SMP Fri Apr 23 09:05:57 PDT 2021\n"

The debug info also contains information about what type of symbol linux_banner actually is:

(gdb) whatis linux_banner
type = const char []

We can also use gdb‘s print command to achieve the same result:

(gdb) print linux_banner
$7 = 0xffffffff81e001c0 <linux_banner> "Linux version 5.4.17-2102.201.3.el8uek.x86_64 (mockbuild@host-100-100-224-44) (gcc version 8.3.1 20190507 (Red Hat 8.3.1-4.5.0.8) (GCC)) #2 SMP Fri Apr 23 09:05:57 PDT 2021\n"

Note that print will not work like this if we just provide the address that we gathered previously:

(gdb) print/c *0xffffffff81e001c0
$15 = 76 'L'

We need to use the * operator here to tell gdb to examine the value at the address, as opposed to just printing the value provided to the command, and we also have to provide the /c bit after print to tell it to treat the retrieved value as a character instead of an integer. We can see that the retrieved character is an L, as expected, but this is obviously not going to be an efficient way to dump out an entire string.

It is possible to make the print command work properly with the address, using a cast:

(gdb) print (const char *) 0xffffffff81e001c0
$14 = 0xffffffff81e001c0 <linux_banner> "Linux version 5.4.17-2102.201.3.el8uek.x86_64 (mockbuild@host-100-100-224-44) (gcc version 8.3.1 20190507 (Red Hat 8.3.1-4.5.0.8) (GCC)) #2 SMP Fri Apr 23 09:05:57 PDT 2021\n"

However, this is obviously not as convenient as leveraging the debug info to do some of this work for us. Note that we don’t use the * operator, or the /c, before the address this time, because the cast explicitly tells gdb to treat the address as a char pointer.

gdb with Qemu’s gdbserver

Using this technique is a natural next step from using gdb directly on the system being debugged. By utilizing Qemu’s built-in ability to communicate with a gdb process running on the host system, we can do everything that’s possible while running gdb against /proc/kcore, and more. Not only can we set breakpoints to halt the system, allowing us to examine things that are not necessarily “slow moving” (i.e. the state of running processes, IO requests, etc.), but Qemu also provides many other VM-specific capabilities through the monitor command.

Configuring the VM to Start a gdb Server

We need to manually edit our VM’s configuration XML to tell qemu to start the gdbserver when the VM starts.

To do this, we first run the following command on the host system to open the configuration XML in our editor:

sudo virsh edit ol8.4

In my case, the first line of the file is:

<domain type='kvm'>

This needs to be mofidied to look like this:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

This change explicitly pulls in the qemu namespace, allowing you to utilize various debugging and experimental features provided by libvirt. These libvirt features are specifically for working with qemu, however they may not be fully supported by your Linux distribution.

After that you can add the necessary qemu:commandline tag directly below that line:

  <qemu:commandline>
    <qemu:arg value='-s'/>
  </qemu:commandline>

I generally try to stick with the same indentation that the XML file had originally, though this does not appear to be critical, as it will correct formatting issues (and re-order the tags beneath domain) when you close and save the file.

This extra tag will add the -s argument to the qemu command line when libvirt starts qemu. This will cause qemu to pause before booting the kernel and wait for a gdb process to attach to a gdbserver running on port 1234.

Restart the VM

If your VM is already running, you’ll need to shut it down and start it again to attach gdb.

It’s generally best to gracefully shut down the VM by logging into as root and running:

shutdown now

The VM should shut down relatively quickly after this. You can confirm that it has shut down by running the following command from the host:

$ sudo virsh list --all
 Id   Name    State
------------------------
 -    ol8.4   shut off

Here we can see that the VM is indeed completely shut down. Now we can start the VM again using:

sudo virsh start ol8.4

At this point qemu will start up, but pause and wait for gdb to connect.

Connect gdb to the VM

To connect gdb to our VM, we need to first start up gdb, pointing it at our kernel’s debuginfo binary:

gdb /usr/lib/debug/usr/lib/modules/5.4.17-2102.201.3.el8uek.x86_64/vmlinux

Now we issue the following commands in gdb:

set architecture i386:x86-64:intel
target remote :1234

If successful, you should see a response similar to this from gdb:

Remote debugging using :1234
0xffffffff819dda92 in native_safe_halt () at ./arch/x86/include/asm/irqflags.h:60
60              asm volatile("sti; hlt": : :"memory");

At this point we can issue gdb‘s continue command to move forward with booting the kernel:

(gdb) continue
Continuing.

Setting a Breakpoint in the VM

At this point, we can do everything that I covered in the section about using gdb on /proc/kcore, but we’re also able to set a breakpoint to halt the system and allow us to examine things.

To set a breakpoint, we first need to pause the VM again from gdb, to allow us to enter some commands. To do this just issue a Ctrl-C to the terminal running your gdb session. gdb will respond with something like:

Thread 1 received signal SIGINT, Interrupt.
0xffffffff819dda92 in native_safe_halt () at ./arch/x86/include/asm/irqflags.h:60
60              asm volatile("sti; hlt": : :"memory");

Now we will set a breakpoint on a function which we can easily cause to be executed from userspace:

(gdb) break show_cpuinfo
Breakpoint 1 at 0xffffffff8104c650: file arch/x86/kernel/cpu/proc.c, line 58.

After this we can issue the continue command again, to allow the VM to continue running.

With this breakpoint set, log into your VM as root via SSH (in a separate terminal, of course), and run the following command:

cat /proc/cpuinfo

Back in your gdb terminal, you should see some output similar to this:

(gdb) [Switching to Thread 1.2]

Thread 2 hit Breakpoint 1, show_cpuinfo (m=0xffff888069f54300, v=0xffff88806c410260) at arch/x86/kernel/cpu/proc.c:58
58      {
(gdb)

You’re now able to examine the state of the system right before show_cpuinfo starts to execute. You can see the backtrace that lead to this breakpoint using gdb‘s bt command:

(gdb) bt
#0  show_cpuinfo (m=0xffff888069f54300, v=0xffff88806c410260) at arch/x86/kernel/cpu/proc.c:58
#1  0xffffffff8130d777 in seq_read (file=<optimized out>, buf=<optimized out>, size=<optimized out>, ppos=<optimized out>) at fs/seq_file.c:229
#2  0xffffffff8137723e in proc_reg_read (file=<optimized out>, buf=<optimized out>, count=<optimized out>, ppos=<optimized out>) at fs/proc/inode.c:223
#3  0xffffffff812e2b3b in __vfs_read (file=<optimized out>, buf=<optimized out>, count=<optimized out>, pos=<optimized out>) at fs/read_write.c:425
#4  0xffffffff812e2bf9 in vfs_read (pos=<optimized out>, count=<optimized out>, buf=<optimized out>, file=<optimized out>) at fs/read_write.c:461
#5  vfs_read (file=0xffff888069a47000, buf=0x7f66bd6c2000 <error: Cannot access memory at address 0x7f66bd6c2000>, count=131072, pos=0xffffc90000367ee8) at fs/read_write.c:446
#6  0xffffffff812e2f71 in ksys_read (fd=<optimized out>, buf=0x7f66bd6c2000 <error: Cannot access memory at address 0x7f66bd6c2000>, count=131072) at fs/read_write.c:587
#7  0xffffffff812e300a in __do_sys_read (count=<optimized out>, buf=<optimized out>, fd=<optimized out>) at fs/read_write.c:597
#8  __se_sys_read (count=<optimized out>, buf=<optimized out>, fd=<optimized out>) at fs/read_write.c:595
#9  __x64_sys_read (regs=<optimized out>) at fs/read_write.c:595
#10 0xffffffff81004500 in do_syscall_64 (nr=<optimized out>, regs=0xffff88806c410260) at arch/x86/entry/common.c:296
#11 0xffffffff81a001b8 in entry_SYSCALL_64 () at arch/x86/entry/entry_64.S:179
Backtrace stopped: Cannot access memory at address 0x1ed80

From here, any number of things are possible. You’re able to step through code or examine anything you’re interested in, all with the system in a halted state, so that nothing is changing while you’re poking around.

gdb Python Scripts

At this point, it’s a good time to mention the gdb Python scripts that are included with the source for recent Linux kernels. These need to be “built” before they can be used, but this is relatively simple.

First go to where you installed your kernel source. Mine is found here:

~/rpmbuild/BUILD/kernel-5.4.17/linux-5.4.17-2102.201.3.el8uek

As mentioned above, you’ll need to copy the appropriate config file into your kernel before running any kind of build:

cp ~/rpmbuild/SOURCES/config-x86_64 ./.config

Note that you may need to look in /boot to find the appropriate config (or search for it elsewhere) if you are using a git repository instead of the source RPM.

Now you can run the following command in your kernel source tree to prepare the copied-in config:

yes '' | make oldconfig

And then run the following to prepare the gdb scripts:

make scripts_gdb

After this, you can pull in the Python scripts from your gdb session. If you’re following along, your VM should still be paused. Otherwise, you’ll need to connect to your VM with gdb and pause it with Ctrl-C before running:

source /path/to/home/rpmbuild/BUILD/kernel-5.4.17/linux-5.4.17-2102.201.3.el8uek/vmlinux-gdb.py

Note that you need to provide the absolute path to vmlinux-gdb.py in order for this to work properly.

At this point, you can use:

apropos lx

To get some info on what commands and functions are provided by vmlinux-gdb.py.

Many of these commands are self-explanatory. For instance the lx-ps command will print out a list of all the running tasks on the system, along with their task_struct address, and PID:

(gdb) lx-ps
0xffffffff82414780 <init_task> 0 swapper/0
0xffff88800fd94800 1 systemd
0xffff88800fd91800 2 kthreadd
0xffff88800fd93000 3 rcu_gp
0xffff88800fd96000 4 rcu_par_gp
...

We can use the lx_task_by_pid function to dump a task_struct for a particular process, using its PID. That function is used like this:

print $lx_task_by_pid(<PID>)

I won’t show example output here, as this will dump a fairly large amount of information.

Many of these commands provide functionality that you can get by just logging into the system and running commands from a shell. However, it’s often valuable to be able to examine some of this information while the system is paused, which is where these commands become useful. These commands can also be used on kernel crash dumps, which can be very useful, as you’re obviously not able to log in and run commands on a core dump.

Shut Down the VM

At this point we’ve covered both of the basic gdb-based tehcniques that I have used for various kernel debugging tasks. Make sure you shut down your VM by logging into it and running:

shutdown now

Next up

In the next post I’ll discuss using the kernel’s built in kdb and kgdboc debugging utilities, as well as a utility called crash, which was specifically written to analyze both kernel core dumps and running kernels.