Monday Jan 16, 2012

Using kexec for fast reboots on Oracle Linux with UEK.

A feature that's not often talked about in Linux is kexec. kexec is part of an infrastructure that allows the Linux kernel to load a kernel directly. Basically jump right into executing the new kernel immediately instead of going to a standard reset -> system power-on -> bios/firmware initialize -> memory/device discovery -> bootloader -> linux kernel.

kexec's mechanism is most commonly used with kdump. Basically with kdump, when a crash or panic occurs, a new kernel is booted after the crash while the memory is preserved from the previous kernel's runtime. The new kernel can then capture this data and generate the dump which then can go to local disk, remote disk or anywhere else for that matter. In order to use kdump, you basically have to allocate/reserve memory for this dump kernel. This is done by adding crashkernel=xxx@yyy to the grub command line when booting. The crash kernel image is then loaded and will be executed when a crash or panic occurs. Even though kdump is a bit cumbersome to set up, it allows for really great flexibility and is very powerful in helping with debugging issues.

For those interested in kdump, there's a good blog out there test kdump on Oracle Linux.
Or for those that just want to read the documentation that's part of the Linux kernel tree : kdump.

Anyway, this entry is not about kdump. kdump is great but I wanted to talk about the use of kexec proper and how it can help with doing fast reboots of your systems. Both Oracle Linux 5 and Oracle Linux 6 have support for reboot to use kexec as the reboot mechanism (see /etc/init.d/halt for details). When a standard reboot command is executed, init goes to 6 and /etc/init.d/halt gets run. This script, when it sees that kexec has been configured with a kernel image, will just execute kexec -e. In a standard reboot (not reboot -f) the normal shutdown scripts get executed and at the end where the system normally does a reset.
This reset then makes the system hard reset, jump into the bios, does a memory test, finds devices, initialize the devices and firmware, boot the bootdevice bootloader, start the kernel.

To set up kexec you should run the following command shortly after you boot the system. If you want to automate this, it makes sense to add this to your rc scripts. We will look at integrating this more into the OS management scripts for Oracle Linux to make it easier for the system administrators.

kexec -l --append="`cat /proc/cmdline`" 
    --initrd=/boot/initrd-`uname -r`.img /boot/vmlinuz-`uname -r`
In my case I am running 2.6.32-200.13.1.el5uek.

Once this is done, the new kernel image is prepared, memory is allocated and you now can do one of 2 things :

- run reboot : halt at the tail end of a normal reboot (shutdown all services) will execute directly into this new kernel image, exactly the same way as you booted the OS to get to this point.

- you wish to do a very fast reboot without shutdown (reboot -f). then you do

sync; umount -a ; kexec -e

In this case, you bypass all the service shutdown scripts and instantly jump start the new kernel, this is by far the fastest way to restart your box.

The total amount of time saved is highly dependent on your server. Basically time a system startup all the way to grub executing the kernel image, that's the amount of time you will save on a subsequent reboot. This can range from a number of seconds (15,20) to, sometimes, several minutes.

One caveat with the use of kexec and instant restarts without going through device resets is that, in some cases, the devices might act badly or the driver might not be doing the right thing. Before you really use this on your system, test it first to ensure that the drivers for the hardware you have and the devices themselves are doing the right thing (tm).

You can find more info about kexec in this article written a number of years ago : kexec article.

I am planning on writing a entry next about how to use this with Oracle VM Server 3.


Wim Coekaerts is the Senior Vice President of Linux and Virtualization Engineering for Oracle. He is responsible for Oracle's complete desktop to data center virtualization product line and the Oracle Linux support program.

You can follow him on Twitter at @wimcoekaerts


« July 2016