X

News, tips, partners, and perspectives for the Oracle Linux operating system and upstream Linux kernel work

Generating a vmcore in OCI

John Falkenthal
Sr. Director, Software Development

 

In this blog, Oracle Linux kernel developer Manjunath Patil demostrates how you can configure your Oracle Linux instances (both bare metal and virtual machine) running in Oracle Cloud for crash dumps.

OCI instances can generate a vmcore using kdump.

kdump is a mechanism to dump the 'memory contents of a system' [vmcore] when the system crashes. The vmcore later can be analyzed using the crash utility to understand the cause of the system crash. The kdump mechanism works by booting a second kernel [called kdump kernel or capture kernel] when the system running first kernel[called panicked kernel] crashes. The kdump kernel runs in its own reserved memory so that it wont affect the memory used by the system.

The OCI systems are all pre-configured with kdump. When an OCI instance crashes, it will generate the vmcore which can be shared with developers to understand the cause of the crash.


How to configure your Oracle Linux system with kdump

1. Pre-requisites

Make sure you have the kexec-tools rpm installed

shell # yum install kexec-tools # yum list installed | grep kexec-tools

This is the main rpm which contains the tools to configure the kdump

2. Reserve memory for kdump kernel

kdump kernel needs its own reserved memory so that when it boots, it won't use the first kernel's memory. The first kernel is told to reserve the memory for kdump kernel using crashkernel=auto kernel parameter. The first kernel needs to be rebooted for the kernel parameter to be effective.

a. Here is how we can check if the memory is reserved
# cat /proc/iomem | grep -i crash
27000000-370fffff : Crash kernel

# dmesg | grep -e "Reserving .* crashkernel"
[0.000000] Reserving 257MB of memory at 624MBfor crashkernel (System RAM: 15356MB)
b. How to set kernel parameters
OL6 systems - update the /etc/grub.conf file
OL7 systems - update the /etc/default/grub file [GRUB_CMDLINE_LINUX= line] and re-generate the grub.cfg [grub2-mkconfig -o /boot/grub2/grub.cfg]
3. Setup the serial console

Setting serial console prints progress of kdump kernel onto serial console. It would also help debugging any of the kdump kernel related issues. This setting is optional.

To set: add 'console=tty0 console=ttyS0,115200n8' kernel parameters. Addition of kernel parameters require a reboot to be effective.
4. Configuring kdump

/etc/kdump.conf is used to configure the kdump. The following are the two main configurations -

a. where to dump the vmcore?
Default location is /var/crash/
To change, update the line starting with 'path'. Make sure the new path has enough space to accommodate vmcore.
b. minimize the size of vmcore

We can reduce the size of vmcore by excluding memory pages such as pages filled with zero, user process data pages, free pages etc. This is controlled by 'core_collector' line in the config file.

Default value is 'core_collector makedumpfile -p --message-level 1 -d 31'

-p = compress the data using snappy
--message-level = print messages on console. Range 0[brevity] to 31[verbose].
-d = dump level = dictates size of vmcore. Range 0[biggest] to 31[smallest]

More on dump level and message-level in 'man makedumpfile'
5. Make kdump service run at boot time
OL6: # chkconfig kdump on; chkconfig kdump --list
OL7: # systemctl enable kdump; systemctl is-enabled kdump
6. Manually crash the system to make sure it's working
# echo c > /proc/sysrq-trigger
[After reboot]
# ls -l /var/crash/*
  1. Keep the system in configured state, so that when system crashes vmcore is collected.
8. Examples
a. OL6U10 - VM
[root@ol6u10-vm ~]# cat /proc/cmdline
ro root=UUID=... crashkernel=auto ... console=tty0 console=ttyS0,9600
 
[root@ol6u10-vm ~]# service kdump status
Kdump is operational
 
[root@ol6u10-vm ~]# cat /proc/iomem | grep -i crash
 27000000-370fffff : Crash kernel
[root@ol6u10-vm ~]# dmesg | grep -e "Reserving .* crashkernel"
[ 0.000000] Reserving 257MB of memory at 624MB for crashkernel (System RAM: 15356MB)
 
[root@ol6u10-vm ~]# echo c > /proc/sysrq-trigger
 
[After reboot]
[root@ol6u10-vm ~]# cat /etc/kdump.conf | grep -v '#' | grep -v '^$'
path /var/crash
core_collector makedumpfile -l --message-level 1 -d 31
[root@ol6u10-vm ~]# ls -lhs /var/crash/127.0.0.1-20.../
total 96M
96M -rw-------. 1 root root 96M Dec 13 00:48 vmcore
44K -rw-r--r--. 1 root root 41K Dec 13 00:48 vmcore-dmesg.txt
[root@ol6u10-vm ~]# free -h
 total used free shared buffers cached
Mem: 14G 395M 14G 208K 10M 114M
-/+ buffers/cache: 270M 14G
Swap: 8.0G 0B 8.0G
b. OL6U10 - BM
[root@ol6u10-bm ~]# cat /proc/cmdline
ro root=UUID=... crashkernel=auto ... console=tty0 console=ttyS0,9600
 
[root@ol6u10-bm ~]# service kdump status
Kdump is operational
 
[root@ol6u10-bm ~]# cat /proc/iomem | grep -i crash
 27000000-37ffffff : Crash kernel
[root@ol6u10-bm ~]# dmesg | grep -e "Reserving .* crashkernel"
[ 0.000000] Reserving 272MB of memory at 624MB for crashkernel (System RAM: 262010MB)
 
[root@ol6u10-bm ~]# echo c > /proc/sysrq-trigger
 
[After Reboot]
[root@ol6u10-bm ~]# cat /etc/kdump.conf | grep -v '#' | grep -v '^$'
path /var/crash
core_collector makedumpfile -l --message-level 1 -d 31
 
[root@ol6u10-bm ~]# ls -lhs /var/crash/127.0.0.1-20.../
total 1.1G
1.1G -rw-------. 1 root root 1.1G Dec 18 05:24 vmcore
 92K -rw-r--r--. 1 root root 90K Dec 18 05:23 vmcore-dmesg.txt
[root@ol6u10-bm ~]# free -h
 total used free shared buffers cached
Mem: 251G 1.4G 250G 224K 13M 157M
-/+ buffers/cache: 1.2G 250G
Swap: 8.0G 0B 8.0G
c. OL7U7 - VM
[root@ol7u7-vm opc]# free -h
 total used free shared buff/cache available
Mem: 14G 290M 13G 16M 274M 13G
Swap: 8.0G 0B 8.0G
 
[root@ol7u7-vm opc]# service kdump status
Redirecting to /bin/systemctl status kdump.service
 kdump.service - Crash recovery kernel arming
 Loaded: loaded (/usr/lib/systemd/system/kdump.service; enabled; vendor preset: enabled)
 Active: active (exited) ...
 ... ol7u7-vm systemd[1]: Started Crash recovery kernel arming.
 
[root@ol7u7-vm opc]# cat /proc/cmdline
BOOT_IMAGE=... crashkernel=auto ... console=tty0 console=ttyS0,9600
 
[root@ol7u7-vm opc]# dmesg | grep -e "Reserving .* crashkernel"
[ 0.000000] Reserving 257MB of memory at 624MB for crashkernel (System RAM: 15356MB)
[root@ol7u7-vm opc]# cat /proc/iomem | grep -i crash
 27000000-370fffff : Crash kernel
 
[root@ol7u7-vm ~]# ls -lhs /var/crash/127.0.0.1-20.../
total 90M
90M -rw-------. 1 root root 90M Dec 18 13:48 vmcore
48K -rw-r--r--. 1 root root 47K Dec 18 13:48 vmcore-dmesg.txt
 
[root@ol7u7-vm ~]# cat /etc/kdump.conf | grep -v '#' | grep -v "^$"
path /var/crash
core_collector makedumpfile -l --message-level 1 -d 31
d. OL7U7-BM
[root@ol7u7-bm opc]# free -h
 total used free shared buff/cache available
Mem: 251G 1.1G 250G 17M 298M 249G
Swap: 8.0G 0B 8.0G

[root@ol7u7-bm opc]# service kdump status
Redirecting to /bin/systemctl status kdump.service
 kdump.service - Crash recovery kernel arming
 Loaded: loaded (/usr/lib/systemd/system/kdump.service; enabled; vendor preset: enabled)
 Active: active (exited) ...
 ... ol7u7-bm systemd[1]: Started Crash recovery kernel arming.

[root@ol7u7-bm opc]# cat /proc/cmdline
BOOT_IMAGE=... crashkernel=auto ... console=tty0 console=ttyS0,9600

[root@ol7u7-bm opc]# cat /proc/iomem | grep -i crash
 25000000-35ffffff : Crash kernel
[root@ol7u7-bm opc]# dmesg | grep -e "Reserving .* crashkernel"
[ 0.000000] Reserving 272MB of memory at 592MB for crashkernel (System RAM: 262010MB)

[root@ol7u7-bm opc]# cat /etc/kdump.conf | grep -v '#' | grep -v '^$'
path /var/crash
core_collector makedumpfile -l --message-level 1 -d 31

[root@ol7u7-bm opc]# echo c > /proc/sysrq-trigger

[root@ol7u7-bm ~]# ls -lhs /var/crash/127.0.0.1-20.../
total 1.1G
1.1G -rw-------. 1 root root 1.1G Dec 18 14:16 vmcore
116K -rw-r--r--. 1 root root 114K Dec 18 14:16 vmcore-dmesg.txt

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.