Kexec/Kdump HOWTO Introduction Kexec and kdump are new features in the 2.6 mainstream kernel. These features are included in Red Hat Enterprise Linux 5. The purpose of these features is to ensure faster boot up and creation of reliable kernel vmcores for diagnostic purposes. Overview Kexec Kexec is a fastboot mechanism which allows booting a Linux kernel from the context of already running kernel without going through BIOS. BIOS can be very time consuming especially on the big servers with lots of peripherals. This can save a lot of time for developers who end up booting a machine numerous times. Kdump Kdump is a new kernel crash dumping mechanism and is very reliable because the crash dump is captured from the context of a freshly booted kernel and not from the context of the crashed kernel. Kdump uses kexec to boot into a second kernel whenever system crashes. This second kernel, often called a capture kernel, boots with very little memory and captures the dump image. The first kernel reserves a section of memory that the second kernel uses to boot. Kexec enables booting the capture kernel without going through BIOS hence contents of first kernel's memory are preserved, which is essentially the kernel crash dump. Kdump is supported on the i686, x86_64, ia64 and ppc64 platforms. The standard kernel and capture kernel are one in the same on i686, x86_64 and ia64, while ppc64 requires a separate capture kernel (provided by the kernel-kdump package) at this time. If you're reading this document, you should already have kexec-tools installed. If not, you install it via the following command: # yum install kexec-tools Now load a kernel with kexec: # kver=`uname -r` # kexec -l /boot/vmlinuz-$kver --initrd=/boot/initrd-$kver.img \ --command-line="`cat /proc/cmdline`" NOTE: The above will boot you back into the kernel you're currently running, if you want to load a different kernel, substitute it in place of `uname -r`. Now reboot your system, taking note that it should bypass the BIOS: # reboot How to configure kdump: Again, we assume if you're reading this document, you should already have kexec-tools installed. If not, you install it via the following command: # yum install kexec-tools If you're on ppc64, you'll first need to install the kernel-kdump package: # yum install kernel-kdump To be able to do much of anything interesting in the way of debug analysis, you'll also need to install the kernel-debuginfo package, of the same arch as your running kernel, and the crash utility: # yum --enablerepo=\*debuginfo install kernel-debuginfo.$(uname -m) crash Next up, we need to modify some boot parameters to reserve a chunk of memory for the capture kernel. For i686 and x86_64, edit /etc/grub.conf, and append "crashkernel=128M@16M" to the end of your kernel line. Similarly, append the same to the append line in /etc/yaboot.conf for ppc64, followed by a /sbin/ybin to load the new configuration (not needed for grub). On ia64, edit /etc/elilo.conf, adding "crashkernel=256M@256M" to the append line for your kernel. Note that the X@Y values are such that X = the amount of memory to reserve for the capture kernel and Y = the offset into memory at which that reservation should start. Examples: # cat /etc/grub.conf # grub.conf generated by anaconda # # Note that you do not have to rerun grub after making changes to this file # NOTICE: You have a /boot partition. This means that # all kernel and initrd paths are relative to /boot/, eg. # root (hd0,0) # kernel /vmlinuz-version ro root=/dev/VolGroup00/root # initrd /initrd-version.img #boot=/dev/hda default=0 timeout=5 splashimage=(hd0,0)/grub/splash.xpm.gz hiddenmenu title Red Hat Enterprise Linux (2.6.17-1.2621.el5) root (hd0,0) kernel /vmlinuz-2.6.17-1.2621.el5 ro root=/dev/VolGroup00/root crashkernel=128M@16M initrd /initrd-2.6.17-1.2621.el5.img # cat /etc/yaboot.conf # yaboot.conf generated by anaconda boot=/dev/sda1 init-message=Welcome to Red Hat Enterprise Linux!\nHit for boot options partition=2 timeout=80 install=/usr/lib/yaboot/yaboot delay=5 enablecdboot enableofboot enablenetboot nonvram fstype=raw image=/vmlinuz-2.6.17-1.2621.el5 label=linux read-only initrd=/initrd-2.6.17-1.2621.el5.img append="root=LABEL=/ crashkernel=128M@16M" # cat /etc/elilo.conf prompt timeout=20 default=2.6.17-1.2621.el5 relocatable image=vmlinuz-2.6.17-1.2621.el5 label=2.6.17-1.2621.el5 initrd=initrd-2.6.17-1.2621.el5.img read-only append="-- root=LABEL=/ crashkernel=256M@256M" After making said changes, reboot your system, so that the X MB of memory starting Y MB into your memory is left untouched by the normal system, reserved for the capture kernel. Take note that the output of 'free -m' will show X MB less memory than without this parameter, which is expected. You may be able to get by with less than 128M, but testing with only 64M has proven unreliable of late. On ia64, as much as 512M may be required. Now that you've got that reserved memory region set up, you want to turn on the kdump init script: # chkconfig kdump on Then, start up kdump as well: # service kdump start This should load your kernel-kdump image via kexec, leaving the system ready to capture a vmcore upon crashing. To test this out, you can force-crash your system by echo'ing a c into /proc/sysrq-trigger: # echo c > /proc/sysrq-trigger You should see some panic output, followed by the system restarting into the kdump kernel. When the boot process gets to the point where it starts the kdump service, your vmcore should be copied out to disk (by default, in /var/crash//vmcore), then the system rebooted back into your normal kernel. Once back to your normal kernel, you can use the previously installed crash kernel in conjunction with the previously installed kernel-debuginfo to perform postmortem analysis: # crash /usr/lib/debug/lib/modules/2.6.17-1.2621.el5/vmlinux /var/crash/2006-08-23-15:34/vmcore crash> bt and so on... Caveats: Console frame-buffers and X are not properly supported. If you typically run with something along the lines of "vga=791" in your kernel config line or have X running, console video will be garbled when a kernel is booted via kexec. Note that the kdump kernel should still be able to create a dump, and when the system reboots, video should be restored to normal. Notes on configuration: The kexec-utils package contains two extra configuration files: /etc/sysconfig/kdump This file allows you to specify an alternate kernel to boot in the event of a panic (other than the kernel running at the moment), and allows you to override or append options on the kernel command line. It also alows you to pass extra options to the kexec utility when the kdump service is starting. See documentation in the template kdump sysconfig file for exact usage /etc/kdump.conf This file allows you to configure how kdump will record your core file. Unlike the stock version of kdump, this version of kdump attempts to record your vmcore file from the initramfs, so as to still function properly in the event that your root file system is corrupted and unmountable. This file is interrogated on kdump service start and is used to populate the initramfs for the kdump kernel with the appropriate data and utilities to copy your core file to the desired location. See documentation in /etc/kdump.conf for available config directives and targets. Note especially the ifc option. kdump will attempt to determine which network interface to use when dumping to a remote server, but due to the possibility of interface renaming, or alternate module load strategies, the interface name may change in the kdump kernel. This option is used to override that guess, so that the appropriate interface will be activated in the kdump kernel.