kdump: Add firmware-assisted dump howto document
Resending this patch by updating 'fadump control flow' to reflect the latest changes. This patch adds fadump howto document to kexec-tools. The document is prepared in reference to kexec-kdump-howto.txt document. Signed-off-by: Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com> Signed-off-by: Hari Bathini <hbathini@linux.vnet.ibm.com> Acked-by: Vivek Goyal <vgoyal@redhat.com>
This commit is contained in:
Hari Bathini
WANG Chao
parent
78589a3207
commit
072fa3a65a
251
fadump-howto.txt
Normal file
251
fadump-howto.txt
Normal file
@ -0,0 +1,251 @@
|
|||||||
|
Firmware assisted dump (fadump) HOWTO
|
||||||
|
|
||||||
|
Introduction
|
||||||
|
|
||||||
|
Firmware assisted dump is a new feature in the 3.4 mainline kernel supported
|
||||||
|
only on powerpc architecture. The goal of firmware-assisted dump is to enable
|
||||||
|
the dump of a crashed system, and to do so from a fully-reset system, and to
|
||||||
|
minimize the total elapsed time until the system is back in production use. A
|
||||||
|
complete documentation on implementation can be found at
|
||||||
|
Documentation/powerpc/firmware-assisted-dump.txt in upstream linux kernel tree
|
||||||
|
from 3.4 version and above.
|
||||||
|
|
||||||
|
Please note that the firmware-assisted dump feature is only available on Power6
|
||||||
|
and above systems with recent firmware versions.
|
||||||
|
|
||||||
|
Overview
|
||||||
|
|
||||||
|
Fadump
|
||||||
|
|
||||||
|
Fadump is a robust kernel crash dumping mechanism to get reliable kernel crash
|
||||||
|
dump with assistance from firmware. This approach does not use kexec, instead
|
||||||
|
firmware assists in booting the kdump kernel while preserving memory contents.
|
||||||
|
Unlike kdump, the system is fully reset, and loaded with a fresh copy of the
|
||||||
|
kernel. In particular, PCI and I/O devices are reinitialized and are in a
|
||||||
|
clean, consistent state. This second kernel, often called a capture kernel,
|
||||||
|
boots with very little memory and captures the dump image.
|
||||||
|
|
||||||
|
The first kernel registers the sections of memory with the Power firmware for
|
||||||
|
dump preservation during OS initialization. These registered sections of memory
|
||||||
|
are reserved by the first kernel during early boot. When a system crashes, the
|
||||||
|
Power firmware fully resets the system, preserves all the system memory
|
||||||
|
contents, save the low memory (boot memory of size larger of 5% of system
|
||||||
|
RAM or 256MB) of RAM to the previous registered region. It will also save
|
||||||
|
system registers, and hardware PTE's.
|
||||||
|
|
||||||
|
Fadump is supported only on ppc64 platform. The standard kernel and capture
|
||||||
|
kernel are one and the same on ppc64.
|
||||||
|
|
||||||
|
If you're reading this document, you should already have kexec-tools
|
||||||
|
installed. If not, you install it via the following command:
|
||||||
|
|
||||||
|
# yum install kexec-tools
|
||||||
|
|
||||||
|
Fadump Operational Flow:
|
||||||
|
|
||||||
|
Like kdump, fadump also exports the ELF formatted kernel crash dump through
|
||||||
|
/proc/vmcore. Hence existing kdump infrastructure can be used to capture fadump
|
||||||
|
vmcore. The idea is to keep the functionality transparent to end user. From
|
||||||
|
user perspective there is no change in the way kdump init script works.
|
||||||
|
|
||||||
|
However, unlike kdump, fadump does not pre-load kdump kernel and initrd into
|
||||||
|
reserved memory, instead it always uses default OS initrd during second boot
|
||||||
|
after crash. Hence, for fadump, we rebuild the new kdump initrd and replace it
|
||||||
|
with default initrd. Before replacing existing default initrd we take a backup
|
||||||
|
of original default initrd for user's reference. The dracut package has been
|
||||||
|
enhanced to rebuild the default initrd with vmcore capture steps. The initrd
|
||||||
|
image is rebuilt as per the configuration in /etc/kdump.conf file.
|
||||||
|
|
||||||
|
The control flow of fadump works as follows:
|
||||||
|
01. System panics.
|
||||||
|
02. At the crash, kernel informs power firmware that kernel has crashed.
|
||||||
|
03. Firmware takes the control and reboots the entire system preserving
|
||||||
|
only the memory (resets all other devices).
|
||||||
|
04. The reboot follows the normal booting process (non-kexec).
|
||||||
|
05. The boot loader loads the default kernel and initrd from /boot
|
||||||
|
06. The default initrd loads and runs /init
|
||||||
|
07. dracut-kdump.sh script present in fadump aware default initrd checks if
|
||||||
|
'/proc/device-tree/rtas/ibm,kernel-dump' file exists before executing
|
||||||
|
steps to capture vmcore.
|
||||||
|
(This check will help to bypass the vmcore capture steps during normal boot
|
||||||
|
process.)
|
||||||
|
09. Captures dump according to /etc/kdump.conf
|
||||||
|
10. Is dump capture successful (yes goto 12, no goto 11)
|
||||||
|
11. Perfom the default action specified in /etc/kdump.conf (Default action
|
||||||
|
is reboot, if unspecified)
|
||||||
|
12. Reboot
|
||||||
|
|
||||||
|
|
||||||
|
How to configure fadump:
|
||||||
|
|
||||||
|
Again, we assume if you're reading this document, you should already have
|
||||||
|
kexec-tools installed. If not, you install it via the following command:
|
||||||
|
|
||||||
|
# yum install kexec-tools
|
||||||
|
|
||||||
|
To be able to do much of anything interesting in the way of debug analysis,
|
||||||
|
you'll also need to install the kernel-debuginfo package, of the same arch
|
||||||
|
as your running kernel, and the crash utility:
|
||||||
|
|
||||||
|
# yum --enablerepo=\*debuginfo install kernel-debuginfo.$(uname -m) crash
|
||||||
|
|
||||||
|
Next up, we need to modify some boot parameters to enable firmware assisted
|
||||||
|
dump. With the help of grubby, it's very easy to append "fadump=on" to the end
|
||||||
|
of your kernel boot parameters. Optionally, user can also append
|
||||||
|
'fadump_reserve_mem=X' kernel cmdline to specify size of the memory to reserve
|
||||||
|
for boot memory dump preservation.
|
||||||
|
|
||||||
|
# grubby --args="fadump=on" --update-kernel=/boot/vmlinuz-`uname -r`
|
||||||
|
|
||||||
|
The term 'boot memory' means size of the low memory chunk that is required for
|
||||||
|
a kernel to boot successfully when booted with restricted memory. By default,
|
||||||
|
the boot memory size will be the larger of 5% of system RAM or 256MB.
|
||||||
|
Alternatively, user can also specify boot memory size through boot parameter
|
||||||
|
'fadump_reserve_mem=' which will override the default calculated size. Use this
|
||||||
|
option if default boot memory size is not sufficient for second kernel to boot
|
||||||
|
successfully.
|
||||||
|
|
||||||
|
After making said changes, reboot your system, so that the specified memory is
|
||||||
|
reserved and left untouched by the normal system. Take note that the output of
|
||||||
|
'free -m' will show X MB less memory than without this parameter, which is
|
||||||
|
expected. If you see OOM (Out Of Memory) error messages while loading capture
|
||||||
|
kernel, then you should bump up the memory reservation size.
|
||||||
|
|
||||||
|
Now that you've got that reserved memory region set up, you want to turn on
|
||||||
|
the kdump init script:
|
||||||
|
|
||||||
|
# systemctl enable kdump.service
|
||||||
|
|
||||||
|
Then, start up kdump as well:
|
||||||
|
|
||||||
|
# systemctl start kdump.service
|
||||||
|
|
||||||
|
This should turn on the firmware assisted functionality in kernel by
|
||||||
|
echo'ing 1 to /sys/kernel/fadump_registered, leaving the system ready
|
||||||
|
to capture a vmcore upon crashing. To test this out, you can force-crash
|
||||||
|
your system by echo'ing a c into /proc/sysrq-trigger:
|
||||||
|
|
||||||
|
# echo c > /proc/sysrq-trigger
|
||||||
|
|
||||||
|
You should see some panic output, followed by the system reset and booting into
|
||||||
|
fresh copy of kernel. When default initrd loads and runs /init, vmcore should
|
||||||
|
be copied out to disk (by default, in /var/crash/<YYYY.MM.DD-HH:MM:SS>/vmcore),
|
||||||
|
then the system rebooted back into your normal kernel.
|
||||||
|
|
||||||
|
Once back to your normal kernel, you can use the previously installed crash
|
||||||
|
kernel in conjunction with the previously installed kernel-debuginfo to
|
||||||
|
perform postmortem analysis:
|
||||||
|
|
||||||
|
# crash /usr/lib/debug/lib/modules/2.6.17-1.2621.el5/vmlinux
|
||||||
|
/var/crash/2006-08-23-15:34/vmcore
|
||||||
|
|
||||||
|
crash> bt
|
||||||
|
|
||||||
|
and so on...
|
||||||
|
|
||||||
|
Saving vmcore-dmesg.txt
|
||||||
|
----------------------
|
||||||
|
Kernel log bufferes are one of the most important information available
|
||||||
|
in vmcore. Now before saving vmcore, kernel log bufferes are extracted
|
||||||
|
from /proc/vmcore and saved into a file vmcore-dmesg.txt. After
|
||||||
|
vmcore-dmesg.txt, vmcore is saved. Destination disk and directory for
|
||||||
|
vmcore-dmesg.txt is same as vmcore. Note that kernel log buffers will
|
||||||
|
not be available if dump target is raw device.
|
||||||
|
|
||||||
|
Dump Triggering methods:
|
||||||
|
|
||||||
|
This section talks about the various ways, other than a Kernel Panic, in which
|
||||||
|
fadump can be triggered. The following methods assume that fadump is configured
|
||||||
|
on your system, with the scripts enabled as described in the section above.
|
||||||
|
|
||||||
|
1) AltSysRq C
|
||||||
|
|
||||||
|
FAdump can be triggered with the combination of the 'Alt','SysRq' and 'C'
|
||||||
|
keyboard keys. Please refer to the following link for more details:
|
||||||
|
|
||||||
|
http://kbase.redhat.com/faq/FAQ_43_5559.shtm
|
||||||
|
|
||||||
|
In addition, on PowerPC boxes, fadump can also be triggered via Hardware
|
||||||
|
Management Console(HMC) using 'Ctrl', 'O' and 'C' keyboard keys.
|
||||||
|
|
||||||
|
2) Kernel OOPs
|
||||||
|
|
||||||
|
If we want to generate a dump everytime the Kernel OOPses, we can achieve this
|
||||||
|
by setting the 'Panic On OOPs' option as follows:
|
||||||
|
|
||||||
|
# echo 1 > /proc/sys/kernel/panic_on_oops
|
||||||
|
|
||||||
|
3) PowerPC specific methods:
|
||||||
|
|
||||||
|
On IBM PowerPC machines, issuing a soft reset invokes the XMON debugger(if
|
||||||
|
XMON is configured). To configure XMON one needs to compile the kernel with
|
||||||
|
the CONFIG_XMON and CONFIG_XMON_DEFAULT options, or by compiling with
|
||||||
|
CONFIG_XMON and booting the kernel with xmon=on option.
|
||||||
|
|
||||||
|
Following are the ways to remotely issue a soft reset on PowerPC boxes, which
|
||||||
|
would drop you to XMON. Pressing a 'X' (capital alphabet X) followed by an
|
||||||
|
'Enter' here will trigger the dump.
|
||||||
|
|
||||||
|
3.1) HMC
|
||||||
|
|
||||||
|
Hardware Management Console(HMC) available on Power4 and Power5 machines allow
|
||||||
|
partitions to be reset remotely. This is specially useful in hang situations
|
||||||
|
where the system is not accepting any keyboard inputs.
|
||||||
|
|
||||||
|
Once you have HMC configured, the following steps will enable you to trigger
|
||||||
|
fadump via a soft reset:
|
||||||
|
|
||||||
|
On Power4
|
||||||
|
Using GUI
|
||||||
|
|
||||||
|
* In the right pane, right click on the partition you wish to dump.
|
||||||
|
* Select "Operating System->Reset".
|
||||||
|
* Select "Soft Reset".
|
||||||
|
* Select "Yes".
|
||||||
|
|
||||||
|
Using HMC Commandline
|
||||||
|
|
||||||
|
# reset_partition -m <machine> -p <partition> -t soft
|
||||||
|
|
||||||
|
On Power5
|
||||||
|
Using GUI
|
||||||
|
|
||||||
|
* In the right pane, right click on the partition you wish to dump.
|
||||||
|
* Select "Restart Partition".
|
||||||
|
* Select "Dump".
|
||||||
|
* Select "OK".
|
||||||
|
|
||||||
|
Using HMC Commandline
|
||||||
|
|
||||||
|
# chsysstate -m <managed system name> -n <lpar name> -o dumprestart -r lpar
|
||||||
|
|
||||||
|
3.2) Blade Management Console for Blade Center
|
||||||
|
|
||||||
|
To initiate a dump operation, go to Power/Restart option under "Blade Tasks" in
|
||||||
|
the Blade Management Console. Select the corresponding blade for which you want
|
||||||
|
to initate the dump and then click "Restart blade with NMI". This issues a
|
||||||
|
system reset and invokes xmon debugger.
|
||||||
|
|
||||||
|
|
||||||
|
Advanced Setups & Default action:
|
||||||
|
|
||||||
|
Kdump and fadump exhibit similar behavior in terms of setup & default action.
|
||||||
|
For fadump advanced setup related information see section "Advanced Setups" in
|
||||||
|
"kexec-kdump-howto.txt" document. Refer to "Default action" section in "kexec-
|
||||||
|
kdump-howto.txt" document for fadump default action related information.
|
||||||
|
|
||||||
|
Compression and filtering
|
||||||
|
|
||||||
|
Refer "Compression and filtering" section in "kexec-kdump-howto.txt" document.
|
||||||
|
Compression and filtering are same for kdump & fadump.
|
||||||
|
|
||||||
|
|
||||||
|
Notes on rootfs mount:
|
||||||
|
Dracut is designed to mount rootfs by default. If rootfs mounting fails it
|
||||||
|
will refuse to go on. So fadump leaves rootfs mounting to dracut currently.
|
||||||
|
We make the assumtion that proper root= cmdline is being passed to dracut
|
||||||
|
initramfs for the time being. If you need modify "KDUMP_COMMANDLINE=" in
|
||||||
|
/etc/sysconfig/kdump, you will need to make sure that appropriate root=
|
||||||
|
options are copied from /proc/cmdline. In general it is best to append
|
||||||
|
command line options using "KDUMP_COMMANDLINE_APPEND=" instead of replacing
|
||||||
|
the original command line completely.
|
||||||
Loading…
Reference in New Issue
Block a user