rpms/kexec-tools
7
0
Fork 0
Code Issues Pull Requests Releases Activity

kexec-kdump-howto.txt: Add some format to the document

Browse Source 
When adding doc for the non-mounted dump target support, I found the
document are a bit uneasy to read due to lack of a proper format, this
commit should it make looks better.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
This commit is contained in:
Kairui Song 2020-05-09 18:25:23 +08:00
parent ca01cbdfd5
commit f2ffa3d340
1 changed files with 127 additions and 83 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

210
kexec-kdump-howto.txt
View File

@ -1,15 +1,22 @@
=================
Kexec/Kdump HOWTO Kexec/Kdump HOWTO
=================
Introduction Introduction
============
Kexec and kdump are new features in the 2.6 mainstream kernel. These features Kexec and kdump are new features in the 2.6 mainstream kernel. These features
are included in Red Hat Enterprise Linux 5. The purpose of these features are included in Red Hat Enterprise Linux 5. The purpose of these features
is to ensure faster boot up and creation of reliable kernel vmcores for is to ensure faster boot up and creation of reliable kernel vmcores for
diagnostic purposes. diagnostic purposes.
Overview Overview
========
Kexec Kexec
-----
Kexec is a fastboot mechanism which allows booting a Linux kernel from the Kexec is a fastboot mechanism which allows booting a Linux kernel from the
context of already running kernel without going through BIOS. BIOS can be very context of already running kernel without going through BIOS. BIOS can be very
@ -17,6 +24,7 @@ time consuming especially on the big servers with lots of peripherals. This can
save a lot of time for developers who end up booting a machine numerous times. save a lot of time for developers who end up booting a machine numerous times.
Kdump Kdump
-----
Kdump is a new kernel crash dumping mechanism and is very reliable because Kdump is a new kernel crash dumping mechanism and is very reliable because
the crash dump is captured from the context of a freshly booted kernel and the crash dump is captured from the context of a freshly booted kernel and
@ -52,7 +60,8 @@ Now reboot your system, taking note that it should bypass the BIOS:
# reboot # reboot
How to configure kdump: How to configure kdump
======================
Again, we assume if you're reading this document, you should already have Again, we assume if you're reading this document, you should already have
kexec-tools installed. If not, you install it via the following command: kexec-tools installed. If not, you install it via the following command:
@ -136,7 +145,9 @@ perform postmortem analysis:
and so on... and so on...
Notes:
Notes on kdump
==============
When kdump starts, the kdump kernel is loaded together with the kdump When kdump starts, the kdump kernel is loaded together with the kdump
initramfs. To save memory usage and disk space, the kdump initramfs is initramfs. To save memory usage and disk space, the kdump initramfs is
@ -152,8 +163,10 @@ recommended to rebuild the initramfs manually with following command:
# kdumpctl rebuild # kdumpctl rebuild
Saving vmcore-dmesg.txt Saving vmcore-dmesg.txt
---------------------- =======================
Kernel log bufferes are one of the most important information available Kernel log bufferes are one of the most important information available
in vmcore. Now before saving vmcore, kernel log bufferes are extracted in vmcore. Now before saving vmcore, kernel log bufferes are extracted
from /proc/vmcore and saved into a file vmcore-dmesg.txt. After from /proc/vmcore and saved into a file vmcore-dmesg.txt. After
@ -161,7 +174,9 @@ vmcore-dmesg.txt, vmcore is saved. Destination disk and directory for
vmcore-dmesg.txt is same as vmcore. Note that kernel log buffers will vmcore-dmesg.txt is same as vmcore. Note that kernel log buffers will
not be available if dump target is raw device. not be available if dump target is raw device.
Dump Triggering methods:
Dump Triggering methods
=======================
This section talks about the various ways, other than a Kernel Panic, in which This section talks about the various ways, other than a Kernel Panic, in which
Kdump can be triggered. The following methods assume that Kdump is configured Kdump can be triggered. The following methods assume that Kdump is configured
@ -268,7 +283,11 @@ to initate the dump and then click "Restart blade with NMI". This issues a
system reset and invokes xmon debugger. system reset and invokes xmon debugger.
Advanced Setups: Advanced Setups
===============
Dump targets
------------
In addition to being able to capture a vmcore to your system's local file In addition to being able to capture a vmcore to your system's local file
system, kdump can be configured to capture a vmcore to a number of other system, kdump can be configured to capture a vmcore to a number of other
@ -295,7 +314,6 @@ which out of the box, is fairly well documented itself. Any alterations to
the changes can be incorporated in the kdump initrd. Restarting the kdump the changes can be incorporated in the kdump initrd. Restarting the kdump
service is as simple as '/sbin/systemctl restart kdump.service'. service is as simple as '/sbin/systemctl restart kdump.service'.
Note that kdump.conf is used as a configuration mechanism for capturing dump Note that kdump.conf is used as a configuration mechanism for capturing dump
files from the initramfs (in the interests of safety), the root file system is files from the initramfs (in the interests of safety), the root file system is
mounted, and the init process is started, only as a last resort if the mounted, and the init process is started, only as a last resort if the
@ -314,7 +332,9 @@ someone uses the filesystem for something else other than dumping vmcore
you can mount it as read-only. Mkdumprd will still remount it as read-write you can mount it as read-only. Mkdumprd will still remount it as read-write
for creating dump directory and will move it back to read-only afterwards. for creating dump directory and will move it back to read-only afterwards.
Raw partition Followings are the available config options for dump targets.
1) Raw partition
Raw partition dumping requires that a disk partition in the system, at least Raw partition dumping requires that a disk partition in the system, at least
as large as the amount of memory in the system, be left unformatted. Assuming as large as the amount of memory in the system, be left unformatted. Assuming
@ -325,7 +345,7 @@ onto partition /dev/vg/lv_kdump. Restart the kdump service via
initrd. Dump target should be persistent device name, such as lvm or device initrd. Dump target should be persistent device name, such as lvm or device
mapper canonical name. mapper canonical name.
Dedicated file system 2) Dedicated file system
Similar to raw partition dumping, you can format a partition with the file Similar to raw partition dumping, you can format a partition with the file
system of your choice, Again, it should be at least as large as the amount system of your choice, Again, it should be at least as large as the amount
@ -349,7 +369,7 @@ Be careful of your filesystem selection when using this target.
It is recommended to use persistent device names or UUID/LABEL for file system It is recommended to use persistent device names or UUID/LABEL for file system
dumps. One example of persistent device is /dev/vg/<devname>. dumps. One example of persistent device is /dev/vg/<devname>.
NFS mount 3) NFS mount
Dumping over NFS requires an NFS server configured to export a file system Dumping over NFS requires an NFS server configured to export a file system
with full read/write access for the root user. All operations done within with full read/write access for the root user. All operations done within
@ -367,7 +387,7 @@ mount the NFS mount and copy out the vmcore to your NFS server. Restart the
kdump service via '/sbin/systemctl restart kdump.service' to commit this change kdump service via '/sbin/systemctl restart kdump.service' to commit this change
to your kdump initrd. to your kdump initrd.
Special mount via "dracut_args" 4) Special mount via "dracut_args"
You can utilize "dracut_args" to pass "--mount" to kdump, see dracut manpage You can utilize "dracut_args" to pass "--mount" to kdump, see dracut manpage
about the format of "--mount" for details. If there is any "--mount" specified about the format of "--mount" for details. If there is any "--mount" specified
@ -385,7 +405,7 @@ dracut_args --mount "192.168.1.1:/share /mnt/test nfs4 defaults"
NOTE: NOTE:
- <mountpoint> must be specified as an absolute path. - <mountpoint> must be specified as an absolute path.
Remote system via ssh/scp 5) Remote system via ssh/scp
Dumping over ssh/scp requires setting up passwordless ssh keys for every Dumping over ssh/scp requires setting up passwordless ssh keys for every
machine you wish to have dump via this method. First up, configure kdump.conf machine you wish to have dump via this method. First up, configure kdump.conf
@ -404,7 +424,8 @@ to send over the necessary ssh key file. Restart the kdump service via
'/sbin/systemctl restart kdump.service' to commit this change to your kdump initrd. '/sbin/systemctl restart kdump.service' to commit this change to your kdump initrd.
Path Path
==== ----
"path" represents the file system path in which vmcore will be saved. In "path" represents the file system path in which vmcore will be saved. In
fact kdump creates a directory $hostip-$date with-in "path" and saves fact kdump creates a directory $hostip-$date with-in "path" and saves
vmcore there. So practically dump is saved in $path/$hostip-$date/. To vmcore there. So practically dump is saved in $path/$hostip-$date/. To
@ -427,25 +448,26 @@ at automatically depending on what's mounted in the current system.
Following are few examples. Following are few examples.
path /var/crash/ - path /var/crash/
----------------
Assuming there is no disk mounted on /var/ or on /var/crash, dump will
be saved on disk backing rootfs in directory /var/crash.
path /var/crash/ (A separate disk mounted on /var) Assuming there is no disk mounted on /var/ or on /var/crash, dump will
-------------------------------------------------- be saved on disk backing rootfs in directory /var/crash.
Say a disk /dev/sdb is mouted on /var. In this case dump target will
become /dev/sdb and path will become "/crash" and dump will be saved
on "sdb:/crash/" directory.
path /var/crash/ (NFS mounted on /var) - path /var/crash/ (A separate disk mounted on /var)
-------------------------------------
Say foo.com:/export/tmp is mounted on /var. In this case dump target is Say a disk /dev/sdb is mouted on /var. In this case dump target will
nfs server and path will be adjusted to "/crash" and dump will be saved to become /dev/sdb and path will become "/crash" and dump will be saved
foo.com:/export/tmp/crash/ directory. on "sdb:/crash/" directory.
- path /var/crash/ (NFS mounted on /var)
Say foo.com:/export/tmp is mounted on /var. In this case dump target is
nfs server and path will be adjusted to "/crash" and dump will be saved to
foo.com:/export/tmp/crash/ directory.
Kdump boot directory Kdump boot directory
==================== --------------------
Usually kdump kernel is the same as 1st kernel. So kdump will try to find Usually kdump kernel is the same as 1st kernel. So kdump will try to find
kdump kernel under /boot according to /proc/cmdline. E.g we execute below kdump kernel under /boot according to /proc/cmdline. E.g we execute below
command and get an output: command and get an output:
@ -456,6 +478,7 @@ However a variable KDUMP_BOOTDIR in /etc/sysconfig/kdump is provided to
user if kdump kernel is put in a different directory. user if kdump kernel is put in a different directory.
Kdump Post-Capture Executable Kdump Post-Capture Executable
-----------------------------
It is possible to specify a custom script or binary you wish to run following It is possible to specify a custom script or binary you wish to run following
an attempt to capture a vmcore. The executable is passed an exit code from an attempt to capture a vmcore. The executable is passed an exit code from
@ -463,6 +486,7 @@ the capture process, which can be used to trigger different actions from
within your post-capture executable. within your post-capture executable.
Kdump Pre-Capture Executable Kdump Pre-Capture Executable
----------------------------
It is possible to specify a custom script or binary you wish to run before It is possible to specify a custom script or binary you wish to run before
capturing a vmcore. Exit status of this binary is interpreted: capturing a vmcore. Exit status of this binary is interpreted:
@ -470,6 +494,7 @@ capturing a vmcore. Exit status of this binary is interpreted:
non 0 - reboot the system non 0 - reboot the system
Extra Binaries Extra Binaries
--------------
If you have specific binaries or scripts you want to have made available If you have specific binaries or scripts you want to have made available
within your kdump initrd, you can specify them by their full path, and they within your kdump initrd, you can specify them by their full path, and they
@ -478,6 +503,7 @@ This may be particularly useful for those running post-capture scripts that
rely on other binaries. rely on other binaries.
Extra Modules Extra Modules
-------------
By default, only the bare minimum of kernel modules will be included in your By default, only the bare minimum of kernel modules will be included in your
kdump initrd. Should you wish to capture your vmcore files to a non-boot-path kdump initrd. Should you wish to capture your vmcore files to a non-boot-path
@ -486,7 +512,8 @@ need to manually specify additional kernel modules to load into your kdump
initrd. initrd.
Failure action Failure action
============== --------------
Failure action specifies what to do when dump to configured dump target Failure action specifies what to do when dump to configured dump target
fails. By default, failure action is "reboot" and that is system reboots fails. By default, failure action is "reboot" and that is system reboots
if attempt to save dump to dump target fails. if attempt to save dump to dump target fails.
@ -494,21 +521,24 @@ if attempt to save dump to dump target fails.
There are other failure actions available though. There are other failure actions available though.
- dump_to_rootfs - dump_to_rootfs
This option tries to mount root and save dump on root filesystem This option tries to mount root and save dump on root filesystem
in a path specified by "path". This option will generally make in a path specified by "path". This option will generally make
sense when dump target is not root filesystem. For example, if sense when dump target is not root filesystem. For example, if
dump is being saved over network using "ssh" then one can specify dump is being saved over network using "ssh" then one can specify
failure action to "dump_to_rootfs" to try saving dump to root failure action to "dump_to_rootfs" to try saving dump to root
filesystem if dump over network fails. filesystem if dump over network fails.
- shell - shell
Drop into a shell session inside initramfs. Drop into a shell session inside initramfs.
- halt - halt
Halt system after failure Halt system after failure
- poweroff - poweroff
Poweroff system after failure. Poweroff system after failure.
Compression and filtering Compression and filtering
-------------------------
The 'core_collector' parameter in kdump.conf allows you to specify a custom The 'core_collector' parameter in kdump.conf allows you to specify a custom
dump capture method. The most common alternate method is makedumpfile, which dump capture method. The most common alternate method is makedumpfile, which
@ -526,22 +556,21 @@ Core collector command format depends on dump target type. Typically for
filesystem (local/remote), core_collector should accept two arguments. filesystem (local/remote), core_collector should accept two arguments.
First one is source file and second one is target file. For ex. First one is source file and second one is target file. For ex.
ex1. - ex1.
---
core_collector "cp --sparse=always"
Above will effectively be translated to: core_collector "cp --sparse=always"
cp --sparse=always /proc/vmcore <dest-path>/vmcore Above will effectively be translated to:
ex2. cp --sparse=always /proc/vmcore <dest-path>/vmcore
---
core_collector "makedumpfile -l --message-level 1 -d 31"
Above will effectively be translated to: - ex2.
makedumpfile -l --message-level 1 -d 31 /proc/vmcore <dest-path>/vmcore core_collector "makedumpfile -l --message-level 1 -d 31"
Above will effectively be translated to:
makedumpfile -l --message-level 1 -d 31 /proc/vmcore <dest-path>/vmcore
For dump targets like raw and ssh, in general, core collector should expect For dump targets like raw and ssh, in general, core collector should expect
one argument (source file) and should output the processed core on standard one argument (source file) and should output the processed core on standard
@ -549,55 +578,56 @@ output (There is one exception of "scp", discussed later). This standard
output will be saved to destination using appropriate commands. output will be saved to destination using appropriate commands.
raw dumps core_collector examples: raw dumps core_collector examples:
---------
ex3.
---
core_collector "cat"
Above will effectively be translated to. - ex3.
cat /proc/vmcore | dd of=<target-device> core_collector "cat"
ex4. Above will effectively be translated to.
---
core_collector "makedumpfile -F -l --message-level 1 -d 31"
Above will effectively be translated to. cat /proc/vmcore | dd of=<target-device>
makedumpfile -F -l --message-level 1 -d 31 | dd of=<target-device> - ex4.
core_collector "makedumpfile -F -l --message-level 1 -d 31"
Above will effectively be translated to.
makedumpfile -F -l --message-level 1 -d 31 | dd of=<target-device>
ssh dumps core_collector examples: ssh dumps core_collector examples:
---------
ex5.
---
core_collector "cat"
Above will effectively be translated to. - ex5.
cat /proc/vmcore | ssh <options> <remote-location> "dd of=path/vmcore" core_collector "cat"
ex6. Above will effectively be translated to.
---
core_collector "makedumpfile -F -l --message-level 1 -d 31"
Above will effectively be translated to. cat /proc/vmcore | ssh <options> <remote-location> "dd of=path/vmcore"
makedumpfile -F -l --message-level 1 -d 31 | ssh <options> <remote-location> "dd of=path/vmcore" - ex6.
core_collector "makedumpfile -F -l --message-level 1 -d 31"
Above will effectively be translated to.
makedumpfile -F -l --message-level 1 -d 31 | ssh <options> <remote-location> "dd of=path/vmcore"
There is one exception to standard output rule for ssh dumps. And that is There is one exception to standard output rule for ssh dumps. And that is
scp. As scp can handle ssh destinations for file transfers, one can scp. As scp can handle ssh destinations for file transfers, one can
specify "scp" as core collector for ssh targets (no output on stdout). specify "scp" as core collector for ssh targets (no output on stdout).
ex7. - ex7.
----
core_collector "scp"
Above will effectively be translated to. core_collector "scp"
scp /proc/vmcore <user@host>:path/vmcore Above will effectively be translated to.
scp /proc/vmcore <user@host>:path/vmcore
About default core collector About default core collector
---------------------------- ----------------------------
Default core_collector for ssh/raw dump is: Default core_collector for ssh/raw dump is:
"makedumpfile -F -l --message-level 1 -d 31". "makedumpfile -F -l --message-level 1 -d 31".
Default core_collector for other targets is: Default core_collector for other targets is:
@ -615,7 +645,9 @@ dump data from stdard input to a normal dumpfile (readable with analysis
tools). tools).
For example: "makedumpfile -R vmcore < vmcore.flat" For example: "makedumpfile -R vmcore < vmcore.flat"
Caveats:
Caveats
=======
Console frame-buffers and X are not properly supported. If you typically run Console frame-buffers and X are not properly supported. If you typically run
with something along the lines of "vga=791" in your kernel config line or with something along the lines of "vga=791" in your kernel config line or
@ -624,7 +656,11 @@ kexec. Note that the kdump kernel should still be able to create a dump,
and when the system reboots, video should be restored to normal. and when the system reboots, video should be restored to normal.
Notes
=====
Notes on resetting video: Notes on resetting video:
-------------------------
Video is a notoriously difficult issue with kexec. Video cards contain ROM code Video is a notoriously difficult issue with kexec. Video cards contain ROM code
that controls their initial configuration and setup. This code is nominally that controls their initial configuration and setup. This code is nominally
@ -646,7 +682,9 @@ Secondly, it may be worth trying to add vga15fb.ko to the extra_modules list in
/etc/kdump.conf. This will attempt to use the video card in framebuffer mode, /etc/kdump.conf. This will attempt to use the video card in framebuffer mode,
which can blank the screen prior to the start of a dump capture. which can blank the screen prior to the start of a dump capture.
Notes on rootfs mount: Notes on rootfs mount
---------------------
Dracut is designed to mount rootfs by default. If rootfs mounting fails it Dracut is designed to mount rootfs by default. If rootfs mounting fails it
will refuse to go on. So kdump leaves rootfs mounting to dracut currently. will refuse to go on. So kdump leaves rootfs mounting to dracut currently.
We make the assumtion that proper root= cmdline is being passed to dracut We make the assumtion that proper root= cmdline is being passed to dracut
@ -656,7 +694,8 @@ options are copied from /proc/cmdline. In general it is best to append
command line options using "KDUMP_COMMANDLINE_APPEND=" instead of replacing command line options using "KDUMP_COMMANDLINE_APPEND=" instead of replacing
the original command line completely. the original command line completely.
Notes on watchdog module handling: Notes on watchdog module handling
---------------------------------
If a watchdog is active in first kernel then, we must have it's module If a watchdog is active in first kernel then, we must have it's module
loaded in crash kernel, so that either watchdog is deactivated or started loaded in crash kernel, so that either watchdog is deactivated or started
@ -670,7 +709,8 @@ not been written in watchdog-core framework then this option will not have
any effect and module will not be added. Please note that only systemd any effect and module will not be added. Please note that only systemd
watchdog daemon is supported as watchdog kick application. watchdog daemon is supported as watchdog kick application.
Notes for disk images: Notes for disk images
---------------------
Kdump initramfs is a critical component for capturing the crash dump. Kdump initramfs is a critical component for capturing the crash dump.
But it's strictly generated for the machine it will run on, and have But it's strictly generated for the machine it will run on, and have
@ -684,7 +724,8 @@ a machine with a disk image which have kdump initramfs embedded, you
should rebuild the initramfs using "kdumpctl rebuild" command manually, should rebuild the initramfs using "kdumpctl rebuild" command manually,
or else kdump may not work as expeceted. or else kdump may not work as expeceted.
Notes on encrypted dump target: Notes on encrypted dump target
------------------------------
Currently, kdump is not working well with encrypted dump target. Currently, kdump is not working well with encrypted dump target.
First, user have to give the password manually in capture kernel, First, user have to give the password manually in capture kernel,
@ -698,7 +739,8 @@ crash kernel according, or update your encryption setup.
It's recommanded to use a non-encrypted target (eg. remote target) It's recommanded to use a non-encrypted target (eg. remote target)
instead. instead.
Notes on device dump: Notes on device dump
--------------------
Device dump allows drivers to append dump data to vmcore, so you can Device dump allows drivers to append dump data to vmcore, so you can
collect driver specified debug info. The drivers could append the collect driver specified debug info. The drivers could append the
@ -715,8 +757,10 @@ the dump target setup will be included. To ensure the device dump data
will be included in the vmcore, you need to force include related will be included in the vmcore, you need to force include related
device drivers by using "extra_modules" option in /etc/kdump.conf device drivers by using "extra_modules" option in /etc/kdump.conf
Parallel Dumping Operation Parallel Dumping Operation
========================== ==========================
Kexec allows kdump using multiple cpus. So parallel feature can accelerate Kexec allows kdump using multiple cpus. So parallel feature can accelerate
dumping substantially, especially in executing compression and filter. dumping substantially, especially in executing compression and filter.
For example: For example:
@ -746,8 +790,10 @@ may lead to panic due to Out Of Memory.
hang, system reset or power-off at boot, depending on your system and runtime hang, system reset or power-off at boot, depending on your system and runtime
situation at the time of crash. situation at the time of crash.
Debugging Tips Debugging Tips
-------------- ==============
- One can drop into a shell before/after saving vmcore with the help of - One can drop into a shell before/after saving vmcore with the help of
using kdump_pre/kdump_post hooks. Use following in one of the pre/post using kdump_pre/kdump_post hooks. Use following in one of the pre/post
scripts to drop into a shell. scripts to drop into a shell.
@ -772,5 +818,3 @@ Debugging Tips
minicom -C /tmp/console-logs minicom -C /tmp/console-logs
Now minicom should be logging serial console in file console-logs. Now minicom should be logging serial console in file console-logs.