Currently kdump is not working well with encrypted targets, add document
about this issue.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Add some note about the limitation of kdumpctl's auto detect and rebuild
feature, and suggest the user to rebuild the initramfs manually on
major system change, and don't include the initramfs in disk images.
Put the note about system change in front part of the document so user
will less likely to miss it.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Previous we rebuild the initramfs when kenrel load module list changed,
but this is not very stable as some async services may load/unload
kernel modules, and cause unnecessary initramfs rebuild.
Instead, it's better to just check if the module required to dump to
the dump target is loaded or not, and rebuild if not loaded. This
avoids most false-positives, and ensure local target change is always
covered.
Currently only local fs dump target is covered, because this check
requires the dump target to be mounted when building the initramfs,
this guarantee that the module is in the loaded kernel module list,
else we may still get some false positive.
dracut-install could be leveraged to combine the modalias list with
kernel loaded module list as a more stable module list in the initramfs,
but upstream dracut change need to be done first.
Passed test on a KVM VM, changing the storage between SATA/USB/VirtIO
will trigger initramfs rebuild and didn't notice any false-positive.
Also passed test on my laptop with no false-positive.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
This reverts commit 6b479b6572.
Check initramfs rebuild by looking at if there is any change of load
kernel modules list is not very stable after all. Previously we are
counting on udev to settle before kdump is started to ensure all modules
is ready, but actually any service may cause a kernel module load, even
after udev is settled.
The previous commit is trying to workaround an issue that VM created
with disk snapshot may fail in the kdump initramfs. The better fix is to
not include the kdump initramfs in the disk snapshot at all, as the
kdump initramfs is not generated for a generic use. And With new added
"kdumpctl reload" command, admins could rebuild the image easily, and
should rebuild the initramfs on hardware change manually.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
On powerpc, after hot add cpu and trigger crash on the hot-added cpu, the
kdump kernel hangs after "I'm in purgatory".
The current udev rules expects the dtb to be rebuit on cpu add/remove event.
But since powerpc does not follow the standard cpu hot add framework, it
only ejects online/offline event to user space when cpu is hot
added/removed, instead of add/remove event. Pingfan tried fixing that but
it didn't please the maintainer as it breaks some old userspace tools.
Due to the failure of dtb's rebuilding, KDump kernel fails to get the
'boot_cpuid' and eventually fails to boot [see early_init_dt_scan_cpus() in
arch/powerpc/kernel/prom.c file] if system crashes on hot-added CPU.
Work around it by changing udev rules on powerpc to onlne/offline.
As for offline message, it is even useless on powerpc, and can be dropped.
See the explain: On powerpc, /sys/devices/system/cpu/cpuX nodes are present
for all "possible", irrespective of whether a CPU is hot-added/removed.
crash_notes are already built for all /sys/devices/system/cpu/cpuX nodes and
these nodes are present for all "possible" CPUs
(online/offline/could-be-hot-removed/could-be-hot-added)
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
kdumpctl fails to load a crash kernel if KASLR is enabled.
The incorrect configuration of the kptr_restrict parameter causes that
the kdumpctl fails to load a crash kernel. The problem occurs when the
Kernel Address Space Layout Randomization (KASLR) mechanism is enabled.
The value of kptr_restrict decides how to expose the content of /proc
files. Inappropriate setting will cause that the /proc/kallsyms file is
being printed as all zeros. This configuration will break the kdump
loading process since it needs access to the conntent of /proc/kallsyms
if KASLR is enabled.
Set kptr_restrict to 1 to avoid the problem described above.
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
These two command are not documented, update the man page to let user
know how to use them.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Use "kdumpctl rebuild" to rebuild the image directly. This could help
admins to rebuild kdump image directly.
Also merge fadump related initramfs backup/restore into setup_initrd,
and do permission only when actually trying to rebuild the image.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Seems some dead codes are left here for historical reason, just remove
them, read the config and strip comments only for once.
This improve the speed by a lot (2.6s -> 0.498s for reading a
simple config in my test case, on HDD) and make the code cleaner.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
With kernel commit 0823c68b054b ("powerpc/fadump: re-register firmware-
assisted dump if already registered") support is enabled to re-register
when FADump is alredy registered. Leverage that option in kdump scripts.
Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
Acked-by: Kairui Song <kasong@redhat.com>
The idea behind adding support for dracut '--rebuild' option was to
ensure the initrd built for fadump takes into consideration all the
build parameters passed to original initrd. Pass original initrd
instead of current default initrd for rebuild as current initrd
might already have build parameters from original initrd along
with parameters from previous fadump intird build making the
build parameters look like this after a few iterations:
-H --persistent-policy 'by-uuid' -f --quiet --hostonly --hostonly-
cmdline --hostonly-i18n --hostonly-mode 'strict' -o 'plymouth dash
resume ifcfg' --mount '/dev/mapper/rhel_zzfp219--lp3-home /kdumproot
//home xfs defaults' -f --kver '4.18.0-60.el8.ppc64le' --quiet
--hostonly --hostonly-cmdline --hostonly-i18n --hostonly-mode 'strict'
-o 'plymouth dash resume ifcfg' --mount '/dev/mapper/rhel_zzfp219--lp3-home
/kdumproot//home xfs defaults' -f --kver '4.18.0-60.el8.ppc64le' --quiet
--hostonly --hostonly-cmdline --hostonly-i18n --hostonly-mode 'strict'
-o 'plymouth dash resume ifcfg' --mount '/dev/mapper/rhel_zzfp219--lp3-home
/kdumproot//home xfs defaults' -f --kver '4.18.0-60.el8.ppc64le' --include
'/tmp/fadump.initramfs' '/etc/fadump.initramfs' --include
'/tmp/fadump.initramfs' '/etc/fadump.initramfs' --include
'/tmp/fadump.initramfs' '/etc/fadump.initramfs'
--
Since it is not desirable to build initrd with stale and/or duplicate
build parameters, use original initrd (backed up) to rebuild fadump
initrd, instead of current default initrd.
Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
Acked-by: Dave Young <dyoung@redhat.com>
eppic project have moved to github, update to latest upstream snapshot,
change source link and tar file naming style to fit github's URL format.
This fix the O0 warning reported by annocheck and passes all distro package
flag checking.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Backport the patches required to make the hardening build flags work with
kexec-tools and makedumpfile, and enabld hardening flags in spec file.
This will make the pacakge pass all warnings for kexec and makedumpfile
reported by annocheck.
Didn't find any issue with basic tests with kexec and makedumpfile.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Currently we use "\b" (word boundary) as the delimiter for ro option,
which is not correct. For mount options like
"defaults,errors=remount-ro" the ro on the tail will also be replaced
and result in an invalid mount option.
So we use a more strict logic on detecting ro mount option. It should
either starts with "," or "^" (begin of line) and ends with "," or "$"
(end of line), and keep the delimiter untouched. This should ensure
only valid mount option got detected and replaced.
This passed following tests:
defaults,ro,noauto,errors=remount-ro,nobootwait,nofail => defaults,rw,errors=remount-ro,
defaults,errors=remount-ro => defaults,errors=remount-ro
defaults,ro,relatime => defaults,rw,relatime
defaults,ro => defaults,rw
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Since early kdump is generally used for capturing vmcore when
boot-time panic occurs, if a system always reboots after capturing
vmcore, it can go into a crash loop.
To avoid this issue, this patch add a note of 'final_action' option
to the early kdump document.
Signed-off-by: Kazuhito Hagio <k-hagio@ab.jp.nec.com>
Cc: Dave Young <dyoung@redhat.com>
Cc: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Bhupesh Sharma <bhsharma@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Signed-off-by: Kairui Song <kasong@redhat.com>
If a crash occurs repeatedly after enabling kdump, the system goes
into a crash loop and the dump target may get filled up by vmcores.
This is likely especially with early kdump.
This patch introduces 'final_action' option to kdump.conf, in order
for users to be able to power off the system even after capturing
a vmcore successfully.
Signed-off-by: Kazuhito Hagio <k-hagio@ab.jp.nec.com>
Cc: Dave Young <dyoung@redhat.com>
Cc: Lianbo Jiang <lijiang@redhat.com>
Cc: Bhupesh Sharma <bhsharma@redhat.com>
Acked-by: Bhupesh Sharma <bhsharma@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Signed-off-by: Kairui Song <kasong@redhat.com>
In preparation for adding 'final_action' option, since it's confusing
to have the 'final_action' and 'default' options at the same time,
this patch introduces 'failure_action' as an alias of the 'default'
option to /etc/kdump.conf, and makes 'default' obsolete to be removed
in the future.
Also, the "default action" term is renamed to "failure action".
Signed-off-by: Kazuhito Hagio <k-hagio@ab.jp.nec.com>
Cc: Dave Young <dyoung@redhat.com>
Cc: Lianbo Jiang <lijiang@redhat.com>
Cc: Bhupesh Sharma <bhsharma@redhat.com>
Acked-by: Bhupesh Sharma <bhsharma@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Signed-off-by: Kairui Song <kasong@redhat.com>
earlykdump is not suppose to be loaded for a kdump initramfs, and user
may add it into dracut's config file so it will be included by default.
It will also make the image building always fail because earlykdump
actually detect if it's being used for kdump image and raise an error if
so.
In that case, we always force drop this module to avoid such problem.
Signed-off-by: Kairui Song <ryncsn@gmail.com>
Acked-by: Dave Young <dyoung@redhat.com>
Previously we handled the case when the installed kernel version for
early kdump is different from dracut target, it will be better to
print a warning even if installation successed, to let the user know
that an different kernel is used.
No warn message will be given if the user specified a KDUMP_KERNELVER
value, as in such case a different kernel version is used on purpose.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Currently when earlykdump failed to install required kernel image or
initramfs, it will still install the earlykdump hook and other utils.
But it won't work due to the absent of kernel image or kdump initramfs,
so the hook and installed utils is meanless.
We can't simply fail dracut building, as if earlykdump is included by
dracut config file, this may fail kernel update, where kernel image is
installed but initramfs failed to generate, and then it will fail
booting.
So this patch let it skip earlydkump install if anything is missing and
give a clean error message to let the user better ware of the situation.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
There is currently a problem with earlykdump image building, when a user
is upgrading kernel, dracut will generate new initramfs for the new
kernel, and earlykdump will install currently running version of kernel
into the initramfs, and remain the version based kernel image naming
untouched. But after a reboot the new kernel is running, and it
will try to load the image corresponding to the new kernel version by
file naming.
This patch fixes the problem by creating a symlink with unified stable
naming to the installed kernel image and initramfs, and use the symlink
instand so it will always work despite the kernel version number change.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Early kdump always fails to load the vmlinuz-xxx after the 'binutils'
package has been installed, and outputs the following messages:
...
dracut-cmdline[309]: Cannot determine the file type of /boot/vmlinuz-4.18.0-51.el8.x86_64
dracut-cmdline[309]: kexec: failed to load early-kdump kernel
...
The reason is that the vmlinuz-xxx image is mistakenly stripped when
using dracut to generate the kdump initrd. Because dracut always find
all executable binary files to strip only if the 'binutils' package
is installed, otherwise it will skip the stripping.
Therefore, remove the executable permissions of the vmlinuz-xxx in
'${initrd}' in order to let dracut skip the mistakenly stripping.
Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
Without this patch, when there are two or more spaces after 'path'
configuration phrase with ssh or nfs setting, SAVE_PATH is set to
'/var/crash' in mkdumprd, and in most cases kdump service fails to
start by checking the /var/crash directory regardless of the path
value.
ssh kdump(a)192.168.122.1
path /kdump
^^
This behavior would be too sensitive and different from the other
configurations. With this patch, mkdumprd allows such spaces.
Signed-off-by: Kazuhito Hagio <k-hagio(a)ab.jp.nec.com>
Acked-by: Kairui Song <kasong@redhat.com>
When someone is using a minimal kernel without squash module installed,
including squash dracut module will either either fail to build or fail to
boot the initramfs.
As kdump always build the image for one single kernel, we can safely just
use modprobe to check if a modules is already built in, or it exists and
loadable for the kernel we are using for kdump image, and don't include
the squash module if they are missing. Everything will still work just
fine without squash module.
We do the check in kdump dracut modules not in squash dracut module
because kdump dracut module could leverage of the KDUMP_KERNELVER variable
to know which kernel it should check against, squash dracut module may be
used to build for a generic image.
And we only check for the kernel module dependency, other binary
dependencies are either well checked or well declared in dracut.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
If default action is poweroff, we can observe that the machine is
rebooted, instead of poweroff. That is due to the following two race
processes:
systemctl poweroff
systemctl reboot -f
which is launched by kdump-error-handle.sh.
Unfortunately, although both of them are executed in systemd block
mode, but due to poweroff will tear down some internal things in
systemd, there is no guarantee for the block mode. As we can see
the msg "Failed to execute operation: Connection reset by peer",
which is thrown by "systemctl reboot -f".
poweroff and reboot share most of code, if one fails, then the other
should also fails, so it is meaningless to use reboot as the backup of
poweroff. Using "systemctl poweroff -f", the sdbus will teared down
immediately, which prevent the following "systemctl reboot -f" from
executing. Meanwhile, as man systemctl says:
-f, --force
When used with enable, overwrite any existing conflicting symlinks.
When used with halt, poweroff, reboot or kexec, execute the selected
operation without shutting down all units. However, all processes will
be killed forcibly and all file systems are unmounted or remounted read-only.
Hence, replacing the 'poweroff' with 'systemctl poweroff -f'
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
In commit 1c97aee and commit 227c185 udev rules was rewritten to use
systemd-run to run in a non-blocking mode. The problem is that it's a
bit noise, especially on machine bootup, systemd will always generate
extra logs for service start, you might see your journal full of lines
like these if you have many CPUs (each CPU generates a udev event on
boot):
...
Nov 22 22:23:05 localhost systemd[1]: Started /usr/lib/udev/kdump-udev-throttler.
Nov 22 22:23:05 localhost systemd[1]: Started /usr/lib/udev/kdump-udev-throttler.
Nov 22 22:23:05 localhost systemd[1]: Started /usr/lib/udev/kdump-udev-throttler.
Nov 22 22:23:05 localhost systemd[1]: Started /usr/lib/udev/kdump-udev-throttler.
...
While system is still booting up, kdump service is not started yet, so
systemd-run calls will end up doing nothing, the throttler being called
by systemd-run will just exit if kdump is not loaded.
This patch avoid systemd-run from being called at first place if kdump
service is not running by checking kdump.service status in udev rule,
so there won't be unnecessary logs.
Also remove the kdump service checking logic in kdump-udev-throttler as
udev is the only expected callee of this script, if it's not being
called at first place when kdump service is running, this checking will
be redundant. And even if any user called this script manually, it will
still work well as this script will call 'kdumpctl reload', it reload
the kdump resource only if kdump is loaded already.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Currently we still don't support multipath route, when parsing multipath
route kdumpctl will wrongly consider 'nexthop' as the destination address,
and raise errors in second kernel.
When multipath route is in use, ip route output should be like this:
$ /sbin/ip route show
default via 192.168.122.1 dev ens1 proto dhcp metric 100
192.168.122.0/24 dev ens1 proto kernel scope link src 192.168.122.161 metric 100
192.168.122.8
nexthop via 192.168.122.1 dev ens1 weight 50
nexthop via 192.168.122.2 dev ens1 weight 5
As we don't care about HA/performance, simply use the rule with highest
weight and ignore the rest.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
nfs service will append extra mount options to kernel mount options.
Those extra options represent current mounting details, but they may
not suitable for the second kernel. IP address may change, and we only
enable a single network stack (v4/v6), if nfs prefered another
network stack, inheriting the options will force nfs service to use
previous network stack and disable nfs's fallback mechanic and fail.
As nfs service have the capability to negotiate required protocols
and detect proper IP address, just drop those options and let nfs
automatically adapt the possible change in the second kernel.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Resolves: bz1561837
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1561837
Currently 'kdump.conf' and 'kdump.conf MAN page' entries state that the
local dump path should be:
<fs type> <partition>
- Will mount -t <fs type> <partition> <mnt>, and copy
/proc/vmcore to <mnt>/<path>/%DATE/.
The correct vmcore path instead should be:
<mnt>/<path>/%HOST_IP-%DATE/
Signed-off-by: Bhupesh Sharma <bhsharma@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
In commit b34ce3a reload support was added to kdumpctl but the usage
info is not updated. Now add reload to usage output to let user aware
of the new command.
Signed-off-by: Kairui Song <kasong@redhat.com>
Previously, kdump will restart / reload for many times on hotplug
event, especially memory hotplug events. Hotplugged memory may
generate many udev event as memory are managed and hotplugged in
small chunks by the kernel.
This results in unnecessary system workload and an actually longer
delay of kdump reload and the hotplug event, as udev will either
get blocked or kdumpctl will be waiting for other triggered operation.
To fix this, introduce a kdump-udev-throttler as an agent which will
be called by udev and merge concurrent kdump restart requests. Tested
with a Hyper-V VM which is failing due to udev timeout previously,
no new issues found.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
According to udev's man page, PROGRAM is either used to determine
device's name or whether the device matches the rule. So we should
use RUN insteand. Meanwhile, both RUN / PROGRAM only accepts very
short-running foreground tasks, but kdump restart may take a long
time if there are any device changes that will lead to image rebuild,
which may lead to buggy behavior.
On the other hand, memory / CPU hot plug should never trigger a
initramfs rebuild.
To solve this problem, we will use new introduced "kdumpctl reload"
instead, and use systemd-run to create a transient service unit for
the reload and run it in no-block mode, so udev won't be blocked by
anything.
We need to make systemd-run execute in non-blocking mode, and do not
synchronously wait for the operation to finish, because udev expect
the command line in RUN to be finished immediately, however, kdumpctl
reload may take 0.5-1s for an ordinary reload, or even slower on some
machines. So we give systemd-run an explicit --no-block option to run
in non-blocking mode. Without --no-blocking, systemd-run will verify,
enqueue and wait for the operation to finish. By using the --no-block
option, systemd-run will only verify and enqueue the unit then
return. In this way, we make sure the command is executed
asynchronously, and the status will be monitored and logged by
systemd, which is reliable and non-blocking.
Another thing to mention is that --no-block is only needed after
systemd-v220, before v220 systemd-run uses non-blocking mode by
default and --no-block option is not available on earlier systemd
versions.
Also reformat the udev rules to a more maintanceable format.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Add reload support to kdumpctl, reload will simply unload current
loaded kexec crash kernel and initramfs, and load it again.
Changes in /etc/sysconfig/kdump will take effect with kdumpctl
reload, but reloading will not check the content of
/etc/kdump.conf and won't rebuild anything. reload is fast, the only
time-consuming part of kdumpctl reload is loading kernel and initramfs
with kexec which is always necessary.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
In dracut-049, a new squash module is introduced, it can reduce the
memory usage of kdump initramfs in the capture kernel, this helps a lot
on lowering the risk of OOM failure.
Tested with latest rawhide with NFS, SSH and local dump.
Signed-off-by: Kairui Song <kasong@redhat.com>
Currently the kdumpctl script doesn't check if the raw device is
formatted which might destroy existing data at the time of dump
capture.
This patch addresses this issue, by ensuring kdumpctl prints
a warning in case it finds the raw device to be formatted.
Signed-off-by: Kenneth D'souza <kdsouza@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
Currently the script does not check if the dump target is read-only and would
always mount to read-write mode. This caused an issue with nfs mount as the
fstab options would be reconsidered while remounting to read-write mode.
The remount would fail with the below error as all options cannot be changed
runtime.
mount.nfs: mount(2): Invalid argument
mount.nfs: an incorrect mount option was specified
Which in result would not save the vmcore on the dump target.
This patch addresses this issue by checking the dump target status for read-only.
If yes, remount to read-write mode without reconsidering the fstab options.
Signed-off-by: Kenneth D'souza <kdsouza@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
