There is currently a problem with earlykdump image building, when a user
is upgrading kernel, dracut will generate new initramfs for the new
kernel, and earlykdump will install currently running version of kernel
into the initramfs, and remain the version based kernel image naming
untouched. But after a reboot the new kernel is running, and it
will try to load the image corresponding to the new kernel version by
file naming.
This patch fixes the problem by creating a symlink with unified stable
naming to the installed kernel image and initramfs, and use the symlink
instand so it will always work despite the kernel version number change.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Early kdump always fails to load the vmlinuz-xxx after the 'binutils'
package has been installed, and outputs the following messages:
...
dracut-cmdline[309]: Cannot determine the file type of /boot/vmlinuz-4.18.0-51.el8.x86_64
dracut-cmdline[309]: kexec: failed to load early-kdump kernel
...
The reason is that the vmlinuz-xxx image is mistakenly stripped when
using dracut to generate the kdump initrd. Because dracut always find
all executable binary files to strip only if the 'binutils' package
is installed, otherwise it will skip the stripping.
Therefore, remove the executable permissions of the vmlinuz-xxx in
'${initrd}' in order to let dracut skip the mistakenly stripping.
Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
Without this patch, when there are two or more spaces after 'path'
configuration phrase with ssh or nfs setting, SAVE_PATH is set to
'/var/crash' in mkdumprd, and in most cases kdump service fails to
start by checking the /var/crash directory regardless of the path
value.
ssh kdump(a)192.168.122.1
path /kdump
^^
This behavior would be too sensitive and different from the other
configurations. With this patch, mkdumprd allows such spaces.
Signed-off-by: Kazuhito Hagio <k-hagio(a)ab.jp.nec.com>
Acked-by: Kairui Song <kasong@redhat.com>
When someone is using a minimal kernel without squash module installed,
including squash dracut module will either either fail to build or fail to
boot the initramfs.
As kdump always build the image for one single kernel, we can safely just
use modprobe to check if a modules is already built in, or it exists and
loadable for the kernel we are using for kdump image, and don't include
the squash module if they are missing. Everything will still work just
fine without squash module.
We do the check in kdump dracut modules not in squash dracut module
because kdump dracut module could leverage of the KDUMP_KERNELVER variable
to know which kernel it should check against, squash dracut module may be
used to build for a generic image.
And we only check for the kernel module dependency, other binary
dependencies are either well checked or well declared in dracut.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
If default action is poweroff, we can observe that the machine is
rebooted, instead of poweroff. That is due to the following two race
processes:
systemctl poweroff
systemctl reboot -f
which is launched by kdump-error-handle.sh.
Unfortunately, although both of them are executed in systemd block
mode, but due to poweroff will tear down some internal things in
systemd, there is no guarantee for the block mode. As we can see
the msg "Failed to execute operation: Connection reset by peer",
which is thrown by "systemctl reboot -f".
poweroff and reboot share most of code, if one fails, then the other
should also fails, so it is meaningless to use reboot as the backup of
poweroff. Using "systemctl poweroff -f", the sdbus will teared down
immediately, which prevent the following "systemctl reboot -f" from
executing. Meanwhile, as man systemctl says:
-f, --force
When used with enable, overwrite any existing conflicting symlinks.
When used with halt, poweroff, reboot or kexec, execute the selected
operation without shutting down all units. However, all processes will
be killed forcibly and all file systems are unmounted or remounted read-only.
Hence, replacing the 'poweroff' with 'systemctl poweroff -f'
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
In commit 1c97aee and commit 227c185 udev rules was rewritten to use
systemd-run to run in a non-blocking mode. The problem is that it's a
bit noise, especially on machine bootup, systemd will always generate
extra logs for service start, you might see your journal full of lines
like these if you have many CPUs (each CPU generates a udev event on
boot):
...
Nov 22 22:23:05 localhost systemd[1]: Started /usr/lib/udev/kdump-udev-throttler.
Nov 22 22:23:05 localhost systemd[1]: Started /usr/lib/udev/kdump-udev-throttler.
Nov 22 22:23:05 localhost systemd[1]: Started /usr/lib/udev/kdump-udev-throttler.
Nov 22 22:23:05 localhost systemd[1]: Started /usr/lib/udev/kdump-udev-throttler.
...
While system is still booting up, kdump service is not started yet, so
systemd-run calls will end up doing nothing, the throttler being called
by systemd-run will just exit if kdump is not loaded.
This patch avoid systemd-run from being called at first place if kdump
service is not running by checking kdump.service status in udev rule,
so there won't be unnecessary logs.
Also remove the kdump service checking logic in kdump-udev-throttler as
udev is the only expected callee of this script, if it's not being
called at first place when kdump service is running, this checking will
be redundant. And even if any user called this script manually, it will
still work well as this script will call 'kdumpctl reload', it reload
the kdump resource only if kdump is loaded already.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Currently we still don't support multipath route, when parsing multipath
route kdumpctl will wrongly consider 'nexthop' as the destination address,
and raise errors in second kernel.
When multipath route is in use, ip route output should be like this:
$ /sbin/ip route show
default via 192.168.122.1 dev ens1 proto dhcp metric 100
192.168.122.0/24 dev ens1 proto kernel scope link src 192.168.122.161 metric 100
192.168.122.8
nexthop via 192.168.122.1 dev ens1 weight 50
nexthop via 192.168.122.2 dev ens1 weight 5
As we don't care about HA/performance, simply use the rule with highest
weight and ignore the rest.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
nfs service will append extra mount options to kernel mount options.
Those extra options represent current mounting details, but they may
not suitable for the second kernel. IP address may change, and we only
enable a single network stack (v4/v6), if nfs prefered another
network stack, inheriting the options will force nfs service to use
previous network stack and disable nfs's fallback mechanic and fail.
As nfs service have the capability to negotiate required protocols
and detect proper IP address, just drop those options and let nfs
automatically adapt the possible change in the second kernel.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Resolves: bz1561837
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1561837
Currently 'kdump.conf' and 'kdump.conf MAN page' entries state that the
local dump path should be:
<fs type> <partition>
- Will mount -t <fs type> <partition> <mnt>, and copy
/proc/vmcore to <mnt>/<path>/%DATE/.
The correct vmcore path instead should be:
<mnt>/<path>/%HOST_IP-%DATE/
Signed-off-by: Bhupesh Sharma <bhsharma@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
In commit b34ce3a reload support was added to kdumpctl but the usage
info is not updated. Now add reload to usage output to let user aware
of the new command.
Signed-off-by: Kairui Song <kasong@redhat.com>
Previously, kdump will restart / reload for many times on hotplug
event, especially memory hotplug events. Hotplugged memory may
generate many udev event as memory are managed and hotplugged in
small chunks by the kernel.
This results in unnecessary system workload and an actually longer
delay of kdump reload and the hotplug event, as udev will either
get blocked or kdumpctl will be waiting for other triggered operation.
To fix this, introduce a kdump-udev-throttler as an agent which will
be called by udev and merge concurrent kdump restart requests. Tested
with a Hyper-V VM which is failing due to udev timeout previously,
no new issues found.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
According to udev's man page, PROGRAM is either used to determine
device's name or whether the device matches the rule. So we should
use RUN insteand. Meanwhile, both RUN / PROGRAM only accepts very
short-running foreground tasks, but kdump restart may take a long
time if there are any device changes that will lead to image rebuild,
which may lead to buggy behavior.
On the other hand, memory / CPU hot plug should never trigger a
initramfs rebuild.
To solve this problem, we will use new introduced "kdumpctl reload"
instead, and use systemd-run to create a transient service unit for
the reload and run it in no-block mode, so udev won't be blocked by
anything.
We need to make systemd-run execute in non-blocking mode, and do not
synchronously wait for the operation to finish, because udev expect
the command line in RUN to be finished immediately, however, kdumpctl
reload may take 0.5-1s for an ordinary reload, or even slower on some
machines. So we give systemd-run an explicit --no-block option to run
in non-blocking mode. Without --no-blocking, systemd-run will verify,
enqueue and wait for the operation to finish. By using the --no-block
option, systemd-run will only verify and enqueue the unit then
return. In this way, we make sure the command is executed
asynchronously, and the status will be monitored and logged by
systemd, which is reliable and non-blocking.
Another thing to mention is that --no-block is only needed after
systemd-v220, before v220 systemd-run uses non-blocking mode by
default and --no-block option is not available on earlier systemd
versions.
Also reformat the udev rules to a more maintanceable format.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Add reload support to kdumpctl, reload will simply unload current
loaded kexec crash kernel and initramfs, and load it again.
Changes in /etc/sysconfig/kdump will take effect with kdumpctl
reload, but reloading will not check the content of
/etc/kdump.conf and won't rebuild anything. reload is fast, the only
time-consuming part of kdumpctl reload is loading kernel and initramfs
with kexec which is always necessary.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
In dracut-049, a new squash module is introduced, it can reduce the
memory usage of kdump initramfs in the capture kernel, this helps a lot
on lowering the risk of OOM failure.
Tested with latest rawhide with NFS, SSH and local dump.
Signed-off-by: Kairui Song <kasong@redhat.com>
Currently the kdumpctl script doesn't check if the raw device is
formatted which might destroy existing data at the time of dump
capture.
This patch addresses this issue, by ensuring kdumpctl prints
a warning in case it finds the raw device to be formatted.
Signed-off-by: Kenneth D'souza <kdsouza@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
Currently the script does not check if the dump target is read-only and would
always mount to read-write mode. This caused an issue with nfs mount as the
fstab options would be reconsidered while remounting to read-write mode.
The remount would fail with the below error as all options cannot be changed
runtime.
mount.nfs: mount(2): Invalid argument
mount.nfs: an incorrect mount option was specified
Which in result would not save the vmcore on the dump target.
This patch addresses this issue by checking the dump target status for read-only.
If yes, remount to read-write mode without reconsidering the fstab options.
Signed-off-by: Kenneth D'souza <kdsouza@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
Resolves: bz1619122
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1619122
This patch fixes the "Unhandled rela relocation: R_X86_64_PLT32" error
that we are seeing with Fedora 29 (and newer kernels > 4.18) which
trying to run kexec/kdump on x86_64 machines.
The patch is being discussed upstream and has been ACK'ed by Baoquan and
myself (see <https://www.spinics.net/lists/kexec/msg21255.html>) and I
have also tested the same on Fedora 29/rawhide x86_64 machine as well:
Before the patch:
----------------
[root@hp-bl480c-01 ~]# kdumpctl restart
kexec: unloaded kdump kernel
Stopping kdump: [OK]
Unhandled rela relocation: R_X86_64_PLT32
kexec: failed to load kdump kernel
Starting kdump: [FAILED]
After the patch:
---------------
[root@hp-bl480c-01 ~]# kdumpctl restart
kexec: unloaded kdump kernel
Stopping kdump: [OK]
kexec: loaded kdump kernel
Starting kdump: [OK]
Suggested Upstream Fix:
In response to a change in binutils, commit b21ebf2fb4c
(x86: Treat R_X86_64_PLT32 as R_X86_64_PC32) was applied to
the linux kernel during the 4.16 development cycle and has
since been backported to earlier stable kernel series. The
change results in the failure message in $SUBJECT when
rebooting via kexec.
Fix this by replicating the change in kexec.
Signed-off-by: Chris Clayton <chris2553@googlemail.com>
Signed-off-by: Bhupesh Sharma <bhsharma@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
Currently the kdumpctl script doesn't check if the path option is
set more than once due to which a vmcore is not captured.
This patch addresses this issue by ensuring that only one path
is specified in /etc/kdump.conf file.
Signed-off-by: Kenneth D'souza <kdsouza@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
If nofail or nobootwait option is used, systemd's local-fs.target won't
wait for the mounting to complete, and kdump might start before the
required mount point is ready and then fail.
The host might use nofail for reasons like the device may get unpluged,
and if the device is not mounted and it is set as kdump target as the same
time then kdump service won't start, we will never enter the capture
kernel. By the time we have entered the capture kernel, the target device
must exist and ready to use, or else kdump would fail anyway. So force
remove nofail and nobootwait option.
Also drop rootflags=nofail option, as we don't depend on rootfs anymore
if the dump target don't required it. So the nofail option is no longer
needed.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
We test if to include the drm module or not by testing if there are any
drm entry in sysfs. But there is an exception for hyper-v, DRM module
take care of hyperv's framebuffer driver as well but hyperv_fb will
not create any drm entry. So currently we got black screen on
hyperv guest.
Fix by detect hyperv's special entry as well.
Signed-off-by: Kairui Song <kasong@redhat.com>
Kdump anaconda has been included as a subpackage for a long time, which
is not a good practice, as the anaconda plugin should be built as
noarch and it does not belong to kexec-tools. We have created a new
package 'kdump-anaconda-addon', so remove it here.
The release version should be bumped later so that kdump-anaconda-addon
could mark previous versions as obsoleted.
Signed-off-by: Kairui Song <kasong@redhat.com>
armv7hl build failed because no makedumpfile* built but the latest commit
tries to install them.
Exclude armv7hl in the code chunk.
Signed-off-by: Dave Young <dyoung@redhat.com>
kexec_test seems to be no longer used upstream, so we had introduced
the 'kexec-tools-2.0.3-disable-kexec-test.patch' earlier to disable the
same from fedora kexec-tools as well.
However an earlier patch "Remove obsolete kdump tool" now explicitly
installs needed files via appropriate logic in .spec file, so we can
drop this patch now to reduce the maintenance burden.
Signed-off-by: Bhupesh Sharma <bhsharma@redhat.com>
Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1441677
Testing: On x86_64 Fedora machine. After this patch kdump utility and related
man page cannot be found on this machine:
[root@tyan-gt24-09 ~]# which kdump
/usr/bin/which: no kdump in
(/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin)
[root@tyan-gt24-09 ~]# man kdump
No manual entry for kdump
Update the fedora 'kexec-tools.spec' to not install the obsolete
kdump tool.
I have submitted an upstream patch to obsolete the kdump tool from
upstream kexec-tools (which has been accepted), but after an internal
discussion we decided not to backport the upstream 'kexec-tools' patch
(which does the same) for fedora, as we would prefer to manage the
changes directly in the .spec file itself.
Signed-off-by: Bhupesh Sharma <bhsharma@redhat.com>
This commit basically reverts commit c755499fad,
and make use of new introduced tri-state hostonly mode.
Following dracut commits merged multipath-hostonly into multipath
module, and introduced a tri-state hostonly mode.
commit 35e86ac117acbfd699f371f163cdda9db0ebc047
Author: Kairui Song <kasong@redhat.com>
Date: Thu Jul 5 16:20:04 2018 +0800
Merge 90-multipath-hostonly and 90-multipath
commit a695250ec7db21359689e50733c6581a8d211215
Author: Kairui Song <kasong@redhat.com>
Date: Wed Jul 4 17:21:37 2018 +0800
Introduce tri-state hostonly mode
multipath-hostonly module was introduced only for kdump, because kdump
need a more strict hostonly policy for multipath device to save memory.
Now multipath module will provide the behave we wanted by setting
hostonly mode to strict.
Currently, we only rebuilt kdump initramfs on config file change,
fs change, or watchdog related change. This will not cover the case
that hardware changed but fs layout and other configurations still
stays the same, and kdump may fail.
To cover such case, we can detect and compare loaded kernel modules,
if a hardware change requires the image to be rebuilt, loaded kernel
modules must have changed.
Starting from commit 7047294 dracut will record loaded kernel modules
when the image is built if hostonly mode is enabled. With this patch,
kdumpctl will compare the recorded value with currently loaded kernel
modules, and rebuild the image on change.
"kdumpctl start" will be a bit slower, as we have to call lsinitrd one
more time to get the loaded kernel modules list. I measure the time
consumption and we have an overall 0.2s increased loading time.
Time consumption of command "kdumpctl restart":
Before:
real 0m0.587s
user 0m0.481s
sys 0m0.102s
After:
real 0m0.731s
user 0m0.591s
sys 0m0.133s
Time comsumption of command "kdumpctl restart" with image rebuild:
Before (force rebuild):
real 0m10.972s
user 0m8.966s
sys 0m1.318s
After (inserted ~100 new modules):
real 0m11.220s
user 0m9.387s
sys 0m1.337s
Signed-off-by: Kairui Song <kasong@redhat.com>
Kdump always use _proto=dhcp for both ipv4 and ipv6. But for ipv6
the dhcp address assignment is not like ipv4, there are different ways
for it, stateless and stateful, see below document:
https://fedoraproject.org/wiki/IPv6Guide
In case stateless, kernel can do the address assignment, dracut use
_proto=auto6; for stateful case, dracut use _proto=dhcp6.
But it is hard to decide whether stateless or stateful takes effect,
hence, dracut introduces ip=either6 option, which can try both of these
method automatically for us. For detail, refer to dracut:
commit 67354ee 40network: introduce ip=either6 option
We do not see bug reports before because for the most auto6 cases
kernel assign ip address before dhclient, kdump just happened to work.
Signed-off-by: Pingfan Liu <piliu@redhat.com>
When using fence_kdump, module-setup will create a kdump.conf with
fence_kdump_nodes. The node name comes from the cluster xml, which may
use the hostname alias. Later in kdump stage, "fence_kdump_send alias_1
alias_2" sends out notification to peers. Hence it requires /etc/hosts
and nsswitch.conf to make alias work.
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Kdump service starts too late, so early crashes will have no chance
to get kdump kernel booting, this will cause crash information to be
lost. It is necessary to add a dracut module in order to load crash
kernel and initramfs as early as possible. You can provide "rd.early
kdump" in grub commandline to enable, then the early kdump will load
those files like the normal kdump, which is disabled by default.
For the normal kdump service, it can check whether the early kdump
has loaded the crash kernel and initramfs. It has no conflict with
the early kdump.
If you rebuild the new initramfs for early kdump, the new initramfs
size will become large, because it will put the vmlinuz and kdump
initramfs into the new initramfs.
In addition, early kdump doesn't support fadump.
Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Reviewed-by: Kazuhito Hagio <khagio@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
we move some common functions from kdumpctl to kdump-lib.sh, the
functions could be used in other modules, such as early kdump.
It has no bad effect.
Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Reviewed-by: Kazuhito Hagio <khagio@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
port from rhel, original patch is contributed by Minfei Huang:
Using /sys to determines crashkernel actual size is confusing since
there is no unit of measure.
Add a new command "kdumpctl showmem" to show the reserved memory kindly.
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Signed-off-by: Minfei Huang <mhuang@redhat.com>
Acked-by: Bhupesh Sharma <bhsharma@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Kairui Song
Lianbo Jiang
Kazuhito Hagio
Pingfan Liu
Bhupesh Sharma
Markus Linnala
Kenneth Dsouza
Dave Young
Fedora Release Engineering