Currently, kexec-tools tries to reset crashkernel when using anaconda to
install the system. But grubby isn't ready and complains that,
10:34:17,014 INF packaging: Configuring (running scriptlet for): kexec-tools-2.0.23-9.el9.x86_64 1646034766 53ff7158f8808774f4e3c3c87e504aa7a6d677b537754dac86c87925c8f0a397
10:34:17,205 INF dnf.rpm: grep: /boot/grub2/grubenv: No such file or directory
grep: /boot/grub2/grubenv: No such file or directory
grep: /boot/grub2/grubenv: No such file or directory
kexec-tools is supposed to update the kernel crashkernel parameter after
package upgrade. Unfortunately, the posttrans RPM scriptlet doesn't
distinguish between package install and upgrade. This patch skips
reset_crashkernel_after_update as similar to e218128e ("Only try to
reset crashkernel for osbuild during package install").
Reported-by: Jan Stodola <jstodola@redhat.com>
Signed-off-by: Coiby Xu <coxu@redhat.com>
When the dump target is lvm2 thinp, if we didn't mount
the dump target first, get_fs_type_from_target will get
empty output:
Before mount:
$ get_fs_type_from_target /dev/vg00/thinlv
After mount:
$ mount /dev/vg00/thinlv /mnt
$ get_fs_type_from_target /dev/vg00/thinlv
ext4
As a result, kdumpctl start will fail with:
$ kdumpctl start
kdump: Dump target is invalid
kdump: Starting kdump: [FAILED]
This patch fix the issue by bypassing check_fs_modified
when the dump target is lvm2 thinp.
Signed-off-by: Tao Liu <ltao@redhat.com>
Reviewed-by: Coiby Xu <prudo@redhat.com>
F36 has dropped support on ifcfg and as a result current network tests
fails. Use .nmconnection to set up test network instead.
Signed-off-by: Coiby Xu <coxu@redhat.com>
Reviewed-by: Philipp Rudo <prudo@redhat.com>
With commit fa9201b2 ("fadump: isolate fadump initramfs image within
the default one"), initramfs image gets to hold two images, one for
production kernel boot purpose and the other for capture kernel boot.
Most files are common among the two images. Retain file modification
time to replace duplicate files with hardlinks and save space. Also,
avoid unnecessarily compressing fadump image that is decompressed
immediately anyway.
Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
Reviewed-by: Philipp Rudo <prudo@redhat.com>
With commit fa9201b2 ("fadump: isolate fadump initramfs image within
the default one"), initramfs image gets to hold two squash images, one
for production kernel boot purpose and the other for capture kernel
boot. Having separate images improved reliability for both production
kernel and capture kernel boot scenarios, but the size of initramfs
image became considerably larger.
Instead of having squash images, compressing $initdir without using
squash images reduced the size of initramfs image for fadump case by
around 30%. So, avoid using squash for fadump case.
Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
Reviewed-by: Philipp Rudo <prudo@redhat.com>
Previously, all files within $TESTCASEDIR/$test_case are regarded
as shell script files for testing. However there might be config
files under the directory. So let's only iterate the .sh files.
Signed-off-by: Tao Liu <ltao@redhat.com>
Reviewed-by: Philipp Rudo <prudo@redhat.com>
The 80lvmthinpool-monitor module is needed for monitor and
autoextend the size of thin pool in 2nd kernel. The module was
integrated in dracut version 057.
If lvmthinpool-monitor module is not found, we will print a warning.
Because we don't want to block the kdump process when the thin pool
capacity is enough and no monitor-and-autoextend actually needed.
Signed-off-by: Tao Liu <ltao@redhat.com>
Reviewed-by: Philipp Rudo <prudo@redhat.com>
lvm2 relies on /etc/lvm/lvm.conf to determine its behaviour. The
important configs such as thin_pool_autoextend_threshold and
thin_pool_autoextend_percent will be used during kdump in 2nd
kernel. So if the file is modified, the initramfs should be
rebuild to include the latest.
Signed-off-by: Tao Liu <ltao@redhat.com>
Reviewed-by: Philipp Rudo <prudo@redhat.com>
We need to check if a directory or a device is lvm2 thinp target.
First, we use get_block_dump_target() to convert dump path into
block device, then we check if the device is lvm2 thinp target by
cmd lvs.
is_lvm2_thinp_device is now located in kdump-lib-initramfs.sh, for it
will be used in 2nd kernel. is_lvm2_thinp_dump_target is located in
kdump-lib.sh, for it is only used in 1st kernel, and it has dependencies
which exist in kdump-lib.sh.
Signed-off-by: Tao Liu <ltao@redhat.com>
Reviewed-by: Philipp Rudo <prudo@redhat.com>
Currently, updating kexec-tools on s390x gives the warning
sed: can't read /etc/default/grub: No such file or directory
This happens because s390x doesn't use GRUB and /etc/default/grub
doesn't exist. We need to skip both reading and writing to
/etc/default/grub.
Reported-by: Jie Li <jieli@redhat.com>
Reviewed-by: Philipp Rudo <prudo@redhat.com>
Signed-off-by: Coiby Xu <coxu@redhat.com>
memory requirement for LUKS-encrypted target
When there are multiple key slots, "kdumpctl estimate" uses the least
memory-consuming key slot. For example, when there are two memory slots
created with --pbkdf-memory=1048576 (1G) and --pbkdf-memory=524288 (512M),
"kdumpctl estimate" thinks the extra memory requirement is only 512M.
This will of course lead to OOM if the user uses the more
memory-consuming key slot. Fix it by sorting in reverse order.
Fixes: e9e6a2c ("kdumpctl: Add kdumpctl estimate")
Signed-off-by: Coiby Xu <coxu@redhat.com>
Reviewed-by: Lichen Liu <lichliu@redhat.com>
Signed-off-by: Coiby Xu <coxu@redhat.com>
Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=2060319
Currently, kexec-tools tries to reset crashkernel when using anaconda to
install the system. But grubby isn't ready and complains that,
10:33:17,631 INF packaging: Configuring (running scriptlet for): kernel-core-5.14.0-70.el9.x86_64 1645746534 03dcd32db234b72440ee6764d59b32347c5f0cd98ac3fb55beb47214a76f33b4
10:34:16,696 INF dnf.rpm: grep: /boot/grub2/grubenv: No such file or directory
grep: /boot/grub2/grubenv: No such file or directory
We only need to try resetting crashkernel for osbuild. Skip it for other
cases. To tell if it's package install instead of package upgrade, make
use of %pre to write a file /tmp/kexec-tools-install when "$1 == 1" [1].
[1] https://docs.fedoraproject.org/en-US/packaging-guidelines/Scriptlets/#_syntax
Reported-by: Jan Stodola <jstodola@redhat.com>
Signed-off-by: Coiby Xu <coxu@redhat.com>
Reviewed-by: Lichen Liu <lichenliu@redhat.com>
Previously kexec-tools will pass "--compress zstd" to dracut. It
will make dracut to decide whether: a) call mksquashfs to make a
zstd format squash-root.img, b) call cmd zstd to make a initramfs.
Since dracut(>= 057) has decoupled the compressor for dracut and
dracut-squash, So in this patch, we will pass the compressor seperately.
Note:
The is_squash_available && !dracut_has_option --squash-compressor
&& !is_zsdt_command_available case is left unprocessed on purpose.
Actually, the situation when we want to call zstd compression is:
1) If squash function OK, we want dracut to invoke mksquashfs to make
a zstd format squash-root.img within initramfs.
2) If squash function is not OK, and cmd zstd presents, we want dracut
to invoke cmd zstd to make a zstd format initramfs.
is_zstd_command_available check can handle case 2 completely.
However, for the is_squash_available check, it cannot handle case 1
completely. It only checks if the kernel supports squashfs, it doesn't
check whether the squash module has been added by dracut when making
initramfs. In fact, in kexec-tools we are unable to do the check,
there are multiple ways to forbit dracut to load a module, such as
"dracut -o module" and "omit_dracutmodules in dracut.conf".
When squash dracut module is omitted, is_squash_available check will
still pass, so "--compress zstd" will be appended to dracut cmdline,
and it will call cmd zstd to do the compression. However cmd zstd may
not exist, so it fails.
The previous "--compress zstd" is ambiguous, after the intro of
"--squash-compressor", "--squash-compressor" only effect for
mksquashfs and "--compress" only effect for specific cmd.
So for the is_squash_available && !dracut_has_option
--squash-compressor && !is_zsdt_command_available case, we just leave
it to be handled the default way.
Reviewed-by: Philipp Rudo <prudo@redhat.com>
Signed-off-by: Tao Liu <ltao@redhat.com>
Previously the sync for dump_fs is problematic, it always
return success according to man 2 sync. So it cannot detect
the error of the dump target is full and not all of vmcore
data been written back the disk, which will leave the vmcore
imcomplete and report misleading log as "saving vmcore
complete".
In this patch, we will use "sync -f vmcore" instead, which
will return error if syncfs on the dump target fails. In
this way, vmcore sync related failures, such as autoextend
of lvm2 thinpool fails, can be detected and handled properly.
Signed-off-by: Tao Liu <ltao@redhat.com>
Reviewed-by: Coiby Xu <coxu@redhat.com>
This patch add virtiofs support for kexec-tools by introducing a new option
for /etc/kdump.conf:
virtiofs myfs
Where myfs is a variable tag name specified in qemu cmdline
"-device vhost-user-fs-pci,tag=myfs".
The patch covers the following cases:
1) Dumping VM's vmcore to a virtiofs shared directory;
2) When the VM's rootfs is a virtiofs shared directory and dumping the
VM's vmcore to its subdirectory, such as /var/crash;
3) The combination of case 1 & 2: The VM's rootfs is a virtiofs shared
directory and dumping the VM's vmcore to another virtiofs shared
directory.
Case 2 & 3 need dracut >= 057, otherwise VM cannot boot from virtiofs
shared rootfs. But it is not the issue of kexec-tools.
Reviewed-by: Philipp Rudo <prudo@redhat.com>
Signed-off-by: Tao Liu <ltao@redhat.com>
Since commit c5bdd2d8f1 ("kdump-lib: use non-debug kernels first"),
non-debug kernel is preferred, over the debug variant, as dump capture
kernel to reduce memory consumption. This works alright for kdump as
the capture kernel is loaded using kexec.
In case of fadump, regular boot loader is used to load the capture
kernel. So, the default kernel needs to be used as capture kernel as
well. But with commit c5bdd2d8f1, initrd of a different kernel is
made dump capture capable, breaking fadump's ability to capture dump
properly. Fix this by sticking with the debug variant in case of
fadump.
Fixes: c5bdd2d8f1 ("kdump-lib: use non-debug kernels first")
Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
Acked-by: Lichen Liu <lichliu@redhat.com>
Acked-by: Coiby Xu <coxu@redhat.com>
When kdump is configured with a NFS location, and the remote directory does
not exist, kdump.service fails with a confusing error message.
kdumpctl[2172]: kdump: Dump path "/tmp/mkdumprd.ftWhOF/target/dumps"
does not exist in dump target "10.111.113.2:/srv/kdump"
We just need to print the remote directory "dumps" in such case, because
"/tmp/mkdumprd.ftWhOF/target" is the local temporary mount point.
Signed-off-by: Lichen Liu <lichliu@redhat.com>
Reviewed-by: Coiby Xu<coxu@redhat.com>
Decrease the risk that of leaking information that could potentially
be used to exploit the crash further (think location of keys).
Signed-off-by: Lichen Liu <lichliu@redhat.com>
Acked-by: Coiby Xu <coxu@redhat.com>
These kdump.sysconfig.* files are almost identical with a bit difference
in several parameters, just use a simple script to generate them upon
packaging. This should make it easier to maintain, updating a comment or
param for a certain arch can be done in one place.
There are only some comment or empty option differences with the generated
version because some arch's sysconfig is not up-to-dated, this actually
fixes the issue, I used the following script to check these differences:
# for arch in aarch64 i386 ppc64 ppc64le s390x x86_64; do
./gen-kdump-sysconfig.sh $arch > kdump.sysconfig.$arch.new
git checkout HEAD^ kdump.sysconfig.$arch &>/dev/null
echo "$arch:"
diff kdump.sysconfig.$arch kdump.sysconfig.$arch.new; echo ""
done; git reset;
Signed-off-by: Kairui Song <kasong@tencent.com>
Reviewed-by: Philipp Rudo <prudo@redhat.com>
So fedpkg will fetch the sources that matches given Fedora version.
Signed-off-by: Coiby Xu <coxu@redhat.com>
Reviewed-by: Philipp Rudo <prudo@redhat.com>
New version of qemu-img requires specifying the backing format for the
backing file otherwise it will abort.
Signed-off-by: Coiby Xu <coxu@redhat.com>
Reviewed-by: Philipp Rudo <prudo@redhat.com>
Fedora 33 and 34 Cloud Base Images have only one partition with the
following directory structure,
.
├── bin -> usr/bin
├── boot
├── dev
├── etc
├── home
├── root
By comparison, Fedora 35, 36 and 37 Cloud Base Images have multiple
partitions. The root partition which is the last partition has the
following directory,
.
├── home
└── root
├── bin -> usr/bin
├── boot
├── dev
├── etc
├── home
├── root
and the 2nd partition is the boot partition.
This patch address the above changes by mounting {LAST_PARTITION}/root as
to TEMP_ROOT and mount SECOND_PARTITION to TEMP_ROOT/boot. So the test
image can be built successfully.
Signed-off-by: Coiby Xu <coxu@redhat.com>
Reviewed-by: Philipp Rudo <prudo@redhat.com>
s390x doesn't use GRUB. To make sure the boot entries are updated, call
zipl after running grubby.
Suggested-by: smitterl@redhat.com
Reviewed-by: Philipp Rudo <prudo@redhat.com>
Signed-off-by: Coiby Xu <coxu@redhat.com>
"grubby --zipl" only takes effect when setting default kernel. It's
useless to add "--zipl" when updating kernel command line. Also rename
_update_grub to _update_kernel_cmdline since s390x doesn't use GRUB.
Reviewed-by: Philipp Rudo <prudo@redhat.com>
Signed-off-by: Coiby Xu <coxu@redhat.com>
Resolves: bz2104534
When running "kdumpctl reset-crashkernel --kernel=ALL" on s390x,
sed: can't read /etc/default/grub: No such file or directory
sed: can't read /etc/default/grub: No such file or directory
This happens because s390x doesn't use the grub bootloader and
/etc/default/grub doesn't exist.
Reported-by: smitterl@redhat.com
Reviewed-by: Philipp Rudo <prudo@redhat.com>
Signed-off-by: Coiby Xu <coxu@redhat.com>
Resolves: bz2092012
According to the ostree team [1], the existence of /run/ostree-booted
> is the most stable way to signal/check that a system has been
> booted in ostree-style. It is also used by rpm-ostree at
> compose/install time in the sandboxed environment where scriptlets run,
> in order to signal that the package is being installed/composed into
> an ostree commit (i.e. not directly on a live system). See
> 8ddf5f40d9/src/libpriv/rpmostree-scripts.cxx (L350-L353)
> for reference.
By checking the existence of /run/ostree-booted, we could skip trying to
update kernel cmdline during OSTree compose time.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=2092012#c3
Reported-by: Luca BRUNO <lucab@redhat.com>
Suggested-by: Luca BRUNO <lucab@redhat.com>
Fixes: 0adb0f4 ("try to reset kernel crashkernel when kexec-tools updates the default crashkernel value")
Signed-off-by: Coiby Xu <coxu@redhat.com>
Reviewed-by: Philipp Rudo <prudo@redhat.com>
Acked-by: Timothée Ravier <siosm@fedoraproject.org>
Resolves: bz2089871
Currently, kexec-tools can't be updated using virt-customize because
older version of kdumpctl can't acquire instance lock for the
get-default-crashkernel subcommand. The reason is /var/lock is linked to
/run/lock which however doesn't exist in the case of virt-customize.
This patch fixes this problem by using /tmp/kdump.lock as the lock
file if /run/lock doesn't exist.
Note
1. The lock file is now created in /run/lock instead of /var/run/lock since
Fedora has adopted adopted /run [2] since F15.
2. %pre scriptlet now always return success since package update won't
be blocked
[1] https://fedoraproject.org/wiki/Features/var-run-tmpfs
Fixes: 0adb0f4 ("try to reset kernel crashkernel when kexec-tools updates the default crashkernel value")
Reported-by: Nicolas Hicher <nhicher@redhat.com>
Suggested-by: Laszlo Ersek <lersek@redhat.com>
Suggested-by: Philipp Rudo <prudo@redhat.com>
Signed-off-by: Coiby Xu <coxu@redhat.com>
Reviewed-by: Philipp Rudo <prudo@redhat.com>
Currently, kdump may experience failure on some aws aarch64 platform.
The final scenario is:
[ 79.145089] printk: console [ttyS0] disabled
Then the system has no response any more. And after reboot, there is no
vmcore generated under /var/crash/. More detail [1].
In a short word, it is caused by the irqpoll policy and some unknown
acpi issue. The serial device is hot-removed as a pci device.
More detailed, the irqpoll policy demands to iterate over all interrupt
handler, if the interrupt line is shared, then the handler is
dispatched. And acpi handler acpi_irq() is on a shared interrupt line,
so it is called. But for some unknown reason, the acpi hardware regs
hold wrong state, and the acpi driver decides that a hot-removed event
happens on a pci slot, which finally removes the pci serial device.
To tackle this issue by removing the irqpoll parameter on aws aarch64
platform, until the real root cause in acpi is found and resolved.
[1]: https://bugzilla.redhat.com/show_bug.cgi?id=2080468#c0
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Coiby Xu <coxu@redhat.com>
Resolves: bz2106645
The code of commit 163c02970e takes effect in rhel firstly, later
pulled to Fedora. However, Fedora OS doesn't have 40-redhat.rules
in systemd-udev package. With this commit applied, a false positive
warning message can always been seen as below.
So fixing it by checking if 40-redhat.rules exists before handling.
With this change, the false warning is gone.
[root@ ~]# kdumpctl restart
kdump: kexec: unloaded kdump kernel
kdump: Stopping kdump: [OK]
kdump: No kdump initial ramdisk found.
kdump: Rebuilding /boot/initramfs-5.19.0-rc6+kdump.img
sed: can't read /var/tmp/dracut.NnAV2g/initramfs/usr/lib/udev/rules.d/40-redhat.rules: No such file or directory
kdump: kexec: loaded kdump kernel
kdump: Starting kdump: [OK]
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
The kernel of CoreOS is not in the standard locations, add
/boot/ostree/* to the boot_dirlist to find the vmlinuz.
Signed-off-by: Lichen Liu <lichliu@redhat.com>
Acked-by: Coiby Xu <coxu@redhat.com>
Currently $boot_img can get bad data if running on a platform
that doesn't set BOOT_IMAGE in the kernel command line. For
example, currently:
- s390x Fedora CoreOS machine:
```
[root@cosa-devsh ~]# sed "s/^BOOT_IMAGE=\((\S*)\)\?\(\S*\) .*/\2/" /proc/cmdline
mitigations=auto,nosmt ignition.platform.id=qemu ostree=/ostree/boot.0/fedora-coreos/2a72567ac8f7ed678c3ac89408f795e6ccd4e97b41e14af5f471b6a807e858b9/0 root=UUID=2a88436a-3b6b-4706-b33a-b8270bd87cde rw rootflags=prjquota boot=UUID=f4b2eaa5-9317-4798-85cf-308c477fee4c crashkernel=600M
```
where on a platform that uses GRUB we get:
- x86_64 Fedora CoreOS machine:
```
[root@cosa-devsh ~]# sed "s/^BOOT_IMAGE=\((\S*)\)\?\(\S*\) .*/\2/" /proc/cmdline
/ostree/fedora-coreos-af4f6cc7b9ff486cfa647680b180e989c72c8eed03a34a42e7328e49332bd20e/vmlinuz-5.18.5-200.fc36.x86_64
```
We should change the setting of the boot_img variable such that it will
be empty if BOOT_IMAGE doesn't exist.
With this change on the s390x machine:
```
[root@cosa-devsh ~]# grep -P -o '^BOOT_IMAGE=(\S+)' /proc/cmdline | sed "s/^BOOT_IMAGE=\((\S*)\)\?\(\S*\)/\2/"
[root@cosa-devsh ~]#
```
This change mattered much more before the change in c5bdd2d which changed
the following line from [[ -n $boot_img ]] to [[ "$boot_img" == *"$kdump_kernelver" ]].
Still I think this change has merit.
Signed-off-by: Dusty Mabe <dusty@dustymabe.com>
Acked-by: Coiby Xu <coxu@redhat.com>
There are many variants on OSTree based systems these days so
we should probably refer to the class of systems as "OSTree
based systems". Also, Atomic Host is dead.
Signed-off-by: Dusty Mabe <dusty@dustymabe.com>
Acked-by: Coiby Xu <coxu@redhat.com>
On RHEL9 and Fedora, the arm64 platform only supports 4KB page size.
the reserved memory size can be aligned to that on x86_64.
Introducing a new formula for 4KB on arm64, which bases on x86_64 plus
extra 64MB.
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
Kdump uses currently running kernel as default, but when currently
running kernel is a debug kernel, it will consume more memory,
which may cause out-of-memory and fail to collect vmcore.
Now we will try to use non-debug kernels first if possible.
Also extract the logic of determine KDUMP_KERNEL from
prepare_kdump_bootinfo into a function. This function will return
KDUMP_KERNEL given a kernel version.
Signed-off-by: Lichen Liu <lichliu@redhat.com>
Acked-by: Coiby Xu <coxu@redhat.com>
For CoreOS based systems we use Ignition for provisioning machines
in the initramfs on first boot. We trigger Ignition right now by
the presence of `ignition.firstboot` in the kernel command line. The
kernel argument is only present on first boot so after a reboot it
no longer is in the kernel command line.
If a kernel crash happens before the first reboot of a machine we
want the `ignition.firstboot` kernel argument to be removed and not
passed on to the crash kernel.
This patch rewrites get_recommend_size to get rid of the following
limitations,
1. only supports ranges in crashkernel sorted in increasing order
2. the first entry of crashkernel should have only a single digit and
it's in gigabytes
Suggested-by: Philipp Rudo <prudo@redhat.com>
Signed-off-by: Coiby Xu <coxu@redhat.com>
Reviewed-by: Philipp Rudo <prudo@redhat.com>
Recently, it's found 'kdumpctl estimate' returns 512M while the system
reserves 1024M kdump memory in a case. This happens because the ranges
in /proc/iomem are inclusively. For example, "0-1: System RAM" means 2
bytes of system memory other than 1 byte. Fix this error by adding one
more byte.
Note
1. the function has been simplified as well.
2. define PROC_IOMEM as /proc/iomem for the sake of unit tests
Reported-by: Ruowen Qin <ruqin@redhat.com>
Fixes: 1813189 ("kdump-lib.sh: introduce functions to return recommened mem size")
Suggested-by: Philipp Rudo <prudo@redhat.com>
Signed-off-by: Coiby Xu <coxu@redhat.com>
Reviewed-by: Philipp Rudo <prudo@redhat.com>
Make log saving the last step of kdump.sh, so it can catch more info,
for example, the output of post.d hooks will be covered by the log now.
Signed-off-by: Kairui Song <kasong@tencent.com>
Reviewed-by: Philipp Rudo <prudo@redhat.com>
Coiby Xu
Tao Liu
Hari Bathini
Lichen Liu
Kairui Song
Pingfan Liu
Baoquan He
Dusty Mabe