When kdump service fails, the current errors do not display the
absolute path of dump location(marked it as "^"), for example:
kdump: kexec: unloaded kdump kernel
kdump: Stopping kdump: [OK]
kdump: Detected change(s) in the following file(s): /etc/kdump.conf
kdump: Rebuilding /boot/initramfs-4.18.0-304.el8.x86_64kdump.img
kdump: Dump path "/var1/crash" does not exist in dump target "UUID=c202ef45-3ac3-4adb-85e7-307a916757f0"
^^^^^^^^^^^
kdump: mkdumprd: failed to make kdump initrd
kdump: Starting kdump: [FAILED]
Here, it should output the absolute path of dump location with this
format: "<mount path>/<path>". To fix it, let's extend the relative
pathname to the absolute pathname in check_user_configured_target().
Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
This patch fixes bz1941106 and bz1941905 which passed empty rd.znet to the
kernel command line in the following cases,
- The IBM (Z15) KVM guest uses virtio for all devices including network
device, so there is no znet device for IBM KVM guest. So we can't
assume a s390x machine always has a znet device.
- When a bridged network is used, kexec-tools tries to obtain the znet
configuration from the ifcfg script of the bridged network rather than
from the ifcfg script of znet device.
We can iterate /sys/bus/ccwgroup/devices to tell if there if there is
a znet network device. By getting an ifname from znet, we can also avoid
mistaking the slave netdev as a znet network device in a bridged network
or bonded network.
Note: This patch also assumes there is only one znet device as commit
7148c0a30d ("add s390x netdev setup")
which greatly simplifies the code. According to IBM [1], there could be
more than znet devices for a z/VM system and a z/VM system may have a
non-znet network device like ConnectX. Since kdump_setup_znet was
introduced in 2012 and so far there is no known customer complaint that
invalidates this assumption I think it's safe to assume an IBM z/VM
system only has one znet device. Besides, there is no z/VM system found
on beaker to test the alternative scenarios.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1941905#c13
Signed-off-by: Coiby Xu <coxu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
Use a modified and minimized version of emergency shell.
The differences of this kdump shell and dracut emergency shell are:
- Kdump shell won't generate a rdsosreport automatically
- Customized prompts
- Never ask root password
- Won't tangle with dracut's emergency_action. If emergency_action is
set, dracut emergency shell will perform dracut's emergency_action
instead of kdump final_action on exit.
- If rd.shell=no is set, kdump shell will still work, dracut emergency
shell won't, even if kdump failure_action is set to shell.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Coiby Xu <coxu@redhat.com>
The wrapper is introduced in commit 002337c, according to the commit
message, the only usage of the wrapper is when dracut-initqueue calls
"systemctl start emergency" directly. In that case, emergency
is started, but not in a isolation mode, which means dracut-initqueue
is still running. On the other hand, emergency will call
"systemctl start dracut-initqueue" again when default action is dump_to_rootfs.
systemd would block on the last dracut-initqueue, waiting for the first
instance to exit, which leaves us hang.
In previous commit we added initqueue status detect in dump_to_rootfs,
so now even without the wrapper, it will not hang.
And actually, previously, with the wrapper, emergency might still hang
for like 30s. When dracut called emergency service because initqueue
timed out, dump_to_rootfs will try start initqueue again and timeout
again. Now with the wrapper removed, we can avoid these two kinds of
hangs, bacause without the isolation we can detect initqueue service
status correctly in such case.
Also remove the invalid header comments in service file, the service
is not part of systemd code. And sync the service spec with dracut.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Coiby Xu <coxu@redhat.com>
kdump's dump_to_rootfs will try to start initqueue unconditionally.
dump_to_rootfs will run after systemd isolate to emergency
target, so this is currently accetable.
But there is a problem when initqueue starts the emergency action
because of initqueue timeout. dump_to_rootfs will start initqueue and
lead to timeout again.
So following patch will remove the previous isolation wrapper, and
detect the service status here. Previous isolation makes the detection
impossible. Now this detection will be valid and helpful to prevent
double timeout or hang.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Coiby Xu <coxu@redhat.com>
Fix the warning observed when KDUMP_KERNELVER is specified:
kdumpctl[10926]: /lib/kdump/kdump-lib.sh: line 697: [: missing `]'
Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
Acked-by: Kairui Song <kasong@redhat.com>
For crashkernel=auto policy, if total RAM size is under a throttle,
there is no memory reserved for kdump.
Also correct a trivial bug by correcting the arch name.
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
Add a rough esitimation support, currently, following memory usage are
checked by this sub command:
- System RAM
- Kdump Initramfs size
- Kdump Kernel image size
- Kdump Kernel module size
- Kdump userspace user and other runtime allocated memory (currently
simply using a fixed value: 64M)
- LUKS encryption memory usage
The output of kdumpctl estimate looks like this:
# kdumpctl estimate
Reserved crashkernel: 256M
Recommanded crashkernel: 160M
Kernel image size: 47M
Kernel modules size: 12M
Initramfs size: 19M
Runtime reservation: 64M
Large modules:
xfs: 1892352
nouveau: 2318336
And if the kdump target is encrypted:
# kdumpctl estimate
Encrypted kdump target requires extra memory, assuming using the keyslot with minimun memory requirement
Reserved crashkernel: 256M
Recommanded crashkernel: 655M
Kernel image size: 47M
Kernel modules size: 12M
Initramfs size: 19M
Runtime reservation: 64M
LUKS required size: 512M
Large modules:
xfs: 1892352
nouveau: 2318336
WARNING: Current crashkernel size is lower than recommanded size 655M.
The "Recommanded" value is calculated based on memory usages mentioned
above, and will be adjusted accodingly to be no less than the value provided
by kdump_get_arch_recommend_size.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
Simplfy the code and also improve the performance. udevadm call is
heavy.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
kexec-tools needs to disable CMA for kdump kernel cmdline,
otherwise kdump kernel may run out of memory.
This patch strips the inherited cma=, hugetlb_cma= cmd
line from 1st kernel, and sets to be 0 for 2nd kernel.
Signed-off-by: Tao Liu <ltao@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
apath (a D-Bus active connection path) is used for nmcli connection operations, e.g.
$ nmcli connection show $apath
Signed-off-by: Coiby Xu <coxu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
nmcli --get-values <field> connection show /org/freedesktop/NetworkManager/ActiveConnection/1
returns the following value for the corresponding field respectively,
Field Value
IP4.DNS "10.19.42.41 | 10.11.5.19 | 10.5.30.160"
802-3-ethernet.s390-subchannels ""
bond.options "mode=balance-rr"
Signed-off-by: Coiby Xu <coxu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
This fixes bz1854037 which happens because kexec-tools generates rd.route for
eth0 instead of for kdump-eth0,
1. "rd.route=168.63.129.16:10.0.0.1:eth0 rd.route=169.254.169.254:10.0.0.1:eth0" is passed to the dracut cmdline by kexec-tools
2. In the 2rd kernel, dracut/modules.d/35network-manager/nm-config.sh calls
/usr/libexec/nm-initrd-generator to generate two .nmconnection files
based on the dracut cmdline, i.e. kdump-eth0.nmconnection and eth0.nmconnection,
- /run/NetworkManager/system-connections/kdump-eth0.nmconnection
[connection]
id=kdump-eth0
uuid=3ef53b1b-3908-437e-a15f-cf1f3ea2678b
type=ethernet
autoconnect-retries=1
interface-name=kdump-eth0
multi-connect=1
permissions=
wait-device-timeout=60000
[ethernet]
mac-address-blacklist=
[ipv4]
address1=10.0.0.4/24,10.0.0.1
dhcp-timeout=90
dns=168.63.129.16;
dns-search=
may-fail=false
method=manual
[ipv6]
addr-gen-mode=eui64
dhcp-timeout=90
dns-search=
method=disabled
[proxy]
- /run/NetworkManager/system-connections/eth0.nmconnection
[connection]
id=eth0
uuid=f224dc22-2891-4d7b-8f66-745029df4b53
type=ethernet
autoconnect-retries=1
interface-name=eth0
multi-connect=1
permissions=
[ethernet]
mac-address-blacklist=
[ipv4]
dhcp-timeout=90
dns=168.63.129.16;
dns-search=
method=auto
route1=168.63.129.16/32,10.0.0.1
route2=169.254.169.254/32,10.0.0.1
[ipv6]
addr-gen-mode=eui64
dhcp-timeout=90
dns-search=
method=auto
[proxy]
3. Since there's eth0.nmconnection, NetworkManager will try to get an IP for eth0 regardless of the fact it's a slave NIC and time out
```
$ ip link show
2: kdump-eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 00:0d:3a:11:86:8b brd ff:ff:ff:ff:ff:ff
3: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master kdump-eth0 state UP mode DEFAULT group default qlen 1000
```
Reported-by: Huijing Hei <hhei@redhat.com>
Signed-off-by: Coiby Xu <coxu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
vmcore-dmesg-incomplete.txt is generated by shell redirection,
which taking the default umask value. When dmesg collector exits
with non-zero, the file will exist and anyone can have access to
it.
This patch fixed the issue by chmod the file, making it accessible
only to its owner.
Signed-off-by: Tao Liu <ltao@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
This reverts commit 5633e83318.
vm.zone_reclaim_mode may cause trashing on some machines. And after
second thought, vm.zone_reclaim_mode is barely helpful for machines
with high mem stress, so just revert it.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
Kdump scirpt already have default values for core_collector, path in
many other place. Empty kdump.conf still works. Fix this corner case and
fix the error message.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
Backport from upstream:
commit 0ef2ca6c9fa2f61f217a4bf5d7fd70f24e12b2eb
Author: Kazuhito Hagio <k-hagio-ab@nec.com>
Date: Thu Feb 4 16:29:06 2021 +0900
[PATCH] Show write byte size in report messages
Show write byte size in report messages. This value can be different
from the size of the actual file because of some holes on dumpfile
data structure.
$ makedumpfile --show-stats -l -d 1 vmcore dump.ld1
...
Total pages : 0x0000000000080000
Write bytes : 377686445
...
# ls -l dump.ld1
-rw------- 1 root root 377691573 Feb 4 16:28 dump.ld1
Note that this value should not be used with /proc/kcore to determine
how much disk space is needed for crash dump, because the real memory
usage when a crash occurs can vary widely.
Signed-off-by: Kazuhito Hagio <k-hagio-ab@nec.com>
Signed-off-by: Tao Liu <ltao@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
Backport from upstream:
commit 6f3e75a558ed50d6ff0b42e3f61c099b2005b7bb
Author: Julien Thierry <jthierry@redhat.com>
Date: Tue Nov 24 10:45:25 2020 +0000
[PATCH 2/2] Add shorthand --show-stats option to show report stats
Provide shorthand --show-stats option to enable report messages
without needing to set a particular value for message-level.
Signed-off-by: Julien Thierry <jthierry@redhat.com>
Signed-off-by: Kazuhito Hagio <k-hagio-ab@nec.com>
Signed-off-by: Tao Liu <ltao@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
Backport from upstream.
commit 3422e1d6bc3511c5af9cb05ba74ad97dd93ffd7f
Author: Julien Thierry <jthierry@redhat.com>
Date: Tue Nov 24 10:45:24 2020 +0000
[PATCH 1/2] Add --dry-run option to prevent writing the dumpfile
Add a --dry-run option to run all operations without writing the
dump to the output file.
Signed-off-by: Julien Thierry <jthierry@redhat.com>
Signed-off-by: Kazuhito Hagio <k-hagio-ab@nec.com>
Signed-off-by: Tao Liu <ltao@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
If the dump target is the root disk, kdump scripts add an entry in
/etc/fstab for root disk with /sysroot as the mount point. The root
disk, passed through root=<> kernel commandline parameter, is mounted
at /sysroot in read-only mode before switching from initial ramdisk.
So, in fadump mode, a remount of /sysroot to read-write mode is needed
to capture dump successfully, because /sysroot is already mounted as
read-only based on root=<> boot parameter.
Commit e8ef4db8ff ("Fix dump_fs mount point detection and fallback
mount") removed initialization of $_op variable, the variable holding
the options the dump target was mounted with, leading to the below
error as remount was skipped:
kdump[586]: saving to /sysroot/var/crash/127.0.0.1-2021-04-22-07:22:08/
kdump.sh[587]: mkdir: cannot create directory '/sysroot/var/crash/127.0.0.1-2021-04-22-07:22:08/': Read-only file system
kdump[589]: saving vmcore failed
Restore $_op variable initialization in dump_fs() function to fix this.
Fixes: e8ef4db8ff ("Fix dump_fs mount point detection and fallback mount")
Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
Acked-by: Kairui Song <kasong@redhat.com>
This fix bz1854037 which happens because kexec-tools generates rd.route for
eth0 instead of for kdump-eth0,
1. "rd.route=168.63.129.16:10.0.0.1:eth0 rd.route=169.254.169.254:10.0.0.1:eth0" is passed to the dracut cmdline by kexec-tools
2. In the 2rd kernel,
- dracut/modules.d/40network/net-lib.sh will write /tmp/net.route.eth0 based on rd.route
- dracut/modules.d/45ifcfg/write-ifcfg.sh will copy /tmp/net.route.eth0 to /tmp/icfg and then copytree /tmp/ifcfg to /run/initramfs/state/etc/sysconfig/network-scripts
3. NetworkManager will try to get an IP for eth0 regardless of the fact it's a slave NIC and time out
```
$ ip link show
2: kdump-eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 00:0d:3a:11:86:8b brd ff:ff:ff:ff:ff:ff
3: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master kdump-eth0 state UP mode DEFAULT group default qlen 1000
```
Reported-by: Huijing Hei <hhei@redhat.com>
Signed-off-by: Coiby Xu <coxu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com
The `/boot` directory on some operating systems might be read-only.
If we cannot write to `$KDUMP_BOOTDIR` when generating the kdump
initrd, attempt to place the generated initrd at `/var/lib/kdump`
instead.
Signed-off by: Kelvin Fan <kelvinfan001@gmail.com>
Acked-by: Kairui Song <kasong@redhat.com>
Recently, dracut-network drops depedency on dhcp-client which requires
ipcalc. Thus the dependency chain
"kexec-tools -> dracut-network -> dhcp-client -> ipcalc"
is broken. When NIC is configured to a static IP, kexec-tools depended
on "ipcalc -m" to get netmask. This commit implements the shell
equivalent of "ipcalc -m".
The following test code shows cal_netmask_by_prefix is consistent with
"ipcalc -m",
#!/bin/bash
. dracut-module-setup.sh
for i in {0..128}; do
mask_expected=$(ipcalc -m fe::/$i| cut -d"=" -f2)
mask_actual=$(cal_netmask_by_prefix $i "-6")
if [[ "$mask_expected" != "$mask_actual" ]]; then
echo "prefix="$i, "expected="$mask_expected, "acutal="$mask_actual
exit
fi
done
echo "IPv6 tests passed"
for i in {0..32}; do
mask_expected=$(ipcalc -m 8.8.8.8/$i| cut -d"=" -f2)
mask_actual=$(cal_netmask_by_prefix $i "")
if [[ "$mask_expected" != "$mask_actual" ]]; then
echo "prefix="$i, "expected="$mask_expected, "acutal="$mask_actual
exit
fi
done
echo "IPv4 tests passed"
i=-2
res=$(cal_netmask_by_prefix "$i" "")
if [[ $? -ne 1 ]]; then
echo "cal_netmask_by_prefix should exit when prefix<0"
exit
fi
res=$(cal_netmask_by_prefix "$i" "")
if [[ $? -ne 1 ]]; then
echo "cal_netmask_by_prefix should exit when prefix<0"
exit
fi
i=33
$(cal_netmask_by_prefix $i "")
if [[ $? -ne 1 ]]; then
echo "cal_netmask_by_prefix should exit when prefix>32 for IPv4"
exit
fi
i=129
$(cal_netmask_by_prefix $i "-6")
if [[ $? -ne 1 ]]; then
echo "cal_netmask_by_prefix should exit when prefix>128 for IPv4"
exit
fi
echo "Bad prefixes tests passed"
echo "All tests passed"
Reported-by: Jie Li <jieli@redhat.com>
Signed-off-by: Coiby Xu <coxu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
die (in dracut-lib.sh) is supposed to be used in the initramfs environment.
Signed-off-by: Coiby Xu <coxu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
On some large systems, /sys/devices is huge and it's not a wise idea to
iterate it. `find` may cause tremendous contention on the kernfs_mutex
when there are already stress on /sys, and it will perform very very
poorly.
Simply check if drm class presents should be good enough.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
Dracut's run-qemu could find which virtualization technology to the
user in the order of kvm, kqemu, userspace. Using run-qemu could allow
running tests where qemu-kvm doesn't exist.
Signed-off-by: Coiby Xu <coxu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
dracut will build the PRMs which will be installed onto the image to run
the tests.
Signed-off-by: Coiby Xu <coxu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
Currently, TEST_RPMS would be only using the last RPM.
Append each RPM path to TEST_RPMs instead,
Signed-off-by: Coiby Xu <coxu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
When nfs-utils is not installed, mounting as NFS fs would fail.
Currently, the error message is not user-friendly,
mount: /tmp/mkdumprd.HyPGpS/target: bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program.
kdump: Failed to mount on xxx for kdump preflight check.
kdump: mkdumprd: failed to make kdump initrd
Prompt the user to install nfs-utilsa in the error message,
kdump: Failed to mount on xxx for kdump preflight check. Please make sure nfs-utils has been installed.
Signed-off-by: Coiby Xu <coxu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
