When `failure_action` is set to `dump_to_rootfs`, the message:
"Waiting for rootfs mount, will timeout after 90 seconds"
is actually wrong. Kdump will simply call `systemctl start sysroot.mount`,
but the timeout value of sysroot.mount depends on the unit service and
dracut parameters. And by default, dracut will set
JobRunningTimeoutSec=0 and JobTimeoutSec=0 for the device units,
which means it will wait forever. (see wait_for_dev function in dracut)
For some devices, this can be fixed by setting rd.timeout=90. But when
initqueue is set enabled during initramfs build, dracut will force set
timeout for host devices to `0`. (see 99base/module-setup.sh).
Depending on dracut / systemd can make things unpredictable and break as
parameters or code change. To make things easy to understand and
maintain, just call `systemctl` with `--no-block` params, and implement
a standalone wait loop. Now `dump_to_rootfs` will actually wait for
90s then timeout.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Coiby Xu <coxu@redhat.com>
Dump capture initramfs needs rebuild after partition migration (LPM).
Use servicelog notification mechanism to invoke kdump rebuild after
migration.
Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
Reviewed-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
"prot_virt" enables the kernel to run Secure Execution virtual machines
on s390. These virtual machines are isolated from the hypervisor and
thus protected against tampering by a malicious host. Enabling
"prot_virt" requires a minimum of ~2.5GB memory which exceeds what is
typically reserved for the crashkernel. Thus remove "prot_virt" from the
command line for the 2nd kernel to prevent it to run out-of-memory.
For more discussions about this, see:
https://lists.fedoraproject.org/archives/list/kexec@lists.fedoraproject.org/thread/QSRRNV4ALKXUJC2VM3US4Z2NSQRHVMXB/
Signed-off-by: Philipp Rudo <prudo@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
Packaging guidelines have been amended to not require systemd for scriptlets,
see https://docs.fedoraproject.org/en-US/packaging-guidelines/Scriptlets/#_scriptlets.
The comment duplicates what the macro contains.
systemd-sysv-convert binary was removed in 2013, trying to call it is
unlikely to succeed.
chkconfig binary is provided by the chkconfig package, which is not in
Requires. (And makes little sense to call nowadays anyway.)
The new `crashkernel.default` file in kernel package can be used as the
ck_cmdline source.
Also keep the legacy code so old kernel packages will still work.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
This reverts commit 073c30973c, i.e.
re-enable the kexec file load by default since this dual signature
issue no longer bothers Fedora 34.
Signed-off-by: Coiby Xu <coxu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
When kdump failed due to initqueue timeout, the sysroot.mount and other
serivces could be stuck in `start` but `dead` status:
Example output of systemctl:
dev-disk-by\x2duuid-530830d1\x2df2c7\x2d4c9a\x2d9a82\x2d148609097521.device loaded inactive dead start
<... snip ...>
squash-root.mount loaded active mounted /squash/root
squash.mount loaded active mounted /squash
sysroot.mount loaded inactive dead start /sysroot
<... snip ...>
dracut-cmdline.service loaded active exited dracut cmdline hook
dracut-initqueue.service loaded activating start start dracut initqueue hook
dracut-mount.service loaded inactive dead start dracut mount hook
At this point calling `systemctl start sysroot.mount` will just hang as
systemd will just wait for the services that are stuck in `start`
status. So call `systemctl cancel` here to cancel all pending jobs and
have a clean start for mounting sysroot.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Coiby Xu <coxu@redhat.com>
To track and manage kernel's crashkernel usage by kernel version,
each kernel package will include a crashkernel.default containing the
default `crashkernel=` value of that kernel. So we can use a hook to
update the kernel cmdline of new installed kernel accordingly.
Put it after all other grub boot loader setup hooks, so it can simply
call grubby to modify the kernel cmdline.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
In newer kernel, crashkernel.default will contain the default
crashkernel value of a kernel build. So introduce a new sub command
to help user reset kernel crashkernel size to the default value.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
Now we need this helper again, for `reset-crashkernel`
This reverts commit ff46cfb19e.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
When running with squash module enabled for both initramfs, /dev and
/run are also mounted by squash-init, so move them to newroot as well,
else they might leak.
Also pass `-d` to umount so loop devices (if used) will be force freed.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Hari Bathini <hbathini@linux.ibm.com>
After fadump embedded the fadump initramfs in the normal initramfs,
kdumpctl will mistakenly rebuild the initramfs everytime.
kdumpctl checks the hostonly-kernel-modules.txt file in initramfs
to check if required drivers are included, but the normal initramfs
is built in non-hostonly mode, so it doesn't have a
hostonly-kernel-modules.txt file. The check will always fail.
So let mkfadumprd make a copy of the hostonly-kernel-modules.txt in the
fadump initramfs and let kdumpctl check that file instead.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Hari Bathini <hbathini@linux.ibm.com>
In case of fadump, the initramfs image has to be built to boot into
the production environment as well as to offload the active crash dump
to the specified dump target (for boot after crash). As the same image
would be used for both boot scenarios, it could not be built optimally
while accommodating both cases.
Use --include to include the initramfs image built for offloading
active crash dump to the specified dump target. Also, introduce a new
out-of-tree dracut module (99zz-fadumpinit) that installs a customized
init program while moving the default /init to /init.dracut. This
customized init program is leveraged to isolate fadump image within
the default initramfs image by kicking off default boot process
(exec /init.dracut) for regular boot scenario and activating fadump
initramfs image, if the system is booting after a crash.
If squash is available, ensure default initramfs image is also built
with squash module to reduce memory consumption in capture kernel.
Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
Change spaces to tab to fix alignment issue.
Fixes: commit 7d47251568
("Iterate /sys/bus/ccwgroup/devices to tell if we should set up rd.znet")
Signed-off-by: Coiby Xu <coxu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
/sys/bus/ccwgroup/devices doesn't exist for non-s390x machines which leads to
the warning "find: '/sys/bus/ccwgroup/devices': No such file or directory".
This warning can be eliminated by checking the existence of
"/sys/bus/ccwgroup/devices" beforehand.
Fixes: commit 7d47251568
("Iterate /sys/bus/ccwgroup/devices to tell if we should set up rd.znet")
Reported-by: Ruowen Qin <ruqin@redhat.com>
BugLink: https://bugzilla.redhat.com/show_bug.cgi?id=1974618
Signed-off-by: Coiby Xu <coxu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
Backport from upstream.
commit 9a6f589d99dcef114c89fde992157f5467028c8f
Author: Tao Liu <ltao@redhat.com>
Date: Fri Jun 18 18:28:04 2021 +0800
[PATCH] check for invalid physical address of /proc/kcore when making ELF dumpfile
Previously when executing makedumpfile with -E option against
/proc/kcore, makedumpfile will fail:
# makedumpfile -E -d 31 /proc/kcore kcore.dump
...
write_elf_load_segment: Can't convert physaddr(ffffffffffffffff) to an offset.
makedumpfile Failed.
It's because /proc/kcore contains PT_LOAD program headers which have
physaddr (0xffffffffffffffff). With -E option, makedumpfile will
try to convert the physaddr to an offset and fails.
Skip the PT_LOAD program headers which have such physaddr.
Signed-off-by: Tao Liu <ltao@redhat.com>
Signed-off-by: Kazuhito Hagio <k-hagio-ab@nec.com>
Signed-off-by: Tao Liu <ltao@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
Backport from upstream.
commit 38d921a2ef50ebd36258097553626443ffe27496
Author: Coiby Xu <coxu@redhat.com>
Date: Tue Jun 15 18:26:31 2021 +0800
[PATCH] check for invalid physical address of /proc/kcore when finding max_paddr
Kernel commit 464920104bf7adac12722035bfefb3d772eb04d8 ("/proc/kcore:
update physical address for kcore ram and text") sets an invalid paddr
(0xffffffffffffffff = -1) for PT_LOAD segments of not direct mapped
regions:
$ readelf -l /proc/kcore
...
Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
NOTE 0x0000000000000120 0x0000000000000000 0x0000000000000000
0x0000000000002320 0x0000000000000000 0x0
LOAD 0x1000000000010000 0xd000000000000000 0xffffffffffffffff
^^^^^^^^^^^^^^^^^^
0x0001f80000000000 0x0001f80000000000 RWE 0x10000
makedumpfile uses max_paddr to calculate the number of sections for
sparse memory model thus wrong number is obtained based on max_paddr
(-1). This error could lead to the failure of copying /proc/kcore
for RHEL-8.5 on ppc64le machine [1]:
$ makedumpfile /proc/kcore vmcore1
get_mem_section: Could not validate mem_section.
get_mm_sparsemem: Can't get the address of mem_section.
makedumpfile Failed.
Let's check if the phys_start of the segment is a valid physical
address to fix this problem.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1965267
Reported-by: Xiaoying Yan <yiyan@redhat.com>
Signed-off-by: Coiby Xu <coxu@redhat.com>
Signed-off-by: Tao Liu <ltao@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
Backport from upstream.
commit 646456862df8926ba10dd7330abf3bf0f887e1b6
Author: Kazuhito Hagio <k-hagio-ab@nec.com>
Date: Wed May 26 14:31:26 2021 +0900
[PATCH] Increase SECTION_MAP_LAST_BIT to 5
* Required for kernel 5.12
Kernel commit 1f90a3477df3 ("mm: teach pfn_to_online_page() about
ZONE_DEVICE section collisions") added a section flag
(SECTION_TAINT_ZONE_DEVICE) and causes makedumpfile an error on
some machines like this:
__vtop4_x86_64: Can't get a valid pmd_pte.
readmem: Can't convert a virtual address(ffffe2bdc2000000) to physical address.
readmem: type_addr: 0, addr:ffffe2bdc2000000, size:32768
__exclude_unnecessary_pages: Can't read the buffer of struct page.
create_2nd_bitmap: Can't exclude unnecessary pages.
Increase SECTION_MAP_LAST_BIT to 5 to fix this. The bit had not
been used until the change, so we can just increase the value.
Signed-off-by: Kazuhito Hagio <k-hagio-ab@nec.com>
Signed-off-by: Tao Liu <ltao@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
For the log entry that has multiple lines, "makedumpfile --dump-dmesg"
would indent the remaining lines while vmcore-dmesg doesn't. For
example, vmcore-dmesg.txt and vmcore-dmesg.txt.2 are the outputs of
vmcore-dmesg and "makedumpfile --dump-dmesg" respectively,
```
diff -u vmcore-dmesg.txt vmcore-dmesg.txt.2
--- vmcore-dmesg.txt 2021-03-28 22:13:09.986000000 -0400
+++ vmcore-dmesg.txt.2 2021-03-28 22:13:39.920106131 -0400
@@ -397,9 +397,9 @@
[ 1.710742] vc vcsa: hash matches
[ 1.711938] RAS: Correctable Errors collector initialized.
[ 1.713736] Unstable clock detected, switching default tracing clock to "global"
-If you want to keep using the local clock, then add:
- "trace_clock=local"
-on the kernel command line
+ If you want to keep using the local clock, then add:
+ "trace_clock=local"
+ on the kernel command line
[ 1.750539] ata1.01: NODEV after polling detection
[ 1.750973] ata1.00: ATA-7: QEMU HARDDISK, 2.5+, max UDMA/100
[ 1.752885] ata1.00: 8388608 sectors, multi 16: LBA48
```
Quite often, all three tests could fail because of the above difference. So
let's ignore all the spaces. This patch could fix bz1952299 [1].
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1952299
Signed-off-by: Coiby Xu <coxu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
test_base_image should depend on EXTRA_RPMS so it gets rebuild when
EXTRA_RPMS changes.
Fixes: commit bbc064f958
("selftest: add EXTRA_RPMs so dracut RPMs can be installed onto
the image to run the tests")
Signed-off-by: Coiby Xu <coxu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
When kdump service fails, the current errors do not display the
absolute path of dump location(marked it as "^"), for example:
kdump: kexec: unloaded kdump kernel
kdump: Stopping kdump: [OK]
kdump: Detected change(s) in the following file(s): /etc/kdump.conf
kdump: Rebuilding /boot/initramfs-4.18.0-304.el8.x86_64kdump.img
kdump: Dump path "/var1/crash" does not exist in dump target "UUID=c202ef45-3ac3-4adb-85e7-307a916757f0"
^^^^^^^^^^^
kdump: mkdumprd: failed to make kdump initrd
kdump: Starting kdump: [FAILED]
Here, it should output the absolute path of dump location with this
format: "<mount path>/<path>". To fix it, let's extend the relative
pathname to the absolute pathname in check_user_configured_target().
Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
This patch fixes bz1941106 and bz1941905 which passed empty rd.znet to the
kernel command line in the following cases,
- The IBM (Z15) KVM guest uses virtio for all devices including network
device, so there is no znet device for IBM KVM guest. So we can't
assume a s390x machine always has a znet device.
- When a bridged network is used, kexec-tools tries to obtain the znet
configuration from the ifcfg script of the bridged network rather than
from the ifcfg script of znet device.
We can iterate /sys/bus/ccwgroup/devices to tell if there if there is
a znet network device. By getting an ifname from znet, we can also avoid
mistaking the slave netdev as a znet network device in a bridged network
or bonded network.
Note: This patch also assumes there is only one znet device as commit
7148c0a30d ("add s390x netdev setup")
which greatly simplifies the code. According to IBM [1], there could be
more than znet devices for a z/VM system and a z/VM system may have a
non-znet network device like ConnectX. Since kdump_setup_znet was
introduced in 2012 and so far there is no known customer complaint that
invalidates this assumption I think it's safe to assume an IBM z/VM
system only has one znet device. Besides, there is no z/VM system found
on beaker to test the alternative scenarios.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1941905#c13
Signed-off-by: Coiby Xu <coxu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
Use a modified and minimized version of emergency shell.
The differences of this kdump shell and dracut emergency shell are:
- Kdump shell won't generate a rdsosreport automatically
- Customized prompts
- Never ask root password
- Won't tangle with dracut's emergency_action. If emergency_action is
set, dracut emergency shell will perform dracut's emergency_action
instead of kdump final_action on exit.
- If rd.shell=no is set, kdump shell will still work, dracut emergency
shell won't, even if kdump failure_action is set to shell.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Coiby Xu <coxu@redhat.com>
The wrapper is introduced in commit 002337c, according to the commit
message, the only usage of the wrapper is when dracut-initqueue calls
"systemctl start emergency" directly. In that case, emergency
is started, but not in a isolation mode, which means dracut-initqueue
is still running. On the other hand, emergency will call
"systemctl start dracut-initqueue" again when default action is dump_to_rootfs.
systemd would block on the last dracut-initqueue, waiting for the first
instance to exit, which leaves us hang.
In previous commit we added initqueue status detect in dump_to_rootfs,
so now even without the wrapper, it will not hang.
And actually, previously, with the wrapper, emergency might still hang
for like 30s. When dracut called emergency service because initqueue
timed out, dump_to_rootfs will try start initqueue again and timeout
again. Now with the wrapper removed, we can avoid these two kinds of
hangs, bacause without the isolation we can detect initqueue service
status correctly in such case.
Also remove the invalid header comments in service file, the service
is not part of systemd code. And sync the service spec with dracut.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Coiby Xu <coxu@redhat.com>
kdump's dump_to_rootfs will try to start initqueue unconditionally.
dump_to_rootfs will run after systemd isolate to emergency
target, so this is currently accetable.
But there is a problem when initqueue starts the emergency action
because of initqueue timeout. dump_to_rootfs will start initqueue and
lead to timeout again.
So following patch will remove the previous isolation wrapper, and
detect the service status here. Previous isolation makes the detection
impossible. Now this detection will be valid and helpful to prevent
double timeout or hang.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Coiby Xu <coxu@redhat.com>
Fix the warning observed when KDUMP_KERNELVER is specified:
kdumpctl[10926]: /lib/kdump/kdump-lib.sh: line 697: [: missing `]'
Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
Acked-by: Kairui Song <kasong@redhat.com>
For crashkernel=auto policy, if total RAM size is under a throttle,
there is no memory reserved for kdump.
Also correct a trivial bug by correcting the arch name.
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
Add a rough esitimation support, currently, following memory usage are
checked by this sub command:
- System RAM
- Kdump Initramfs size
- Kdump Kernel image size
- Kdump Kernel module size
- Kdump userspace user and other runtime allocated memory (currently
simply using a fixed value: 64M)
- LUKS encryption memory usage
The output of kdumpctl estimate looks like this:
# kdumpctl estimate
Reserved crashkernel: 256M
Recommanded crashkernel: 160M
Kernel image size: 47M
Kernel modules size: 12M
Initramfs size: 19M
Runtime reservation: 64M
Large modules:
xfs: 1892352
nouveau: 2318336
And if the kdump target is encrypted:
# kdumpctl estimate
Encrypted kdump target requires extra memory, assuming using the keyslot with minimun memory requirement
Reserved crashkernel: 256M
Recommanded crashkernel: 655M
Kernel image size: 47M
Kernel modules size: 12M
Initramfs size: 19M
Runtime reservation: 64M
LUKS required size: 512M
Large modules:
xfs: 1892352
nouveau: 2318336
WARNING: Current crashkernel size is lower than recommanded size 655M.
The "Recommanded" value is calculated based on memory usages mentioned
above, and will be adjusted accodingly to be no less than the value provided
by kdump_get_arch_recommend_size.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
Simplfy the code and also improve the performance. udevadm call is
heavy.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
kexec-tools needs to disable CMA for kdump kernel cmdline,
otherwise kdump kernel may run out of memory.
This patch strips the inherited cma=, hugetlb_cma= cmd
line from 1st kernel, and sets to be 0 for 2nd kernel.
Signed-off-by: Tao Liu <ltao@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
apath (a D-Bus active connection path) is used for nmcli connection operations, e.g.
$ nmcli connection show $apath
Signed-off-by: Coiby Xu <coxu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
nmcli --get-values <field> connection show /org/freedesktop/NetworkManager/ActiveConnection/1
returns the following value for the corresponding field respectively,
Field Value
IP4.DNS "10.19.42.41 | 10.11.5.19 | 10.5.30.160"
802-3-ethernet.s390-subchannels ""
bond.options "mode=balance-rr"
Signed-off-by: Coiby Xu <coxu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
This fixes bz1854037 which happens because kexec-tools generates rd.route for
eth0 instead of for kdump-eth0,
1. "rd.route=168.63.129.16:10.0.0.1:eth0 rd.route=169.254.169.254:10.0.0.1:eth0" is passed to the dracut cmdline by kexec-tools
2. In the 2rd kernel, dracut/modules.d/35network-manager/nm-config.sh calls
/usr/libexec/nm-initrd-generator to generate two .nmconnection files
based on the dracut cmdline, i.e. kdump-eth0.nmconnection and eth0.nmconnection,
- /run/NetworkManager/system-connections/kdump-eth0.nmconnection
[connection]
id=kdump-eth0
uuid=3ef53b1b-3908-437e-a15f-cf1f3ea2678b
type=ethernet
autoconnect-retries=1
interface-name=kdump-eth0
multi-connect=1
permissions=
wait-device-timeout=60000
[ethernet]
mac-address-blacklist=
[ipv4]
address1=10.0.0.4/24,10.0.0.1
dhcp-timeout=90
dns=168.63.129.16;
dns-search=
may-fail=false
method=manual
[ipv6]
addr-gen-mode=eui64
dhcp-timeout=90
dns-search=
method=disabled
[proxy]
- /run/NetworkManager/system-connections/eth0.nmconnection
[connection]
id=eth0
uuid=f224dc22-2891-4d7b-8f66-745029df4b53
type=ethernet
autoconnect-retries=1
interface-name=eth0
multi-connect=1
permissions=
[ethernet]
mac-address-blacklist=
[ipv4]
dhcp-timeout=90
dns=168.63.129.16;
dns-search=
method=auto
route1=168.63.129.16/32,10.0.0.1
route2=169.254.169.254/32,10.0.0.1
[ipv6]
addr-gen-mode=eui64
dhcp-timeout=90
dns-search=
method=auto
[proxy]
3. Since there's eth0.nmconnection, NetworkManager will try to get an IP for eth0 regardless of the fact it's a slave NIC and time out
```
$ ip link show
2: kdump-eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 00:0d:3a:11:86:8b brd ff:ff:ff:ff:ff:ff
3: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master kdump-eth0 state UP mode DEFAULT group default qlen 1000
```
Reported-by: Huijing Hei <hhei@redhat.com>
Signed-off-by: Coiby Xu <coxu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
