This patch setup kdump network by directly copying NM connection profile(s)
for different network setup including bond, bridge, vlan, and team. For
vlan network, rename phydev to parent_netif to improve code readability.
With the new approach, the related code to build up dracut cmdline
parameter such rd.route, ip and etc can be cleaned up. And there is no
need to setup dns when copying .nmconnection directly to initrd
either. Note the bootdev dracut command line parameter is only used by
dracut's 35network-legacy and network-manager doesn't use it, remove
related code as well.
Note
1. kdump_setup_vlan/bond/... are no longer called in subshells in order
to modify global variables like unique_netifs
2. The original kdump_install_net is renamed to better reflect its
current function
Signed-off-by: Coiby Xu <coxu@redhat.com>
Reviewed-by: Thomas Haller <thaller@redhat.com>
Reviewed-by: Philipp Rudo <prudo@redhat.com>
kexec-tools depends on dracut's 35network-manager module which will
call nm-initrd-generator. We don't want nm-initrd-generator to generate
connection profiles since we will copy them from 1st kernel to
kdump kernel initramfs. NetworkManager >= 1.35.2 won't generate connection
profiles if there's a connection dir with rd.neednet. For Fedora/RHEL,
this connection dir is /etc/NetworkManager/system-connections. For the
details, please refer to the NetworkManager commit 79885656d3
("initrd: don't add a connection if there's a connection dir with
rd.neednet") [1]. Before the release of NetworkManager >= 1.35.2, we
need to mask /usr/libexec/nm-initrd-generator.
[1] https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/1010
Signed-off-by: Coiby Xu <coxu@redhat.com>
Reviewed-by: Thomas Haller <thaller@redhat.com>
Reviewed-by: Philipp Rudo <prudo@redhat.com>
nm-wait-online-initrd.service installed by dracut's 35-networkmanager
module calls nm-online with "-s" which means it returns immediately when
NetworkManager logs "startup complete" after certain timeouts are
reached. "startup complete" doesn't necessarily network connectivity has
been established. nm-initrd-generator has a set of timeouts that in most
of cases when applied, "startup-complete" means network connectivity has
been established. So apply it when setting up kdump network.
Suggested-by: Thomas Haller <thaller@redhat.com>
Signed-off-by: Coiby Xu <coxu@redhat.com>
Reviewed-by: Thomas Haller <thaller@redhat.com>
Reviewed-by: Philipp Rudo <prudo@redhat.com>
According to `man nm-online`,
"By default, connections have the ipv4.may-fail and
ipv6.may-fail properties set to yes; this means that
NetworkManager waits for one of the two address families to
complete configuration before considering the connection
activated. If you need a specific address family configured
before network-online.target is reached, set the corresponding
may-fail property to no."
If a NIC has an IPv4 or IPv6 address, set the corresponding may-fail
property to no. Otherwise, dumping vmcore over IPv6 could fail because
only IPv4 network is ready or vice versa.
Also disable IPv6 if only IPv4 is used and vice versa.
Signed-off-by: Coiby Xu <coxu@redhat.com>
Reviewed-by: Thomas Haller <thaller@redhat.com>
Reviewed-by: Philipp Rudo <prudo@redhat.com>
Each network interface is manged by a NM connection. Given a list of
network interface names, copy the NetworkManager (NM) connection
profiles i.e. .nmconnection files to the kdump initramfs.
Before copying a connection file, clone it to automatically convert a
legacy ifcfg-*[1] file to a .nmconnection file and for the convenience of
editing the connection profile.
[1] https://fedoraproject.org/wiki/Changes/NetworkManager_keyfile_instead_of_ifcfg_rh
Signed-off-by: Coiby Xu <coxu@redhat.com>
Reviewed-by: Thomas Haller <thaller@redhat.com>
Reviewed-by: Philipp Rudo <prudo@redhat.com>
6f9235887f ("module-setup.sh: enable
vlan on team interface") skips establishing teaming network by mistake.
Although it could use one of slave netifs to establish connection
to transfer vmcore to remote fs, it breaks the implicit assumption of
creating an identical network topology to the 1st kernel.
Fixes: 6f92358 ("module-setup.sh: enable vlan on team interface")
Signed-off-by: Coiby Xu <coxu@redhat.com>
Reviewed-by: Thomas Haller <thaller@redhat.com>
Reviewed-by: Philipp Rudo <prudo@redhat.com>
The 80lvmthinpool-monitor module is needed for monitor and
autoextend the size of thin pool in 2nd kernel. The module was
integrated in dracut version 057.
If lvmthinpool-monitor module is not found, we will print a warning.
Because we don't want to block the kdump process when the thin pool
capacity is enough and no monitor-and-autoextend actually needed.
Signed-off-by: Tao Liu <ltao@redhat.com>
Reviewed-by: Philipp Rudo <prudo@redhat.com>
This patch add virtiofs support for kexec-tools by introducing a new option
for /etc/kdump.conf:
virtiofs myfs
Where myfs is a variable tag name specified in qemu cmdline
"-device vhost-user-fs-pci,tag=myfs".
The patch covers the following cases:
1) Dumping VM's vmcore to a virtiofs shared directory;
2) When the VM's rootfs is a virtiofs shared directory and dumping the
VM's vmcore to its subdirectory, such as /var/crash;
3) The combination of case 1 & 2: The VM's rootfs is a virtiofs shared
directory and dumping the VM's vmcore to another virtiofs shared
directory.
Case 2 & 3 need dracut >= 057, otherwise VM cannot boot from virtiofs
shared rootfs. But it is not the issue of kexec-tools.
Reviewed-by: Philipp Rudo <prudo@redhat.com>
Signed-off-by: Tao Liu <ltao@redhat.com>
Resolves: bz2106645
The code of commit 163c02970e takes effect in rhel firstly, later
pulled to Fedora. However, Fedora OS doesn't have 40-redhat.rules
in systemd-udev package. With this commit applied, a false positive
warning message can always been seen as below.
So fixing it by checking if 40-redhat.rules exists before handling.
With this change, the false warning is gone.
[root@ ~]# kdumpctl restart
kdump: kexec: unloaded kdump kernel
kdump: Stopping kdump: [OK]
kdump: No kdump initial ramdisk found.
kdump: Rebuilding /boot/initramfs-5.19.0-rc6+kdump.img
sed: can't read /var/tmp/dracut.NnAV2g/initramfs/usr/lib/udev/rules.d/40-redhat.rules: No such file or directory
kdump: kexec: loaded kdump kernel
kdump: Starting kdump: [OK]
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
Onlining secondary cpus breaks kdump completely on KVM on Power hosts
Though we use maxcpus=1 by default but 40-redhat.rules will bring up all
possible cpus by default.
Thus before we get the kernel fix and the systemd rule fix let's remove
the cpu rule in 40-redhat.rules for ppc64/ppc64le kdump initramfs.
This is back ported from RHEL, and original credit goes to Dave Young
<dyoung@redhat.com>
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Tao Liu <ltao@redhat.com>
When there more than one binaries, quoting "$val" would make
dracut-install treat multiple binaries as one binary. Take
"extra_bins /usr/sbin/ping /usr/sbin/ip" as an example, the
following error would occur when building initrd,
dracut-install: ERROR: installing '/usr/sbin/ping /usr/sbin/ip'
dracut: FAILED: /usr/lib/dracut/dracut-install -D /var/tmp/dracut.ODrioZ/initramfs -a /usr/sbin/ping /usr/sbin/ip
Fix it by not quoting the variable and bypassing SC2086 shellcheck.
Fixes: commit 86538ca6e2
("bash scripts: fix variable quoting issue")
Acked-by: Tao Liu <ltao@redhat.com>
Signed-off-by: Coiby Xu <coxu@redhat.com>
This fixes word splitting issue with nmcli args. Current kexec-tools
scripts won't call nmcli with correct arguments when there are space in
network interface name.
nmcli expects multiple parameters, but get_nmcli_value_by_field only
accepts two params and depends on shell word splitting to split the
_nm_show_cmd into multiple params, which is very fragile.
So switch the param order, simplified this function and now multiple
params can be used properly.
And get_nmcli_connection_show_cmd_by_ifname returns multiple
nmcli params in a single variable, it depend on shell word splitting to
split the words when calling nmcli. But this is very fragile and break
easily when there are any special character in the connection path.
This function is only introduced to get and cache the nmcli command
which contains the "connection name".
Actually only cache the "connection path" is enough. Callers should
just call get_nmcli_connection_apath_by_ifname to cache the path, and
a new helper get_nmcli_field_by_conpath is introduced here to get value
from nmcli. This way "connection path" can contain any character.
Also get rid of another nmcli_cmd usage in
get_nmcli_connection_apath_by_ifname which stores multiple params in a
single bash variable separated by space.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Philipp Rudo <prudo@redhat.com>
kdump-error-handler.sh does nothing except calling three functions,
it can be easily merged into kdump.sh by using a parameter to run the
error handling routine.
kdump-lib-initramfs.sh was created to hold the three shared functions
and related code, so by merging these two files, kdump-lib-initramfs.sh
can be simplified by a lot.
Following up commits will clean up kdump-lib-initramfs.sh.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Philipp Rudo <prudo@redhat.com>
Move all functions needed in the second kernel from kdump-lib.sh
to kdump-lib-initramfs.sh, and update shebang headers.
Now, kdump-lib-initramfs.sh is an independent lib script, no longer
depend on kdump-lib.sh, and kdump-lib.sh is no longer needed for
the second kernel.
In later commits, functions in kdump-lib-initramfs.sh will be reworked
to be POSIX compatible, kdump-lib.sh will contain bash only functions.
POSIX shell have very limited features, eg. `local` keyword doesn't
exist in POSIX but we rely on that heavily. So kdump-lib.sh will
use bash syntax and contain the most complex helper and codes.
kdump-lib-initramfs.sh will contain the minimum set of helpers,
and be shared by both the first and second kernel.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Philipp Rudo <prudo@redhat.com>
This is a batch update done with:
shfmt -s -w mkfadumprd mkdumprd kdumpctl *-module-setup.sh
Clean up code style and reduce code base size, no behaviour change.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Philipp Rudo <prudo@redhat.com>
Declare and assign separately to avoid masking return values:
https://github.com/koalaman/shellcheck/wiki/SC2155
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Philipp Rudo <prudo@redhat.com>
kdumpctl, mkdumprd, *-module-setup.sh only target bash, since they
only run in first kernel and depend on dracut, and dracut depends
on bash. So use '[[ ]]' to replace '[ ]'.
This is a batch update done with following command:
`sed -i -e 's/\(\s\)\[\s\([^]]*\)\s\]/\1\[\[\ \2 \]\]/g' kdumpctl, mkdumprd, *-module-setup.sh`
and replaced [ ... -a ... ] with [[ ... ]] && [[ ... ]] manually.
See https://tldp.org/LDP/abs/html/testconstructs.html for more details
on '[[ ]]', it's more versatile, safer, and slightly faster than '[ ]'.
This will also help shfmt to clean up the code in later commits.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Philipp Rudo <prudo@redhat.com>
This is a batch update done with following command:
`sed -i -e 's/`\([^`]*\)`/\$(\1)/g' mkfadumprd mkdumprd \
kdumpctl dracut-module-setup.sh dracut-fadump-module-setup.sh \
dracut-early-kdump-module-setup.sh`
And manually converted some corner cases. This fixes
all related issues detected by shellcheck.
Make it easier to do clean up in later commits.
Check following link for reasons to switch to the new syntax:
https://github.com/koalaman/shellcheck/wiki/SC2006
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Philipp Rudo <prudo@redhat.com>
Use bash builtin string substitution instead, as suggested by:
https://github.com/koalaman/shellcheck/wiki/SC2001
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Philipp Rudo <prudo@redhat.com>
Some `cat` calls are useless, remove them to make it cleaner.
See: https://github.com/koalaman/shellcheck/wiki/SC2002
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Philipp Rudo <prudo@redhat.com>
Some functions are executed in subshell to avoid variable environment
pollution. But the surrounding $() is not needed, and it may lead to
executing output which is unexpected here.
See: https://github.com/koalaman/shellcheck/wiki/SC2091
Signed-off-by: Kairui Song <kasong@redhat.com>
Suggested-by: Coiby Xu <coxu@redhat.com>
Wrap the variable with {...}, else it may get interpreted as array due
to the '[' char next to it.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Philipp Rudo <prudo@redhat.com>
As suggested by:
https://github.com/koalaman/shellcheck/wiki/SC2199
The array is not quoted here but implicitly concatenate still happens,
could be harmless but shellcheck complains about it so fix it.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Philipp Rudo <prudo@redhat.com>
Add a helper kdump_get_conf_val to replace get_option_value.
It can help cover more corner cases in the code, like when there are
multiple spaces in config file, config value separated by a tab,
heading spaces, or trailing comments.
And this uses "sed group command" and "sed hold buffer", make it much
faster than previous `grep <config> | tail -1`.
This helper is supposed to provide a universal way for kexec-tools
scripts to read in config value. Currently, different scripts are
reading the config in many different fragile ways.
For example, following codes are found in kexec-tools script code base:
1. grep ^force_rebuild $KDUMP_CONFIG_FILE
echo $_force_rebuild | cut -d' ' -f2
2. grep ^kdump_post $KDUMP_CONFIG_FILE | cut -d\ -f2
3. awk '/^sshkey/ {print $2}' $conf_file
4. grep ^path $KDUMP_CONFIG_FILE | cut -d' ' -f2-
1, 2, and 4 will fail if the space is replaced by, e.g. a tab
1 and 2 might fail if there are multiple spaces between config name
and config value:
"kdump_post /var/crash/scripts/kdump-post.sh"
A space will be read instead of config value.
1, 2, 3 will fail if there are space in file path, like:
"kdump_post /var/crash/scripts dir/kdump-post.sh"
4 will fail if there are trailing comments:
"path /var/crash # some comment here"
And all will fail if there are heading space,
" path /var/crash"
And all will most likely cause problems if the config file contains
the same option more than once.
And all of them are slower than the new sed call. Old get_option_value
is also very slow and doesn't handle heading space.
Although we never claim to support heading space or tailing comments
before, it's harmless to be more robust on config reading, and many
conf files in /etc support heading spaces. And have a faster and
safer config reading helper makes it easier to clean up the code.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Philipp Rudo <prudo@redhat.com>
Add a helper `kdump_read_conf` to replace read_strip_comments.
`kdump_read_conf` does a few more things:
- remove trailing spaces.
- format the content, remove duplicated spaces between name and value.
- read from KDUMP_CONFIG_FILE (/etc/kdump.conf) directly, avoid pasting
"/etc/kdump.conf" path everywhere in the code.
- check if config file exists, just in case.
Also unify the environmental variable, now KDUMP_CONFIG_FILE stands for
the default config location.
This helps avoid some shell pitfalls about spaces when reading config.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Philipp Rudo <prudo@redhat.com>
On s390x KVM machines, the following errors would show when building kdump
initramfs that dumps vmcore to a remote target,
$ kdumpctl rebuild
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh: line 475: /sys/bus/ccwgroup/devices/online: No such file or directory
/usr/lib/dracut/modules.d/99kdumpbase/module-setup.sh: line 476: [: -ne: unary operator expected
This happens because s390x KVM machines use virtual network and
/sys/bus/ccwgroup/devices/ exists but is empty. Fix it by check
the existence of file "/sys/bus/ccwgroup/devices/*/online".
Fixes: commit 7d47251568
("Iterate /sys/bus/ccwgroup/devices to tell if we should set up rd.znet")
BugLink: https://bugzilla.redhat.com/show_bug.cgi?id=1982474
Reported-by: Jie Li <jieli@redhat.com>
Signed-off-by: Coiby Xu <coxu@redhat.com>
t Acked-by: Kairui Song <kasong@redhat.com>
In case of fadump, the initramfs image has to be built to boot into
the production environment as well as to offload the active crash dump
to the specified dump target (for boot after crash). As the same image
would be used for both boot scenarios, it could not be built optimally
while accommodating both cases.
Use --include to include the initramfs image built for offloading
active crash dump to the specified dump target. Also, introduce a new
out-of-tree dracut module (99zz-fadumpinit) that installs a customized
init program while moving the default /init to /init.dracut. This
customized init program is leveraged to isolate fadump image within
the default initramfs image by kicking off default boot process
(exec /init.dracut) for regular boot scenario and activating fadump
initramfs image, if the system is booting after a crash.
If squash is available, ensure default initramfs image is also built
with squash module to reduce memory consumption in capture kernel.
Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
Change spaces to tab to fix alignment issue.
Fixes: commit 7d47251568
("Iterate /sys/bus/ccwgroup/devices to tell if we should set up rd.znet")
Signed-off-by: Coiby Xu <coxu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
/sys/bus/ccwgroup/devices doesn't exist for non-s390x machines which leads to
the warning "find: '/sys/bus/ccwgroup/devices': No such file or directory".
This warning can be eliminated by checking the existence of
"/sys/bus/ccwgroup/devices" beforehand.
Fixes: commit 7d47251568
("Iterate /sys/bus/ccwgroup/devices to tell if we should set up rd.znet")
Reported-by: Ruowen Qin <ruqin@redhat.com>
BugLink: https://bugzilla.redhat.com/show_bug.cgi?id=1974618
Signed-off-by: Coiby Xu <coxu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
This patch fixes bz1941106 and bz1941905 which passed empty rd.znet to the
kernel command line in the following cases,
- The IBM (Z15) KVM guest uses virtio for all devices including network
device, so there is no znet device for IBM KVM guest. So we can't
assume a s390x machine always has a znet device.
- When a bridged network is used, kexec-tools tries to obtain the znet
configuration from the ifcfg script of the bridged network rather than
from the ifcfg script of znet device.
We can iterate /sys/bus/ccwgroup/devices to tell if there if there is
a znet network device. By getting an ifname from znet, we can also avoid
mistaking the slave netdev as a znet network device in a bridged network
or bonded network.
Note: This patch also assumes there is only one znet device as commit
7148c0a30d ("add s390x netdev setup")
which greatly simplifies the code. According to IBM [1], there could be
more than znet devices for a z/VM system and a z/VM system may have a
non-znet network device like ConnectX. Since kdump_setup_znet was
introduced in 2012 and so far there is no known customer complaint that
invalidates this assumption I think it's safe to assume an IBM z/VM
system only has one znet device. Besides, there is no z/VM system found
on beaker to test the alternative scenarios.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1941905#c13
Signed-off-by: Coiby Xu <coxu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
The wrapper is introduced in commit 002337c, according to the commit
message, the only usage of the wrapper is when dracut-initqueue calls
"systemctl start emergency" directly. In that case, emergency
is started, but not in a isolation mode, which means dracut-initqueue
is still running. On the other hand, emergency will call
"systemctl start dracut-initqueue" again when default action is dump_to_rootfs.
systemd would block on the last dracut-initqueue, waiting for the first
instance to exit, which leaves us hang.
In previous commit we added initqueue status detect in dump_to_rootfs,
so now even without the wrapper, it will not hang.
And actually, previously, with the wrapper, emergency might still hang
for like 30s. When dracut called emergency service because initqueue
timed out, dump_to_rootfs will try start initqueue again and timeout
again. Now with the wrapper removed, we can avoid these two kinds of
hangs, bacause without the isolation we can detect initqueue service
status correctly in such case.
Also remove the invalid header comments in service file, the service
is not part of systemd code. And sync the service spec with dracut.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Coiby Xu <coxu@redhat.com>
This fixes bz1854037 which happens because kexec-tools generates rd.route for
eth0 instead of for kdump-eth0,
1. "rd.route=168.63.129.16:10.0.0.1:eth0 rd.route=169.254.169.254:10.0.0.1:eth0" is passed to the dracut cmdline by kexec-tools
2. In the 2rd kernel, dracut/modules.d/35network-manager/nm-config.sh calls
/usr/libexec/nm-initrd-generator to generate two .nmconnection files
based on the dracut cmdline, i.e. kdump-eth0.nmconnection and eth0.nmconnection,
- /run/NetworkManager/system-connections/kdump-eth0.nmconnection
[connection]
id=kdump-eth0
uuid=3ef53b1b-3908-437e-a15f-cf1f3ea2678b
type=ethernet
autoconnect-retries=1
interface-name=kdump-eth0
multi-connect=1
permissions=
wait-device-timeout=60000
[ethernet]
mac-address-blacklist=
[ipv4]
address1=10.0.0.4/24,10.0.0.1
dhcp-timeout=90
dns=168.63.129.16;
dns-search=
may-fail=false
method=manual
[ipv6]
addr-gen-mode=eui64
dhcp-timeout=90
dns-search=
method=disabled
[proxy]
- /run/NetworkManager/system-connections/eth0.nmconnection
[connection]
id=eth0
uuid=f224dc22-2891-4d7b-8f66-745029df4b53
type=ethernet
autoconnect-retries=1
interface-name=eth0
multi-connect=1
permissions=
[ethernet]
mac-address-blacklist=
[ipv4]
dhcp-timeout=90
dns=168.63.129.16;
dns-search=
method=auto
route1=168.63.129.16/32,10.0.0.1
route2=169.254.169.254/32,10.0.0.1
[ipv6]
addr-gen-mode=eui64
dhcp-timeout=90
dns-search=
method=auto
[proxy]
3. Since there's eth0.nmconnection, NetworkManager will try to get an IP for eth0 regardless of the fact it's a slave NIC and time out
```
$ ip link show
2: kdump-eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 00:0d:3a:11:86:8b brd ff:ff:ff:ff:ff:ff
3: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master kdump-eth0 state UP mode DEFAULT group default qlen 1000
```
Reported-by: Huijing Hei <hhei@redhat.com>
Signed-off-by: Coiby Xu <coxu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
This reverts commit 5633e83318.
vm.zone_reclaim_mode may cause trashing on some machines. And after
second thought, vm.zone_reclaim_mode is barely helpful for machines
with high mem stress, so just revert it.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
