Dracut has switch network-legacy to network-manager by default, which makes
vlan on team easy. So it can be enabled.
Testing network topology with two VMs.
VM1
ens2-\ /----> VLAN8 (192.168.120.50)
---> team0
ens3-/ (192.168.122.10)
VM2
ens2-\ /----> VLAN8 (192.168.120.100)
---> team0
ens3-/ (192.168.122.20)
Both of ens2/ens3 in VM1/VM2 are connected to virbr0.
During test, dump target is set as root@192.168.120.100:/var/crash
then crashing in VM1
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Lianbo Jiang <lijiang@redhat.com>
Let's add the rd.kdumploglvl option to control log level in the second
kernel, which can make us avoid rebuilding the kdump initramfs after we
change the log level in /etc/sysconfig/kdump.
Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
depend() in module-setup.sh is a better place to setup dracut module
dependency, it will do early check, and fail early if needed module is
missing. Also remove a unneeded helper add_dracut_module.
Also remove the unnecessary return in depend() function.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Lianbo Jiang <lijiang@redhat.com>
Previously journalctl logs are directly dropped to save memory, but this
make journalctl unusable in kdump kernel and diffcult to debug. So
instead just don't let it read kmsg but keep other logs stored as volatile.
Kernel message are already stored in the kernel log ring buffer,
no need to let journalctl make a copy, especially when in kdump
kernel, ususlly there won't be too much kernel log overlapping
the old ring buffer.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Lianbo Jiang <lijiang@redhat.com>
Let's use the logger in the second kernel and collect the kernel ring
buffer(dmesg) of the second kernel.
Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
If failure action is set to "shell", user will need more debug info
available in kdump kernel. Especially when serial console is not
available, manually retrieve the log from journalctl is very useful
for debugging kdump issue.
Else, we can still drop journalctl content to save memory assuming
nothing will use it.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Lianbo Jiang <lijiang@redhat.com>
Even if the directory "/etc/kdump/pre.d/" is empty, the following false
alarm can be observed during building kdump.initrd:
"/etc/kdump/pre.d/* is not executable"
Suppress it.
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
Dracut get_persistent_dev function don't recognize UUID= or LABEL=
format, so caller should conver it to the path to the block device
before calling it. There is already such a helper
"kdump_get_persistent_dev", just move it to kdump-lib.sh and rename
it to reuse it,
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
This patch installs the binary and script files under /etc/kdump/{pre.d,post.d}
into new initramfs of kdump.
Signed-off-by: Shinichi Onitsuka <onitsuka.shinic@fujitsu.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
Switch root is never used for kdump image, and this will be helpful to
reduce the initramfs size.
Also increase dracut dependency version and the function is
dracut_no_switch_root is new introduced.
This commit is applied to RHEL some time ago, but missing in Fedora as
Fedora's Dracut didn't backport this feature at that time. Now apply
this missing commit.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
Upstream dracut now use network-manager module by default and since
upstream commit 3dcaa97, network-manager expects user to pass "rd.neednet"
to indicate network is required.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
This reverts commit cee618593c.
Upstream dracut have provided a parameter for adding mandantory network
requirement by appending "rd.neednet" parameter, so we should use that
instead.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
In most cases, it always provides a persistent MAC address. But for
the s390 Arch, sometimes, kernel could run in the LPAR mode and it
doesn't provide a persistent MAC address, which caused the kdump
failure.
Currently, some rules rely on the persistent MAC address, for the
above case, which won't work in kdump kernel because non-persistent
MAC could not match with udev rules.
To fix this issue, need to add a new option 'rd.znet_ifname' in order
to provide extra parameters such as 'ifname' and 'subchannels' for
some rules, which ensures kdump can also work appropriately without
the persistent MAC. Please refer to the following commit in dracut:
872eb69936bd ("95znet: Add a rd.znet_ifname= option")
Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
pcs cluster and cluster cib-upgrade may throw some information and disturb
the parsing. Mute them
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
There is a race issue between "pcs" and "kdumpctl restart"
-1. set up cluster
# pcs cluster setup --start mycluster node1 node2
# pcs stonith create kdump fence_kdump pcmk_reboot_action="off"
# pcs stonith level add 1 node1 kdump
# pcs stonith level add 1 node2 kdump
-2. Then here comes the command _immediately_ in kdumpctl
# pcs cluster cib
But due to some pcs internal mechanism, "pcs cluster cib" can not
fetch the updated info in time.
Fix these issue by forcing the upgrade of cib.
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
If user configured target is used, path should be used as the absolute
path within the dump target direct, and user should be fully aware of
the path structure within the target device. The adjust_bind_mount_path
call here make it very hard to control the behavior.
Especially, if it's a cross device bind mount, this will likely create a
invalid path in the target. And for atomic case, adjust_bind_mount_path call
here assumes user will always pass root device as the explicitly configured
dump target, which is not true.
If user configured target device is used, the path is always be the
absolute path inside of given target. If user don't know about the path
structure in the target device, then user should either use the path
based config, or carefully exam the target device before using it as a
dump target.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Lianbo Jiang <lijiang@redhat.com>
This commit remove almost all special workaround for atomic, and treat
all bind mounts in any environment equally.
Use a helper get_bind_mount_directory_from_path to get the bind mount
source path of given path.
is_atomic function now only used to determine the right /boot path
for atomic/silverblue environment.
And remove get_mntpoint_from_path(), it's the only function that never
ignore bind mount, and it have no caller after this clean up.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Lianbo Jiang <lijiang@redhat.com>
In /etc/hosts, the alias name can come at the 2nd column, regardless of the
recommendation.
E.g. the following format is valid although not recommended
cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.22.21 fastvm-rhel-7-6-21 fastvm-rhel-7-6-21.localdomain
192.168.22.22 fastvm-rhel-7-6-22 fastvm-rhel-7-6-22.localdomain
192.168.22.21 node1_hb
192.168.22.22 node2_hb
So filtering out both 2nd and 3rd column for matching.
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
Process substitution is not POSIX standard syntax, so if bash is configured
to strictly follow POSIC, this will fail.
Just use a POSIX friendly syntax instead.
Fixes: bz1708321
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Lianbo Jiang <lijiang@redhat.com>
This help remove redundant spaces and tailing comment in installed
kdump.conf, currently installed kdump.conf always contain extra empty
lines.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Lianbo Jiang <lijiang@redhat.com>
Latest dracut release stopped creating
$systemdsystemunitdir/initrd.target.wants dir for us, so ensure it
exists before creating the symlink.
Signed-off-by: Kairui Song <kasong@redhat.com>
Tested-and-Reviewed-by: Bhupesh Sharma <bhsharma@redhat.com>
Currently while trying to save vmcore via vlan eth interface, the Kdump
kernel fails with network unreachable message.
This is because mkdumprd produces a vlan config that does not get
ip address for vlan on eth device.
Fix the same via this patch.
Signed-off-by: Bhupesh Sharma <bhsharma@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
The dracut initqueue may quit immediately and won't trigger any hook if
there is no "finished" hook still pending (finished hook will be deleted
once it return 0).
This issue start to appear with latest dracut, latest dracut use
network-manager to configure the network,
network-manager module only install "settled" hook, and we didn't
install any other hook. So NFS/SSH dump will fail. iSCSI dump works
because dracut iscsi module will install a "finished" hook to detect if
the iscsi target is up.
So for NFS/SSH we keep initqueue running until the host successfully get
a valid IP address, which means the network is ready.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
sed and awk is heavily used everywhere in the code, but it's not
explicitely installed by kdump dracut module. If the module in dracut
stop installing them (which already happened with latest dracut
upstream), kdump will break.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
For ssh/nfs dump, kdump need the 'ip' tool to get the host ip address
for naming the vmcore. But kdump-module-setup.sh never installed this
tool. kdump-module-setup.sh worked so far as dracut network module will
help install it.
After dracut changed to use 35network-manager for network setup, "ip"
command won't be installed in second kernel by default. So need to
ensure "ip" is installed when installing kdump dracut module.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
This help deduplicate the code. Use a single function instead of
repeat the same logic.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
By default kernel have vm.zone_reclaim_mode = 0 and large page
allocation might fail as kernel is very conservative on memory
reclaiming. If the page allocation failure is not handled carefully
it could lead to more serious problems.
This issue can be reproduced by change with following steps:
- Fill up page cache use:
# dd if=/dev/urandom of=/test bs=1M count=1300
- Now the memory is filled with write cache:
# free -m
total used free shared buff/cache available
Mem: 1790 184 132 2 1473 1348
Swap: 2119 7 2112
- Insert a module which simply calls "kmalloc(SZ_1M, GFP_KERNEL)" for
512 times: (Notice: vmalloc don't have such problem)
# insmod debug_module.ko
- Got following allocation failure:
insmod: page allocation failure: order:8, mode:0x40cc0(GFP_KERNEL|__GFP_COMP), nodemask=(null),cpuset=/,mems_allowed=0
- Clean up and repeat again with vm.zone_reclaim_mode = 3, OOM is not
observed.
In kdump kernel there is usually only one online CPU and limited memory,
so we set vm.zone_reclaim_mode = 3 to let kernel reclaim memory more
aggresively to avoid such issue.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
Merge kdump_setup_netdev into kdump_install_net.
kdump_install_net is a wrapper of calling kdump_setup_netdev, and
it do following three extra things:
1. Sanitize and resolve the hostname
2. Resolve the route to the destination
3. Set the default gateway for once
There is currently only one caller of kdump_setup_netdev, the iscsi
network setup code, and it's doing 1 and 2 by itself. And there should
only be one default gateway in kdump enviroment, so applying 3 here is
fine.
And the comment of kdump_install_net is wrong and obsoleted, update the
comment too.
Just merge kdump_setup_netdev into kdump_install_net and always use
kdump_install_net instead.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
In commit a431a7e354 (module-setup: fix 99kdumpbase network dependency),
the statement for OR operation is still wrong.
The OR condition statement should be: if a || b
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
Bond options in ifcfg is space separated, dracut expected it to be comma
separated, so it have to be parsed and converted during initramfs
building.
The currently parsing and convert pattern is flawed, for example:
" downdelay=0 miimon=100 mode=802.3ad updelay=0 "
is converted to :
":,downdelay=0 miimon=100 mode=802.3ad updelay=0 "
should be:
":downdelay=0,miimon=100,mode=802.3ad,updelay=0"
So fix this issue by using more simple but robust method for processing
the options.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
The localhost is filtered out in case of is_pcs_fence_kdump, do it too in
case of is_generic_fence_kdump.
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
'hostname -A' can not get the alias, meanwhile 'hostname -a' is deprecated.
So we should do it by ourselves.
The parsing is based on the format of /etc/hosts, i.e.
IP_address canonical_hostname [aliases...]
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
fadump will alter the normal boot initramfs and we don't want a normal
boot to foward and drop the journalctl logs.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
The current code only exclude the hostname, while localhost can have alias in
/etc/hosts. All of the alias should be excluded from the fence dump node to
avoid deadlock issue.
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
Squash module is used to save memory. For fadump this is not neccessary
and may slow down the build time, and make it more fragile.
fadump initramfs is used for normal boot as well, although squash module
is capable of being used for generic normal boot, but there are cases
where is doesn't work well. So disable it and make fadump more robust.
Signed-off-by: Kairui Song <kasong@redhat.com>
Tested-by: Hari Bathini <hbathini@linux.ibm.com>
Acked-by: Dave Young <dyoung@redhat.com>
Don't use any log storage and forward to console directly, this make
console output more useful, and also save more memory. On a fresh
installed Fedora 30 it saved ~5M of memory, and the amount of log being
printed to console is still accetable.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
When reading kdump configs, a single parsing should be enough and this
saves a lot of duplicated striping call which speed up the total load
speed.
Speed up about 2 second when building and 0.1 second for reload in my
tests.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
When someone is using a minimal kernel without squash module installed,
including squash dracut module will either either fail to build or fail to
boot the initramfs.
As kdump always build the image for one single kernel, we can safely just
use modprobe to check if a modules is already built in, or it exists and
loadable for the kernel we are using for kdump image, and don't include
the squash module if they are missing. Everything will still work just
fine without squash module.
We do the check in kdump dracut modules not in squash dracut module
because kdump dracut module could leverage of the KDUMP_KERNELVER variable
to know which kernel it should check against, squash dracut module may be
used to build for a generic image.
And we only check for the kernel module dependency, other binary
dependencies are either well checked or well declared in dracut.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Currently we still don't support multipath route, when parsing multipath
route kdumpctl will wrongly consider 'nexthop' as the destination address,
and raise errors in second kernel.
When multipath route is in use, ip route output should be like this:
$ /sbin/ip route show
default via 192.168.122.1 dev ens1 proto dhcp metric 100
192.168.122.0/24 dev ens1 proto kernel scope link src 192.168.122.161 metric 100
192.168.122.8
nexthop via 192.168.122.1 dev ens1 weight 50
nexthop via 192.168.122.2 dev ens1 weight 5
As we don't care about HA/performance, simply use the rule with highest
weight and ignore the rest.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
In dracut-049, a new squash module is introduced, it can reduce the
memory usage of kdump initramfs in the capture kernel, this helps a lot
on lowering the risk of OOM failure.
Tested with latest rawhide with NFS, SSH and local dump.
Signed-off-by: Kairui Song <kasong@redhat.com>
We test if to include the drm module or not by testing if there are any
drm entry in sysfs. But there is an exception for hyper-v, DRM module
take care of hyperv's framebuffer driver as well but hyperv_fb will
not create any drm entry. So currently we got black screen on
hyperv guest.
Fix by detect hyperv's special entry as well.
Signed-off-by: Kairui Song <kasong@redhat.com>
This commit basically reverts commit c755499fad,
and make use of new introduced tri-state hostonly mode.
Following dracut commits merged multipath-hostonly into multipath
module, and introduced a tri-state hostonly mode.
commit 35e86ac117acbfd699f371f163cdda9db0ebc047
Author: Kairui Song <kasong@redhat.com>
Date: Thu Jul 5 16:20:04 2018 +0800
Merge 90-multipath-hostonly and 90-multipath
commit a695250ec7db21359689e50733c6581a8d211215
Author: Kairui Song <kasong@redhat.com>
Date: Wed Jul 4 17:21:37 2018 +0800
Introduce tri-state hostonly mode
multipath-hostonly module was introduced only for kdump, because kdump
need a more strict hostonly policy for multipath device to save memory.
Now multipath module will provide the behave we wanted by setting
hostonly mode to strict.
Kdump always use _proto=dhcp for both ipv4 and ipv6. But for ipv6
the dhcp address assignment is not like ipv4, there are different ways
for it, stateless and stateful, see below document:
https://fedoraproject.org/wiki/IPv6Guide
In case stateless, kernel can do the address assignment, dracut use
_proto=auto6; for stateful case, dracut use _proto=dhcp6.
But it is hard to decide whether stateless or stateful takes effect,
hence, dracut introduces ip=either6 option, which can try both of these
method automatically for us. For detail, refer to dracut:
commit 67354ee 40network: introduce ip=either6 option
We do not see bug reports before because for the most auto6 cases
kernel assign ip address before dhclient, kdump just happened to work.
Signed-off-by: Pingfan Liu <piliu@redhat.com>
When using fence_kdump, module-setup will create a kdump.conf with
fence_kdump_nodes. The node name comes from the cluster xml, which may
use the hostname alias. Later in kdump stage, "fence_kdump_send alias_1
alias_2" sends out notification to peers. Hence it requires /etc/hosts
and nsswitch.conf to make alias work.
Signed-off-by: Pingfan Liu <piliu@redhat.com>
This reverts commit 2f4149f276.
It is not proved to be right to get auto6 or dhcpv6 in 1st kernel,
pingfan is working on a dracut fix to do some fallback in 2nd kernel initramfs.
So revert this commit
Kdump always use _proto=dhcp for both ipv4 and ipv6. But for ipv6
the dhcp address assignment is not like ipv4, there are different ways
for it, stateless and stateful, see below document:
https://fedoraproject.org/wiki/IPv6Guide
In case stateless, kernel can do the address assignment, dracut use
_proto=auto6; for stateful case, dracut use _proto=dhcp6.
We do not see bug reports before because for the most auto6 cases
kernel assign ip address before dhclient, kdump just happened to work.
Here we use auto6 if possible first. And we take the assumption that
host use auto6 if /proc/sys/net/ipv6/conf/$netdev/autoconf is enabled
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Due to the following commit in dracut, which splits out hostonly modules
commit 5ce7cc7337a4c769b223152c083914f2052aa348
Author: Harald Hoyer <harald@redhat.com>
Date: Mon Jul 10 13:28:40 2017 +0200
add 90multipath-hostonly module
hardcoding the wwid of the drives in the initramfs causes problems
when the drives are cloned to a system with the same hardware, but
different disk wwid's
https://bugzilla.redhat.com/show_bug.cgi?id=1457311
So kdump should decide whether to include the hostonly module.
The multipath-hostonly can help kdump to include only the needed mpath device,
in order to use less memory by 2nd kernel.
---- The performance -----
before this patch
[root@localhost ~]# time kdumpctl start
Detected change(s) in the following file(s):
/etc/kdump.conf
Rebuilding /boot/initramfs-4.13.9-300.fc27.x86_64kdump.img
kexec: loaded kdump kernel
Starting kdump: [OK]
real 0m12.485s
user 0m10.096s
sys 0m1.887s
after this patch
root@localhost ~]# time kdumpctl start
Detected change(s) in the following file(s):
/etc/kdump.conf
Rebuilding /boot/initramfs-4.13.9-300.fc27.x86_64kdump.img
kexec: loaded kdump kernel
Starting kdump: [OK]
real 0m15.839s
user 0m13.015s
sys 0m1.853s
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
