This commit basically reverts commit c755499fad,
and make use of new introduced tri-state hostonly mode.
Following dracut commits merged multipath-hostonly into multipath
module, and introduced a tri-state hostonly mode.
commit 35e86ac117acbfd699f371f163cdda9db0ebc047
Author: Kairui Song <kasong@redhat.com>
Date: Thu Jul 5 16:20:04 2018 +0800
Merge 90-multipath-hostonly and 90-multipath
commit a695250ec7db21359689e50733c6581a8d211215
Author: Kairui Song <kasong@redhat.com>
Date: Wed Jul 4 17:21:37 2018 +0800
Introduce tri-state hostonly mode
multipath-hostonly module was introduced only for kdump, because kdump
need a more strict hostonly policy for multipath device to save memory.
Now multipath module will provide the behave we wanted by setting
hostonly mode to strict.
Kdump always use _proto=dhcp for both ipv4 and ipv6. But for ipv6
the dhcp address assignment is not like ipv4, there are different ways
for it, stateless and stateful, see below document:
https://fedoraproject.org/wiki/IPv6Guide
In case stateless, kernel can do the address assignment, dracut use
_proto=auto6; for stateful case, dracut use _proto=dhcp6.
But it is hard to decide whether stateless or stateful takes effect,
hence, dracut introduces ip=either6 option, which can try both of these
method automatically for us. For detail, refer to dracut:
commit 67354ee 40network: introduce ip=either6 option
We do not see bug reports before because for the most auto6 cases
kernel assign ip address before dhclient, kdump just happened to work.
Signed-off-by: Pingfan Liu <piliu@redhat.com>
When using fence_kdump, module-setup will create a kdump.conf with
fence_kdump_nodes. The node name comes from the cluster xml, which may
use the hostname alias. Later in kdump stage, "fence_kdump_send alias_1
alias_2" sends out notification to peers. Hence it requires /etc/hosts
and nsswitch.conf to make alias work.
Signed-off-by: Pingfan Liu <piliu@redhat.com>
This reverts commit 2f4149f276.
It is not proved to be right to get auto6 or dhcpv6 in 1st kernel,
pingfan is working on a dracut fix to do some fallback in 2nd kernel initramfs.
So revert this commit
Kdump always use _proto=dhcp for both ipv4 and ipv6. But for ipv6
the dhcp address assignment is not like ipv4, there are different ways
for it, stateless and stateful, see below document:
https://fedoraproject.org/wiki/IPv6Guide
In case stateless, kernel can do the address assignment, dracut use
_proto=auto6; for stateful case, dracut use _proto=dhcp6.
We do not see bug reports before because for the most auto6 cases
kernel assign ip address before dhclient, kdump just happened to work.
Here we use auto6 if possible first. And we take the assumption that
host use auto6 if /proc/sys/net/ipv6/conf/$netdev/autoconf is enabled
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Due to the following commit in dracut, which splits out hostonly modules
commit 5ce7cc7337a4c769b223152c083914f2052aa348
Author: Harald Hoyer <harald@redhat.com>
Date: Mon Jul 10 13:28:40 2017 +0200
add 90multipath-hostonly module
hardcoding the wwid of the drives in the initramfs causes problems
when the drives are cloned to a system with the same hardware, but
different disk wwid's
https://bugzilla.redhat.com/show_bug.cgi?id=1457311
So kdump should decide whether to include the hostonly module.
The multipath-hostonly can help kdump to include only the needed mpath device,
in order to use less memory by 2nd kernel.
---- The performance -----
before this patch
[root@localhost ~]# time kdumpctl start
Detected change(s) in the following file(s):
/etc/kdump.conf
Rebuilding /boot/initramfs-4.13.9-300.fc27.x86_64kdump.img
kexec: loaded kdump kernel
Starting kdump: [OK]
real 0m12.485s
user 0m10.096s
sys 0m1.887s
after this patch
root@localhost ~]# time kdumpctl start
Detected change(s) in the following file(s):
/etc/kdump.conf
Rebuilding /boot/initramfs-4.13.9-300.fc27.x86_64kdump.img
kexec: loaded kdump kernel
Starting kdump: [OK]
real 0m15.839s
user 0m13.015s
sys 0m1.853s
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
This commit eliminates a redundant kdump_get_mac_addr call in
kdump_setup_netdev.
Signed-off-by: Ziyue Yang <ziyang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
After adding "--hostonly-cmdline", besides 99kdump, 95iscsi
also generates iscsi related cmdline. IOW, we have duplicate
software iscsi cmdlines.
95iscsi generated software iscsi cmdline doesn't work, so we
remove that of 95iscsi and use that of 99kdump which has been
well tested.
We can change to use 95iscsi when possible in the future.
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Currently, we throw the error message at the very beginning, as
a result on a pure-hardware(all-offload) iscsi machine with many
iscsi partitions, we suffered from too much noise as follows:
iscsiadm: No records found
Unable to find iscsi record for
/sys/devices/pci0000:00/0000:00:06.0/0000:07:00.0/0000:08:01.1/host0/session1
iscsiadm: No records found
Unable to find iscsi record for
/sys/devices/pci0000:00/0000:00:06.0/0000:07:00.0/0000:08:01.3/host1/session2
iscsiadm: No records found
Unable to find iscsi record for
/sys/devices/pci0000:00/0000:00:06.0/0000:07:00.0/0000:08:01.1/host0/session1
iscsiadm: No records found
Unable to find iscsi record for
/sys/devices/pci0000:00/0000:00:06.0/0000:07:00.0/0000:08:01.3/host1/session2
iscsiadm: No records found
Unable to find iscsi record for
/sys/devices/pci0000:00/0000:00:06.0/0000:07:00.0/0000:08:01.1/host0/session1
iscsiadm: No records found
Unable to find iscsi record for
/sys/devices/pci0000:00/0000:00:06.0/0000:07:00.0/0000:08:01.3/host1/session2
iscsiadm: No records found
Unable to find iscsi record for
/sys/devices/pci0000:00/0000:00:06.0/0000:07:00.0/0000:08:01.1/host0/session1
iscsiadm: No records found
Unable to find iscsi record for
/sys/devices/pci0000:00/0000:00:06.0/0000:07:00.0/0000:08:01.3/host1/session2
iscsiadm: No records found
Unable to find iscsi record for
/sys/devices/pci0000:00/0000:00:06.0/0000:07:00.0/0000:08:01.1/host0/session1
iscsiadm: No records found
Unable to find iscsi record for
/sys/devices/pci0000:00/0000:00:06.0/0000:07:00.0/0000:08:01.3/host1/session2
kexec: loaded kdump kernel
Starting kdump: [OK]
There's no need to know the very early error messages, we can
remove the error output which is actually normal for the pure
hardware iscsi.
As for unexpected errors, we kept the error outputs in the
succeeding kdump_iscsi_get_rec_val() calls by not appending
"2>/dev/null".
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Currently, systemd uses 90s as the default mount unit timeout
(see "man 5 systemd-system.conf " for "DefaultTimeoutStartSec"),
in some cases, although it works well in 1st kernel, it's not
enough under kdump and results in mount timeout, further results
in kdump dumping failure.
We've met several such issues, we decided to enlarge this default
value a little for kdump.
We know that dracut has a default initqueue timeout value of 180s
("rd.retry"), we finalized a little larger value 300s as kdump's
default timeout if there is no explicit "DefaultTimeoutStartSec=X,
specified by users.
"DefaultTimeoutStartSec=X" can be overridden by individual mount
option "x-systemd.device-timeout=X", users can specify their own
values as needed.
This patch achieves the purpose by creating a dedicated conf file
"/etc/systemd/system.conf.d/kdump.conf" which has the content of
"DefaultTimeoutStartSec=300s", this is based on the fact that all
the conf files will be parsed by systemd and the last parsed one
will be used if there are duplicate definitions.
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
I noticed that network is still enabled for local dumping,
like the following kdump boot message on my test machine
using local disk as the dump target:
tg3.c:v3.137 (May 11, 2014)
tg3 0000:02:00.0 eth2: Tigon3 [partno(BCM95720) rev
(PCI Express) MAC address c8:1f:66:c9:35:0d
tg3 0000:02:00.0 eth2: attached PHY is 5720C
After some debugging, found it due to a misuse in code below:
if [ is_generic_fence_kdump -o is_pcs_fence_kdump ]; then
_dep="$_dep network"
fi
The "if" condition always results in "true", and should be
changed as follows:
if is_generic_fence_kdump -o is_pcs_fence_kdump; then
_dep="$_dep network"
fi
After this, network won't be involved in non-network dumping,
as for dumpings require network such as nfs/ssh/iscsi/fcoe/etc,
dracut will add network accordingly. And kdump initramfs size
can be reduced from 24MB to 17MB tested on some real hardware,
and from 19MB to 14MB on my kvm. Moreover, it could avoid the
network (driver) initialization thereby saving us more memory.
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
The /sys/modules/*/drivers sysfs entries do not exist anymore on newer
kernels which means that the DRM moduels would never be included.
Instead check if there is any device with a "drm" sysfs directory to
decide on whether DRM modules need to be included.
Acked-by: Dave Young <dyoung@redhat.com>
Since the current dracut of Fedora already supports not always
mounting root device, we can remove "root=X" from the command
line directly, and always get the dump target specified in
"/etc/kdump.conf" and mount it. If the dump target is located
at root filesystem, we will add the root mount info explicitly
from kdump side instead of from dracut side.
For example, in case of nfs/ssh/usb/raw/etc(non-root) dumping,
kdump will not mount the unnecessary root fs after this change.
This patch removes "root=X" via the "KDUMP_COMMANDLINE_REMOVE"
(if "default dump_to_rootfs" is specified, don't remove "root=X"),
and mounts non-root target under "/kdumproot", the root target
still under "/sysroot"(to be align with systemd sysroot.mount).
After removing "root=X", we now add root fs mount information
explicitly from the kdump side.
Changed check_dump_fs_modified() a little to avoid rebuild when
dump target is root, since we add root fs mount explicitly now.
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Pratyush Anand <panand@redhat.com>
Acked-by:Dave Young <dyoung@redhat.com>
We met a problem that the kdump emergency service failed to
start when the target dump timeout(we passed "rd.timeout=30"
to kdump), it reported "Transaction is destructive" messages:
[ TIME ] Timed out waiting for device dev-mapper-fedora\x2droot.device.
[DEPEND] Dependency failed for Initrd Root Device.
[ SKIP ] Ordering cycle found, skipping System Initialization
[DEPEND] Dependency failed for /sysroot.
[DEPEND] Dependency failed for Initrd Root File System.
[DEPEND] Dependency failed for Reload Configuration from the Real Root.
[ SKIP ] Ordering cycle found, skipping System Initialization
[ SKIP ] Ordering cycle found, skipping Initrd Default Target
[DEPEND] Dependency failed for File System Check on /dev/mapper/fedora-root.
[ OK ] Reached target Initrd File Systems.
[ OK ] Stopped dracut pre-udev hook.
[ OK ] Stopped dracut cmdline hook.
Starting Setup Virtual Console...
Starting Kdump Emergency...
[ OK ] Reached target Initrd Default Target.
[ OK ] Stopped dracut initqueue hook.
Failed to start kdump-error-handler.service: Transaction is destructive.
See system logs and 'systemctl status kdump-error-handler.service' for details.
[FAILED] Failed to start Kdump Emergency.
See 'systemctl status emergency.service' for details.
[DEPEND] Dependency failed for Emergency Mode.
This is because in case of root failure, initrd-root-fs.target
will trigger systemd emergency target which requires the systemd
emergency service actually is kdump-emergency.service, then our
kdump-emergency.service starts kdump-error-handler.service with
"systemctl isolate"(see 99kdumpbase/kdump-emergency.service, we
replace systemd's with this one under kdump).
This will lead to systemd two contradictable jobs queued as an
atomic transaction:
job 1) the emergency service gets started by initrd-root-fs.target
job 2) the emergency service gets stopped due to "systemctl isolate"
thereby throwing "Transaction is destructive".
In order to solve it, we can utilize "IgnoreOnIsolate=yes" for both
kdump-emergency.service and kdump-emergency.target. Unit with attribute
"IgnoreOnIsolate=yes" won't be stopped when isolating another unit,
they can keep going as expected in case be triggered by any failure.
We add kdump-emergency.target dedicated to kdump the similar way
as did for kdump-emergency.service(i.e. will replace systemd's
emergency.target with kdump-emergency.target under kdump), and
adds "IgnoreOnIsolate=yes" into both of them.
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Acked-by: Pratyush Anand <panand@redhat.com>
[bhe: improve the patch log about IgnoreOnIsolate="]
We replace "reserved_memory = XXXX"(default value is 8192) with
"reserved_memory = 1024" in /etc/lvm/lvm.conf used by "lvm2", it
can save 7MB peak memory consumption, so lower the possibility of
OOM under kdump.
For kdump, we don't have too many lvm targets, lvm2 locates in the
RAM(rootfs), so don't need that much memory, as discussed with lvm
people, they agreed that we use 1MB under kdump as long as there
are not that many lvm targets invloved.
We modify /etc/lvm/lvm.conf when "99kdumpbase" install() is executed,
because it is parsed after "90lvm" by dracut.
We add the code unconditionally with &>/dev/null to ignore errors, it
doesn't matter in case of "lvm" not included(i.e. there is no lvm.conf).
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
kdump_to_udev_name function name cause confusion to people, change it to
kdump_get_persistent_dev which sounds better.
Signed-off-by: Dave Young <dyoung@redhat.com>
Reviewed-by: Xunlei Pang <xlpang@redhat.com>
Although we use by-id in mkdumprd as persistent policy for the dump target
checking, finally it is not used in kdump 2nd kernel because we call dracut
function in module-setup.sh without persistent policy specified that means
kdump will copy default "by-uuid" dev name.
Though by-uuid usually works and it is still better to fix it as raw disk
uuid make no sense.
Also do not need to call bind mount adjust function for raw dump, here add
another switch case for raw dump and cleanup the functions with short
variable names to keep code shorter.
Signed-off-by: Dave Young <dyoung@redhat.com>
Reviewed-by: Xunlei Pang <xlpang@redhat.com>
When a remote dump target is specified, kdump dracut module prefixes
'kdump-' to network interface name (ifname) as kernel assigned names
are not persistent. In fadump mode, kdump dracut module is added to
the default initrd, which adds the 'kdump-' prefix to the ifname of
the prodcution kernel itself. If fadump mode is disabled after this,
kdump dracut module picks the ifname that is already prefixed with
'kdump-' in the production kernel and adds another 'kdump-' to it,
making the ifname something like kdump-kdump-eth0 for kdump kernel.
Eventually, kdump kernel fails with below traces:
dracut-initqueue[246]: RTNETLINK answers: Network is unreachable
dracut-initqueue[246]: arping: Device kdump-kdump-eth0 not available.
The ip command shows the below:
kdump:/# ip addr show kdump-kdump-eth0
2: kdump-kdump-eth: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 \
qdisc pfifo_fast state UNKNOWN qlen 1000
link/ether 22:82:87:7b:98:02 brd ff:ff:ff:ff:ff:ff
inet6 2002:903:15f:550:2082:87ff:fe7b:9802/64 scope global \
mngtmpaddr dynamic
valid_lft 2591890sec preferred_lft 604690sec
inet6 fe80::2082:87ff:fe7b:9802/64 scope link
valid_lft forever preferred_lft forever
kdump:/#
The trailing 0 from kdump-kdump-eth0 is missing in the ifname, probably
truncated owing to ifname length limit, while setting.
This patch fixes this by avoiding addition of the prefix 'kdump-' when
such prefix is already present in the ifname.
Signed-off-by: Hari Bathini <hbathini@linux.vnet.ibm.com>
Acked-by: Dave Young <dyoung@redhat.com>
There are some complaints about nfs kdump that users must mount
nfs beforehand, which may cause some overhead to nfs server.
For example, there're thounsands of diskless clients deployed with
nfs dumping, each time the client is boot up, it will trigger
kdump rebuilding so will mount nfs, thus resulting in thousands
of nfs request concurrently imposed on the same nfs server.
We introduce a new way of specifying mount information via the
already-existent "dracut_args" directive(so avoid adding extra
directives in /etc/kdump.conf), we will skip all the filesystem
mounting and checking stuff for it. So it can be used in the
above-mentioned nfs scenario to avoid severe nfs server overhead.
Specifically, if there is any "--mount" information specified via
"dracut_args" in /etc/kdump.conf, always use it as the final mount
without any validation(mounting or checking like mount options,
fs size, etc), so users are expected to ensure its correctness.
NOTE:
-Only one mount target is allowed using "dracut_args" globally.
-Dracut will create <mountpoint> if it doesn't exist in kdump kernel,
<mountpoint> must be specified as an absolute path.
-Users should do a test first and ensure it works because kdump does
not prepare the mount or check all the validity.
Reviewed-by: Pratyush Anand <panand@redhat.com>
Suggested-by: Dave Young <dyoung@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Now dracut takes care to add module for active watchdog. Therefore we do
not need to pass iTCO_wdt and lpc_ich module in rd.driver.pre specifically
here.
Signed-off-by: Pratyush Anand <panand@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
There are several kinds of iSCSI mode rhel support currently.
- Pure hardware iSCSI
- iBFT iSCSI
- Pure software iSCSI
Except for the 1st one that firmware takes care of everything to
make it behave like a local disk, both iBFT and pure software iSCSI
mode need pass information to kdump kernel for configuring them
correctly.
Currently kdump takes iBFT mode as a software iSCSI and collects
the related information to set up software iSCSI in 2nd kernel,
though dracut can detect and collect information to set up iBFT
iSCSI of 2nd kernel. This brings up 2 problems:
1) Redundent information about the related iSCSI is collected. One
is done by kdump, the other is from dracut.
2) These 2 sessions of 2nd kernel for a certain session of 1st kernel
could contain two "ip=xxx" cmdline option. This will cause cmdline
handling error in dracut.
The 1st one is not critical while the 2nd is. In order to avoid above
2 problems, kdump need detect iBFT mode iSCSI and leave it to dracut.
This is what is done in this patch.
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
The ifcfg file name of <netif> under "/etc/sysconfig/network-scripts/"
may not be "ifcfg-<netif>". For example, for "enp0s25" we are able to
generate its ifcfg like "/etc/sysconfig/network-scripts/ifcfg-enp0s25test"
via network-manager. If we alway assume "ifcfg-<netif>" is there, we will
got the wrong result in some cases.
The issue can be resolved by using the new get_ifcfg_filename() introduced
by PATCH "kdump-lib: Add get_ifcfg_filename() to get the proper ifcfg file",
so we hereby change all the "ifcfg-<netif>" users to use get_ifcfg_filename().
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
dracut will place the config in the random path during generating the
initramfs. Remove the duplicate prefix path ${initdir}.
Signed-off-by: Minfei Huang <mhuang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
The system may have multiple default route entry. Following is an
example to show the details.
# ip -6 route list dev eth0
2620:52:0:1040::/64 proto kernel metric 256 expires 2591978sec fe80::/64 proto kernel metric 256
default via fe80:52:0:1040::1 proto ra metric 1024 expires 1778sec hoplimit 64
default via fe80:52:0:1040::2 proto ra metric 1024 expires 1778sec hoplimit 64
Choose the first matched entry.
Signed-off-by: Minfei Huang <mhuang@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
Dracut will die in the situation that dracut detects to use dhcp to
setup ip address, but kdump passes the ip address to it.
In commit 7ea50dc7a3, we start to use
option permanent to get the ip address in kdump_static_ip. If the
network is setuped by static, we will get the ip address, otherwise
getting none.
In commit c994a80698 which it used to
support ipv6 protocol, I miss the option permanent.
This patch is not a fixing patch, just pulls back something to make
kdump work as original.
Signed-off-by: Minfei Huang <mhuang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Due to the different format between ipv4 and ipv6 protocol, quote the
ipv6 address with bracket "[]" to make dracut notify.
Signed-off-by: Minfei Huang <mhuang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Kdump will parse the hostname to get the ip address, if hostname is
specfied in /etc/kdump.conf. We will get the ip address(ipv4 or ipv6,
according to the DNS server) by using "getent hosts".
For now, it is more reasonable that we shall get all of the ip
address(including ipv4 and ipv6 address) which point to the hostname by
using "getent ahosts". And we will prefer to use the ipv4 address, if
both ipv4 and ipv6 address work.
The reason why we choose the ipv4 as preferred address is to solve the
issue kdump will fail to connect the hostname machine(parsed as ipv6
address), due to the DNS server is ipv4 address in 2nd kernel.
Signed-off-by: Minfei Huang <mhuang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Previously, Kdump will save route to setup the network route in the 2nd
kernel for ipv4 protocol. To support ipv6 protocol, make Kdump fetch
correct nexthop, since the ruturning format is different.
In order to enhance kdump to support ipv6, support the static ip for
ipv6 protocol, which ipv4 has supported already.
Introduce a new lib function get_remote_host which is used to factor out
the ip address(ipv4 or ipv6) and hostname in /etc/kdump.conf.
Introduce a new lib function is_ipv6_address which is used to make sure
whether the passed ip address is ipv4 or ipv6.
Introduce a new lib function is_hostname which is used to confirm
whether the passed parameter is hostname, not the ip address.
Introduce a new function get_ip_route_field which is used to factor out
the specified string in ip route info.
Due to the different format between ipv4 and ipv6 protocol, quote the
ipv6 address with bracket "[]" to make dracut notify.
Signed-off-by: Minfei Huang <mhuang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
For now, Kdump will use ipv4 address as dump directory, and it works, if
ipv4 is enabled.
Once Kdump start to support ipv6 protocol, we may only setup the ipv6
address exclusively. Modify the code to make Kdump work in either ipv4
and ipv6 protocol.
Signed-off-by: Minfei Huang <mhuang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Now Kdump will ingore the DNS config in /etc/resolv.conf, when it
generates the initram. And most users do not concern about this issue,
because they never use deployment tools to configure machines
environment, like puppet.
It is more convenient to add the DNS config to /etc/resolv.conf for
people who use deployment tools to configure machines concurrently.
Signed-off-by: Minfei Huang <mhuang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
We have added *wdt in kdump initramfs, but to improve it more we can do below
(1) load wdt drivers as early as possible so that we can save time before wdt timeout
some drivers like iTCO_wdt can stop the watchdog while driver initialization, so
it can give more chance for kdump.
It can save time especially in case some drivers take long time to init, like
some storage and networking cards.
(2) add only used wdt drivers in kdump initrd instead of add *wdt
wdt driver layer need a change so that we can get the proper driver name from
/dev/watchdog. Question to this is are we sure 1st kernel use /dev/watchdog
instead of /dev/watchdog1? It need more investigation.
(3) in case a driver can not stop (nowayout?) during module_init, we need load it
as early as possible and kick the watchdog. Likely we can use systemd default
watchdog functionality.
This patch is about to address (1), and specially for iTCO_wdt, we only tested
iTCO_wdt, thus in this patch only add this driver, need investigate on other drivers
later to see if other drivers works in this way.
Signed-off-by: Dave Young <dyoung@redhat.com>
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: Minfei Huang <mhuang@redhat.com>
The ipv6 patchset is still under review, previously the commit was mistakenly
merged, thus let's revert it.
Revert "dracut-kdump: Use proper the known hosts entry in the file known_hosts"
This reverts commit 63476302aa.
Conflicts:
kdump-lib.sh
Signed-off-by: Minfei Huang <mhuang@redhat.com>
Signed-off-by: Dave Young <dyoung@redhat.com>
Kdump will dump the vmcore in incorrect target directory, if the target
is bind mounted.
As commented in the previous patch, we can construct the real path in
Atomic, which contains two part, one bind mounted path, the other
specified dump path. Then replace the path as the real path in
/etc/kdump.conf.
findmnt can find the real path for nfs, although the path is in bind
mode. So nfs can work well with the path in bind mode.
Signed-off-by: Minfei Huang <mhuang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
Now kdump cannt parse the path correctly, if the path contains
duplicated "/". Following is an example to explain it detail. (the
directory /mnt is a mount point which is mounted a block device)
path //mnt/var/crash
Then the warning will raise.
Force rebuild /boot/initramfs-3.19.1kdump.img
Rebuilding /boot/initramfs-3.19.1kdump.img
df: ‘/mnt///mnt/var/crash’: No such file or directory
/sbin/mkdumprd: line 239: [: -lt: unary operator expected
kexec: loaded kdump kernel
Starting kdump: [OK]
For above case, kdump fails to check the fs size, due to the incorrect
path.
In kdump code flow, we will cut out the mount point(/mnt) from the
path(//mnt/var/crash). But the mount point cannt match the path, because
of the duplicated "/".
To fix it, we will strip the duplicated "/" firstly.
Signed-off-by: Minfei Huang <mhuang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
Harald warned it's dangerous to use /tmp/*$$* in shell scripts of dracut
modules.
Quote his saying as below:
***************************
This can be exploited so easily and used to overwrite e.g. /etc/shadow.
The only thing you have to do is waiting until the next time the kdump
initramfs is generated on a kernel update.
If at all, please use "$initdir/tmp/" because $initdir is a mktemp generated
directory with a non-guessable name!
**************************
So make a clean up in this patch.
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Steve found a bug. When mount a disk in /var and not specify path
in /etc/kdump.conf, the vmcore will be dumped into /var/crash of
that disk, but not /crash on that disk.
This is because when write the parsed path into /tmp/$$-kdump.conf
in default_dump_target_install_conf() of mkdumprd, it uses below
sed command. So if no path specified at all, this sed command won't
add it to /tmp/$$-kdump.conf. Then in 2nd kernel it will take default
path, namely "/var/crash" as path if no path in /etc/kdump.conf in
2nd kernel.
sed -i -e "s#$_save_path#$_path#" /tmp/$$-kdump.conf
According to Dave Young's suggestion, erase the old path line and then
insert the parsed path. This can fix it.
v2->v3:
erase the old path line and then insert the parsed path.
sed -i -e "s#^path[[:space:]]\+$_save_path##" /tmp/$$-kdump.conf
echo "path $_path" >> /tmp/$$-kdump.conf
v3->v4:
Change the sed pattern, erase lines starting with "path" and then
insert the parsed path.
sed -i -e "s#^path.*##" /tmp/$$-kdump.conf
echo "path $_path" >> /tmp/$$-kdump.conf
v4->v5:
Chaowang suggested using sed command d to remove the whole line
like below:
sed -i "/^path/d" /tmp/$$-kdump.conf
echo "path $_path" >> /tmp/$$-kdump.conf
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: WANG Chao <chaowang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Chao pointed out that it's better to use get_option_value to get
get a specific config_val.
And also there's a potential risk when use below sed command to
do the replacement.
sed -i -e "s#$_save_path#$_path#" /tmp/$$-kdump.conf
Say user configure kdump.conf like the following. Then sed may
replace "/var/crash/post.sh" with something else, depanding on
mount point.
kdump_post /var/crash/post.sh
path /var/crash
So in this patch clean them up.
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: WANG Chao <chaowang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Certain kernel parameters like min_free_kbytes can be configured at runtime
using sysctl. While this is useful in first kernel, it can lead to unnecessary
failures like OOM in kdump kernel. This patch enforces default vaules for all
sysctl parameters, in kdump kernel, by removing sysctl.conf & sysctl.d/* files.
Signed-off-by: Hari Bathini <hbathini@linux.vnet.ibm.com>
Acked-by: Dave Young <dyoung@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
Acked-by: WANG Chao <chaowang@redhat.com>
Once login using ssh, the ssh will store the known hosts entry to the
local ~/.ssh/known_hosts. From now, we can login using ssh automaticly.
The ssh will check the ~/ssh/.known_hosts entry, if set the option
StrictHostKeyChecking=yes/ask in the config or command line, when you
want to login the target. the default value of StrictHostKeyChecking is
ask.
And the kdump using the ssh will append the option
StrictHostKeyChecking=yes in the command line.
We can using following ip to connect peer machine, if enable the ipv6.
fe80::5054:ff:fe48:ca80%eth0
Obviously, above ip contains the ethX.
Kdump will add the prefix "kdump-" before ethX to avoid flowing
netdevice name in case netdevice names ethX in the 2nd kernel. So the
ip address will change to fe80::5054:ff:fe48:ca80%kdump-eth0.
Kdump will login the target manully in the 2nd kernel, because of the
option StrictHostKeyChecking=yes and inexistence known hosts entry
in the local ~/.ssh/known_hosts. Hence dumping core will fail.
In order to login automaticly using ssh, we should add the prefix
"kdump-" before ethX in the local ~/.ssh/known_hosts.
Signed-off-by: Minfei Huang <mhuang@redhat.com>
For ethX, it may fail to setup the network in the 2nd kernel due to the
mapping of ethernet device name and MAC changes.
The commit(ba7660f37e) has fixed this
issue by add the prefix "kdump-" before ethX. But the network will fail
to work in the static route mode because of this commit.
Here is the config which is used to setup the static route:
rd.route=192.168.201.215:192.168.200.137:eth1
Obviously, the static route config comtains the ethX. But the network
device names kdump-ethX in the 2nd kernel, so the static route config
will fail to execute. To fix it, we should identify the network device.
Add the prefix "kdump-" before the ethX in the static route config to
setup it successfully in the 2nd kernel.
Signed-off-by: Minfei Huang <mhuang@redhat.com>
Acked-by: WANG Chao <chaowang@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
It is boring that internal result is shown in the terminal. Do not print
anything to standard output by using the command "grep -q".
Signed-off-by: Minfei Huang <mhuang@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
Acked-by: WANG Chao <chaowang@redhat.com>
Previously for solving static route issues, all routes which go
through a specific dev will be saved in 1st kernel, and then added
in 2nd kernel. Because we use below search pattern, an exception
will happen:
/sbin/ip route show | grep -v default | grep "^[[:digit:]].*via.* $_netdev"
That exception is a corner case which happened when 2 machines connected
directly by cable and the 2 network interfaces are configured in
different network subnets. E.g there are 2 machines A and B:
A:ens10 < ------ > B:ens9
A:ens10 inet 192.168.100.111/24 scope global ens10
route need be added in A:
192.168.110.0/24 dev ens10
B:ens9 inet 192.168.110.222/24 scope global ens9
route need be added in B
192.168.100.0/24 dev ens9
Now if A want to dump to B, the route "192.168.110.0/24 dev ens10"
has to be saved and added in 2nd kernel.
So in this patch "ip route get to $target" command is executed, then
an exact route can be got for going to that target. By this, static
route works and the corner case can be fixed too.
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: Marc Milgram <mmilgram@redhat.com>
Acked-by: WANG Chao <chaowang@redhat.com>
In case of iscsi boot, kernel cmdline will contain ip=xxx kernel
parameter for dracut setting up iscsi root in initramfs. For example:
"root=xxx ip=192.168.3.26:::255.255.255.0:localhost.localdomain:eno19:none ..."
dracut doesn't allow duplicate ip conf for the same network card. dracut
will not ignore the either of the duplicate. Instead, it refuses to
continue:
[ 15.876306] dracut: FATAL: For argument 'ip=192.168.3.26:::255.255.255.0:localhost.localdomain:eno19:none'\n
Duplication configurations for 'eno19'
[ 16.055513] dracut: Refusing to continue ev argument for multiple ip= lines
That's why in our code we don't add a duplicate ip conf when handling
the same network card the second time. But we never consider the case
that ip conf is already added in kernel cmdline for some special
purpose, for example, iscsi boot.
Now we also look up /proc/cmdline for ip conf. If it exists, we use the
existing one. The existing one should work out of box because dracut
will handle it in second kernel like it does for first kernel. That
said, the network card will be brought up and root disk will be mounted
under /sysroot.
Signed-off-by: WANG Chao <chaowang@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
Node could be referenced by short hostname (hostname -s) in cluster
configuration:
[root@virt-068 /]# pcs status nodes
Pacemaker Nodes:
Online: virt-066 virt-067 virt-068
Standby:
Offline:
We didn't know it before. Martin noticed the kdump failure, and provide
this fix. Thanks to Martin.
Signed-off-by: WANG Chao <chaowang@redhat.com>
Tested-by: Martin Juricek <mjuricek@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
If one target address is not local and its route is different than
default gateway, the specific route to this target address need be
added. E.g, target is 192.168.200.222.
sh> ip route show
default via 192.168.122.1 dev eth0 proto static metric 1024
192.168.200.0/24 via 192.168.100.222 dev ens10 proto static metric 1
In this patch, get the route to the specific target address and store
it as cmdline, here is /etc/cmdline.d/45-route-static.conf. And the
route options are separated by semicolon like below. Then the stored
route can be parsed when kdump kernel boot up.
192.168.200.0/24:192.168.100.222:ens10
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
This patch introduce a new kdump-capture.service which is used to run
kdump.sh.
kdump-capture.service has OnFailure=emergency.target and
OnFailureIsolate=yes set. When kdump.sh fails, the kdump emergency
service will be triggered and enter the error handling path.
In 2nd kernel, the default target for systemd is initrd.target, so we
put kdump-capture.service in initrd.target.wants/ and by that, system
will start kdump-capture as part of the boot process.
kdump.sh used to run in dracut-pre-pivot hook. Now kdump-capture.service
is placed after dracut-pre-pivot.service and other dependencies are all
copied from dracut-pre-pivot.service. So the start point of
kdump.sh will be almost the same as it used to be.
Signed-off-by: WANG Chao <chaowang@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Now upon failure kdump script might not be called at all and it might
not be able to execute default action. It results in a hang.
Because we disable emergency shell and rely on kdump.sh being invoked
through dracut-pre-pivot hook. But it might happen that we never call
into dracut-pre-pivot hook because certain systemd targets could not
reach due to failure in their dependencies. In those cases error
handling code does not run and system hangs. For example:
sysroot-var-crash.mount --> initrd-root-fs.target --> initrd.target \
--> dracut-pre-pivot.service --> kdump.sh
If /sysroot/var/crash mount fails, initrd-root-fs.target will not be
reached. And then initrd.target will not be reached,
dracut-pre-pivot.service wouldn't run. Finally kdump.sh wouldn't run.
To solve this problem, we need to separate the error handling code from
dracut-pre-pivot hook, and every time when a failure shows up, the
separated code can be called by the emergency service.
By default systemd provides an emergency service which will drop us into
shell every time upon a critical failure. It's very convenient for us to
re-use the framework of systemd emergency, because we don't have to
touch the other parts of systemd. We can use our own script instead of
the default one.
This new scheme will overwrite emergency shell and replace with kdump
error handling code. And this code will do the error handling as needed.
Now, we will not rely on dracut-pre-pivot hook running always. Instead
whenever error happens and it is serious enough that emergency shell
needed to run, now kdump error handler will run.
dracut-emergency is also replaced by kdump error handler and it's
enabled again all the way down. So all the failure (including systemd
and dracut) in 2nd kernel could be captured, and trigger kdump error
handler.
dracut-initqueue is a special case, which calls "systemctl start
emergency" directly, not via "OnFailure=emergency". In case of failure,
emergency is started, but not in a isolation mode, which means
dracut-initqueue is still running. On the other hand, emergency will
call dracut-initqueue again when default action is dump_to_rootfs.
systemd would block on the last dracut-initqueue, waiting for the first
instance to exit, which leaves us hang. It looks like the following:
dracut-initqueue (running)
--> call dracut-emergency:
--> dracut-emergency (running)
--> kdump-error-handler.sh (running)
--> call dracut-initqueue:
--> blocking and waiting for the original instance to exit.
To fix this, I'd like to introduce a wrapper emergency service. This
emegency service will replace both the systemd and dracut emergency. And
this service does nothing but to isolate to real kdump error handler
service:
dracut-initqueue (running)
--> call dracut-emergency:
--> dracut-emergency isolate to kdump-error-handler.service
--> dracut-emergency and dracut-initqueue will both be stopped
and kdump-error-handler.service will run kdump-error-handler.sh.
In a normal failure case, this still works:
foo.service fails
--> trigger emergency.service
--> emergency.service isolates to kdump-error-handler.service
--> kdump-error-handler.service will run kdump-error-handler.sh
Signed-off-by: WANG Chao <chaowang@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Extract functions from kdump.sh, and construct kdump-lib-initramfs.sh as
kdump common functions/varaibles library.
kdump-lib-initramfs.sh will include kdump-lib.sh, because it will use
the functions from there. IOW, kdump-lib-initramfs.sh will be a superset
of kdump-lib.sh
So after this cleanup:
- scripts running in 1st kernel only have to include kdump-lib.sh
- scripts running in 2nd kernel only have to include kdump-lib-initramfs.sh
Signed-off-by: WANG Chao <chaowang@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
