In RHEL6, vmcore file was created if panic occurred during shutdown. This was
very useful for analyzing problems during the shutdown sequence.
However, in RHEL7, vmcore will not be created after kdump service is stopped.
This will make it very difficult to solve problems during shutodwn.
The reason why kdump fails to dump vmcore is kdump is stopped too early during
the power is off.
If add "DefaultDependencies=no" to the [Unit] of kdump.service , kdump will
not be stopped by systemd after shutdown command.
The manpage of systemd.unit about the DefaultDependencies:
If true, (the default), a few default dependencies will implicitly be created
for the unit. The actual dependencies created depend on the unit type. For
example, for service units, these dependencies ensure that the service is
started only after basic system initialization is completed and is properly
terminated on system shutdown.
The manpage about basic.target:
A special target unit covering basic boot-up.
systemd automatically adds dependencies of the types Requires= and After= for
this target unit to all services (except for those with DefaultDependencies=no).
Usually this should pull-in all mount points, swap devices, sockets, timers,
and path units and other basic initialization necessary for general purpose
daemons.
So "DefaultDependencies=no" can keep kdump not stopped too early. But to make
it start when power on, add After=basic.target will be better.
The systemd-devel mailed to me: using DefaultDependencies=no but also
After=basic.target will make sure the service isn't started too early (but kept
until systemd's final process killing spree).
Signed-off-by: Chao Fan <cfan@redhat.com>
Acked-by: Minfei Huang <mhuang@redhat.com>
Customer found when specify "noauto" option in fstab for nfs mount,
dump failed.
The reason is if "noauto" option is specified in fstab, the mount entry
in fstab related to dump target will passed to dracut and stored in
kdump initrd. Then during kdump kernel boots this entry containing
"noauto" will be ignored by mount service. This cause dump failing.
In fact with "noauto" not only nfs dump will fail, non-root disk dump
will fail too. root disk dump can dump successfully since root disk can
always be mounted by systemd.
So now "noauto" need be filtered out when the fstab entry corresponding
to dump target contains "noauto".
Changelog:
v4 -> v5
code comment is not clear enough. supplement it.
Signed-off-by: Qiao Zhao <qzhao@redhat.com>
Acked-by: Minfei Huang <mhuang@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
Beginning from f23 program hardening become the defaults for all packages.
Details can be checked from below link:
https://fedoraproject.org/wiki/Changes/Harden_All_Packages
Adding this to makedumpfile CFLAGS, otherwise makedumpfile building will
fail on koji.
Signed-off-by: Baoquan He <bhe@redhat.com>
Kdump will dump the vmcore in incorrect target directory, if the target
is bind mounted.
As commented in the previous patch, we can construct the real path in
Atomic, which contains two part, one bind mounted path, the other
specified dump path. Then replace the path as the real path in
/etc/kdump.conf.
findmnt can find the real path for nfs, although the path is in bind
mode. So nfs can work well with the path in bind mode.
Signed-off-by: Minfei Huang <mhuang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
kdump will raise the warning in Atomic, if the path is bind mounted
directory. The reason why causes this issue is kdump cannt parse the
bind mounted directory.
To correct dumping target, we can construct the real dumping path in
Atomic, which contains two part, one bind mounted path, the other
specified dump target.
Following is an example:
-bash-4.2# cat /etc/kdump.conf | grep ^path
path /var/crash
-bash-4.2# findmnt /var | tail -n 1 | awk '{print $2}'
/dev/mapper/atomicos-root[/ostree/deploy/rhel-atomic-host/var]
-bash-4.2# findmnt -v /var | tail -n 1 | awk '{print $2}'
/dev/mapper/atomicos-root
Then we can found it that the real path of dumping vmcore is
/ostree/deploy/rhel-atomic-host/var/crash.
To fix this issue, we can replace the target path as the real path which
is from above parsing.
Signed-off-by: Minfei Huang <mhuang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
Any block device can be mounted multiply on the different mount point.
Once a mount point is mounted in bind mode, the general mount point can
be unmounted. Thus kdump would not find the general mount point[1] to
handle the path.
The mount point, which is as general mount point, will be got by
"fintmnt" previously. But the mntpoint may be incorrect, if the mntpoint
is bind mount.
In order to fix it to support bind mounted in atomic, we will add a
judgement to comfirm the mntpoint is bind mounted, or not.
For general mount, returning path is like following, if we use
"findmnt". The returning is same as "findmnt -v".
-bash-4.2# findmnt /var | tail -n 1 | awk '{print $2}'
/dev/mapper/atomicos-root
But for bind mount, returning path is like following, if we use
"fintmnt".
-bash-4.2# findmnt /var | tail -n 1 | awk '{print $2}'
/dev/mapper/atomicos-root[/ostree/deploy/rhel-atomic-host/var]
Use "findmnt -v" is like this:
-bash-4.2# findmnt -v /var | tail -n 1 | awk '{print $2}'
/dev/mapper/atomicos-root
So we can determine the bind mount, if the returning is different
between "findmnt" and "findmnt -v".
[1] general mount point is a directory without being in bind mounted
mode, just a normal directory.
Signed-off-by: Minfei Huang <mhuang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
For Atomic system, the cmdline will contain the specific string
"ostree". So we can filter out the "ostree" to judge the system is
Atomic or not.
Signed-off-by: Minfei Huang <mhuang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
findmnt uses the option "-v, --nofsroot" to exclude the [/dir] in the
SOURCE column for bind-mounts, then if $_mntpoint equals to
$_mntpoint_nofsroot, the mountpoint is not bind mounted directory.
the value of $_mntpoint may be like
/dev/mapper/atomicos-root[/ostree/deploy/rhel-atomic-host/var], if the
directory is bind mounted. The former part represents the device path, the
rest part is the bind mounted directory which quotes by bracket "[]".
Signed-off-by: Minfei Huang <mhuang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
Now kdump cannt parse the path correctly, if the path contains
duplicated "/". Following is an example to explain it detail. (the
directory /mnt is a mount point which is mounted a block device)
path //mnt/var/crash
Then the warning will raise.
Force rebuild /boot/initramfs-3.19.1kdump.img
Rebuilding /boot/initramfs-3.19.1kdump.img
df: ‘/mnt///mnt/var/crash’: No such file or directory
/sbin/mkdumprd: line 239: [: -lt: unary operator expected
kexec: loaded kdump kernel
Starting kdump: [OK]
For above case, kdump fails to check the fs size, due to the incorrect
path.
In kdump code flow, we will cut out the mount point(/mnt) from the
path(//mnt/var/crash). But the mount point cannt match the path, because
of the duplicated "/".
To fix it, we will strip the duplicated "/" firstly.
Signed-off-by: Minfei Huang <mhuang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
This reverts commit f4c45236bf.
Since that commit will change the behaviour of kdump_post. That is not
good.
Signed-off-by: Baoquan He <bhe@redhat.com>
Harald warned it's dangerous to use /tmp/*$$* in shell scripts of dracut
modules.
Quote his saying as below:
***************************
This can be exploited so easily and used to overwrite e.g. /etc/shadow.
The only thing you have to do is waiting until the next time the kdump
initramfs is generated on a kernel update.
If at all, please use "$initdir/tmp/" because $initdir is a mktemp generated
directory with a non-guessable name!
**************************
So make a clean up in this patch.
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Now we use the pattern:
<machine/ipaddr>-YYYY.MM.DD-HH:MM:SS
while rhel6 uses the following:
<machine/ipaddr>-YYYY-MM-DD-HH:MM:SS
This change may break someone's script and we should change it back to
keep consistent between releases.
Signed-off-by: WANG Chao <chaowang@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
Acked-by: Minfei Huang <mhuang@redhat.com>
User complains that kdump_post script doesn't execute after mount
failed. This happened since mount failure will trigger
kdump-error-handler.service, and then start kdump-error-handler.sh.
However in kdump-error-handler.sh it doesn't execute kdump_post.
Hence add it in this patch.
Surely the function do_kdump_post need be moved into kdump-lib-initramfs.sh
to be a common function.
v1->v2:
Add a return value to do_kdump_post when invoked in kdump_error-handler.sh.
And call do_kdump_post earlier than do_default_action, otherwise
it may not execute if reboot/poweroff/halt.
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Acked-by: Meifei Huang <mhuang@redhat.com>
Previously /boot is asumed as the default dir where kernel and initrd
is put. However, the directory containing the running kernel image
on Atomic systems differs in each installation. Usually something like:
/boot/ostree/rhel-atomic-host-b50a015b637c353dc6554c851f8a1212b60d6121a7316715e4a63e2a4113cd72
This means that kdump will not find vmlinuz when installed on an
Atomic host, and thus the kdump service will fail to start.
In this patch, the kdump boot dir finding behaviour is a little changed.
Firstly check whether user has specify a directory explicitly in
/etc/sysconfig/kdump. If yes that is respected. Otherwise we assume
1st kernel and kdump kernel are put in the same place under /boot.
Then find it according /proc/cmdline and append it to /boot/
Note:
So now the KDUMP_BOOTDIR in /etc/sysconfig/kdump is set as empty
by default. If user set KDUMP_BOOTDIR to a directory, then he need to
take care of all related things himself. otherwise kdump script handle
it automatically.
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Acked-by: Minfei Huang <mhuang@redhat.com>
Steve found a bug. When mount a disk in /var and not specify path
in /etc/kdump.conf, the vmcore will be dumped into /var/crash of
that disk, but not /crash on that disk.
This is because when write the parsed path into /tmp/$$-kdump.conf
in default_dump_target_install_conf() of mkdumprd, it uses below
sed command. So if no path specified at all, this sed command won't
add it to /tmp/$$-kdump.conf. Then in 2nd kernel it will take default
path, namely "/var/crash" as path if no path in /etc/kdump.conf in
2nd kernel.
sed -i -e "s#$_save_path#$_path#" /tmp/$$-kdump.conf
According to Dave Young's suggestion, erase the old path line and then
insert the parsed path. This can fix it.
v2->v3:
erase the old path line and then insert the parsed path.
sed -i -e "s#^path[[:space:]]\+$_save_path##" /tmp/$$-kdump.conf
echo "path $_path" >> /tmp/$$-kdump.conf
v3->v4:
Change the sed pattern, erase lines starting with "path" and then
insert the parsed path.
sed -i -e "s#^path.*##" /tmp/$$-kdump.conf
echo "path $_path" >> /tmp/$$-kdump.conf
v4->v5:
Chaowang suggested using sed command d to remove the whole line
like below:
sed -i "/^path/d" /tmp/$$-kdump.conf
echo "path $_path" >> /tmp/$$-kdump.conf
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: WANG Chao <chaowang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Chao pointed out that it's better to use get_option_value to get
get a specific config_val.
And also there's a potential risk when use below sed command to
do the replacement.
sed -i -e "s#$_save_path#$_path#" /tmp/$$-kdump.conf
Say user configure kdump.conf like the following. Then sed may
replace "/var/crash/post.sh" with something else, depanding on
mount point.
kdump_post /var/crash/post.sh
path /var/crash
So in this patch clean them up.
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: WANG Chao <chaowang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
kdumpctl now parses mount points in determining the partition to
save the dump to. So /etc/fstab can be considered a configuration
file for kdump.
Change adds an additional depenedency check on /etc/fstab when
kdump is restarted.
Signed-off-by: Jerry Hoemann <jerry.hoemann@hp.com>
Acked-by: WANG Chao <chaowang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Jerry Hoemann reported a bug that a mount will fail when he installed
a system with separate "/" "/var" and "/var/crash". That means root
disk /dev/a mounted on /, and another disk /dev/b is mounted on /var,
then the 3rd disk /dev/c mounted on /var/crash. Then kdump will fail
since mount will fail.
This is because the mount information will be written into
/$mntimage/etc/fstab like below. And the dump target is /dev/c. However
/dev/b is not related in kdump, its mount info is not necessary and not
saved. So when go into kdump kernel, it will find there's not a crash
dir under /sysroot/var. And in current implementation, if not a root
disk dump, sysroot is a read-only mount, no dir can be created in this
situation.
/dev/disk/by-uuid/cdcf007a-b623-45ee-8d73-a8be0d372440 /sysroot/var/crash xfs rw,relatime,...,x-initrd.mount 0 2
So in this patch, change the mount behavior to fix this bug. If dump
target is a root disk, mount point is /sysroot. If dump target is not
root, just mount it to /kdumproot/$_target. Now it works.
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Acked-by: WANG Chao <chaowang@redhat.com>
Tested-by: Jerry Hoemann <jerry.hoemann@hp.com>
With the inclusion of 'panic_on_warn',
http://marc.info/?l=linux-api&m=141570937328528&w=2
and which is now staged in Andrew Morton's tree, we need to remove
'panic_on_warn' from the 2nd kernel's cmdline. If it is included it is
possible a non-fatal warning could panic the second kernel.
Before:
[ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0+
root=/dev/mapper/rhel_intel--canoepass--05-root ro
rd.lvm.lv=rhel_intel-canoepass-05/root
rd.lvm.lv=rhel_intel-canoepass-05/swap console=ttyS0,115200n81
LANG=en_US.UTF-8 systemd.debug panic_on_warn=1 irqpoll nr_cpus=1
reset_devices cgroup_disable=memory mce=off numa=off udev.children-max=2
panic=10 rootflags=nofail acpi_no_memhotplug disable_cpu_apicid=0
elfcorehdr=839092K
After:
[ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0+
root=/dev/mapper/rhel_intel--canoepass--05-root ro
rd.lvm.lv=rhel_intel-canoepass-05/root
rd.lvm.lv=rhel_intel-canoepass-05/swap console=ttyS0,115200n81
LANG=en_US.UTF-8 systemd.debug irqpoll nr_cpus=1 reset_devices
cgroup_disable=memory mce=off numa=off udev.children-max=2 panic=10
rootflags=nofail acpi_no_memhotplug disable_cpu_apicid=0
elfcorehdr=839092K
Signed-off-by: Prarit Bhargava <prarit@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
Certain kernel parameters like min_free_kbytes can be configured at runtime
using sysctl. While this is useful in first kernel, it can lead to unnecessary
failures like OOM in kdump kernel. This patch enforces default vaules for all
sysctl parameters, in kdump kernel, by removing sysctl.conf & sysctl.d/* files.
Signed-off-by: Hari Bathini <hbathini@linux.vnet.ibm.com>
Acked-by: Dave Young <dyoung@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
Acked-by: WANG Chao <chaowang@redhat.com>
Once login using ssh, the ssh will store the known hosts entry to the
local ~/.ssh/known_hosts. From now, we can login using ssh automaticly.
The ssh will check the ~/ssh/.known_hosts entry, if set the option
StrictHostKeyChecking=yes/ask in the config or command line, when you
want to login the target. the default value of StrictHostKeyChecking is
ask.
And the kdump using the ssh will append the option
StrictHostKeyChecking=yes in the command line.
We can using following ip to connect peer machine, if enable the ipv6.
fe80::5054:ff:fe48:ca80%eth0
Obviously, above ip contains the ethX.
Kdump will add the prefix "kdump-" before ethX to avoid flowing
netdevice name in case netdevice names ethX in the 2nd kernel. So the
ip address will change to fe80::5054:ff:fe48:ca80%kdump-eth0.
Kdump will login the target manully in the 2nd kernel, because of the
option StrictHostKeyChecking=yes and inexistence known hosts entry
in the local ~/.ssh/known_hosts. Hence dumping core will fail.
In order to login automaticly using ssh, we should add the prefix
"kdump-" before ethX in the local ~/.ssh/known_hosts.
Signed-off-by: Minfei Huang <mhuang@redhat.com>
For ethX, it may fail to setup the network in the 2nd kernel due to the
mapping of ethernet device name and MAC changes.
The commit(ba7660f37e) has fixed this
issue by add the prefix "kdump-" before ethX. But the network will fail
to work in the static route mode because of this commit.
Here is the config which is used to setup the static route:
rd.route=192.168.201.215:192.168.200.137:eth1
Obviously, the static route config comtains the ethX. But the network
device names kdump-ethX in the 2nd kernel, so the static route config
will fail to execute. To fix it, we should identify the network device.
Add the prefix "kdump-" before the ethX in the static route config to
setup it successfully in the 2nd kernel.
Signed-off-by: Minfei Huang <mhuang@redhat.com>
Acked-by: WANG Chao <chaowang@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
It is boring that internal result is shown in the terminal. Do not print
anything to standard output by using the command "grep -q".
Signed-off-by: Minfei Huang <mhuang@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
Acked-by: WANG Chao <chaowang@redhat.com>
In ssh or raw dump case, if user do not specify "core_collector" in
kdump.conf, kdump will fail. Because global DEFAULT_CORE_COLLECTOR
variable isn't applied to CORE_COLLECTOR. Now fix it and clean up the
duplicate code in kdump.sh.
Signed-off-by: WANG Chao <chaowang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
I forgot to add kdump.sysconfig.ppc64le to "Source" directive to
kexec-tools.spec. And on ppc64le, the default kdump.sysconfig will be
installed to /etc/sysconfig/kdump. Now fix it.
Signed-off-by: WANG Chao <chaowang@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
get_option_value() is used to get the value of $1 configured in
/etc/kdump.conf. But when we use "get_option_value ssh", it can get the
value of "sshkey" instead of "ssh".
Fix the regexp pattern to get an exact match.
Signed-off-by: WANG Chao <chaowang@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
Previously for solving static route issues, all routes which go
through a specific dev will be saved in 1st kernel, and then added
in 2nd kernel. Because we use below search pattern, an exception
will happen:
/sbin/ip route show | grep -v default | grep "^[[:digit:]].*via.* $_netdev"
That exception is a corner case which happened when 2 machines connected
directly by cable and the 2 network interfaces are configured in
different network subnets. E.g there are 2 machines A and B:
A:ens10 < ------ > B:ens9
A:ens10 inet 192.168.100.111/24 scope global ens10
route need be added in A:
192.168.110.0/24 dev ens10
B:ens9 inet 192.168.110.222/24 scope global ens9
route need be added in B
192.168.100.0/24 dev ens9
Now if A want to dump to B, the route "192.168.110.0/24 dev ens10"
has to be saved and added in 2nd kernel.
So in this patch "ip route get to $target" command is executed, then
an exact route can be got for going to that target. By this, static
route works and the corner case can be fixed too.
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: Marc Milgram <mmilgram@redhat.com>
Acked-by: WANG Chao <chaowang@redhat.com>
With fadump support, dracut-kdump.sh script is installed into default
initrd to capture vmcore generated by firmware assisted dump. Thus in
fadump case, the same initrd is being used for normal boot as well as
boot after system crash. Hence a device node, added by firmware while
system crashes, is checked to identify if it is a normal boot or boot
after crash to determine whether or not capture vmcore. While testing
fadump in fedora21 alpha, observed that vmcore capture is initiated
even during normal boot, inspite of this check, with the below error:
"kdump.sh[451]: /bin/kdump.sh: line 5: return: can only `return'
from a function or sourced script"
The below patch tries to fix this issue.
Signed-off-by: Hari Bathini <hbathini@linux.vnet.ibm.com>
Acked-by: Dave Young <dyoung@redhat.com>
Acked-by: WANG Chao <chaowang@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
Remove this package and put eppic_makedumpfile.so and its sample
scripts in kexec-tools package.
makedumpfile does dlopen() on eppic_makedumpfile.so and that does not
enforce any choice. One could either ship it in kexec-tools package or
in a subpackage. Both will work.
The real reason was that code for eppic_makedumpfile.so
(extension_eppic.c) and some eppic scripts are in upstream makedumpfile
project. And that project is distributed as part of kexec-tools package.
Now breaking down that makedumpfile in two parts and shipping all
eppic specific bits in a separate subpackage was creating confusion
everytime we did some changes.
So to avoid that confusion and to keep all of the makedumpfile related
bits in a single package, this change is being done.
Signed-off-by: WANG Chao <chaowang@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
By default on powerpc platform, kvm will reserve a relatively large CMA
(128M aligned) at early boot. In kdump kernel, even KVM sounds useless
but still it reserves 128M and makes kdump kernel fail to boot.
Now fix this by adding the following to kernel command line:
"kvm_cma_resv_ratio=0"
which disable the CMA reservation.
Signed-off-by: WANG Chao <chaowang@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
