For fadump, this helps to reduce the risk of boot failure, and
may also help speed up the boot by a bit.
For normal kdump, this will delay the dump target mounting, and no
longer depend on systemd to do the mounting job.
And currently there is a failure that caused by some mount handling
bug with kernel and systemd that is failing the system booting:
[FAILED] Failed to mount /kdumproot/home.
See 'systemctl status kdumproot-home.mount' for details.
[DEPEND] Dependency failed for Local File Systems.
[ OK ] Reached target Remote File Systems (Pre).
[ OK ] Reached target Remote File Systems.
Starting udev Coldplug all Devices...
Starting Create Volatile Files and Directories...
Starting Kdump Emergency...
This patch can bypass it. The fix of root cause is still WIP, but this
patch itself is a nice to have optimization so it's reasonable to do so.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
When trying to setup kdump for fedora-coreos, kdumpctl start fails to
find the correct boot directory since BOOT_IMAGE start with the grub
device name
Signed-off-by: Yuval Turgeman <yturgema@redhat.com>
Print some message during the long wait period to reflect the process.
The message will look like:
Network dump target is not usable, waiting for it to be ready
...
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
Backport from the makedumpfile devel branch in upstream.
commit 8425342a52b23d462f10bceeeb1c8a3a43d56bf0
Author: Kazuhito Hagio <k-hagio@ab.jp.nec.com>
Date: Fri Sep 6 09:50:34 2019 -0400
[PATCH] Fix inconsistent return value from find_vmemmap()
When -e option is given, the find_vmemmap() returns FAILED(1) if
it failed on x86_64, but on architectures other than that, it is
stub_false() and returns FALSE(0).
if (info->flag_excludevm) {
if (find_vmemmap() == FAILED) {
ERRMSG("Can't find vmemmap pages\n");
#define find_vmemmap() stub_false()
As a result, on the architectures other than x86_64, the -e option
does some unnecessary processing with no effect, and marks the dump
DUMP_DH_EXCLUDED_VMEMMAP unexpectedly.
Also, the functions for the -e option return COMPLETED or FAILED,
which are for command return value, not for function return value.
So let's fix the issue by following the common style that returns
TRUE or FALSE, and avoid confusion.
Signed-off-by: Kazuhito Hagio <k-hagio@ab.jp.nec.com>
Signed-off-by: Kazuhito Hagio <k-hagio@ab.jp.nec.com>
Acked-by: Kairui Song <kasong@redhat.com>
Backport from the makedumpfile devel branch in upstream.
commit b461971bfac0f193a0c274c3b657d158e07d4995
Author: Kazuhito Hagio <k-hagio@ab.jp.nec.com>
Date: Thu Aug 29 14:51:56 2019 -0400
[PATCH] Fix exclusion range in find_vmemmap_pages()
In the function, since pfn ranges are literally start and end, not start
and end+1, if the struct page of endpfn is at the last in a vmemmap page,
the vmemmap page is dropped by the following code, and not excluded.
npfns_offset = endpfn - vmapp->rep_pfn_start;
vmemmap_offset = npfns_offset * size_table.page;
// round down to page boundary
vmemmap_offset -= (vmemmap_offset % pagesize);
We can use (endpfn+1) here to fix.
Signed-off-by: Kazuhito Hagio <k-hagio@ab.jp.nec.com>
Signed-off-by: Kazuhito Hagio <k-hagio@ab.jp.nec.com>
Acked-by: Kairui Song <kasong@redhat.com>
Backport from the makedumpfile devel branch in upstream.
commit aa5ab4cf6c7335392094577380d2eaee8a0a8d52
Author: Kazuhito Hagio <k-hagio@ab.jp.nec.com>
Date: Thu Aug 29 12:26:34 2019 -0400
[PATCH] x86_64: Fix incorrect exclusion by -e option with KASLR
The -e option uses info->vmemmap_start for creating a table to determine
the positions of page structures that should be excluded, but it is a
hardcoded value even with KASLR-enabled vmcore. As a result, the option
excludes incorrect pages from it.
To fix this, get the vmemmap start address from info->mem_map_data.
Signed-off-by: Kazuhito Hagio <k-hagio@ab.jp.nec.com>
Signed-off-by: Kazuhito Hagio <k-hagio@ab.jp.nec.com>
Acked-by: Kairui Song <kasong@redhat.com>
On a host with ipaddr not ready before kdump service, ssh return errno 255.
While if no ssh-key, ssh also return errno 255. For both of cases, the
current kdump code promote user to run 'kdumpctl propagate'. This confuses
user who already installs ssh-key.
In order to tell these two cases from each other, the ssh warning message
should be involved, and parsed.
For the no ssh-key case , warning message is "Permission denied" or "No
such file or directory". For the other, warning message is "Network
Unreachable"
This patch also does a slight change to enlarge the timeout from 60s to
180s. This value can meet test at the time being
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
Currently there are two issues with device dump:
- It may use too much memory
- kdump won't automatically include required driver in second kernel
User should manually reserve enough memory, and include the required
driver by using extra_modules. Add some notes about the issues in
kexec-kdump-howto.txt
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Device dump may use a log of memory and cause OOM issue, so append
"novmcoredd" option for second kernel and disable it by default.
To use device dump, user should remove the vmcoredd parameter
manually.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Bond options in ifcfg is space separated, dracut expected it to be comma
separated, so it have to be parsed and converted during initramfs
building.
The currently parsing and convert pattern is flawed, for example:
" downdelay=0 miimon=100 mode=802.3ad updelay=0 "
is converted to :
":,downdelay=0 miimon=100 mode=802.3ad updelay=0 "
should be:
":downdelay=0,miimon=100,mode=802.3ad,updelay=0"
So fix this issue by using more simple but robust method for processing
the options.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
The localhost is filtered out in case of is_pcs_fence_kdump, do it too in
case of is_generic_fence_kdump.
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
'hostname -A' can not get the alias, meanwhile 'hostname -a' is deprecated.
So we should do it by ourselves.
The parsing is based on the format of /etc/hosts, i.e.
IP_address canonical_hostname [aliases...]
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
If dump target is ipv6 address, a host should have ipv6 address ready
before starting kdump service. Otherwise, kdump service fails to start due
to the failure "ssh dump_server_ip mkdir -p $SAVE_PATH".
And user can see message like:
"Could not create root@2620:52:0:10da:46a8:42ff:fe23:3272/var/crash"
I observe a long period (about 30s) on some machine before they got ipv6
address dynamiclly, which is never seen on ipv4 host.
Hence kdump service has a dependency on ipv6 address. But there is no good
way to resolve it. One way is asking user to run the cmd "nmcli connection
modify eth0 ipv6.may-fail false". But this will block systemd until ipv6
address is ready. Despite doing so, kdump can try its best (wait 1 minutes
after it starts up) before failure.
How to implement the wait is arguable. It will involve too many technique
details if explicitly waiting on ipv6 address, instead, just lean on 'ssh'
return value to see the availability of network.
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
Backport from the makedumpfile devel branch in upstream.
commit 7bdb468c2c99dd780c9a5321f93c79cbfdce2527
Author: Kazuhito Hagio <k-hagio@ab.jp.nec.com>
Date: Tue Jul 23 12:24:47 2019 -0400
[PATCH] Increase SECTION_MAP_LAST_BIT to 4
kernel commit 326e1b8f83a4 ("mm/sparsemem: introduce a SECTION_IS_EARLY
flag") added the flag to mem_section->section_mem_map value, and it caused
makedumpfile an error like the following:
readmem: Can't convert a virtual address(fffffc97d1000000) to physical address.
readmem: type_addr: 0, addr:fffffc97d1000000, size:32768
__exclude_unnecessary_pages: Can't read the buffer of struct page.
create_2nd_bitmap: Can't exclude unnecessary pages.
To fix this, SECTION_MAP_LAST_BIT needs to be updated. The bit has not
been used until the addition, so we can just increase the value.
Signed-off-by: Kazuhito Hagio <k-hagio@ab.jp.nec.com>
Signed-off-by: Kazuhito Hagio <k-hagio@ab.jp.nec.com>
Acked-by: Dave Young <dyoung@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
Backport from the makedumpfile devel branch in upstream.
commit c1b834f80311706db2b5070cbccdcba3aacc90e5
Author: Kazuhito Hagio <k-hagio@ab.jp.nec.com>
Date: Tue Jul 23 11:50:52 2019 -0400
[PATCH] Do not proceed when get_num_dumpable_cyclic() fails
Currently, when get_num_dumpable_cyclic() fails and returns FALSE in
create_dump_bitmap(), info->num_dumpable is set to 0 and makedumpfile
proceeds to write a broken dumpfile slowly with incorrect progress
indicator due to the value.
It should not proceed when get_num_dumpable_cyclic() fails.
Signed-off-by: Kazuhito Hagio <k-hagio@ab.jp.nec.com>
Signed-off-by: Kazuhito Hagio <k-hagio@ab.jp.nec.com>
Acked-by: Dave Young <dyoung@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
fadump will alter the normal boot initramfs and we don't want a normal
boot to foward and drop the journalctl logs.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
When building for i386, an error occured:
kexec/arch/i386/kexec-x86.c:39:22: error: 'multiboot2_x86_probe'
undeclared here (not in a function); did you mean 'multiboot_x86_probe'?
39 | { "multiboot2-x86", multiboot2_x86_probe, multiboot2_x86_load,
| ^~~~~~~~~~~~~~~~~~~~
| multiboot_x86_probe
kexec/arch/i386/kexec-x86.c:39:44: error: 'multiboot2_x86_load'
undeclared here (not in a function); did you mean 'multiboot_x86_load'?
39 | { "multiboot2-x86", multiboot2_x86_probe, multiboot2_x86_load,
| ^~~~~~~~~~~~~~~~~~~
| multiboot_x86_load
kexec/arch/i386/kexec-x86.c:40:4: error: 'multiboot2_x86_usage'
undeclared here (not in a function); did you mean 'multiboot_x86_usage'?
40 | multiboot2_x86_usage },
| ^~~~~~~~~~~~~~~~~~~~
| multiboot_x86_usage
Fix this issue by putting the definition in the right header, also tidy
up Makefile.
Signed-off-by: Kairui Song <kasong@redhat.com>
The current code only exclude the hostname, while localhost can have alias in
/etc/hosts. All of the alias should be excluded from the fence dump node to
avoid deadlock issue.
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
Squash module is used to save memory. For fadump this is not neccessary
and may slow down the build time, and make it more fragile.
fadump initramfs is used for normal boot as well, although squash module
is capable of being used for generic normal boot, but there are cases
where is doesn't work well. So disable it and make fadump more robust.
Signed-off-by: Kairui Song <kasong@redhat.com>
Tested-by: Hari Bathini <hbathini@linux.ibm.com>
Acked-by: Dave Young <dyoung@redhat.com>
Don't use any log storage and forward to console directly, this make
console output more useful, and also save more memory. On a fresh
installed Fedora 30 it saved ~5M of memory, and the amount of log being
printed to console is still accetable.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Some firmware will provide a ACPI HEST table with massive amount of
entries, and the way how kernel handles these entries will consume a lot
of memory which will lead to OOM issue in kdump kernel. During testing
on certain machine, disable HEST saved ~60M of memory.
Kdump is only for emergency use in case of a kernel panic, so temporarily
disable hardware error report & recovery related feature is acceptable
in general. So disable HEST support in kdump kernel to save memory.
Currently such issue is only observed on x86_64, so limit this change to
x86_64 only.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
systemd has the following message "OnFailureIsolate is deprecated. Please
use OnFailureJobMode= instead"
Changing the file to meet systemd's requirement
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
Backport from the makedumpfile devel branch in upstream.
commit d222b01e516bba73ef9fefee4146734a5f260fa1 (HEAD -> devel)
Author: Lianbo Jiang <lijiang@redhat.com>
Date: Wed Jan 30 10:48:53 2019 +0800
[PATCH] x86_64: Add support for AMD Secure Memory Encryption
On AMD machine with Secure Memory Encryption (SME) feature, if SME is
enabled, page tables contain a specific attribute bit (C-bit) in their
entries to indicate whether a page is encrypted or unencrypted.
So get NUMBER(sme_mask) from vmcoreinfo, which stores the value of
the C-bit position, and drop it to obtain the true physical address.
Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
Looking at the difference of the x86_64 and aarch64 kdump.sysconfig
options for Fedora, one can see the following options which are
different:
Present in kdump.sysconfig.x86_64 but not in kdump.sysconfig.aarch64:
---------------------------------------------------------------------
cgroup_disable=memory
mce=off
numa=off
udev.children-max=2
panic=10
acpi_no_memhotplug
transparent_hugepage=never
nokaslr
Present in kdump.sysconfig.aarch64 but not in kdump.sysconfig.x86_64:
---------------------------------------------------------------------
swiotlb=noforce
After going through all the options, it makes sense to add the
following options added to kdump.sysconfig.aarch64:
KDUMP_COMMANDLINE_APPEND="cgroup_disable=memory udev.children-max=2
panic=10 irqpoll nr_cpus=1
swiotlb=noforce reset_devices"
This has helped reduce the memory footprint of crashkernel on several
aarch64 machines available in the beaker lab. For e.g. I was seeing
OOM issues on large aws ec2 instances with the default crashkernel size
of 512M, and I had to use an increased crashkernel size of 786M on the
same to boot the crash dump kernel.
Signed-off-by: Bhupesh Sharma <bhsharma@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
'maxcpus' setting normally don't work on several kdump enabled systems
due to a known udev issue.
Currently the fedora kdump configuration is set as the following on the
aarch64 systems:
# cat /etc/sysconfig/kdump
<..snip..>
# This variable lets us append arguments to the current kdump
# commandline after processed by KDUMP_COMMANDLINE_REMOVE
# KDUMP_COMMANDLINE_APPEND="irqpoll maxcpus=1 reset_devices"
<..snip..>
Since the 'maxcpus' setting doesn't limit the number of SMP CPUs,
so the kdump kernel still boots with all CPUs available on the system.
For e.g on the qualcomm amberwing its 46 CPUs:
# lscpu
Architecture: aarch64
Byte Order: Little Endian
CPU(s): 46
On-line CPU(s) list: 0-45
Thread(s) per core: 1
Core(s) per socket: 46
Socket(s): 1
NUMA node(s): 1
Vendor ID: Qualcomm
Model: 1
Model name: Falkor
Stepping: 0x0
CPU max MHz: 2600.0000
CPU min MHz: 600.0000
BogoMIPS: 40.00
L1d cache: 32K
L1i cache: 64K
L2 cache: 512K
L3 cache: 58880K
NUMA node0 CPU(s): 0-45
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid asimdrdm
This causes the memory consumption in the kdump kernel to swell up and
we can end up having OOM issues in the kdump kernel boot.
Whereas if we use 'nr_cpus=1' in the bootargs, the number of SMP CPUs in
the kdump kernel get limited to 1.
The 'swiotlb=noforce' setting in bootargs provide us extra guarding, to
ensure the crash kernel size requirements do not swell on systems
which support swiotlb.
With the above settings, crashkernel boots properly (without OOM) on all
the aarch64 boards I could test on - qualcomm amberwings, hp-moonshots
and hpe-apache (thunderx2) for crash dump saving on local disk.
Signed-off-by: Bhupesh Sharma <bhsharma@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
Currently kdumpctl rebuild will simply rebuild the initramfs, and
only perform basic config syntax check. But it should also check if the
target path is available when using SSH target, else kdump may fail.
is second kernel. kdumpctl rebuild should cover this case, and create
the path if it doesn't exist.
This patch make rebuild and restart behaves the same, rebuild is
now equal to restart, except it won't check config change or reload
kdump resource.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Although "kdumpctl rebuild" is introduced to help user rebuild the
initramfs without modifying the kdump.conf, if the kdump.conf is
modified and "kdumpctl rebuild" is called, a initramfs with a faulty
kdump.conf will be built.
Kdump will refuse to load the initramfs when restarted, but kdumpctl
reload may load the faulty initramfs. So need to make sure the faulty
build won't be generate in the first place.
Check for kdump.conf error before building the initramfs to ensure such
failure won't happen.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
We don't necessarily have to always rebuild the initramfs when
extra_modules is set. Instead, just detect if any module is updated,
and only rebuild initramfs if found any updated kernel module.
Tested with in-tree kernel modules, out-of-tree kernel modules, weak
modules, all worked as expected.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Previously only the symlink's timestamp is used for checking if file are
modified, this will not trigger a rebuild if the symlink target it
modified.
So check both symlink timestamp and symlink target timestamp, rebuild
the initramfs on both symlink changed and target changed.
Also give a proper error message if the file doesn't exist.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
When reading kdump configs, a single parsing should be enough and this
saves a lot of duplicated striping call which speed up the total load
speed.
Speed up about 2 second when building and 0.1 second for reload in my
tests.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Early kdump inherits the settings of normal kdump, so any changes that
caused normal kdump rebuilding also require rebuilding the system initramfs
to make sure that the changes take effect for early kdump.
Therefore, when the early kdump is enabled, provide a prompt message after
the rebuilding of kdump initramfs is completed.
Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Currently kdump is not working well with encrypted targets, add document
about this issue.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Add some note about the limitation of kdumpctl's auto detect and rebuild
feature, and suggest the user to rebuild the initramfs manually on
major system change, and don't include the initramfs in disk images.
Put the note about system change in front part of the document so user
will less likely to miss it.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Previous we rebuild the initramfs when kenrel load module list changed,
but this is not very stable as some async services may load/unload
kernel modules, and cause unnecessary initramfs rebuild.
Instead, it's better to just check if the module required to dump to
the dump target is loaded or not, and rebuild if not loaded. This
avoids most false-positives, and ensure local target change is always
covered.
Currently only local fs dump target is covered, because this check
requires the dump target to be mounted when building the initramfs,
this guarantee that the module is in the loaded kernel module list,
else we may still get some false positive.
dracut-install could be leveraged to combine the modalias list with
kernel loaded module list as a more stable module list in the initramfs,
but upstream dracut change need to be done first.
Passed test on a KVM VM, changing the storage between SATA/USB/VirtIO
will trigger initramfs rebuild and didn't notice any false-positive.
Also passed test on my laptop with no false-positive.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
This reverts commit 6b479b6572.
Check initramfs rebuild by looking at if there is any change of load
kernel modules list is not very stable after all. Previously we are
counting on udev to settle before kdump is started to ensure all modules
is ready, but actually any service may cause a kernel module load, even
after udev is settled.
The previous commit is trying to workaround an issue that VM created
with disk snapshot may fail in the kdump initramfs. The better fix is to
not include the kdump initramfs in the disk snapshot at all, as the
kdump initramfs is not generated for a generic use. And With new added
"kdumpctl reload" command, admins could rebuild the image easily, and
should rebuild the initramfs on hardware change manually.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
On powerpc, after hot add cpu and trigger crash on the hot-added cpu, the
kdump kernel hangs after "I'm in purgatory".
The current udev rules expects the dtb to be rebuit on cpu add/remove event.
But since powerpc does not follow the standard cpu hot add framework, it
only ejects online/offline event to user space when cpu is hot
added/removed, instead of add/remove event. Pingfan tried fixing that but
it didn't please the maintainer as it breaks some old userspace tools.
Due to the failure of dtb's rebuilding, KDump kernel fails to get the
'boot_cpuid' and eventually fails to boot [see early_init_dt_scan_cpus() in
arch/powerpc/kernel/prom.c file] if system crashes on hot-added CPU.
Work around it by changing udev rules on powerpc to onlne/offline.
As for offline message, it is even useless on powerpc, and can be dropped.
See the explain: On powerpc, /sys/devices/system/cpu/cpuX nodes are present
for all "possible", irrespective of whether a CPU is hot-added/removed.
crash_notes are already built for all /sys/devices/system/cpu/cpuX nodes and
these nodes are present for all "possible" CPUs
(online/offline/could-be-hot-removed/could-be-hot-added)
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
kdumpctl fails to load a crash kernel if KASLR is enabled.
The incorrect configuration of the kptr_restrict parameter causes that
the kdumpctl fails to load a crash kernel. The problem occurs when the
Kernel Address Space Layout Randomization (KASLR) mechanism is enabled.
The value of kptr_restrict decides how to expose the content of /proc
files. Inappropriate setting will cause that the /proc/kallsyms file is
being printed as all zeros. This configuration will break the kdump
loading process since it needs access to the conntent of /proc/kallsyms
if KASLR is enabled.
Set kptr_restrict to 1 to avoid the problem described above.
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
These two command are not documented, update the man page to let user
know how to use them.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Use "kdumpctl rebuild" to rebuild the image directly. This could help
admins to rebuild kdump image directly.
Also merge fadump related initramfs backup/restore into setup_initrd,
and do permission only when actually trying to rebuild the image.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
