Commit Graph

1258 Commits

Author SHA1 Message Date
Kairui Song
7d861422fa Rebase makedumpfile to 1.6.8
makedumpfile have moved to Github, so update the upstream URLs.
2020-11-30 16:48:02 +08:00
Hari Bathini
7a77d5a267 fadump-howto: update about 'nocma' and 'off' options for 'fadump=' parameter
Along with 'on' option, 'fadump=' kernel parameter also supports
'nocma' & 'off' options. Update about these missing options in the
fadump-howto.txt document.

Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
2020-11-30 15:29:11 +08:00
Pingfan Liu
6f9235887f module-setup.sh: enable vlan on team interface
Dracut has switch network-legacy to network-manager by default, which makes
vlan on team easy. So it can be enabled.

Testing network topology with two VMs.
VM1
            ens2-\         /----> VLAN8 (192.168.120.50)
                  ---> team0
            ens3-/     (192.168.122.10)

VM2
            ens2-\         /----> VLAN8 (192.168.120.100)
                  ---> team0
            ens3-/      (192.168.122.20)

Both of ens2/ens3 in VM1/VM2 are connected to virbr0.

During test, dump target is set as root@192.168.120.100:/var/crash
then crashing in VM1

Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Lianbo Jiang <lijiang@redhat.com>
2020-11-30 15:27:00 +08:00
Kairui Song
9966b0a12e kdump-lib: Fix get_bind_mount_source to support btrfs and fstab
Currently get_bind_mount_source will not work on btrfs, that's because
this function relies on findmnt to detect bind mount.

For a bind mount, findmnt will return different value with "-v" option.

For example, we have /dev/sdc mounted on /mnt/source, and then bind
mount /mnt/source/sub/path to /mnt/bind:

$ findmnt /mnt/bind
  TARGET    SOURCE              FSTYPE OPTIONS
  /mnt/bind /dev/sdc[/sub/path] ext4   rw,relatime,seclabel

$ findmnt -v /mnt/bind
  TARGET    SOURCE   FSTYPE OPTIONS
  /mnt/bind /dev/sdc ext4   rw,relatime,seclabel

But findmnt also return similiar result for btrfs, on a fresh installed
Fedora 33:

$ findmnt /
  TARGET SOURCE           FSTYPE OPTIONS
  /      /dev/sdb7[/root] btrfs  rw,relatime,seclabel,ssd,space_cache,subvolid=256,subvol=/root

$ findmnt -v /
  TARGET SOURCE    FSTYPE OPTIONS
  /      /dev/sdb7 btrfs  rw,relatime,seclabel,ssd,space_cache,subvolid=256,subvol=/root

The [...] indicator will contain the subvol of btrfs as well. And if
it's bind mounted under btrfs, it will contain a mixup of btrfs subvol
and the actuall fsroot.

And also, if the bind mount source device is not mounted on /,
get_bind_mount_source will also not work.

So rewrite the get_bind_mount_source function, make it work in every
cases.

Tested with:
 - Silverblue's bind mount
 - Bind mount with source device mounted not under /
 - Btrfs
 - Bind mount and source device is Btrfs

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
2020-11-30 15:26:53 +08:00
Kairui Song
08d9846eba Make get_mount_info work with bind mount
Remove the --real when calling findmnt.

The option is only useful in capture kernel, to avoid
`findmnt` returning the pseudo 'rootfs' for non mounted path.

example, when /kdumproot/mnt/ is not mounted:
kdump:/# findmnt --target /kdumproot/mnt
TARGET SOURCE FSTYPE OPTIONS
/      rootfs rootfs rw,size=61368k,nr_inodes=15342

kdump:/# findmnt --target /kdumproot/mnt
<return 1 and empty output>

But this function will make findmnt also return empty value for bind
mount. So remove it and add an extra if statement for second kernel.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
2020-11-30 15:26:47 +08:00
Kairui Song
d551516f52 Set watchdog's pretimeout to zero in kdump kernel
Most watchdogs have a parameter pretimeout, if set to non-zero, it means
before the watchdog really reset the system, it will try to panic the
kernel first, so kdump could kick in, or, just print a panic stacktrace
and then kernel should reset it self.

If we are already in kdump kernel, this is not really helpful, only
increase kernel hanging chance. And it also make thing become complex
as some watchdog triggers the kernel panic in NMI context, which
could also hang the kernel in strange ways, and fail the watchdog it
self. So just disable this parameter.

Also for hpwdt, it have another parameter kdumptimeout, which is
just designed for first kernel. The default behaviour is the watchdog
will simply stop working if timeouted, trigger a panic, and leave the
kernel to kdump. Again, if we are already in kdump this is not helpful.
So also disable that.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Lianbo Jiang <lijiang@redhat.com>
2020-11-30 15:25:36 +08:00
Kairui Song
4464bcf8f3 kdump-lib.sh: Use a more generic helper to detect omitted dracut module
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Lianbo Jiang <lijiang@redhat.com>
2020-11-30 15:25:26 +08:00
Kairui Song
647aa56b53 Fix the watchdog drivers detection code
Currently the watchdog detection code is broken already, it
get the list of active watchdog drivers, then check if they are
set in the /etc/cmdline.d/* as preload module. But after we
switched to use squash module, /etc/cmdline.d/* is not directly visible.

So just detect whether current needed driver is installed.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Lianbo Jiang <lijiang@redhat.com>
2020-11-30 15:25:19 +08:00
Kairui Song
320bd209fe Add a helper for detecting watchdog drivers
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Lianbo Jiang <lijiang@redhat.com>
2020-11-30 15:25:13 +08:00
Kairui Song
276de0f810 Remove a redundant nfs check
In check_fs_modified, is_nfs_dump_target is already called, the dump
target can't be nfs. No need to check here.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Lianbo Jiang <lijiang@redhat.com>
2020-11-30 15:25:06 +08:00
Kairui Song
d54e5bab0f kdumpctl: split the driver detection from fs dection function
The driver detection have nothing to do with fs detection, and currently
if the dump target is raw, the block driver detection is skipped which
is wrong. Just split it out and run the block driver detection when dump
target is fs or raw.

Also simplfied the code a bit.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Lianbo Jiang <lijiang@redhat.com>
2020-11-30 15:24:45 +08:00
Kairui Song
4f492cf73e Release 2.0.20-20
Also fix some changelog issue.

Signed-off-by: Kairui Song <kasong@redhat.com>
2020-11-19 00:38:23 +08:00
Kairui Song
7b7e5d0743 selftest: Fix several test issue with Fedora 33
- ssh-copy-id is bugged and not working, use a more robust way to sync
  ssh keys
- systemd-resolvd will bind on port 53 so DHCP server won't work,
  disable systemd-resolvd's builtin DNS server

Signed-off-by: Kairui Song <kasong@redhat.com>
2020-11-18 23:57:13 +08:00
Kairui Song
616d359c5e selftest: add more detailed log and fix a test failure issue
Signed-off-by: Kairui Song <kasong@redhat.com>
2020-11-18 23:57:02 +08:00
Kairui Song
13ac244630 selftest: Update test base image to Fedora 33 2020-11-18 17:15:18 +08:00
Kairui Song
f85a291fcb selftest: Fix qcow2 image format detect
qemu-img will report "qcow2" or "qcow2 backing qcow2" for qcow2 image,
cover both case.

Signed-off-by: Kairui Song <kasong@redhat.com>
2020-11-18 01:53:01 +08:00
Kairui Song
aced2c06a0 selftest: Always use the get_image_fmt helper
Avoid code duplication.
2020-11-18 01:53:01 +08:00
Lianbo Jiang
638167358f Doc: improve the usage documentation of the logger
Let's remove some redundant descriptions in the usage documentation
of the logger, and make it clear.

Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
2020-11-13 02:49:56 +08:00
Lianbo Jiang
8b0a755b82 Update the kdump sysconfig
Some unused log levels have been removed, and kdump has used the
different options to control the log levels for the first kernel
and the second kernel. Therefore, let's update the kdump sysconfig
accordingly.

Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
2020-11-13 02:49:56 +08:00
Lianbo Jiang
0098b5d9f3 Capitalize the configuration name of log level
In the /etc/sysconfig/kdump, we usually use the uppercase configuration
name for all options. So let's use the same method to handle this.

Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
2020-11-13 02:49:35 +08:00
Lianbo Jiang
e345ed18e2 Add the rd.kdumploglvl option to control log level in the second kernel
Let's add the rd.kdumploglvl option to control log level in the second
kernel, which can make us avoid rebuilding the kdump initramfs after we
change the log level in /etc/sysconfig/kdump.

Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
2020-11-13 02:43:49 +08:00
Lianbo Jiang
239d64f6da Appropriately converts logger numeric level to syslog log level
The kdump-logger will be used by the system service(daemons), so let's
appropriately convert the logger numeric level to syslog level with the
facility(daemon). The number is constructed by multiplying the facility
by 8 and then adding the level.

About The Syslog Protocol, please refer to the RFC5424 for more details.

Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
2020-11-13 02:43:30 +08:00
Lianbo Jiang
5b2b7ec08b Remove unused log levels for kdump logger
Previously, the range of log level is from 1 to 6, and the TRACE
level and FATAL level are not used, therefore, let's remove these
unused log levels.

Now it has only four log levels: error(1), warn(2), info(3)
and debug(4). We have to remap the numeric log level to the logger
priority or syslog log level, which is finished in kdump-logger.sh
module, it is invisible for user.

Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
2020-11-13 02:42:48 +08:00
Lianbo Jiang
2dea15c4fd Add sanity checks for the log levels
Let's add sanity checks for the log levels in order to avoid
passing illegal log levels to the logger.

Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
2020-11-13 02:42:19 +08:00
Kairui Song
69bf81bc8b Move watchdog detect and install code to module-setup.sh
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Lianbo Jiang <lijiang@redhat.com>
2020-11-12 14:03:40 +08:00
Kairui Song
bc639c9763 Add a helper to omit non-mandatory dracut module
Use dracut_args to omit some non-mandatory modules.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Lianbo Jiang <lijiang@redhat.com>
2020-11-12 14:03:35 +08:00
Kairui Song
08de712528 Move some dracut module dependencies checks to module-setup.sh
depend() in module-setup.sh is a better place to setup dracut module
dependency, it will do early check, and fail early if needed module is
missing. Also remove a unneeded helper add_dracut_module.

Also remove the unnecessary return in depend() function.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Lianbo Jiang <lijiang@redhat.com>
2020-11-12 14:03:19 +08:00
Lianbo Jiang
cd85fe9165 Add code comments to help better understanding
Let's add some code comments to help better understanding, and
no code changes.

Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
2020-11-12 13:59:21 +08:00
Kairui Song
bfd06661e8 Release 2.0.20-19
Signed-off-by: Kairui Song <kasong@redhat.com>
2020-11-05 01:11:32 +08:00
Kairui Song
1b8e0325f2 Fix comment about ssh dump target
When using ssh dump target, scp is always used, correct the comment.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
2020-11-05 01:10:14 +08:00
Kairui Song
a1664c63d0 mkdumprd: Ensure kdumpbase is added
Dracut only check if a module failed installtion if the module is listed
in --add params. Without this param, if kdumpbase failed to install due
to any reason, dracut will still build the initramfs only print a
warning. Add this param to ensure it fail early.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
2020-11-05 01:10:14 +08:00
Jonathan Lebon
0084806493 kdump.service: use ConditionKernelCommandLine=crashkernel
Because otherwise, `kdumpctl start` will fail anyway. This makes it
easier to enable kdump by simply adding the mandatory karg and leaving
the service enabled.
2020-10-27 12:22:36 -04:00
Lianbo Jiang
12e72b9dd6 Revert "Revert "s390x: enable the kexec file load by default""
This reverts commit fa8aa52d94.

For the s390x, the vmlinuz image has only single signature according
to the kernel.spec. The dual signature issue doesn't happens on s390x,
therefore, let's restore it in order to enable the file load on s390x
by default.

Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
2020-10-27 17:45:37 +08:00
Lianbo Jiang
3221f4e91f increase makdumpfile default message level to 7
Currently, the makedumpfile option '--message-level' is set to 1 when
dumping the vmcore, it only displays the progress indicator message,
but there are no common message and error message, it is important to
report some additional messages, especially for the error message,
which is very useful for the debugging.

In view of this, let's change the message level to 7 by default.

Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
2020-10-27 17:45:32 +08:00
Kairui Song
b9a1f461a8 Fix error when using raw target with opalcore
Commit 08276e9 wrongly raise this warning message to error level, fix
this.

Fixes: 08276e9 ('Rework check_config and warn on any duplicated option')
Signed-off-by: Kairui Song <kasong@redhat.com>
2020-10-27 17:37:14 +08:00
Kairui Song
46cc7f46b2 module-setup.sh: Instead of drop journalctl log, just don't read kmsg
Previously journalctl logs are directly dropped to save memory, but this
make journalctl unusable in kdump kernel and diffcult to debug. So
instead just don't let it read kmsg but keep other logs stored as volatile.

Kernel message are already stored in the kernel log ring buffer,
no need to let journalctl make a copy, especially when in kdump
kernel, ususlly there won't be too much kernel log overlapping
the old ring buffer.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Lianbo Jiang <lijiang@redhat.com>
2020-10-27 17:34:15 +08:00
Lianbo Jiang
d4c6f56456 Doc: add a documentation for the usage of logger
Because the logger is introduced to output the kdump logs, need to
add a documentation for this change and describe how to use it.

Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
2020-10-27 17:34:11 +08:00
Lianbo Jiang
d7054f4cd8 Improve debugging in the kdump kernel
Let's use the logger in the second kernel and collect the kernel ring
buffer(dmesg) of the second kernel.

Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
2020-10-27 17:34:07 +08:00
Lianbo Jiang
88a8b94de9 kdumpctl: add the '-d' option to enable the kexec loading debugging messages
Currently, the kexec option '--debug/-d' is not enabled by default, which
means that users need to set it manually and wait for the next failure to
capture the additional information.

Therefore, let's enable the option '-d' for kexec loading by default.

Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
2020-10-27 17:34:03 +08:00
Lianbo Jiang
d001bd177f kdump.sysconfig: add the kdump logger configurations
The kdump logger has the default values of the log levels, but
sometimes, need to change the value of log level in order to
get more debugging messages for troubleshooting.

Here, user will have a chance to reconfigure it.

Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
2020-10-27 17:33:58 +08:00
Lianbo Jiang
3b743ae6ae enable the logger for kdump
Since the logger was introduced into kdump, let's enable it for kdump
so that we can output kdump messages according the log level and save
these messages for debugging.

Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
2020-10-27 17:33:54 +08:00
Lianbo Jiang
41b3da3996 introduce the kdump logger from the dracut
Currently, all messages are directly printed to the console, sometimes,
we also need to output these messages to the journal log according to
the log level.

In view of this, introduce the kdump logger from the dracut module.

Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
2020-10-27 17:31:54 +08:00
Kairui Song
08276e9f7a Rework check_config and warn on any duplicated option
Instead of read and parse the kdump.conf multiple times, only read once
and use a single loop to handle the error check, which is faster.

Also check for any duplicated config otion, and error out if there are
duplicated ones.

Now it checks for following errors, most are unchanged from before:
 - Any duplicated config options. (New added)
 - Deprecated/Invalid kdump config option.
 - Duplicated kdump target, will have a different error message of
   other duplicated config options.
 - Duplicated --mount options in dracut_args.
 - Empty config values. All kdump configs should be in
   "<config_opt> <config_value>" format.
 - Check If raw target is used in fadump mode.

And removed detect of lines start with space, it will not break kdump
anyway.

The performance is measurable better than before for the check_config
function.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
2020-10-27 17:07:54 +08:00
Pingfan Liu
21f5f95b06 kdump-lib.sh: detect secure boot on s390
On s390, if Secure-IPL is enabled, then "kexec -s -l" is required.
Otherwise kdump kernel can not be loaded.

Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
2020-10-19 16:45:17 +08:00
Kairui Song
041ba89902 Don't drop journalctl content if failure action is "shell"
If failure action is set to "shell", user will need more debug info
available in kdump kernel. Especially when serial console is not
available, manually retrieve the log from journalctl is very useful
for debugging kdump issue.

Else, we can still drop journalctl content to save memory assuming
nothing will use it.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Lianbo Jiang <lijiang@redhat.com>
2020-09-17 10:43:07 +08:00
Kairui Song
bcaa4358b1 dracut-module-install: Move systemd conf install code to a function
No feature change.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Lianbo Jiang <lijiang@redhat.com>
2020-09-17 10:43:07 +08:00
Kairui Song
c44cdb6703 selftest: Show the path of dumped vmcore on test end
Make the test script print following line when the test is finished and vmcore is successfully dumped:

  You can retrive the verify the vmcore file using following command:
  ./scripts/copy-from-image.sh \
      /home/kasong/fedpkg/kexec-tools/tests/output/ssh-kdump/0-server.img \
      /var/crash/192.168.77.62-2020-09-02-05:16:26/vmcore.flat ./
  Kernel package verion is: kernel-core-5.6.6-300.fc32.x86_64

Also add a helper to copy files out of the VM image.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
2020-09-17 10:43:02 +08:00
Kairui Song
978a849765 selftest: Add document for selftests
Acked-by: Dave Young <dyoung@redhat.com>
2020-09-17 10:42:58 +08:00
Kairui Song
a8dbd281f7 selftest: Add basic test framework
Now, by execute `make test-run` in tests/, kexec-tools sanity tests
will be performed using VMs. There are currently 3 test cases, for local
kdump, nfs kdump and ssh kdump.

For each test VM, the selftest framework will create a snapshot layer,
do setup as required by the test case, this ensure each test runs in a
clean VM.

This framework will install a custom systemd service that starts when
system have finished booting, and the service will do basic routine
(fetch and set boot counter, etc..), then call the test case which is
installed in /kexec-kdump-test/test.sh in VM.

Each VM will have two serial consoles, one for ordinary console usage,
one for the test communication and log. The test script will watch the
second test console to know the test status.

The test cases are located in tests/scripts/testcases, documents about
the test cases structure will be provided in following commits.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
2020-09-17 10:42:54 +08:00
Kairui Song
2457f22baf selftest: Add basic infrastructure to build test image
The Makefile In tests/ could help build a VM image using Fedora cloud
image as base image, or, user can specify a base image using
BASE_IMAGE=<path/to/file>. The current repo will be packeged and
installed in the image, so the image could be used as a test image to
test kexec-tools.

The image building is splited into two steps:
The first step, it either convert the base image to qcow2 or create
a snapshot on it, and install basic packages (dracut, grubby, ...)
and do basic setups (setup crashkernel=, disable selinux, ...).
See tests/scripts/build-scripts/base-image.sh for detail.

The second step, it creates a snapshot on top of the image produced by
the previous step, and install the packaged kexec-tools of current
repo.  See tests/scripts/build-scripts/test-base-image.sh for detail.

In this way, if repo's content is changes, `make` will detect it and
only rebuild the second snapshot which speed up the rebuild by a lot.

The image will be located as tests/output/test-base-image, and in qcow2
format. And default user/password is set to root/fedora.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
2020-09-17 10:42:34 +08:00