Commit Graph

1191 Commits

Author SHA1 Message Date
Pingfan Liu
8bef2d943b kexec-tools.spec: make the existence of pre.d and post.d mandatory
If the directory /etc/kdump/pre.d is optional, then it is hard
to tell between the following cases:
  -1. no directory "/etc/kdump/pre.d"
  -2. "rm -rf /etc/kdump/pre.d", which removes all scripts under pre.d
For the second case, kdump.img should be rebuilt.

To bail out from this corner case, always creating pre.d and post.d
during rpm installation.

Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
2020-07-20 16:12:04 +08:00
Hari Bathini
5521a70cc7 ppc64/kdump: use kexec_file_load when secureboot is enabled
In secure boot mode, kexec_load syscall is disabled. So, if in secure
boot mode, load kdump kernel with kexec_file_load syscall instead.

Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
2020-07-20 15:21:54 +08:00
Kairui Song
070d362846 Release 2.0.20-14
Also fix a timestamp issue.

Signed-off-by: Kairui Song <kasong@redhat.com>
2020-07-01 17:15:38 +08:00
Lianbo Jiang
8884af04b6 s390x: enable the kexec file load by default
Let's use the option KEXEC_ARGS="-s" to enable the kexec file load
by default.

Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
2020-07-01 17:07:46 +08:00
Lianbo Jiang
e5cbf4d1ad x86_64: enable the kexec file load by default
Let's use the option KEXEC_ARGS="-s" to enable the kexec file load
by default.

Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
2020-07-01 17:07:46 +08:00
Lianbo Jiang
bf20a948ef Revert "s390x: add kdump sysconfig option to use the kexec_file_load() syscall"
This reverts commit 66ff48ed68.

Since the code logic is changed(which related to the option
'KDUMP_FILE_LOAD=on|off'), accordingly, need to remove the
option from the sysconfig file.

Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
2020-07-01 17:07:46 +08:00
Lianbo Jiang
073646998f Revert "kdump-lib: switch to the kexec_file_load() syscall on x86_64 by default"
This reverts commit 6a20bd5447.

Let's restore the logic of secureboot status check, and remove the
option 'KDUMP_FILE_LOAD=on|off'. We will use the option KEXEC_ARGS="-s"
to enable the kexec file load later, which can avoid failures when
the secureboot is enabled.

Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
2020-07-01 17:07:46 +08:00
Pingfan Liu
f2824e7b74 kdump.conf: fix a grammar issue
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
2020-07-01 17:07:03 +08:00
Pingfan Liu
6dd2b9fb41 man: improve description about /etc/kdump/{pre.d,post.d}interface
The description is identical to the notes in kdump.conf

Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
2020-07-01 16:49:41 +08:00
Kairui Song
61d4fa72b5 mkdumprd: Improve the error message for umounted dump target
Since commit ca01cbd, mkdumprd can generate the initramfs even if dump
target is not mounted. But it will refuse to work if the dump target is
not marked as noauto, and not mounted.

If a target is not marked as noauto in fstab, it should be mounted
automatically on boot. If it's not mounted, most likely something is
wrong and user should fix that. So mkdumprd will just show an error
message and quit to avoid further failure.

This commit make the error message more clear and less confusing.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
2020-06-24 14:15:49 +08:00
Kairui Song
807f0b5e48 mkdumprd: Fix nfs detection in to_mount
If target is in format of UUID= or LABEL=, current code will fail as it
expects target to be the absoluta path to the block device. Check for
the fstype instead.

Also simplify the code logic.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
2020-06-23 09:34:53 +08:00
Kairui Song
a29de38da5 Always wrap up call to dracut get_persistent_dev function
Dracut get_persistent_dev function don't recognize UUID= or LABEL=
format, so caller should conver it to the path to the block device
before calling it. There is already such a helper
"kdump_get_persistent_dev", just move it to kdump-lib.sh and rename
it to reuse it,

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
2020-06-22 19:58:08 +08:00
Lianbo Jiang
66ff48ed68 s390x: add kdump sysconfig option to use the kexec_file_load() syscall
Currently, kernel has supported the kexec_file_load() syscall on s390x,
so I follow this commit:
6a20bd5447 ("kdump-lib: switch to the kexec_file_load() syscall on
x86_64 by default")

And also enable the kexec_file_load() syscall for kdump on s390x by
default.

Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
2020-06-15 18:20:20 +08:00
Kairui Song
5d4a5896c9 mkdumprd: Fix dracut error on multiple extra_modules
Dracut expects --add-drivers with a single parameter, so mkdumprd
should quota the module list.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
2020-06-15 17:42:35 +08:00
Kairui Song
1a5d44d7f4 Fix kdump failure when mount target specified by dracut_args
commit 61e0169 changed definition of dump_fs function, so
need to do a mount target conversion before calling it.

Signed-off-by: Kairui Song <kasong@redhat.com>
2020-06-11 14:00:54 +08:00
onitsuka.shinic@fujitsu.com
077a7f19bb kdump.conf: Specify /etc/kdump/{pre.d,post.d}interface
This patch discribe /etc/kdump/{pre.d,post.d} interface.

Signed-off-by: Shinichi Onitsuka <onitsuka.shinic@fujitsu.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
2020-06-11 13:18:26 +08:00
onitsuka.shinic@fujitsu.com
45e02e73fa dracut-kdump.sh: Execute the binary and script filesin /etc/kdump/{pre.d,post.d}
This patch executes the binary and script files in /etc/kdump/{pre.d,post.d}
just like kdump_pre or kdump_post directive written in /etc/kdump.conf.

Signed-off-by: Shinichi Onitsuka <onitsuka.shinic@fujitsu.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
2020-06-11 12:59:21 +08:00
onitsuka.shinic@fujitsu.com
bdd57a5864 kdumpctl: Check the update of the binary and script files in /etc/kdump/{pre.d,post.d}
This patch adds the binary and script files in /etc/kdump/{pre.d,post.d}
to modified checklist in order to update kdump initramfs when one adds
new scripts or binaries or removes the existing ones under
/etc/kdump/{pre.d, post.d}.

Signed-off-by: Shinichi Onitsuka <onitsuka.shinic@fujitsu.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
2020-06-11 12:59:15 +08:00
onitsuka.shinic@fujitsu.com
4246f26725 dracut-module-setup.sh: Install files under /etc/kdump/{pre.d,post.d} into kdump initramfs
This patch installs the binary and script files under /etc/kdump/{pre.d,post.d}
into new initramfs of kdump.

Signed-off-by: Shinichi Onitsuka <onitsuka.shinic@fujitsu.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
2020-06-11 12:58:48 +08:00
Kairui Song
e05c550144 Drop switch root capability for non fadump initramfs
Switch root is never used for kdump image, and this will be helpful to
reduce the initramfs size.

Also increase dracut dependency version and the function is
dracut_no_switch_root is new introduced.

This commit is applied to RHEL some time ago, but missing in Fedora as
Fedora's Dracut didn't backport this feature at that time. Now apply
this missing commit.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
2020-06-10 22:37:34 +08:00
Hari Bathini
8fcbb4d265 fadump: update fadump-howto.txt with some more troubleshooting help
On journaling filesystems like XFS, bootloader is likely to pick up
older initrd (without vmcore capture scripts) if system crashes right
after initrd update, as the bootloader (read GRUB) may not replay
filesystem log before reading the initrd from disk. Added steps to
workaround that problem.

Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
2020-06-04 11:45:05 +08:00
Pingfan Liu
47bdeee35d fadump-howto.txt: source it in spec file
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
2020-06-03 11:34:37 +08:00
Kairui Song
134c578463 Don't inherit swiotlb parameter form 1st kernel by default
Usually kdump kernel don't need a swiotlb. There are two cases known
common case a swiotlb is needed:

 - SEV/SME is active.
 - crashkernel high reservation is used, or there are crashkernel memory
   above 4G.

For both case, if swiotlb is set to a large value in first kernel, kdump
kernel will likely to fail with OOM. So don't inherit swiotlb parameter
by default.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
2020-05-29 10:56:31 +08:00
Kairui Song
0cc3b85d0d module-setup.sh: Add "rd.neednet" parameter if network is needed
Upstream dracut now use network-manager module by default and since
upstream commit 3dcaa97, network-manager expects user to pass "rd.neednet"
to indicate network is required.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
2020-05-28 16:26:06 +08:00
Kairui Song
cfd93e2b7e Revert "Add a hook to wait for kdump target in initqueue"
This reverts commit cee618593c.

Upstream dracut have provided a parameter for adding mandantory network
requirement by appending "rd.neednet" parameter, so we should use that
instead.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
2020-05-28 16:26:00 +08:00
Lianbo Jiang
a81ec90ed8 kdump.sysconfig: Remove the option 'log_buf_len' from kdump command line
The option 'log_buf_len' in the kernel command line will make kernel
dynamically allocate memory (system ram) from memblock and always hold
the memory allocated. If the size of this option is misused such as
'log_buf_len=64M'. This may cause a short of memory for kdump kernel,
because the size of memory is usually limited in kdump kernel.

In order to avoid this problem, need to remove this option from the
kdump kernel command line.

Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
2020-05-28 10:11:15 +08:00
Kairui Song
0094fb496d Release 2.0.20-13
Add support for using unmounted device as dump target, update docs, and
fix several known issue.

Signed-off-by: Kairui Song <kasong@redhat.com>
2020-05-22 19:29:19 +08:00
Kairui Song
73da3224be Fix an error in commit ca01cbd
Previous commit ca01cbd ('mkdumprd: generate usable kdump initramfs
even target is not mounted') applied a wrong version of patch, let me
just fix it.

Signed-off-by: Kairui Song <kasong@redhat.com>
2020-05-22 19:29:19 +08:00
Kairui Song
846e23a2d5 Update docs for the new noauto dump target support
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
2020-05-22 16:16:01 +08:00
Kairui Song
f2ffa3d340 kexec-kdump-howto.txt: Add some format to the document
When adding doc for the non-mounted dump target support, I found the
document are a bit uneasy to read due to lack of a proper format, this
commit should it make looks better.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
2020-05-22 16:14:13 +08:00
Kairui Song
ca01cbdfd5 mkdumprd: generate usable kdump initramfs even target is not mounted
Currently kexec-tools always depend on dump target to be mounted, which
caused some inconvenience for setup.

So for user configured target, allow kdump to start and build initramfs
even if target is not mounted.

When a mounted user configured target is used, the behavior is not
changed.

When a unmounted user configured target is used, mkdumprd will look for
corresponding mount info in fstab, and a entry with noauto option is
founded, mkdumprd will try to mount it inplace with optoins specified
in fstab and do basic checks on the device, then umount it.

If there is no fstab entry, mkdumprd will try to mount it in temporary
path with defaults option, do same basic check and umount it.

If there is a fstab entry but "noauto" option is not used, then there
must be some reason that the target device is not mounted, mkdumprd will
error out.

When path based target is used, there is no behavior change.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
2020-05-22 16:14:07 +08:00
Kairui Song
61e016939c User get_mount_info to replace findmnt calls
Use get_mount_info so that fstab is used as a failback when look for
mount info.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
2020-05-22 16:14:02 +08:00
Kairui Song
be578f6382 kdump-lib.sh: add fstab failback helper for getting mount info
This allows look up mount info even if target is not mounted.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
2020-05-22 16:13:57 +08:00
Kairui Song
70deeb474b Allow calling mkdumprd from kdumpctl even if targat not mounted
Ignore mount check in kdumpctl, mkdumprd will still fail building and
exit if target is not mounted.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
2020-05-22 16:13:49 +08:00
Kairui Song
0624148414 Add a is_mounted helper
Use is_mounted helper instaed of calling findmnt directly or checking if
"mount" value is empty.

If findmnt looks for fstab as well, some non mounted entry will also
return value. Required to support non-mounted target.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
2020-05-22 16:13:24 +08:00
Kairui Song
43ea36b3e8 Introduce get_kdump_mntpoint_from_target and fix duplicated /
User a helper to get the path to mount dump target in kdump kernel, and
fix duplicated '/' in the mount path problem.

Fixes: bz1785371
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
2020-05-22 16:13:02 +08:00
Kairui Song
539bff4083 Append both nofail and x-systemd.before to kdump mount target
By this point, there is still an unresolved vfs kernel issue that blocks
systemd from mounting the dump target properly from time to time. To
prevent systemd from failing by mounting the dump target, we can add
nofail option to the kdump mount point.

But adding nofail will wipe out default dependency of the mount point,
see commit 94a7b43, so systemd randomize the order of calling kdump.sh
and mounting the dump target and lead to unexpected behavior.
However we can use x-systemd.before to ensure the mount is done
in right order.

In dracut-kdump-capture.service, we have "After=initrd.target",
and look at dracut.bootup.7, systems start processing of fstab by
initrd-fs.target, so set "After=initrd.target" could ensure the mount
is ready before pre-pivot and kdump-capture service.

Now with both nofail and x-systemd.before=initrd.target, systemd
will try to mount the dump target before calling kdump, and even if the
mount failed, kdump.sh will still be called and try to mount again. See
dump_fs function, which will try to mount if the target is not mounted.
Kdump will only fail if both mount attemp fails.

Else if the kdump target mount failed or unstable, systemd will directly
jump to kdump failure action, and kdump fails.

This should improve the robustness in general with no other risk.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
2020-05-15 17:17:27 +08:00
Kairui Song
33e79681d9 Fix the problem that kdump prints redundant /
In second kernel, kdump always prints redundant '/':

kdump: saving to /sysroot//var/crash/127.0.0.1-2020-03-12-21:32:54/

Just trim it.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
2020-05-11 14:22:22 +08:00
Kairui Song
b82c35c842 Partially Revert "Don't mount the dump target unless needed"
This partially reverts commit 6dee286467.

There are reports that NFSv3 is failing after this commit, and after
more debug, I found NFSv4 may not work properly if
"nfs4_disable_idmapping" is set to 0.

The cause of the failure is that kdump.sh runs after dracut's pre-pivot
and clean up hook, many dracut module will install hooks to kill some
running services, so if the dump target requires a service to be running
but it's killed, mount will fail.

Dracut ensures the configured mount points are ready before pre-pivot.
After pre-pivot, any further mounting operation may not work as expected.

Although there is no report of other type of dump target failure except
NFSv3, it's better to revert this, to avoid other potential risk, and wait
for a proper fix for that systemd/kernel issue.

Else, this may bring more trouble for further development.

But still keep the change in kdump-lib-initramfs.sh for better
robustness.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
2020-05-11 14:22:22 +08:00
Hari Bathini
d624a0326f fadump: update fadump-howto.txt with some troubleshooting help
Add recommendations on how much memory is required for FADump. Also,
mention the optimizations applied to default initrd when FADump is
used and how to workaround it.

Signed-off-by: Hari Bathini <hbathini@linux.ibm.com>
Acked-by: Pingfan Liu <piliu@redhat.com>
2020-05-11 14:08:14 +08:00
Lianbo Jiang
ce0305d4f9 Add a new option 'rd.znet_ifname' in order to use it in udev rules
In most cases, it always provides a persistent MAC address. But for
the s390 Arch, sometimes, kernel could run in the LPAR mode and it
doesn't provide a persistent MAC address, which caused the kdump
failure.

Currently, some rules rely on the persistent MAC address, for the
above case, which won't work in kdump kernel because non-persistent
MAC could not match with udev rules.

To fix this issue, need to add a new option 'rd.znet_ifname' in order
to provide extra parameters such as 'ifname' and 'subchannels' for
some rules, which ensures kdump can also work appropriately without
the persistent MAC. Please refer to the following commit in dracut:

872eb69936bd ("95znet: Add a rd.znet_ifname= option")

Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
2020-04-27 18:21:22 +08:00
Kazuhito Hagio
74081a2b64 Don't unmount the dump target just after saving vmcore
Since commit 6dee286467 ("Don't mount the dump target unless needed"),
dump_fs function unmounts the dump target just after saving vmcore.

This broke the condition that it's mounted when executing "kdump_post",
which had been stable since RHEL5, and a certain tool which uses the
kdump_post hook to save information of 2nd kernel to the dump target
started to fail.

As unmounting it is done by systemd-shutdown before reboot without
the umount command as below, so let's don't unmount it in dump_fs.

  systemd-shutdown[1]: Unmounting file systems.
  [547]: Remounting '/sysroot' read-only in with options '(null)'.
  EXT4-fs (dm-0): re-mounted. Opts: (null)
  [548]: Unmounting '/sysroot'.

Signed-off-by: Kazuhito Hagio <k-hagio-ab@nec.com>
Acked-by: Kairui Song <kasong@redhat.com>
2020-04-27 16:30:59 +08:00
Pingfan Liu
f33f30eb61 dracut-module-setup.sh: fix breakage in get_pcs_fence_kdump_nodes()
pcs cluster and cluster cib-upgrade may throw some information and disturb
the parsing. Mute them

Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
2020-04-20 11:24:46 +08:00
Pingfan Liu
6348398743 dracut-module-setup.sh: ensure cluster info is ready before query
There is a race issue between "pcs" and "kdumpctl restart"

-1. set up cluster
 # pcs cluster setup --start mycluster node1 node2
 # pcs stonith create kdump fence_kdump pcmk_reboot_action="off"
 # pcs stonith level add 1 node1 kdump
 # pcs stonith level add 1 node2 kdump

-2. Then here comes the command _immediately_ in kdumpctl
 # pcs cluster cib

But due to some pcs internal mechanism, "pcs cluster cib" can not
fetch the updated info in time.

Fix these issue by forcing the upgrade of cib.

Signed-off-by: Pingfan Liu <piliu@redhat.com>
Acked-by: Kairui Song <kasong@redhat.com>
2020-04-08 15:46:06 +08:00
Kairui Song
b5b252ae27 Release 2.0.20-12
Signed-off-by: Kairui Song <kasong@redhat.com>
2020-04-02 01:48:46 +08:00
Kairui Song
3b09c4910d Remove adjust_bind_mount_path call
If user configured target is used, path should be used as the absolute
path within the dump target direct, and user should be fully aware of
the path structure within the target device. The adjust_bind_mount_path
call here make it very hard to control the behavior.

Especially, if it's a cross device bind mount, this will likely create a
invalid path in the target. And for atomic case, adjust_bind_mount_path call
here assumes user will always pass root device as the explicitly configured
dump target, which is not true.

If user configured target device is used, the path is always be the
absolute path inside of given target. If user don't know about the path
structure in the target device, then user should either use the path
based config, or carefully exam the target device before using it as a
dump target.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Lianbo Jiang <lijiang@redhat.com>
2020-03-30 22:06:46 +08:00
Kairui Song
bde4b7af3b No longer treat atomic/silverblue specially
This commit remove almost all special workaround for atomic, and treat
all bind mounts in any environment equally.

Use a helper get_bind_mount_directory_from_path to get the bind mount
source path of given path.

is_atomic function now only used to determine the right /boot path
for atomic/silverblue environment.

And remove get_mntpoint_from_path(), it's the only function that never
ignore bind mount, and it have no caller after this clean up.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Lianbo Jiang <lijiang@redhat.com>
2020-03-30 22:06:37 +08:00
Kairui Song
b5b0b90521 mkdumprd: Simplify handling of user specified target
For user specified target, the config value is used as the dump target,
and SAVE_PATH (path in kdump.conf) value is used as the dump path within
the dump target, no need to do anything extra with the path value.

Current code logic is not only complicated, it also wrongly generate
an redundantly long path in atomic/silverblue environment.

The right way is only check two things, and do nothing else:

 1. The path exists within the target;
 2. The target is large enough to hold to contain the vmcore.

Currently checking the target still requires it to be mounted so it will
error out if it's not mounted. Will implement some auto mount as next
step.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Lianbo Jiang <lijiang@redhat.com>
2020-03-30 22:06:28 +08:00
Kairui Song
fd7e7be483 mkdumprd: Use get_save_path instead of parsing config
get_save_path provides default value fail back and error check, no need
to repeat it again.

Also remove a redundant echo and grep in get_save_path

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Lianbo Jiang <lijiang@redhat.com>
2020-03-30 22:06:07 +08:00
Kairui Song
c1c7f004c8 Remove is_dump_target_configured
It's basically same with is_user_configured_dump_target and only have
one caller. And the name is confusing, the dump target is always
configured, it's either user configured or path based.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Lianbo Jiang <lijiang@redhat.com>
2020-03-30 22:05:31 +08:00