rpms/kexec-tools
7
0
Fork 0
Code Issues Pull Requests Releases Activity

Throttle kdump reload request triggered by udev event

Browse Source 
Previously, kdump will restart / reload for many times on hotplug
event, especially memory hotplug events. Hotplugged memory may
generate many udev event as memory are managed and hotplugged in
small chunks by the kernel.

This results in unnecessary system workload and an actually longer
delay of kdump reload and the hotplug event, as udev will either
get blocked or kdumpctl will be waiting for other triggered operation.

To fix this, introduce a kdump-udev-throttler as an agent which will
be called by udev and merge concurrent kdump restart requests. Tested
with a Hyper-V VM which is failing due to udev timeout previously,
no new issues found.

Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
This commit is contained in:
Kairui Song 2018-10-30 18:35:34 +08:00
parent 227c18506c
commit 1c97aee728
3 changed files with 51 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
98-kexec.rules
View File

@ -7,6 +7,6 @@ GOTO="kdump_reload_end"
LABEL="kdump_reload"
RUN+="/usr/bin/systemd-run --no-block /usr/bin/kdumpctl reload"
RUN+="/usr/bin/systemd-run --no-block /usr/lib/udev/kdump-udev-throttler"
LABEL="kdump_reload_end"

47
kdump-udev-throttler Executable file
View File

@ -0,0 +1,47 @@
#!/bin/bash
# This util helps to reduce the workload of kdump service restarting
# on udev event. When hotplugging memory / CPU, multiple udev
# events may be triggered concurrently, and obviously, we don't want
# to restart kdump service for each event.
# This script will be called by udev, and make sure kdump service is
# restart after all events we are watching are settled.
# On each call, this script will update try to aquire the $throttle_lock
# The first instance acquired the file lock will keep waiting for events
# to settle and then reload kdump. Other instances will just exit
# In this way, we can make sure kdump service is restarted immediately
# and for exactly once after udev events are settled.
throttle_lock="/var/lock/kdump-udev-throttle"
interval=2
# Don't reload kdump service if kdump service is not started by systemd
systemctl is-active kdump.service &>/dev/null || exit 0
exec 9>$throttle_lock
if [ $? -ne 0 ]; then
echo "Failed to create the lock file! Fallback to non-throttled kdump service restart"
/bin/kdumpctl reload
exit 1
fi
flock -n 9
if [ $? -ne 0 ]; then
echo "Throttling kdump restart for concurrent udev event"
exit 0
fi
# Wait for at least 1 second, at most 4 seconds for udev to settle
# Idealy we will have a less than 1 second lag between udev events settle
# and kdump reload
sleep 1 && udevadm settle --timeout 3
# Release the lock, /bin/kdumpctl will block and make the process
# holding two locks at the same time and we might miss some events
exec 9>&-
/bin/kdumpctl reload
exit 0

3
kexec-tools.spec
View File

@ -29,6 +29,7 @@ Source24: kdump.sysconfig.ppc64le
Source25: kdumpctl.8
Source26: live-image-kdump-howto.txt
Source27: early-kdump-howto.txt
Source28: kdump-udev-throttler
#######################################
# These are sources for mkdumpramfs
@ -171,6 +172,7 @@ install -m 755 %{SOURCE23} $RPM_BUILD_ROOT%{_prefix}/lib/kdump/kdump-lib-initram
# For s390x the ELF header is created in the kdump kernel and therefore kexec
# udev rules are not required
install -m 644 %{SOURCE14} $RPM_BUILD_ROOT%{_udevrulesdir}/98-kexec.rules
install -m 755 %{SOURCE28} $RPM_BUILD_ROOT%{_udevrulesdir}/../kdump-udev-throttler
%endif
install -m 644 %{SOURCE15} $RPM_BUILD_ROOT%{_mandir}/man5/kdump.conf.5
install -m 644 %{SOURCE16} $RPM_BUILD_ROOT%{_unitdir}/kdump.service
@ -298,6 +300,7 @@ done
%config(noreplace,missingok) %{_sysconfdir}/kdump.conf
%ifnarch s390x
%config %{_udevrulesdir}
%{_udevrulesdir}/../kdump-udev-throttler
%endif
%{dracutlibdir}/modules.d/*
%dir %{_localstatedir}/crash