https://bugzilla.redhat.com/show_bug.cgi?id=1474706
Makedumpfile failed with below error messages, which is caused by kernel changes 65ade2f872b474fa8a04c2d397783350326634e6:
Buffer size for the cyclic mode: 95992
vtop4_x86_64: Can't get the symbol of init_level4_pgt.
readmem: Can't convert a virtual address(ffffffff8fe18284) to physical address.
readmem: type_addr: 0, addr:ffffffff8fe18284, size:390
check_release: Can't get the address of system_utsname.
Pull in Pratyush's fix in upstream makedumpfile (not merged yet but acked by
maintainer)
Signed-off-by: Dave Young <dyoung@redhat.com>
After the following systemd commit, "x-initrd.mount"
option became useless actually, we can safely remove
it now.
commit ce3f6d82b003f365f718f24e48f55b8a0372b924
Author: nmartensen <nis.martensen@web.de>
Date: Fri Jan 15 07:55:25 2016 +0100
fstab-generator: remove bogus condition
The sysroot mount is already taken care of by the
add_sysroot_mount function. With this condition
left in, we can we can get something like this:
initrd-root-fs.target.requires
`-- usr.mount -> /run/systemd/generator/usr.mount
in the main system (i.e., not in the initramfs). In
the initramfs, the previous condition already kicks in.
[snip]
"mount_in_initrd(me)" is true with "x-initrd.mount" option,
the behaviour of systemd fstab generator changed after the
above-mentioned patch, it always generates local mount units
required by local-fs.target regardless of "x-initrd.mount".
After failure, it enters dracut emergency, further triggers
kdump emergency service, thus there is no problem.
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
We met a problem on AMD machines, when using "nr_cpus=4" for
kdump, and crash happens on cpus other than cpu0, kdump kernel
will fail to boot and eventually reset.
After some debugging, we found that it stuck at the kernel path
do_boot_cpu()-> ... ->wakeup_secondary_cpu_via_init():
apic_icr_write(APIC_INT_LEVELTRIG|APIC_INT_ASSERT|APIC_DM_INIT,
phys_apicid);
that is, it stuck at sending INIT from AP to BP and reset, which
is actually what "disable_cpu_apicid=X" tries to solve. Printing
the value of @phys_apicid showed that it was the value of "apicid"
other that of "initial apicid" showed by /proc/cpuinfo.
As described in x86 specification:
"In MP systems, the local APIC ID is also used as a processor ID by the
BIOS and the operating system. Some processors permit software to modify
the APIC ID. However, the ability of software to modify the APIC ID is
processor model specific. Because of this, operating system software
should avoid writing to the local APIC ID register. The value returned by
bits 31-24 of the EBX register (when the CPUID instruction is executed with a
source operand value of 1 in the EAX register) is always the Initial APIC ID
(determined by the platform initialization). This is true even if software
has changed the value in the Local APIC ID register."
From kernel commit 151e0c7de("x86, apic, kexec: Add disable_cpu_apicid
kernel parameter"), we can see in generic_processor_info(), it uses
a)read_apic_id() and b)@apicid to compare with @disabled_cpu_apicid.
a)@apicid which is actually @phys_apicid above-mentioned is from the
following calltrace(on the problematic AMD machine):
generic_processor_info+0x37/0x300
acpi_register_lapic+0x30/0x90
acpi_parse_lapic+0x40/0x50
acpi_table_parse_entries_array+0x171/0x1de
acpi_boot_init+0xed/0x50f
The value of @apicid(from acpi MADT) is equal to the value of "apicid"
showed by /proc/cpuinfo as proved by our debug printk.
b)read_apic_id() gets the value from LAPIC ID register which is "apicid"
as well.
While the value of "initial apicid" is from cpuid instruction.
One example of "apicid" and "initial apicid" of cpu0 from /proc/cpuinfo
on AMD machine:
apicid : 32
initial apicid : 0
Therefore, we should assign /proc/cpuifo "apicid" to "disable_cpu_apicid=X".
We've never met such issue before, because we usually tested "nr_cpus=1",
and mostly on Intel machines, and "apicid" and "initial apicid" have the
same value in most cases on Intel machines.
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
I noticed that network is still enabled for local dumping,
like the following kdump boot message on my test machine
using local disk as the dump target:
tg3.c:v3.137 (May 11, 2014)
tg3 0000:02:00.0 eth2: Tigon3 [partno(BCM95720) rev
(PCI Express) MAC address c8:1f:66:c9:35:0d
tg3 0000:02:00.0 eth2: attached PHY is 5720C
After some debugging, found it due to a misuse in code below:
if [ is_generic_fence_kdump -o is_pcs_fence_kdump ]; then
_dep="$_dep network"
fi
The "if" condition always results in "true", and should be
changed as follows:
if is_generic_fence_kdump -o is_pcs_fence_kdump; then
_dep="$_dep network"
fi
After this, network won't be involved in non-network dumping,
as for dumpings require network such as nfs/ssh/iscsi/fcoe/etc,
dracut will add network accordingly. And kdump initramfs size
can be reduced from 24MB to 17MB tested on some real hardware,
and from 19MB to 14MB on my kvm. Moreover, it could avoid the
network (driver) initialization thereby saving us more memory.
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
In case of only network target, we can clearly and safely
remove more unnecessary modules to reduce initramfs size,
and to enhance stability.
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
In case of on dm related target, we can clearly and safely
remove many unnecessary modules to reduce initramfs size,
and to enhance stability.
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Resolves: bz1451717
https://bugzilla.redhat.com/1451717
When there is no crypt related kdump target, we can safely
omit "crypt" dracut module, this can avoid the pop asking
disk password during kdump boot in some cases.
This patch introduces omit_dracut_modules() before calling
dracut, we can omit more modules to reduce initrd size in
the future.
We don't want to omit any module for fadump, thus we move
is_fadump_capable() into kdump-lib.sh as a helper to use.
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Resolves: bz1451717
https://bugzilla.redhat.com/1451717
When there is any "rd.lvm.lv=X", "lvm" dracut module will
try to recognize all the lvm volumes which is unnecessary
and probably cause trouble for us.
See https://bugzilla.redhat.com/show_bug.cgi?id=1451717#c2
Remove all the rd.lvm.lv=X inherited from the kernel cmdline,
and generate the corresponding cmdline as needed for kdump.
Because prepare_cmdline() is only used by kdump, we don't need
to add any fadump judgement(also remove the existing judgement
in passing).
Currently, we don't handle "rd.lvm.vg=X", we can add it in
when there is some bug reported in the future.
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Resolves: bz1451717
https://bugzilla.redhat.com/1451717
Now that we have get_kdump_targets(), use it to simplify the code.
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Resolves: bz1451717
https://bugzilla.redhat.com/1451717
We need to know all the kdump targets including the dump
target and root in case of "dump_to_rootfs".
This is useful for us to do some extra work related to the
type of different targets.
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Resolves: bz1451717
https://bugzilla.redhat.com/1451717
This patch improves get_block_dump_target as follows:
-Consider block device in the special "--dracut-args --mount ..."
in get_user_configured_dump_disk().
-Consider save path instead of root fs in get_block_dump_target(),
and move it into kdump-lib.sh because we will have another user
in the following patch.
-For nfs/ssh dumping, there is no need to check the root device.
-Move get_save_path into kdump-lib.sh.
After this patch, get_block_dump_target() can always return the
correct block dump target specified.
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Description of problem
(https://bugzilla.redhat.com/show_bug.cgi?id=1465735):
Run `kdumpctl status` as normal user, get below error messages:
Another app is currently holding the kdump lock; waiting for it to exit...
flock: 9: Bad file descriptor
Another app is currently holding the kdump lock; waiting for it to exit...
flock: 9: Bad file descriptor
...
The bug is caused by behavior difference between bash
and sh (bash in posix).
In the function single_instance_lock in kdumpctl script,
there is
exec 9>/var/lock/kdump
which will fail in user mode. However, this fail will cause
script exiting under bash but not exiting under sh, causing
infinite loop because the flock will always fail.
According to the 16th item in
ftp://ftp.gnu.org/old-gnu/Manuals/bash-2.02/html_node/bashref_66.html
If a POSIX.2 special builtin returns an error status, a non-
interactive shell exits.
And according to
https://www.gnu.org/software/bash/manual/html_node/Special-Builtins.html
exec is one of the POSIX.2 special builtin's.
This patch fixes the bug by checking exec return value.
Fixes: 9fb2996d05 ("kdumpctl: change the shebang header to use /bin/bash")
Signed-off-by: Ziyue Yang <ziyang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Reviewed-by: Xunlei Pang <xlpang@redhat.com>
This patch fixes the whitespace errors reported by
'rpmlint' or 'fedpkg lint' when they are run on kexec-tools srpm:
kexec-tools.spec:242: W: mixed-use-of-spaces-and-tabs (spaces: line 107,
tab: line 242)
Signed-off-by: Bhupesh Sharma <bhsharma@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
The /sys/modules/*/drivers sysfs entries do not exist anymore on newer
kernels which means that the DRM moduels would never be included.
Instead check if there is any device with a "drm" sysfs directory to
decide on whether DRM modules need to be included.
Acked-by: Dave Young <dyoung@redhat.com>
fence_kdump_nodes should include list of cluster node(s) except localhost.
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Reviewed-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
kdump should not send fence_kdump notifications to local host, because
the role of the falied node (i.e local host) is to send fence_kdump
notifications to other nodes to tell them I'm kdumping, tell to itself is
nonsense. And we have excluded hostname of local host but when one use ip
address we also need exclude it.
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Reviewed-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
We met one issue that when changing softlink of "/usr/bin/sh"
to point to "ksh" instead of the default "bash", kdumpctl will
not work and go wrong.
kdumpctl is expected to run under bash like dracut, we should
change its shebang header from "#!/bin/sh" to "#!/bin/bash".
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
The "time kdumpctl start" command shows that strip_comments()
consumes lots of cpu time. By only calling it when necessary,
it saves us nearly half second.
Tested on my Fedora kvm machine.
Before this patch:
$ time kdumpctl start
kexec: loaded kdump kernel
Starting kdump: [OK]
real 0m1.849s
user 0m1.497s
sys 0m0.462s
After this patch:
$ time kdumpctl start
kexec: loaded kdump kernel
Starting kdump: [OK]
real 0m1.344s
user 0m1.195s
sys 0m0.195s
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Resolves: bz1449801
"cat $KDUMP_CONFIG_FILE|grep -v "^#"|while read ..." use pipes
to invoke subshells, as a result we met "the dreaded inaccessible
variables within a subshell problem" as described in the book
"Advanced Bash-Scripting Guide".
It cause regressions, so revert it.
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
I found using "cat $KDUMP_CONFIG_FILE|grep -v "^#"|while read ..."
instead of "while read ... do ...; done < $KDUMP_CONFIG_FILE" will
make the script run faster, it saves us nearly half second.
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Pratyush Anand <panand@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
We use faster "lsinitrd XXX -f usr/lib/dracut/build-parameter.txt"
instead of "lsinitrd XXX | grep "^Arguments:" | head -1", this can
save us around one second.
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Pratyush Anand <panand@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Use the logic of dracut 04watchdog/module-setup.sh to check,
then we only need to compare the content of 00-watchdog.conf,
so we can save one operation of lsinitrd.
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Pratyush Anand <panand@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
handle_mode_switch() can ensure the correct logic, so remove
the needless is_mode_switched(). This helps to save one slow
lsinitrd operation for each boot.
Improved backup_default_initrd() to judge DEFAULT_INITRD.
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Pratyush Anand <panand@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Some cloud people complained that VM boot speed is slower than
Ubuntu and other distributions, after some debugging, we found
that one of the causes is kdump service starts too slow(7seconds
according to the test result on the test VM), actually there is
no "crashkernel=X" specified. Although kdump service is parallel,
it affects the speed more or less especially on VMs with few cpus,
which is unacceptable. It is even worse when kdump initramfs is
built out in case of no reserved memory at first boot.
Commit afa4a35d3 ("kdumpctrl: kdump feasibility should fail if no
crash memory") can actually solve this issue.
This patch is a supplement of above-mentioned commit, we bail out
start() even earlier in case of no reserved memory.
Also made some cosmatic changes for check_crash_mem_reserved().
1) Before this patch
$ time kdumpctl start
No memory reserved for crash kernel.
Starting kdump: [FAILED]
real 0m0.282s
user 0m0.184s
sys 0m0.146s
2) After this patch
$ time kdumpctl start
No memory reserved for crash kernel
Starting kdump: [FAILED]
real 0m0.010s
user 0m0.008s
sys 0m0.001s
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Pratyush Anand <panand@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
This patch introduces the 'force_no_rebuild' option
inside the 'kdump.conf' and its handling inside the 'kdumpctl'
script.
There might be several use cases, where a system admin
decides that he doesn't need to rebuild the kdump initrd
and wants to use an existing version of the same. In such cases,
he can set the 'force_no_rebuild' option inside 'kdump.conf'
to 1, to force the 'kdumpctl' script not to rebuild the kdump
initrd.
Signed-off-by: Bhupesh Sharma <bhsharma@redhat.com>
Acked-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
The kdump-capture.service will fail, if the following conds are meet up.
-1. boot up a VM with the following cmd:
qemu-kvm -name 'avocado-vt-vm1' -sandbox off -machine pc -nodefaults -vga cirrus \
-drive id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=$guest_img \
-device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=04 \
-device virtio-net-pci,mac=9a:4d:4e:4f:50:51,id=id3DveCw,vectors=4,netdev=idgW5YRp,bus=pci.0,addr=05 \
-netdev tap,id=idgW5YRp \
-m 2048 \
-smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \
-cpu 'SandyBridge',+kvm_pv_unhalt \
-vnc :0 \
-rtc base=utc,clock=host,driftfix=slew \
-boot order=cdn,once=c,menu=off,strict=off \
-enable-kvm \
-monitor stdio \
-qmp tcp:localhost:4444,server,nowait
-2. in kernel cmdline with the following options: console=tty0 console=ttyS0,
Because the "-nodefaults" option in qemu cmd excludes the emulation of serial port, the ttyS0 will
have no real backend device. We can observe such issue in 1st kernel by:
echo teststring > /dev/console or
echo teststring > /dev/ttyS0,
It gets the error "-bash: echo: write error: Input/output error".
Such conds cause small issue in 1st kernel, but it is a big problem for kdump-capture and emergency
service.
This patch aims to work aroundthe issue in kdump-capture service:
dump_fs() return value will affect the following code in dracut-kdump.sh
DUMP_RETVAL=$? <---
do_kdump_post $DUMP_RETVAL
if [ $? -ne 0 ]; then
echo "kdump: kdump_post script exited with non-zero status!"
fi
Although kdump-capture saves the vmcore successfully, but it exit 1 and
fall on emergency service.
Signed-off-by: Pingfan Liu <piliu@redhat.com>
Reviewed-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
KASLR is to enhance security on OS kernel. While kdump kernel is
working after normal kernel corrupted. There's no need to do kaslr
in kdump kernel, so add 'nokaslr' to disable kaslr.
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
This reverts commit 892bea7aa
We already eliminated the root filesystem by removing "root=X"
in case of non-root dumping, and for livecd it must be non-root
dumping according to "live-image-kdump-howto.txt".
So it's time to revert this commit.
Also update "live-image-kdump-howto.txt", make sure users do not
configure "default dump_to_rootfs".
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Pratyush Anand <panand@redhat.com>
Acked-by:Dave Young <dyoung@redhat.com>
Since the current dracut of Fedora already supports not always
mounting root device, we can remove "root=X" from the command
line directly, and always get the dump target specified in
"/etc/kdump.conf" and mount it. If the dump target is located
at root filesystem, we will add the root mount info explicitly
from kdump side instead of from dracut side.
For example, in case of nfs/ssh/usb/raw/etc(non-root) dumping,
kdump will not mount the unnecessary root fs after this change.
This patch removes "root=X" via the "KDUMP_COMMANDLINE_REMOVE"
(if "default dump_to_rootfs" is specified, don't remove "root=X"),
and mounts non-root target under "/kdumproot", the root target
still under "/sysroot"(to be align with systemd sysroot.mount).
After removing "root=X", we now add root fs mount information
explicitly from the kdump side.
Changed check_dump_fs_modified() a little to avoid rebuild when
dump target is root, since we add root fs mount explicitly now.
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Pratyush Anand <panand@redhat.com>
Acked-by:Dave Young <dyoung@redhat.com>
For the following scripts:
cmdline="root=/dev/mapper/fedora-root rd.lvm.lv=fedora/root rw"
remove_cmdline_param $cmdline "root"
cmdline="root=nfs4:192.168.122.9:/ ip=ens3:dhcp rw"
remove_cmdline_param $cmdline "root"
The current implementation will get the wrong results:
"rd.lvm.lv=fedora/ rw"
":/ ip=ens3:dhcp rw"
After this patch we can get the correct results:
"rd.lvm.lv=fedora/root rw"
"ip=ens3:dhcp rw"
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Pratyush Anand <panand@redhat.com>
Acked-by:Dave Young <dyoung@redhat.com>
When kexec_crash_loaded does not exist, means kdump was not enabled in
kernel we get
$ kdumpctl status
cat: /sys/kernel/kexec_crash_loaded: No such file or directory
/usr/bin/kdumpctl: line 879: [: ==: unary operator expected
Kdump is not operational
After this patch:
$ kdumpctl status
Perhaps CONFIG_CRASH_DUMP is not enabled in kernel
Kdump is not operational
Signed-off-by: Pratyush Anand <panand@redhat.com>
Reviewed-by: Bhupesh Sharma <bhsharma@redhat.com>
Reviewed-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
We met a problem that the kdump emergency service failed to
start when the target dump timeout(we passed "rd.timeout=30"
to kdump), it reported "Transaction is destructive" messages:
[ TIME ] Timed out waiting for device dev-mapper-fedora\x2droot.device.
[DEPEND] Dependency failed for Initrd Root Device.
[ SKIP ] Ordering cycle found, skipping System Initialization
[DEPEND] Dependency failed for /sysroot.
[DEPEND] Dependency failed for Initrd Root File System.
[DEPEND] Dependency failed for Reload Configuration from the Real Root.
[ SKIP ] Ordering cycle found, skipping System Initialization
[ SKIP ] Ordering cycle found, skipping Initrd Default Target
[DEPEND] Dependency failed for File System Check on /dev/mapper/fedora-root.
[ OK ] Reached target Initrd File Systems.
[ OK ] Stopped dracut pre-udev hook.
[ OK ] Stopped dracut cmdline hook.
Starting Setup Virtual Console...
Starting Kdump Emergency...
[ OK ] Reached target Initrd Default Target.
[ OK ] Stopped dracut initqueue hook.
Failed to start kdump-error-handler.service: Transaction is destructive.
See system logs and 'systemctl status kdump-error-handler.service' for details.
[FAILED] Failed to start Kdump Emergency.
See 'systemctl status emergency.service' for details.
[DEPEND] Dependency failed for Emergency Mode.
This is because in case of root failure, initrd-root-fs.target
will trigger systemd emergency target which requires the systemd
emergency service actually is kdump-emergency.service, then our
kdump-emergency.service starts kdump-error-handler.service with
"systemctl isolate"(see 99kdumpbase/kdump-emergency.service, we
replace systemd's with this one under kdump).
This will lead to systemd two contradictable jobs queued as an
atomic transaction:
job 1) the emergency service gets started by initrd-root-fs.target
job 2) the emergency service gets stopped due to "systemctl isolate"
thereby throwing "Transaction is destructive".
In order to solve it, we can utilize "IgnoreOnIsolate=yes" for both
kdump-emergency.service and kdump-emergency.target. Unit with attribute
"IgnoreOnIsolate=yes" won't be stopped when isolating another unit,
they can keep going as expected in case be triggered by any failure.
We add kdump-emergency.target dedicated to kdump the similar way
as did for kdump-emergency.service(i.e. will replace systemd's
emergency.target with kdump-emergency.target under kdump), and
adds "IgnoreOnIsolate=yes" into both of them.
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Acked-by: Pratyush Anand <panand@redhat.com>
[bhe: improve the patch log about IgnoreOnIsolate="]
We replace "reserved_memory = XXXX"(default value is 8192) with
"reserved_memory = 1024" in /etc/lvm/lvm.conf used by "lvm2", it
can save 7MB peak memory consumption, so lower the possibility of
OOM under kdump.
For kdump, we don't have too many lvm targets, lvm2 locates in the
RAM(rootfs), so don't need that much memory, as discussed with lvm
people, they agreed that we use 1MB under kdump as long as there
are not that many lvm targets invloved.
We modify /etc/lvm/lvm.conf when "99kdumpbase" install() is executed,
because it is parsed after "90lvm" by dracut.
We add the code unconditionally with &>/dev/null to ignore errors, it
doesn't matter in case of "lvm" not included(i.e. there is no lvm.conf).
Signed-off-by: Xunlei Pang <xlpang@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
Dave Young
Fedora Release Engineering
Florian Weimer
Xunlei Pang
Ziyue Yang
Pratyush Anand
Bhupesh Sharma
Benjamin Berg
Pingfan Liu
Baoquan He