Chaowang measured the selinux load_policy memory usage, it need ~50M
It's too much under kdump 2nd kernel, it cause more OOM then before.
Here is the findings from Vivek:
- If we don't load policy or don't do restorecon, kernel automatically
uses a label for file as specified by file
/sys/fs/selinux/initial_contexts/file
On my system this value is "system_u:object_r:file_t:s0". Kernel
enforces this label on a file if it is not labeled. That's the reason
that you see above label on vmcore file when selinux policy was not
loaded in second kernel or restorecon was not done.
Note: I did some testing with rhel6 and there also I see file_t context.
Not sure why that's the case.
- Relabeling of root file system over boot happens if there is a file
/.autorelabel present. This file is touched by systemd service
fedora-autorelabel-mark.service. And this file comes from initscritps
package.
So if this service thinks that system was booted with selinux disabled
it will put this file on root and when next time system boots with
selinux enabled, relabeling is enforced by fedora-autorelabel.service
service.
- In our case relabeling is not happening after saving vmcore because
there does not seem be any fedora-autorelabel-mark.service running
from initramfs context. Looks like this service runs after switching
to real root.
Aug 08 10:44:13 vm9-f19 systemd[1]: Started Mark the need to relabel after reboot.
- selinux poicy is now loaded by systemd after root switch has taken
place.
Aug 08 10:44:10 vm9-f19 systemd[1]: Successfully loaded SELinux policy in 357.693ms.
So now we know that why selinux relabeling is not taking place. Reason
being that systemd service which marks the file system for autorelabeling
does not run from initramfs context.
And it might not make to run this service from initramfs context before
switch root. In general it makes sense to first switch to root, load
selinux policy if needed and then check whether to mark this filesystem
for relabel or not. Ideally root is mourted read only before that. It is
just that we break this rule for kdump. So as long as we make sure we
relabel files created by kdump after booting back, things should be fine.
Since we will relabel the vmcore dir after reboot so let's remove
the selinux dracut module dependency to avoid load_policy in 2nd kernel.
If in the future load_policy memory usage shrinks to an acceptable level
or there's a better solution we can add selinux load_policy back later.
Signed-off-by: Dave Young <dyoung@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
do_dump() takes care of dump procedure. It'll error out if failing to
save vmcore.
Signed-off-by: WANG Chao <chaowang@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
Currently in initrd, hardware clock is always considered to use UTC time
format and system time zone is also UTC. Thus system time isn't correct
if hw clock is localtime or we're using other time zone in real root.
To fix this, install /etc/adjtime and /etc/localtime to initrd.
Previously, this functionality was implemented in dracut base module:
commit 77364fd
Author: WANG Chao <chaowang@redhat.com>
base: setup correct system time and time zone in initrd
But some people complains about a normal boot initrd needs to rebuild
every time if time zone is changed. So let's fix it on our side.
Signed-off-by: WANG Chao <chaowang@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
2nd kernel has very limited memory. Allocating huge pages will probably
trigger OOM. So let's remove hugepages and hugepagesz kernel parameters
for 2nd kernel when 1st kernel are using them.
If user wants huge pages cmdline in 2nd kernel, he/she can still specify
it through KERNEL_COMMANDLINE_APPEND in /etc/sysconfig/kdump.
This patch adds a new function remove_cmdline_param(). It takes a list
of kernel parameters as its arguments and remove them from given kernel
cmdline.
update:
1. Add description of remove_cmdline_param() per Vivek.
2. Remove_cmdline_param() will take kernel cmdline as $1, then strip it
and print the result.
Signed-off-by: WANG Chao <chaowang@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
Currently when action_on_fail is enabled, the emergency_shell won't be called
either. In kdump even though user specify the default action as emergency_shell,
dracut still skip it. Now change the implementation of action_on_fail to depend
on a file which is created by kdump when making kdump initrd, then remove it
at the beginning of kdump. This can solve the explicit emergency_shell problem.
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
Acked-by: WANG Chao <chaowang@redhat.com>
This reverts commit 9e964ff4c6.
Currently, because of dracut implementation, in kdump 'default shell' will
call emergency_shell of dracut. If action_on_fail is enabled, emergency_shell
is skipped. Then 'default shell' won't work either.
Here revert the old commit 9e964ff4 so that take other implementation.
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
Acked-by: WANG Chao <chaowang@redhat.com>
Currently in kdump.sh, we redirect stdout to stderr, because dracut
pre-pivot service (which kdump.sh is running within) only output stderr
to console. That behavior is defined in dracut-pre-pivot.service:
[Service]
...
StandardInput=null
StandardOutput=syslog
StandardError=syslog+console
...
But during testing, it has been observed that systemd will cache stderr
buffer, and first record to syslog (and it's own journal), then copy the
logs to /dev/console. And this practice is somehow unexpected in our
kdump script. We may have suppressed stdout/stderr that hasn't been
write to /dev/console before we run a force reboot.
With this change of redirecting stdout/stderr to /dev/console, kdump.sh
will output everything immediately to console, not cached/hidden by
systemd.
Signed-off-by: WANG Chao <chaowang@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
Build on ppc/ppc64 failed after makedumpfile-1.5.4 is pulled, since the
variable vmap_area_list is not defined. Back port below commit from
upstream to add it.
commit 150b58eb299066c65ef7713a93effc35c00be03a
Author: Baoquan He <bhe@redhat.com>
Date: Mon Jul 15 20:37:14 2013 +0800
[PATCH] Add vmap_area_list definition for ppc/ppc64.
vmap_area_list is added to get vmalloc_start for ppc/ppc64, but its
definition is missing, now add them.
Signed-off-by: Baoquan He <bhe@redhat.com>
Currently some functions are used in subshell to assign string to a
variable. For example:
_mnt=$(to_mount "$1")
In this case if we call perror_exit in the subshell, subshell will exit
1, but the parent process (mkdumprd) won't exit.
So we should handle the exit code of a subshell if the subshell calls
perror_exit over a failure.
Signed-off-by: WANG Chao <chaowang@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
kvm virtio-blk device, for example /dev/vdb, doesn't have serial id by
default. So there's no persistent device node under /dev/disk/ for
/dev/vdb.
In case no persistent dev for dump target, we should use the original
device name directly, not failing the mkdumprd.
v2: update warn message from Vivek.
Signed-off-by: WANG Chao <chaowang@redhat.com>
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
dump_to_rootfs is a special handling of dump_fs. It's better we merge them
together to cleaup code.
Now dump_fs() function takes two types of $1, a mount point like
/sysroot or a dump target device like /dev/mapper/vg-lv_kdump.
v2: remove -F option in makedumpfile case from Vivek
Signed-off-by: WANG Chao <chaowang@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
After dump the vmcore, explicitly commit changed cache to disk in case
umount fail or chances we'll have an incomplete vmcore.
Signed-off-by: WANG Chao <chaowang@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
Percent signs in .spec files get expanded as macros. Currently in kexec-tools.spec,
'%{dist}' are appended to changelog item. This older changelog is not correctly for
rhel7 with this. Let's remove it to make it clearer.
stdout if line buffered, thus even it's redirect to stderr, it will not show
on console automaticly. Because monitor_dd_progress is only for rawdump
currently, so I think we can just use "echo" instead of "echo -n".
Another problem is sometimes CURRENT_SIZE does not get value when it's used
in $(($CURRENT_SIZE / 1048576)), fix this issue as well.
Signed-off-by: Dave Young <dyoung@redhat.com>
Acked-by: WANG Chao <chaowang@redhat.com>
When using makedumpfile as core_collector, makedumpfile will show its
own progress bar, it will mix with the monitor_dd_progress and cause confusion.
In this patch just call monitor_dd_progress when core_collector is not
makedumpfile
Signed-off-by: Dave Young <dyoung@redhat.com>
Acked-by: WANG Chao <chaowang@redhat.com>
Fedora 18 changes the way how to work with services in spec files.
It introduces new macros - %systemd_post, %systemd_preun and
%systemd_postun. These macros are functionally equivalent to the
manual scriptlets used in older versions of Fedora.
By using the new unified RPM macros the .spec file code is
simplified a lot.
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: WANG Chao <chaowang@redhat.com>
Currently umount fs happens right after saving vmcore. Therefore vmcore
isn't directly accessible in kdump_post script or shell. This patch moves
the umount fs operation down to the very last part, right before kdump
exits.
The patch adds a global variable MOUNTS to keep track which filesystem
is used. And umount these filesystems at do_default_action() and
do_final_action().
Signed-off-by: WANG Chao <chaowang@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
Certain dracut module will mount fs under real root(/sysroot/ or
$NEWROOT/). Thus root fs can not be umounted by `umount /sysroot/`.
We should use `umount -R /sysroot/` to recursively umount root and
its submounts.
v2: do the same for dump_fs() from Baoquan
Signed-off-by: WANG Chao <chaowang@redhat.com>
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
Replace $1/$2 with local variable names in dump_raw() and dump_ssh()
Signed-off-by: WANG Chao <chaowang@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
When makedumpfile failed, it could still generate a invalid vmcore. It's
better to suffix these invalid vmcore files with "-incomplete", as we do
in RHEL6.
Signed-off-by: WANG Chao <chaowang@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
set -x is removed and we'll have little output about the dumping
progress. So it's best to output some messages on the top level to let
user know what's going on.
Signed-off-by: WANG Chao <chaowang@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
Add a section for debugging tips. I had to spend some time to figure out
how to drop into a shell from kdump_pre/kdump_post scripts. So thought of
documenting that so that somebdoy else can save time.
Also add a method to log serial consoles for virtual machines. I was
loosing some of the messages over reboot. Logging to file is fine. This
can be helful in debugging.
v2: Add method to log serial console for virtual machines
v3: Fix serial console logging instructions.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
When directory is empty, echo * will output *, not empty string. That's
not intended.
Also it looks a little bit nicer now.
Signed-off-by: WANG Chao <chaowang@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
kernel has exported mac address for each interface, we can get it
directly instead of parsing the output from ip address show.
Signed-off-by: WANG Chao <chaowang@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
Currently we use runtime mac addr to 2nd kernel to setup bonding
interface. But Bonding master will modify its slaves' mac addr and
incorrect mac addr is passed to 2nd kernel. Thus dracut in 2nd kernel
can't find expected slaves and bonding will fail.
Fix this issue by using perm address.
Tested in Fedora 19 KVM guest configured bonding with two slaves.
Signed-off-by: WANG Chao <chaowang@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
Currently "set -x" is specified in dracut-kdump.sh and I see the script
execution commands by default on console while testing with F19. That's
not right. This should be done only if user asked for it. Remove it.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Acked-by: WANG Chao <chaowang@redhat.com>
When 1st kernel is using KMS and crash, 2nd kernel can't reset to
nomodeset and the screen is black. In this case, user can't observe the
boot/dump progress and run commands in shell.
So let's pull in drm dracut module to fix this.
Signed-off-by: WANG Chao <chaowang@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
When ssh dump, if user doesn't have write permission on save path
of server, the crash kernel can be loaded successfully, but finally
kdump will fail because write is not allowed.
Let's check it in the service start phase, if no write permission
print error message and exit.
For differentiation, change the name of old function mkdir_save_path
to mkdir_save_path_fs.
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
We shouldn't output what dracut module are used when rebuilding kdump
initrd. It's confusing to user.
And since we've introduced dracut_args in kdump.conf, we can safely
remove this mandatory -M and let user add as his/her need.
Signed-off-by: WANG Chao <chaowang@redhat.com>
Acked-by: Baoquan He <bhe@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
Dracut root fs is always mounted, but it's not guaranteed to success
because we are in crash/kdump context. So selinux policy can not only
depends on chroot load_policy.
Per discussion with Vivek and Selinux people, relabel kdump files
when the service restart.
Currently only below cases are considerd:
1. target mounted in 1st kernel
2. target mounted as rw, if user mount it as 'ro' they will have to
relabel the files by themselves.
3. save path is not masked, this means if /var/crash is mount to another
disk which is different from dump target it will not visible to user
so user need manually relabel them.
4. only local filesystem based targets.
Tested on F19 machine.
Tested local fs dump and network dump along with different save path
to address above mentioned cases.
Vivek: use function name is_dump_target_configured
use getfattr -m "security.selinux" instead of ".*"
Daniel: use restorecon instead of chcon.
dyoung: keep minix in local fs list since it has not been deperacated yet.
Vivek: wrap is_dump_target_configured checking in function path_to_be_relabeled
dyoung: use awk instead of cut to print config value for different
space delimeters
dyoung: mute df error message: `df $_mnt/$_path 2>/dev/null`
For nfs restorecon, since it will be in 3.11 kernel, we can add it when it's
ok in Fedora.
Signed-off-by: Dave Young <dyoung@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
depends on 98selinux if 1st kernel selinux is enabled so we can load_policy
and correctly label the vmcore/vmcore-dmesg files.
Since dracut always mount rootfs, the 98selinux will chroot and load_policy,
so this will be ok for Fedora. In case rootfs mount failure we have to check
and relable vmcore files, will add the kdumpctl relabeling code in another
patch.
add 'dracut_args --printsize' to /etc/kdump.conf, it shows below added size:
selinux install size: 16k
Tested on F19:
With this patch applied, vmcore selinux attr is ok.
v1->v2: use sestatus 2>/dev/null to mute error messages
Signed-off-by: Dave Young <dyoung@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
In dracut-kdump.sh, kdump did not umount rootfs after dump_to_rootfs, just
like dump_fs does. And in kdump, the FINAL_ACTION is "reboot -f", no umount
action is taken.
Even though "sync" has been executed, it's safer to take a "umount rootfs"
action. Anyway no harm to umount.
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
When doing kdump If the capture kernel crashes for some reason, the default
behavior appears to be hanging the system without rebooting. We at least
need an option to reset if the capture kernel crashes. Business critical
customers tend to want the system to reboot without manual intervention.
Kernel provides a parameter “panic=n” to solve such problem. If this parameter
is given, the capture kernel will reboot after n seconds in case it panics.
Now add this parameter into “KDUMP_COMMANDLINE_APPEND”, and set the default
waiting time value as 10 seconds.
It's tested on KVM f19, and passed.
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
For reserving a chunk of memory for kdump kernel, args need be
appended to kernel cmdline. For different Arch, different
bootloaders and related config files are used, it's a little
annoying. Using grubby, it can be very easy to append a single
arg to kernel cmdline, and can save words in howto document.
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: WANG Chao <chaowang@redhat.com>
Currenty the kexec-tools package contains udev rules for kdump
that reload kdump in case of memory or CPU hotplug:
$ cat /etc/udev/rules.d/98-kexec.rules
SUBSYSTESUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump.service"
SUBSYSTEM=="cpu", ACTION=="offline", PROGRAM="/bin/systemctl try-restart kdump.service"
SUBSYSTEM=="memory", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump.service"
SUBSYSTEM=="memory", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump.service"
On other architectures the rules are necessary because the memory
and CPU layout stored in the kdump in-memory ELF header at kdump
load time. Therefore the kdump kernel has to be reloaded each
time when the CPU or memory configuration changes.
This has drawbacks:
1. During kdump reload the system can't be dumped.
2. On systems with many hotplug events (e.g. on s390 with cpuplugd)
this creates significant overhead
The reload is not necessary on s390 because there the ELF header is
created in the 2nd (kdump) kernel. Therefore, to improve things,
remove the rules for s390.
Log is from IBM, and patch has been tested by IBM and work well.
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
Upon encountering a failure, dracut can drop user to emergency shell. But
in kdump environment kdump module wants to do the error handling and
wants to handle error as sepecified by user in kdump.conf file (halt,
reboot etc). Now dracut has provided an option action_on_fail=continue
which means dracut just ignores the failure and continues and expects
module to handle the error.
Modify kdump.sysconfig to pass action_on_fail=continue to dracut.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: WANG Chao <chaowang@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
In kdump.conf, space key is used as delimiter by default.
In kdump_install_conf of dracut-module-setup.sh, if specify
core_collector with a tab delimiter, the tool may not be
copied into kdump-initrd.
E.g, core_collector scp -v
And in dump_ssh of dracut-kdump.sh, dumping will fail caused
by tab key in core_collector.
Here change code to allow tab key as delimiter when specifying
core_collector.
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
dracut pre-pivot systemd service has below settings:
StandardOutput=syslog
StandardError=syslog+console
Thus kdump_pre/kdump_post output will disapear. Because the output is useful
for users, in case any failure user can watch the console log to see what's
wrong.
Dracut/Systemd people do not want to change the service settings.
So let's redirect the stdout to stderr fo fix it.
Per vivek: redirect whole kdump.sh stdout to stderr instead of only fix
for kdump_pre and kdump_post.
Tested on F19.
Signed-off-by: Dave Young <dyoung@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
Save vmcore-dmesg.txt before saving vmcore. For ssh targets, it assumes
that ssh is enabled. No scp logic as I don't have a local copy of
kernel log buffers and saving one will consume extra memory. We
can possibly enhance this logic to save kernel log buffers first locally
and then scp it (For setups which allow scp but disable ssh access).
(log is from Vivek Goyal <vgoyal@redhat.com>)
And add 1 section to describe it in kexec-kdump-howto.txt
v3->v4:
Remove old description of dmesg in kexec-kdump-howto.txt, now
add a new section to describe it, and note user kernel log
buffers won't be available if dump target is raw device.
Signed-off-by: Baoquan He <bhe@redhat.com>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
