Commit Graph

2095 Commits

Author SHA1 Message Date
Brian C. Lane
e4fe1aab32 lorax: Move default tmp dir to /var/tmp/lorax
If systemd's tmpfiles.d timer is executed while lorax is running it will
remove any files and directories older than 30 days. This is what has
been causing the occasional error where /proc/ would seem to vanish
during the install.

Upstream has proposed this solution, https://github.com/systemd/systemd/pull/11482
but until that is released we need a work-around to protect the lorax
files.

This commit does several things:

* Move the default tmpdir from /var/tmp/ to /var/tmp/lorax/
* Add a lorax.conf tmpfiles.d file that prevents systemd-tmpfiles from
  removing anything under /var/tmp/lorax/
* Add an exit handler to lorax so that temporary directories are removed on
  exit or on a python traceback.
* Use flock to lock access to the tempdir while lorax is running.
* Remove any unlocked tempdirs named /var/tmp/lorax/lorax.* at startup

Note that the exit handler will not remove the tempdir if lorax is
killed with a signal -- those are being caught by dnf and prevent the
exit handler from running.

systemd-tmpfiles cannot clean up the tempdirs at boot time because they
contain files labeled as shadow_t, so we have to remove those when lorax
runs. It uses the flock to prevent removing any directories created by
parallel instances of lorax and only removes ones that are unlocked.
Worst case they will be around until the first run of lorax after a
reboot.

If you want to keep the working directory around for debugging purposes
use --workdir /var/tmp/lorax/my-workdir and it won't be removed by
lorax.
2019-01-29 13:57:12 -08:00
Jan Stodola
cc6fdb2fac Export OS_PROJECT_NAME variable in openstack scripts
The OS_PROJECT_NAME (or OS_TENANT_NAME) environment variable needs to be defined.
Use the OS_PROJECT_NAME, since it is recommended in the documentation instead of
the older OS_TENANT_NAME.
2019-01-29 12:51:30 +02:00
Jan Stodola
67dcff18d9 Collect results from all cleanup scripts
Correctly detect if any of the executed cleanup scripts didn't pass.
2019-01-29 10:43:43 +02:00
Alexander Todorov
d8439bdc11 Typo in PR template 2019-01-28 11:17:58 +02:00
Jan Stodola
961ea18145 Expand parameters as separate words 2019-01-25 21:34:25 +02:00
Alexander Todorov
de6419f0d1 Add PR template with instructions for repo members
this will serve as a reminder that sometimes Jenkins jobs can be
missing or failing and also lists the comments which team members
can use to trigger Jenkins jobs, especially for PRs from
non-members.
2019-01-24 22:50:00 +02:00
Jan Stodola
6680e2ffec Add script for removing old artifacts from OpenStack 2019-01-24 14:12:27 +02:00
Jan Stodola
677c858f60 Add script for removing old artifacts from AWS 2019-01-21 10:54:59 +02:00
Brian C. Lane
612f043b51 Automatic commit of package [lorax] release [30.12-1].
Created by command:

/bin/tito tag
2019-01-18 09:15:37 -08:00
Adam Williamson
2d3f266373 Don't exclude /dev from the setfiles in novirt_install
After a novirt disk image install, we run `setfiles` in the
install root to ensure some SELinux contexts are correct. /dev
is currently excluded from this run. However, as reported and
discussed in https://bugzilla.redhat.com/show_bug.cgi?id=1663040
it seems that with a recent systemd change, startup of many
services will fail if /dev itself is incorrectly labelled, and
in current Rawhide live images, it *is* incorrectly labelled.
Including `/dev` in this setfiles command appears to resolve the
problem in my testing.

Resolves: rhbz#1663040

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-18 09:13:52 -08:00
Brian C. Lane
4885434ac9 Automatic commit of package [lorax] release [30.11-1].
Created by command:

/bin/tito tag
2019-01-18 09:10:12 -08:00
Brian C. Lane
8a2e2a7a36 dracut-fips is no longer a subpackage, it is included in dracut. 2019-01-18 08:24:25 -08:00
Brian C. Lane
f9ade497d3 Automatic commit of package [lorax] release [30.10-1].
Created by command:

/bin/tito tag
2019-01-08 11:38:50 -08:00
Brian C. Lane
2950f2641b Remove unneeded else from for/else loop. It confuses pylint 2019-01-08 11:38:26 -08:00
Brian C. Lane
4fe21135e3 Turn off pylint warning about docstring with backslash 2019-01-08 11:37:56 -08:00
Brian C. Lane
e85bcf4359 Turn off smartquotes in Sphinx documentation
It mangles the double dashes in the docstrings, and should close #155
2019-01-08 10:20:38 -08:00
Anthony F McInerney
f66bff5aa7 fixes #543 qemu -nodefconfig deprecated 2019-01-08 10:19:00 -08:00
Anthony F McInerney
6bb64f94ff fix spinx build warnings 2019-01-08 10:17:44 -08:00
Brian C. Lane
7ee3b3d661 Revert "lorax-composer: Cancel running Anaconda process"
Drop running pkill. This causes problems if more than one is running on
a system (eg. in parallel using mock). It can kill off other processes
unrelated to this instance of anaconda.

This reverts commit 6b5c4df8b5.
2019-01-08 10:14:29 -08:00
Dan Horák
c53de4e63c set inst.stage2 for ppc64le image (rhbz#1577587) 2019-01-08 08:49:42 -08:00
David Shea
35ab6a1336 Allow customizations to be specified as a toml list
Support both

  [customizations]
  hostname = "whatever"

and

  [[customizations]]
  hostname = "whatever"

in the blueprint data. The [[ syntax matches the other customization
directives (user, group, sshkey), and as such it's easy to accidentally
use it for the hostname without even realizing it's specifying something
different.

Add some tests for converting customizations to kickstarts.
2019-01-08 10:21:07 -05:00
Brian C. Lane
ca2c3d9e77 Make sure cancel_func is not None 2019-01-07 13:53:44 -08:00
Dan Horák
3817671587 drop ppc/ppc64 from tests 2019-01-07 13:52:46 -08:00
Dan Horák
f9de60b2b7 drop ppc/ppc64 from spec 2019-01-07 13:52:46 -08:00
Dan Horák
e939469524 all supported arches have docker 2019-01-07 13:52:46 -08:00
Dan Horák
a477db33d6 drop big endian ppc/ppc64 support 2019-01-07 13:52:46 -08:00
Dan Horák
34ab13b68a add qemu command mapping for ppc64le 2019-01-07 13:52:46 -08:00
Dan Horák
d34acfc14c don't reduce initrd size on ppc64/ppc64le 2019-01-07 13:52:46 -08:00
Dan Horák
3625e8e27c fbset has been retired 2019-01-07 13:52:46 -08:00
Brian C. Lane
fd173f7265 Add timestamps to program.log and dnf.log
In lorax, lorax-composer, and livemedia-creator.
2018-12-19 12:13:24 -08:00
Brian C. Lane
3d1d79eea5 Automatic commit of package [lorax] release [30.9-1].
Created by command:

/bin/tito tag
2018-12-17 16:42:57 -08:00
Brian C. Lane
6f141658d4 lorax: Save information about rootfs filesystem size and usage
Run df on the filesystem image after it has been created.
Output will be in program.log, eg:

Running... df /var/tmp/lorax.imgutils.wm04pg_v
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/loop0       1998672 1619508    362780  82% /var/tmp/lorax.imgutils.wm04pg_v
Return code: 0
2018-12-17 16:24:23 -08:00
Brian C. Lane
fee5d4594c Turn on signed tags when using tito.
This requires a patched version of tito at the moment, and doesn't hurt
to have it set -- it will just continue not signing the tags.
2018-12-17 16:23:25 -08:00
Brian C. Lane
6b5c4df8b5 lorax-composer: Cancel running Anaconda process
It ends up that this isn't as easy as you'd think. Anaconda sets up some
signal handlers to handle cleanly exiting, but they are not being run
when sent a TERM after package installation has started. I think DNF
resets them causing it to get ignored.

When the cancel is sent it can take several minutes for it to have an
effect. In my testing it usually takes around 2 minutes for anaconda to
notice and exit.

This sends a TERM to the process and then waits for it to exit. When it
returns it then removed any device-mapper devices that were setup for
image installations, removes any hanging loop devices.

It then kills off any process with pyanaconda. in the cmdline, and
anaconda-bus.conf (because anaconda starts a bunch of helpers and if it
doesn't shut down cleanly they remain running).

Resolves: rhbz#1656691
2018-12-17 16:10:04 -08:00
Brian C. Lane
26a684932b Add cancel_func to virt and novirt_install functions
In addition to monitoring the logs for errors, call a function (or
functions) that tell it to cancel the anaconda process and cleanup.

Also check for a cancel after creating the squashfs image for live-iso
since that's a long running process.

This required adding a new argument to a number of existing functions,
passing it down to QEMUInstall and novirt_install where the function is
called.

Resolves: rhbz#1656691
2018-12-17 16:10:04 -08:00
Brian C. Lane
3762264f56 lorax-composer: Check for STATUS before deleting
When there is no run or new symlink do one last check to make sure no
STATUS file was written. If it is missing, go ahead and remove the
results directory.

Related: rhbz#1656691
2018-12-17 16:10:04 -08:00
Brian C. Lane
fa461dc837 Check for existing CANCEL request, and exit on FINISHED
If another CANCEL request has already been made just exit from
uuid_cancel. If the build is FINISHED before it times out just exit,
don't remove the finished results.

Related: rhbz#1656691
2018-12-17 16:10:04 -08:00
Alexander Todorov
883a959442 tests: use the first IP address if more than 1 retruned from OpenStack 2018-12-08 02:31:06 +02:00
Alexander Todorov
6563a33f63 tests: remove a debugging command 2018-12-08 02:31:06 +02:00
David Shea
d619430d6d Add openstack to the image type list in the docs 2018-12-07 15:11:27 -05:00
Brian C. Lane
6524bb0985 Automatic commit of package [lorax] release [30.8-1].
Created by command:

/bin/tito tag
2018-12-06 16:09:24 -08:00
Brian C. Lane
d18934775c lorax-composer: Handle packages with multiple builds
When the repository has multiple arches, eg. i686 and x86_64, it should
add a new entry to the project's builds list, not create a new project
in the list.

This handles that by adding a modified insort_left function and
examining the packages returned from dnf to make sure they aren't
already listed in the results. It also handles adding them in sorted
order so that no further sorting needs to be done on the results.

Resolves: rhbz#1656642
2018-12-06 15:48:58 -08:00
Brian C. Lane
4dd9004d13 lorax-composer: Check the queue and results at startup
If the system ran out of space, or was rebooted unexpectedly, the state
of the queue symlinks, or the results STATUS files may be inconsistent.
This checks them and:
 * Removes broken symlinks from queue/new and queue/run
 * Removes symlinks from run and sets the build to FAILED
 * Sets builds w/o a STATUS to FAILED
 * Sets builds with STATUS of RUNNING to FAILED
 * Creates missing queue/new symlinks to results with STATUS of WAITING

So, any builds that were running during the reboot will be FAILED, and
any that were waiting to be started will be started upon rebooting.

Resolves: rhbz#1647985
2018-12-06 15:13:28 -08:00
Alexander Todorov
642b909d24 Teach OpenStack test to distinguish between RHEL and Fedora 2018-12-06 22:15:49 +02:00
Alexander Todorov
543a4d88d9 Use full path for Azure playbook as well 2018-12-06 22:15:49 +02:00
Alexander Todorov
16260c1569 Use a temporary dir for ssh keys during testing
resolves path problems with sudo, we already have the same change
in the AWS test script.
2018-12-06 22:15:49 +02:00
Alexander Todorov
59050df710 Update V_DATASTORE b/c defaults appear to have been changed 2018-12-06 22:15:49 +02:00
Alexander Todorov
e0c8cdc5a8 Clone pyvmomi samples in the correct directory 2018-12-06 22:15:49 +02:00
Alexander Todorov
8f430be2a6 Use full path when pushing toml files during testing 2018-12-06 22:15:49 +02:00
Alexander Todorov
57b4f2e8f3 Add empty ci_after_success target for Jenkins
Jenkins uses templates to define all jobs which means they need to
have the same make targets even if the targets don't do anything.
2018-12-03 11:36:29 +02:00