Commit Graph

2001 Commits

Author SHA1 Message Date
Brian C. Lane
ef45d169b0 Add some tests for creator.py
This is complicated by the fact that much of this module requires mount.
So for now just test the things that don't need mount.

(cherry picked from commit 134a333d92)
2019-05-03 16:38:47 -07:00
Brian C. Lane
c63b2e3881 tests: Add executils test
(cherry picked from commit fb87db0e1e)
2019-05-03 16:38:38 -07:00
Brian C. Lane
c6b9a72778 tests: Add sysutils test
(cherry picked from commit cc69478249)
2019-05-03 16:38:33 -07:00
Brian C. Lane
c34a0fb535 tests: Add discinfo test
(cherry picked from commit 2e36e941ce)
2019-05-03 16:38:28 -07:00
Brian C. Lane
c5e839838f tests: Add treeinfo test
(cherry picked from commit e8fd8a496b)
2019-05-03 16:38:22 -07:00
Brian C. Lane
4c71adadd5 Stop using build to run the tests, allow using podman
To use podman run the tests like this:
DOCKER=podman make test-in-docker

This now builds the welder/lorax-tests image as a separate step from
running the tests.

Running the tests uses the welder/lorax-tests image and mounts the
source directory read-only, copies it into /lorax-test/ and runs the
tests from there.

(cherry picked from commit 8a26d0648e)
2019-05-03 16:38:08 -07:00
Alexander Todorov
a2797e8f1b new test: build and deploy an image in OpenStack
(cherry picked from commit 6a42570eea)
2019-05-03 16:37:16 -07:00
Alexander Todorov
cbebbce0d5 Fix typos in VM_NAME and cleanup command
(cherry picked from commit 6005f98123)
2019-05-03 16:37:10 -07:00
Alexander Todorov
71ccb55985 new test: build and deploy images on vSphere
(cherry picked from commit 9906ad6da3)
2019-05-03 16:37:04 -07:00
Alexander Todorov
4c41780856 Update docs with info about ssh keys
it looks like the key value expects the entire content of
~/.ssh/id_rsa.pub

(cherry picked from commit 6631002a81)
2019-05-03 16:36:58 -07:00
Alexander Todorov
c0d51e9e99 new test: build and deploy images on AWS
(cherry picked from commit efaf2bf793)
2019-05-03 16:36:50 -07:00
Alexander Todorov
019aa20d9f Disable execution of new tests which need Docker privileged mode
(cherry picked from commit 7d2b9d2d30)
2019-05-03 16:36:44 -07:00
Alexander Todorov
7c38d74c00 New tests: build ext4-filesystem and partitioned-disk composes
for the moment we just make sure these two can be built without
visible errors.

(cherry picked from commit 2eb62014db)
2019-05-03 16:36:38 -07:00
Marek Marczykowski-Górecki
a8d8a4a2ef Add --squashfs-only option to drop inner rootfs.img layer
Make runtime directly into squashfs image. This reduces largely
unreproducible ext4 layer, but requires anaconda's dracut module
modification to properly mount the image.

(cherry picked from commit 27e611629f)
2019-05-03 16:34:50 -07:00
Brian C. Lane
ccb11427d0 lorax: Log when SOURCE_DATE_EPOCH is used for the current time
(cherry picked from commit 5409748e75)
2019-05-03 16:34:42 -07:00
Marek Marczykowski-Górecki
e2366fd0c0 Drop non-determinism from default templates
Some files are created in non-reproducible way, including including
random data explicitly (/etc/machine-id), timestamps (fontconfig cache,
ldconfig aux-cache, certs cache), or entries in random order (groups,
systemd catalog, package list).
Fix this by either making the files reproducible, or removing them.

(cherry picked from commit fa2158c7a9)
2019-05-03 16:34:37 -07:00
Marek Marczykowski-Górecki
df5f67e690 Use SOURCE_DATE_EPOCH for volumeid of efi boot image
By default mkfs.mksdos choose volume id based on current time. If
SOURCE_DATE_EPOCH is set, use that instead.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
(cherry picked from commit de8124366e)
2019-05-03 16:34:32 -07:00
Marek Marczykowski-Górecki
d04d176cb4 Preserve timestamps when building fs image
Even when FS do not support owner/modes, preserve timestamps.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
(cherry picked from commit e7f45d333f)
2019-05-03 16:34:26 -07:00
Marek Marczykowski-Górecki
376eda543c Use SOURCE_DATE_EPOCH for metadata timestamps
This include .buildinfo, .treeinfo and .discinfo.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
(cherry picked from commit 876ec52215)
2019-05-03 16:34:21 -07:00
Brian C. Lane
ab1effb98d Automatic commit of package [lorax] release [29.27-1].
Created by command:

/bin/tito tag
2019-02-28 10:55:13 -08:00
Brian C. Lane
7248355756 Add some extra cancel_func protection to QEMUInstall
In livemedia-creator's usage of this it can never pass in None, but if
someone were to import the library and use it, it would crash with
NoneType. So add the extra checks to make sure cancel_func isn't None,
just in case.

(cherry picked from commit 9041174142)
2019-02-28 09:38:15 -08:00
Yuval Turgeman
1c5b0d1237 installer: make sure cancel_func has a value (#612)
When using LMC to virt-install a system to an image, cancel_func is not
provided in run_creator, causing a TypeError (NoneType object is not
callable).

Signed-off-by: Yuval Turgeman <yturgema@redhat.com>
(cherry picked from commit 1c731b5618)
2019-02-28 09:38:06 -08:00
Brian C. Lane
03a2c09301 Automatic commit of package [lorax] release [29.26-1].
Created by command:

/bin/tito tag
2019-01-30 12:04:05 -08:00
Brian C. Lane
47a909209f Remove duplicate repositories from the sources list
In some cases when the host has, for whatever reason, multiple copies of
the same repo listed the build may fail with an error about running out
of space.

So this commit removes duplicate entries after the host's repos have been
loaded. It also adjusts some of the test repos to use different
temporary repo names for the tests.

(cherry picked from commit 98482e444d)
2019-01-30 08:43:37 -08:00
Brian C. Lane
80e35d8d0e lorax: Move default tmp dir to /var/tmp/lorax
If systemd's tmpfiles.d timer is executed while lorax is running it will
remove any files and directories older than 30 days. This is what has
been causing the occasional error where /proc/ would seem to vanish
during the install.

Upstream has proposed this solution, https://github.com/systemd/systemd/pull/11482
but until that is released we need a work-around to protect the lorax
files.

This commit does several things:

* Move the default tmpdir from /var/tmp/ to /var/tmp/lorax/
* Add a lorax.conf tmpfiles.d file that prevents systemd-tmpfiles from
  removing anything under /var/tmp/lorax/
* Add an exit handler to lorax so that temporary directories are removed on
  exit or on a python traceback.
* Use flock to lock access to the tempdir while lorax is running.
* Remove any unlocked tempdirs named /var/tmp/lorax/lorax.* at startup

Note that the exit handler will not remove the tempdir if lorax is
killed with a signal -- those are being caught by dnf and prevent the
exit handler from running.

systemd-tmpfiles cannot clean up the tempdirs at boot time because they
contain files labeled as shadow_t, so we have to remove those when lorax
runs. It uses the flock to prevent removing any directories created by
parallel instances of lorax and only removes ones that are unlocked.
Worst case they will be around until the first run of lorax after a
reboot.

If you want to keep the working directory around for debugging purposes
use --workdir /var/tmp/lorax/my-workdir and it won't be removed by
lorax.

(cherry picked from commit e4fe1aab32)
2019-01-29 13:58:19 -08:00
Brian C. Lane
97fa245e0c Automatic commit of package [lorax] release [29.25-1].
Created by command:

/bin/tito tag
2019-01-18 09:35:12 -08:00
Adam Williamson
36aeffc337 Don't exclude /dev from the setfiles in novirt_install
After a novirt disk image install, we run `setfiles` in the
install root to ensure some SELinux contexts are correct. /dev
is currently excluded from this run. However, as reported and
discussed in https://bugzilla.redhat.com/show_bug.cgi?id=1663040
it seems that with a recent systemd change, startup of many
services will fail if /dev itself is incorrectly labelled, and
in current Rawhide live images, it *is* incorrectly labelled.
Including `/dev` in this setfiles command appears to resolve the
problem in my testing.

Resolves: rhbz#1663040

Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit 2d3f266373)
2019-01-18 09:34:35 -08:00
Brian C. Lane
e57ec6a566 dracut-fips is no longer a subpackage, it is included in dracut. 2019-01-18 09:05:43 -08:00
Brian C. Lane
635c4adefd Automatic commit of package [lorax] release [29.24-1].
Created by command:

/bin/tito tag
2019-01-08 14:37:23 -08:00
Brian C. Lane
76071db226 Remove unneeded else from for/else loop. It confuses pylint
(cherry picked from commit 2950f2641b)
2019-01-08 13:47:21 -08:00
Brian C. Lane
486dd3f629 Turn off pylint warning about docstring with backslash
(cherry picked from commit 4fe21135e3)
2019-01-08 13:47:12 -08:00
Brian C. Lane
06db78c9c4 Turn off smartquotes in Sphinx documentation
It mangles the double dashes in the docstrings, and should close #155

(cherry picked from commit e85bcf4359)
2019-01-08 13:47:07 -08:00
Anthony F McInerney
efcfdbff4f fixes #543 qemu -nodefconfig deprecated
(cherry picked from commit f66bff5aa7)
2019-01-08 13:47:00 -08:00
Anthony F McInerney
eb9a591fbc fix spinx build warnings
(cherry picked from commit 6bb64f94ff)
2019-01-08 13:46:51 -08:00
David Shea
cfff807df8 Allow customizations to be specified as a toml list
Support both

  [customizations]
  hostname = "whatever"

and

  [[customizations]]
  hostname = "whatever"

in the blueprint data. The [[ syntax matches the other customization
directives (user, group, sshkey), and as such it's easy to accidentally
use it for the hostname without even realizing it's specifying something
different.

Add some tests for converting customizations to kickstarts.

(cherry picked from commit 35ab6a1336)
2019-01-08 13:46:08 -08:00
Brian C. Lane
9af706dbe3 Revert "lorax-composer: Cancel running Anaconda process"
Drop running pkill. This causes problems if more than one is running on
a system (eg. in parallel using mock). It can kill off other processes
unrelated to this instance of anaconda.

This reverts commit 42addfc2b5.
2019-01-08 13:45:42 -08:00
Brian C. Lane
6be0ca9bb0 Make sure cancel_func is not None
(cherry picked from commit ca2c3d9e77)
2019-01-07 14:11:35 -08:00
Brian C. Lane
bef4d9f4a8 Automatic commit of package [lorax] release [29.23-1].
Created by command:

/bin/tito tag
2018-12-17 16:48:29 -08:00
Brian C. Lane
251306e1ab lorax: Save information about rootfs filesystem size and usage
Run df on the filesystem image after it has been created.
Output will be in program.log, eg:

Running... df /var/tmp/lorax.imgutils.wm04pg_v
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/loop0       1998672 1619508    362780  82% /var/tmp/lorax.imgutils.wm04pg_v
Return code: 0
2018-12-17 16:24:49 -08:00
Brian C. Lane
1607e33227 Turn on signed tags when using tito.
This requires a patched version of tito at the moment, and doesn't hurt
to have it set -- it will just continue not signing the tags.
2018-12-17 16:23:31 -08:00
Brian C. Lane
42addfc2b5 lorax-composer: Cancel running Anaconda process
It ends up that this isn't as easy as you'd think. Anaconda sets up some
signal handlers to handle cleanly exiting, but they are not being run
when sent a TERM after package installation has started. I think DNF
resets them causing it to get ignored.

When the cancel is sent it can take several minutes for it to have an
effect. In my testing it usually takes around 2 minutes for anaconda to
notice and exit.

This sends a TERM to the process and then waits for it to exit. When it
returns it then removed any device-mapper devices that were setup for
image installations, removes any hanging loop devices.

It then kills off any process with pyanaconda. in the cmdline, and
anaconda-bus.conf (because anaconda starts a bunch of helpers and if it
doesn't shut down cleanly they remain running).

Resolves: rhbz#1656691
2018-12-17 16:22:23 -08:00
Brian C. Lane
850ad9845a Add cancel_func to virt and novirt_install functions
In addition to monitoring the logs for errors, call a function (or
functions) that tell it to cancel the anaconda process and cleanup.

Also check for a cancel after creating the squashfs image for live-iso
since that's a long running process.

This required adding a new argument to a number of existing functions,
passing it down to QEMUInstall and novirt_install where the function is
called.

Resolves: rhbz#1656691
2018-12-17 16:22:23 -08:00
Brian C. Lane
df6d7654bd lorax-composer: Check for STATUS before deleting
When there is no run or new symlink do one last check to make sure no
STATUS file was written. If it is missing, go ahead and remove the
results directory.

Related: rhbz#1656691
2018-12-17 16:22:23 -08:00
Brian C. Lane
5bf223c21a Check for existing CANCEL request, and exit on FINISHED
If another CANCEL request has already been made just exit from
uuid_cancel. If the build is FINISHED before it times out just exit,
don't remove the finished results.

Related: rhbz#1656691
2018-12-17 16:22:23 -08:00
Vendula Poncova
8721ce765d Fix vhd images
Images don't work without these fixes:

* Enable Network Manager.
* Disable cloud-init.
* Add Hyper-V modules into initramfs.

Fixes specific for RHEL:

* Create ifcfg-eth0 required by waagent.
* Install python3 and net-tools required by waagent.

Recommended changes:

* Use recommended kernel boot args.
* Disable kdump.
2018-12-11 08:30:06 -08:00
Brian C. Lane
40cfaa515c Automatic commit of package [lorax] release [29.22-1].
Created by command:

/bin/tito tag
2018-12-06 16:14:24 -08:00
Brian C. Lane
3f034a981d lorax-composer: Handle packages with multiple builds
When the repository has multiple arches, eg. i686 and x86_64, it should
add a new entry to the project's builds list, not create a new project
in the list.

This handles that by adding a modified insort_left function and
examining the packages returned from dnf to make sure they aren't
already listed in the results. It also handles adding them in sorted
order so that no further sorting needs to be done on the results.

Resolves: rhbz#1656642
(cherry picked from commit d18934775c)
2018-12-06 16:06:45 -08:00
Brian C. Lane
574360f7d0 lorax-composer: Check the queue and results at startup
If the system ran out of space, or was rebooted unexpectedly, the state
of the queue symlinks, or the results STATUS files may be inconsistent.
This checks them and:
 * Removes broken symlinks from queue/new and queue/run
 * Removes symlinks from run and sets the build to FAILED
 * Sets builds w/o a STATUS to FAILED
 * Sets builds with STATUS of RUNNING to FAILED
 * Creates missing queue/new symlinks to results with STATUS of WAITING

So, any builds that were running during the reboot will be FAILED, and
any that were waiting to be started will be started upon rebooting.

Resolves: rhbz#1647985
(cherry picked from commit 4dd9004d13)
2018-12-06 15:26:40 -08:00
Brian C. Lane
1d21957036 Automatic commit of package [lorax] release [29.21-1].
Created by command:

/bin/tito tag
2018-11-29 14:46:42 -08:00
Brian C. Lane
0c042afd34 New lorax documentation - 29.21 2018-11-29 11:12:11 -08:00