Commit Graph

2073 Commits

Author SHA1 Message Date
Alexander Todorov
7b306e2e84 Use full path for Azure playbook as well
(cherry picked from commit 543a4d88d9)
2019-05-03 16:47:52 -07:00
Alexander Todorov
9584cbd96f Use a temporary dir for ssh keys during testing
resolves path problems with sudo, we already have the same change
in the AWS test script.

(cherry picked from commit 16260c1569)
2019-05-03 16:47:46 -07:00
Alexander Todorov
c4f3ceb0ec Update V_DATASTORE b/c defaults appear to have been changed
(cherry picked from commit 59050df710)
2019-05-03 16:47:38 -07:00
Alexander Todorov
41aa84ba04 Clone pyvmomi samples in the correct directory
(cherry picked from commit e0c8cdc5a8)
2019-05-03 16:47:10 -07:00
Alexander Todorov
a8ff02113b Use full path when pushing toml files during testing
(cherry picked from commit 8f430be2a6)
2019-05-03 16:47:00 -07:00
Alexander Todorov
80c18a6e63 Add empty ci_after_success target for Jenkins
Jenkins uses templates to define all jobs which means they need to
have the same make targets even if the targets don't do anything.

(cherry picked from commit 57b4f2e8f3)
2019-05-03 16:46:54 -07:00
Alexander Todorov
efa9974609 Implicitly specify ssh key directory/files for testing
this is to work around identity/sudo problems

(cherry picked from commit 4b4f4bd0e0)
2019-05-03 16:46:48 -07:00
Alexander Todorov
a907a3757d [test] Clean up containers.json
(cherry picked from commit ed63611cb7)
2019-05-03 16:46:39 -07:00
Alexander Todorov
eda242623b Teach AWS test to distinguish between RHEL and Fedora
(cherry picked from commit ab048947cb)
2019-05-03 16:46:32 -07:00
Alexander Todorov
8c409c8263 Use a temporary shared dir when testing
otherwise composer-cli is unable to glob() the kickstart
files and we're left without supported compose types. Seen
during AWS testing for example.

Helps with running some of the tests via sudo b/c this is
what Jenkins requires.

(cherry picked from commit b88466fd74)
2019-05-03 16:43:41 -07:00
Alexander Todorov
9c17c0cc6c Copy blueprints used for testing to temporary directory
If trying to execute test_cli.sh inside a git checkout
we are going to get the following exception:

Traceback (most recent call last):
  File "./src/sbin/lorax-composer", line 251, in <module>
    repo = open_or_create_repo(server.config["REPO_DIR"])
  File "/home/jenkins/lorax/src/pylorax/api/recipes.py", line 306, in open_or_create_repo
gi.repository.GLib.Error: ggit-error: failed to stat '/home/jenkins/lorax/tests/pylorax/blueprints': Permission denied (-1)
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib64/python3.7/multiprocessing/popen_fork.py", line 54, in _send_signal
    os.kill(self.pid, sig)

From what I can tell open_or_create_repo() is trying to initialize
a git repository inside the blueprints directory which fails when
we have an active git checkout.

This doesn't happen when we run the tests in Travis CI because
rsync excludes .git/ inside the Docker container.

(cherry picked from commit c9d706a382)
2019-05-03 16:43:35 -07:00
Alexander Todorov
178eb927d6 Add make targets for Jenkins
these targets help hooking up things in Jenkins and enable us to
perform build & deploy tests for cloud images.

NOTE: use sudo -E to preserve the environment
(cherry picked from commit 366ae55abe)
2019-05-03 16:43:28 -07:00
Alexander Todorov
2184cff59c Teach test_cli.sh to execute test scripts via arguments
this will be used to invoke scripts that build/push cloud images
without having to duplicate the setup/teardown/report parts!

(cherry picked from commit af2ae790ce)
2019-05-03 16:41:00 -07:00
Alexander Todorov
63e35921fa new test: build an image and deploy it on Azure
(cherry picked from commit 13fbd0861b)
2019-05-03 16:40:54 -07:00
Alexander Todorov
9b31c3cad9 Fix typo in comment
(cherry picked from commit d6dcb38a78)
2019-05-03 16:40:48 -07:00
Brian C. Lane
a0a76a800a Fix reporting of coverage results to coverall.io
When I re-arranged the test-in-docker I didn't realize how .travis.yml
was extracting the results. This should fix it.

When running with test-in-docker we mount the source read-only on
/linux-ro/ inside the container and copy it over to /lorax/ for running
the tests.

The local directory ./.test-results/ is mounted on /test-results/ in the
container and the .coverage file is copied into there so that it is
available on the host.

(cherry picked from commit b61a91954a)
2019-05-03 16:40:42 -07:00
Alexander Todorov
3fd9699e6d For OpenStack build image with rng-tools installed
appears to be a decent workaround for slow boot likely caused
by lack of entropy for ssh-keygen, see:
https://bugzilla.redhat.com/show_bug.cgi?id=1639326

(cherry picked from commit 6ae78fbd47)
2019-05-03 16:39:25 -07:00
Brian C. Lane
51a9c28bc6 Add tests for partitioned disk images
The setup for this is a bit complex, so it really ends up testing things
twice.

(cherry picked from commit fb4e6f2588)
2019-05-03 16:39:18 -07:00
Brian C. Lane
faad43ced8 Create a kpartx_disk_img function
So that it can be used in tests as well as in imgutils

(cherry picked from commit d69f01d4a8)
2019-05-03 16:39:12 -07:00
Brian C. Lane
e7310c7260 Add tests for pylorax.imgutils
Some of these can only run as root on a real system with access to loop
devices. They are skipped when running in a container.

(cherry picked from commit 063a1770e1)
2019-05-03 16:39:05 -07:00
Brian C. Lane
0aa206f661 Add tests to test_creator.py
Add a /.in-container file to the container root so that tests requiring root
and loop device support will be skipped when running in a container.

(cherry picked from commit bab4b20d0d)
2019-05-03 16:39:01 -07:00
Brian C. Lane
5fa5b8e142 Fix make_appliance and the libvirt.tmpl
Apparently nobody has used these since the switch to py3, xrange is now
range and it needs to read the file in binary mode when generating the
sha256.

(cherry picked from commit 8e749efbbf)
2019-05-03 16:38:54 -07:00
Brian C. Lane
ef45d169b0 Add some tests for creator.py
This is complicated by the fact that much of this module requires mount.
So for now just test the things that don't need mount.

(cherry picked from commit 134a333d92)
2019-05-03 16:38:47 -07:00
Brian C. Lane
c63b2e3881 tests: Add executils test
(cherry picked from commit fb87db0e1e)
2019-05-03 16:38:38 -07:00
Brian C. Lane
c6b9a72778 tests: Add sysutils test
(cherry picked from commit cc69478249)
2019-05-03 16:38:33 -07:00
Brian C. Lane
c34a0fb535 tests: Add discinfo test
(cherry picked from commit 2e36e941ce)
2019-05-03 16:38:28 -07:00
Brian C. Lane
c5e839838f tests: Add treeinfo test
(cherry picked from commit e8fd8a496b)
2019-05-03 16:38:22 -07:00
Brian C. Lane
4c71adadd5 Stop using build to run the tests, allow using podman
To use podman run the tests like this:
DOCKER=podman make test-in-docker

This now builds the welder/lorax-tests image as a separate step from
running the tests.

Running the tests uses the welder/lorax-tests image and mounts the
source directory read-only, copies it into /lorax-test/ and runs the
tests from there.

(cherry picked from commit 8a26d0648e)
2019-05-03 16:38:08 -07:00
Alexander Todorov
a2797e8f1b new test: build and deploy an image in OpenStack
(cherry picked from commit 6a42570eea)
2019-05-03 16:37:16 -07:00
Alexander Todorov
cbebbce0d5 Fix typos in VM_NAME and cleanup command
(cherry picked from commit 6005f98123)
2019-05-03 16:37:10 -07:00
Alexander Todorov
71ccb55985 new test: build and deploy images on vSphere
(cherry picked from commit 9906ad6da3)
2019-05-03 16:37:04 -07:00
Alexander Todorov
4c41780856 Update docs with info about ssh keys
it looks like the key value expects the entire content of
~/.ssh/id_rsa.pub

(cherry picked from commit 6631002a81)
2019-05-03 16:36:58 -07:00
Alexander Todorov
c0d51e9e99 new test: build and deploy images on AWS
(cherry picked from commit efaf2bf793)
2019-05-03 16:36:50 -07:00
Alexander Todorov
019aa20d9f Disable execution of new tests which need Docker privileged mode
(cherry picked from commit 7d2b9d2d30)
2019-05-03 16:36:44 -07:00
Alexander Todorov
7c38d74c00 New tests: build ext4-filesystem and partitioned-disk composes
for the moment we just make sure these two can be built without
visible errors.

(cherry picked from commit 2eb62014db)
2019-05-03 16:36:38 -07:00
Marek Marczykowski-Górecki
a8d8a4a2ef Add --squashfs-only option to drop inner rootfs.img layer
Make runtime directly into squashfs image. This reduces largely
unreproducible ext4 layer, but requires anaconda's dracut module
modification to properly mount the image.

(cherry picked from commit 27e611629f)
2019-05-03 16:34:50 -07:00
Brian C. Lane
ccb11427d0 lorax: Log when SOURCE_DATE_EPOCH is used for the current time
(cherry picked from commit 5409748e75)
2019-05-03 16:34:42 -07:00
Marek Marczykowski-Górecki
e2366fd0c0 Drop non-determinism from default templates
Some files are created in non-reproducible way, including including
random data explicitly (/etc/machine-id), timestamps (fontconfig cache,
ldconfig aux-cache, certs cache), or entries in random order (groups,
systemd catalog, package list).
Fix this by either making the files reproducible, or removing them.

(cherry picked from commit fa2158c7a9)
2019-05-03 16:34:37 -07:00
Marek Marczykowski-Górecki
df5f67e690 Use SOURCE_DATE_EPOCH for volumeid of efi boot image
By default mkfs.mksdos choose volume id based on current time. If
SOURCE_DATE_EPOCH is set, use that instead.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
(cherry picked from commit de8124366e)
2019-05-03 16:34:32 -07:00
Marek Marczykowski-Górecki
d04d176cb4 Preserve timestamps when building fs image
Even when FS do not support owner/modes, preserve timestamps.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
(cherry picked from commit e7f45d333f)
2019-05-03 16:34:26 -07:00
Marek Marczykowski-Górecki
376eda543c Use SOURCE_DATE_EPOCH for metadata timestamps
This include .buildinfo, .treeinfo and .discinfo.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
(cherry picked from commit 876ec52215)
2019-05-03 16:34:21 -07:00
Brian C. Lane
ab1effb98d Automatic commit of package [lorax] release [29.27-1].
Created by command:

/bin/tito tag
2019-02-28 10:55:13 -08:00
Brian C. Lane
7248355756 Add some extra cancel_func protection to QEMUInstall
In livemedia-creator's usage of this it can never pass in None, but if
someone were to import the library and use it, it would crash with
NoneType. So add the extra checks to make sure cancel_func isn't None,
just in case.

(cherry picked from commit 9041174142)
2019-02-28 09:38:15 -08:00
Yuval Turgeman
1c5b0d1237 installer: make sure cancel_func has a value (#612)
When using LMC to virt-install a system to an image, cancel_func is not
provided in run_creator, causing a TypeError (NoneType object is not
callable).

Signed-off-by: Yuval Turgeman <yturgema@redhat.com>
(cherry picked from commit 1c731b5618)
2019-02-28 09:38:06 -08:00
Brian C. Lane
03a2c09301 Automatic commit of package [lorax] release [29.26-1].
Created by command:

/bin/tito tag
2019-01-30 12:04:05 -08:00
Brian C. Lane
47a909209f Remove duplicate repositories from the sources list
In some cases when the host has, for whatever reason, multiple copies of
the same repo listed the build may fail with an error about running out
of space.

So this commit removes duplicate entries after the host's repos have been
loaded. It also adjusts some of the test repos to use different
temporary repo names for the tests.

(cherry picked from commit 98482e444d)
2019-01-30 08:43:37 -08:00
Brian C. Lane
80e35d8d0e lorax: Move default tmp dir to /var/tmp/lorax
If systemd's tmpfiles.d timer is executed while lorax is running it will
remove any files and directories older than 30 days. This is what has
been causing the occasional error where /proc/ would seem to vanish
during the install.

Upstream has proposed this solution, https://github.com/systemd/systemd/pull/11482
but until that is released we need a work-around to protect the lorax
files.

This commit does several things:

* Move the default tmpdir from /var/tmp/ to /var/tmp/lorax/
* Add a lorax.conf tmpfiles.d file that prevents systemd-tmpfiles from
  removing anything under /var/tmp/lorax/
* Add an exit handler to lorax so that temporary directories are removed on
  exit or on a python traceback.
* Use flock to lock access to the tempdir while lorax is running.
* Remove any unlocked tempdirs named /var/tmp/lorax/lorax.* at startup

Note that the exit handler will not remove the tempdir if lorax is
killed with a signal -- those are being caught by dnf and prevent the
exit handler from running.

systemd-tmpfiles cannot clean up the tempdirs at boot time because they
contain files labeled as shadow_t, so we have to remove those when lorax
runs. It uses the flock to prevent removing any directories created by
parallel instances of lorax and only removes ones that are unlocked.
Worst case they will be around until the first run of lorax after a
reboot.

If you want to keep the working directory around for debugging purposes
use --workdir /var/tmp/lorax/my-workdir and it won't be removed by
lorax.

(cherry picked from commit e4fe1aab32)
2019-01-29 13:58:19 -08:00
Brian C. Lane
97fa245e0c Automatic commit of package [lorax] release [29.25-1].
Created by command:

/bin/tito tag
2019-01-18 09:35:12 -08:00
Adam Williamson
36aeffc337 Don't exclude /dev from the setfiles in novirt_install
After a novirt disk image install, we run `setfiles` in the
install root to ensure some SELinux contexts are correct. /dev
is currently excluded from this run. However, as reported and
discussed in https://bugzilla.redhat.com/show_bug.cgi?id=1663040
it seems that with a recent systemd change, startup of many
services will fail if /dev itself is incorrectly labelled, and
in current Rawhide live images, it *is* incorrectly labelled.
Including `/dev` in this setfiles command appears to resolve the
problem in my testing.

Resolves: rhbz#1663040

Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit 2d3f266373)
2019-01-18 09:34:35 -08:00
Brian C. Lane
e57ec6a566 dracut-fips is no longer a subpackage, it is included in dracut. 2019-01-18 09:05:43 -08:00