Commit Graph

1966 Commits

Author SHA1 Message Date
Alexander Todorov
551723816c Allow overriding $CLI outside test scripts
this will allow you to test against installed RPM like so:

    # export CLI="/usr/bin/composer-cli"
    # make test_images

If you already have lorax-composer running then you can directly
execute test scripts:

    # ./tests/cli/test_build_and_deploy_aws.sh

Related: rhbz#1678937
2019-03-15 11:03:38 +02:00
Alexander Todorov
05f3bf0f97 Use make ci inside test-in-copy target
this minimizes the possibility of these two to diverge over time.
make ci is the default for Jenkins and will also be used for
internal gating tests.

Related: rhbz#1678937
2019-03-15 11:03:38 +02:00
Alexander Todorov
82492a54bc New test: Build live-iso and boot with KVM
explicitly enables sshd for live-iso during testing

Related: rhbz#1653934
2019-03-04 22:08:22 +02:00
Alexander Todorov
e78e0e610c New test: Build qcow2 compose and test it with QEMU-KVM
Related: rhbz#1653934
2019-03-04 13:18:35 +02:00
Jakub Vávra
5b4a2a47b8 Removed remnants of fedora branding. (#605)
Removed mentions of fedora in example live and atomic kickstarts.

Resolves: rhbz#1672583
2019-02-21 11:17:10 -08:00
Brian C. Lane
05aeb8fc24 Drop auth from the kickstart examples
System defaults should be sufficient these days.

Resolves: rhbz#1672583
2019-02-21 11:13:24 -08:00
Alexander Todorov
50febb4381 New test: Verify tar images with Docker and systemd-nspawn
- on some arches (also Fedora x86_64) systemd-nspawn may not be
  available
- delete composes from other tests in rlPhaseStartCleanup because
  we're seeing the tar compose kind of hanging in Jenkins and that
  test script is executed last so the slave may be running out of
  disk space. Be a good citizen and clean up after the previous
  tests.

Related: rhbz#1653934
2019-02-19 21:16:40 +02:00
Alexander Todorov
b6e86b4216 Update OpenStack flavor and network settings in tests
b/c we've migrated to Upshift we must use different instance type,
specify the desired network to connect to and update how we get
the ip address of the launched VM.

Related: rhbz#1653934
2019-02-18 12:15:37 +02:00
Jan Stodola
6d477a403a Use existing storage account
To avoid creating a new storage account for every new VM.

Related: rhbz#1653934
2019-02-06 19:19:35 +01:00
Jan Stodola
9d34481d9b Record date/time of VM creation
Related: rhbz#1653934
2019-02-06 19:19:35 +01:00
Jiri Kortus
e019ec2822 Make sure compose build tests run with SELinux in enforcing mode
Related: rhbz#1645189
2019-02-06 12:16:25 +01:00
Brian C. Lane
7d4a9ca0b1 Automatic commit of package [lorax] release [28.14.23-1].
Created by command:

/bin/tito tag
2019-01-30 10:08:44 -08:00
Brian C. Lane
6a55184c66 lorax: Move default tmp dir to /var/tmp/lorax
If systemd's tmpfiles.d timer is executed while lorax is running it will
remove any files and directories older than 30 days. This is what has
been causing the occasional error where /proc/ would seem to vanish
during the install.

Upstream has proposed this solution, https://github.com/systemd/systemd/pull/11482
but until that is released we need a work-around to protect the lorax
files.

This commit does several things:

* Move the default tmpdir from /var/tmp/ to /var/tmp/lorax/
* Add a lorax.conf tmpfiles.d file that prevents systemd-tmpfiles from
  removing anything under /var/tmp/lorax/
* Add an exit handler to lorax so that temporary directories are removed on
  exit or on a python traceback.
* Use flock to lock access to the tempdir while lorax is running.
* Remove any unlocked tempdirs named /var/tmp/lorax/lorax.* at startup

Note that the exit handler will not remove the tempdir if lorax is
killed with a signal -- those are being caught by dnf and prevent the
exit handler from running.

systemd-tmpfiles cannot clean up the tempdirs at boot time because they
contain files labeled as shadow_t, so we have to remove those when lorax
runs. It uses the flock to prevent removing any directories created by
parallel instances of lorax and only removes ones that are unlocked.
Worst case they will be around until the first run of lorax after a
reboot.

If you want to keep the working directory around for debugging purposes
use --workdir /var/tmp/lorax/my-workdir and it won't be removed by
lorax.

Resolves: rhbz#1668408
2019-01-30 09:19:24 -08:00
Jan Stodola
595f11cc10 Expand parameters as separate words
Related: rhbz#1653934
2019-01-28 12:28:23 +02:00
Alexander Todorov
bee2593a13 Add /usr/local/bin to PATH for tests
pip3 will install commands there and later scripts will not be able
to find them.

Related rhbz#1653934
2019-01-24 12:51:01 +02:00
Alexander Todorov
fa8a29972a Do not generate journal.xml from beakerlib
bacause this requires additional Python modules and we don't
really use it!

Related: rhbz#1653934
2019-01-24 12:51:01 +02:00
Alexander Todorov
f54c0bf104 Use any tmux version
the unittest suite is currently executed on Fedora where tmux is
version 2.8 but on RHEL 8 it is 2.7 and depsolve fails when
building images.

Related: rhbz#1653934
2019-01-24 12:51:01 +02:00
Alexander Todorov
b4c1e7c6a8 Make test scripts executable with chmod +x
because this is how Jenkins expects them to be

Related: rhbz#1653934
2019-01-24 12:51:01 +02:00
Brian C. Lane
9caa35776b Automatic commit of package [lorax] release [28.14.22-1].
Created by command:

/bin/tito tag
2019-01-11 14:50:49 -08:00
Brian C. Lane
c0c664bc5a Report an error if the blueprint doesn't exist
composer-cli uses TOML for 'blueprints save' which was returning an
empty 200 response if the blueprint didn't exist. Change this to return
a standard 400 error response if the blueprint doesn't exist.

composer-cli is already setup to handle receiving json when an error is
returned so just the toml API response for `blueprints/save` needed to
be changed.

Related: rhbz#1653934
2019-01-11 14:37:01 -08:00
Alexander Todorov
d61ba3a2cc tmux is version 2.8 on Fedora 28
Related: rhbz#1653934
2019-01-11 14:37:01 -08:00
Alexander Todorov
d72ee7bd89 Disable pylint no-member errors for 2 dnf constants
Related: rhbz#1653934
2019-01-11 14:37:01 -08:00
Alexander Todorov
55bdb8a27a Backport cloud image tests to RHEL 8
Related: rhbz#1653934

note: for now use Fedora 28 for the Docker container until
CentOS 8 is released or we figure out how to use subscriptions
for official RHEL 8 images.
2019-01-11 14:37:01 -08:00
Brian C. Lane
7dad328424 Remove unneeded else from for/else loop. It confuses pylint
(cherry picked from commit 2950f2641b)

Related: rhbz#1664485
2019-01-09 16:26:35 -08:00
David Shea
59e2da0a96 Allow customizations to be specified as a toml list
Support both

  [customizations]
  hostname = "whatever"

and

  [[customizations]]
  hostname = "whatever"

in the blueprint data. The [[ syntax matches the other customization
directives (user, group, sshkey), and as such it's easy to accidentally
use it for the hostname without even realizing it's specifying something
different.

Add some tests for converting customizations to kickstarts.

(cherry picked from commit 35ab6a1336)

Resolves: rhbz#1664485
2019-01-09 16:22:21 -08:00
Brian C. Lane
14d43a3d47 New lorax documentation - 28.14.21
Related: rhbz#1650295
2019-01-09 10:23:13 -08:00
Brian C. Lane
1a8591fdd5 Update the example livemedia-creator kickstarts for RHEL8
Rename them to rhel-*, update urls to reference BaseOS and AppStream.
rhel-livemedia will boot into a live GNOME desktop.

Resolves: rhbz#1650295
2019-01-09 09:03:27 -08:00
Brian C. Lane
f3638762fe Change default releasever to 8
Related: rhbz#1650295
2019-01-09 09:03:27 -08:00
Brian C. Lane
bcb8058dce Automatic commit of package [lorax] release [28.14.20-1].
Created by command:

/bin/tito tag
2019-01-08 16:15:32 -08:00
Brian C. Lane
bdc09df37b Revert "lorax-composer: Cancel running Anaconda process"
Drop running pkill. This causes problems if more than one is running on
a system (eg. in parallel using mock). It can kill off other processes
unrelated to this instance of anaconda.

This reverts commit 0ce6d3882b.

Related: rhbz#1656691
2019-01-08 14:47:25 -08:00
Brian C. Lane
0ed7167a03 Make sure cancel_func is not None
Related: rhbz#1656691

(cherry picked from commit ca2c3d9e77)
2019-01-07 14:10:11 -08:00
Brian C. Lane
47eecdfc85 Increase boot.iso rootfs to 3GiB
PPC64 runs out of space when installing into a 2GiB rootfs. Other arches
may also be getting close to running out of space.

Resolves: rhbz#1661169
2019-01-07 08:29:31 -08:00
Brian C. Lane
4f33181f2f Automatic commit of package [lorax] release [28.14.19-1].
Created by command:

/bin/tito tag
2018-12-18 15:17:10 -08:00
Brian C. Lane
efc57eae4c lorax: Save information about rootfs filesystem size and usage
Run df on the filesystem image after it has been created.
Output will be in program.log, eg:

Running... df /var/tmp/lorax.imgutils.wm04pg_v
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/loop0       1998672 1619508    362780  82% /var/tmp/lorax.imgutils.wm04pg_v
Return code: 0

Resolves: rhbz#1659625
2018-12-18 08:49:39 -08:00
Brian C. Lane
0ce6d3882b lorax-composer: Cancel running Anaconda process
It ends up that this isn't as easy as you'd think. Anaconda sets up some
signal handlers to handle cleanly exiting, but they are not being run
when sent a TERM after package installation has started. I think DNF
resets them causing it to get ignored.

When the cancel is sent it can take several minutes for it to have an
effect. In my testing it usually takes around 2 minutes for anaconda to
notice and exit.

This sends a TERM to the process and then waits for it to exit. When it
returns it then removed any device-mapper devices that were setup for
image installations, removes any hanging loop devices.

It then kills off any process with pyanaconda. in the cmdline, and
anaconda-bus.conf (because anaconda starts a bunch of helpers and if it
doesn't shut down cleanly they remain running).

Resolves: rhbz#1656691
2018-12-12 16:44:05 -08:00
Brian C. Lane
4b84475612 Add cancel_func to virt and novirt_install functions
In addition to monitoring the logs for errors, call a function (or
functions) that tell it to cancel the anaconda process and cleanup.

Also check for a cancel after creating the squashfs image for live-iso
since that's a long running process.

This required adding a new argument to a number of existing functions,
passing it down to QEMUInstall and novirt_install where the function is
called.

Resolves: rhbz#1656691
2018-12-11 11:37:21 -08:00
Brian C. Lane
a8f616c6da lorax-composer: Check for STATUS before deleting
When there is no run or new symlink do one last check to make sure no
STATUS file was written. If it is missing, go ahead and remove the
results directory.

Related: rhbz#1656691
2018-12-10 10:35:00 -08:00
Brian C. Lane
962d10b3b7 Check for existing CANCEL request, and exit on FINISHED
If another CANCEL request has already been made just exit from
uuid_cancel. If the build is FINISHED before it times out just exit,
don't remove the finished results.

Related: rhbz#1656691
2018-12-07 16:47:19 -08:00
Brian C. Lane
b2e1aec635 Automatic commit of package [lorax] release [28.14.18-1].
Created by command:

/bin/tito tag
2018-12-07 13:31:42 -08:00
Brian C. Lane
90b6cc7707 New lorax documentation - 28.14.18
Related: rhbz#1656642
2018-12-07 13:30:17 -08:00
David Shea
833b1bba43 Add openstack to the image type list in the docs
(cherry picked from commit d619430d6d)

Related: rhbz#1628645
2018-12-07 15:11:56 -05:00
Brian C. Lane
663a0dcd73 lorax-composer: Handle packages with multiple builds
When the repository has multiple arches, eg. i686 and x86_64, it should
add a new entry to the project's builds list, not create a new project
in the list.

This handles that by adding a modified insort_left function and
examining the packages returned from dnf to make sure they aren't
already listed in the results. It also handles adding them in sorted
order so that no further sorting needs to be done on the results.

Resolves: rhbz#1656642
2018-12-07 11:54:02 -08:00
Brian C. Lane
36fb75abd2 Adjust test_drtfr_gpgkey to pass on Fedora 28 and RHEL 8
Depending on which version of dnf is installed it may return a tuple, or
may return a list. Work around this by only comparing the 1st element of
it.

Related: rhbz#1655876
2018-12-07 11:52:22 -08:00
Brian C. Lane
660e250804 Update the projects tests to use DNF Repo object
Stop using fake dnf object, use the real thing. This will help catch
problems with dnf returning unexpected types like VectorString.

(cherry picked from commit 27aff75aa3)

Related: rhbz#1655876
2018-12-07 11:52:22 -08:00
Brian C. Lane
2ad4b20a91 dnf changed the type of gpgkey to a tuple
And in an intermediate version it returns a VectorString object which
isn't serializable by the json or toml modules.

So convert it to a list so that the type is consistent in the sources
code.

(cherry picked from commit e9e5139750)

Resolves: rhbz#1655876
2018-12-07 11:52:22 -08:00
Brian C. Lane
6a2f574ed7 lorax-composer: Add CDN repo checks to startup and compose start.
This will block starting a compose if the sources only reference
cdn.redhat.com urls.

Resolves: rhbz#1655623
2018-12-07 11:50:52 -08:00
Brian C. Lane
f1af108e5b lorax-composer: Check for CDN only repos
Anaconda is currently not able to handle cdn repo urls in the kickstart.
Add a new function that checks for extra repos and returns True.

Related: rhbz#1655623
2018-12-07 11:50:52 -08:00
Brian C. Lane
15c64d9f8d There is no support for edk2-ovmf on arm/arm64
Resolves: rhbz#1655512
2018-12-07 11:48:59 -08:00
Brian C. Lane
f0bac40d7f lorax-composer: Check the queue and results at startup
If the system ran out of space, or was rebooted unexpectedly, the state
of the queue symlinks, or the results STATUS files may be inconsistent.
This checks them and:
 * Removes broken symlinks from queue/new and queue/run
 * Removes symlinks from run and sets the build to FAILED
 * Sets builds w/o a STATUS to FAILED
 * Sets builds with STATUS of RUNNING to FAILED
 * Creates missing queue/new symlinks to results with STATUS of WAITING

So, any builds that were running during the reboot will be FAILED, and
any that were waiting to be started will be started upon rebooting.

Resolves: rhbz#1647985
(cherry picked from commit 4dd9004d13)
2018-12-06 15:27:48 -08:00
Brian C. Lane
580e771dea Automatic commit of package [lorax] release [28.14.17-1].
Created by command:

/bin/tito tag
2018-11-29 13:21:21 -08:00