Commit Graph

2553 Commits

Author SHA1 Message Date
Alexander Todorov
eb8ca5b5bf Remove quotes around file test in make vm
otherwise make thinks credentials file doesn't exist and
doesn't include it in the VM image causing all cloud tests to
fail.
2019-06-06 14:56:40 +02:00
Lars Karlitski
33151372df test: Don't wait on --sit when test succeeds
Python's `filter()` returns a "filter object", which is truthy.
2019-06-04 20:13:18 +02:00
Lars Karlitski
42c7c0691c Monkey-patch beakerlib to fail on first assert
Beakerlib upstream can't do this yet, but might at some point:

https://github.com/beakerlib/beakerlib/issues/42

This is only enabled in combination with the `--sit` option of the
`test/check-*` scripts. It leaves the system in exacly the state it was
in when an assertion failed. Finishing the test run would run cleanup as
well (such as deleting created images). It also takes longer.
2019-06-04 20:12:54 +02:00
Lars Karlitski
a4dcc34396 test_cli.sh: Return beakerlib's exit code
Read the return status directly from beakerlib's TestResults file. This
is more robust when running the same test multiple times.
2019-06-04 20:12:54 +02:00
Lars Karlitski
4bd03cb8f6 Don't send CORS headers
These are meant for web applications that are accessed by browsers, not
REST APIs.
2019-06-04 20:10:03 +02:00
Lars Karlitski
a164ed3d7d tests: Set BLUEPRINTS_DIR in all cases
`setup_tests()` expected BLUEPRINTS_DIR to be set, but it wasn't when
running in automated mode (with $CLI set).

Fix this and move share and blueprint dirs to function arguments.
2019-06-04 14:07:36 +03:00
Lars Karlitski
ce3a277100 tests: Fail on script errors
Fixes #727
2019-06-04 14:07:36 +03:00
Lars Karlitski
b50f1967c6 Add API integration test
Test the HTTP API directly from outside the VM by forwarding
/run/weldr/api.socket to a TCP port on the host.

This is only a start to test that the API always returns JSON (see
previous commit).
2019-06-04 11:50:25 +03:00
Lars Karlitski
8ed910b29a composer: Set up a custom HTTP error handler
Override flask's default error handler, because that return html. Return
JSON instead with the usual { "status": false, "errors": [ ... ] }
pattern.
2019-06-04 11:50:25 +03:00
Alexander Todorov
3587ed3663 Split live-iso and qcow2 and update test scenario execution
this makes it possible to have more granular test execution
reported as separate statuses on GitHub. ATM we will have:

- cockpit/fedora-30
- cockpit/fedora-30/live-iso
- cockpit/fedora-30/qcow2
- cockpit/fedora-30/aws
- cockpit/fedora-30/azure
- cockpit/fedora-30/openstack
- cockpit/fedora-30/vmware
2019-05-31 12:57:23 +02:00
Alexander Todorov
b71e8f74d8 Configure $PACKAGE for beakerlib reports
this makes it more clear that we are testing the installed RPM
instead of sources
2019-05-31 12:57:23 +02:00
Alexander Todorov
b67ce5379c Use cloud credentials during test if they exist 2019-05-31 12:57:23 +02:00
Alexander Todorov
6c2b34bf15 Don't execute compose/blueprint sanity tests in Travis CI
instead they will be executed in Cockpit CI
2019-05-31 12:57:23 +02:00
Lars Karlitski
c5ae344b4f test: Add --list option to test/check* scripts
Helps in figuring out which tests are in a file without having to open
it. Use like this:

    $ test/check-cli -l
    TestImages.test_live_iso
    TestImages.test_partitioned_disk
    TestImages.test_qcow2
    TestImages.test_tar
    TestSanity.test_blueprint_sanity
    TestSanity.test_compose_sanity

Names of classes containing multiple tests can be given, just like
normal:

    $ test/check-cli -l TestSanity
    TestSanity.test_blueprint_sanity
    TestSanity.test_compose_sanity
2019-05-29 13:53:02 +02:00
Lars Karlitski
5dda214c39 test: Add --sit argument to check-* scripts
Cockpit has this. It's very useful for debugging a failing test locally.
2019-05-24 17:27:50 +02:00
Lars Karlitski
29cf2c4e8c test: Custom main() function
It's not as full featured as unittest.main(), but this will allow us to
add custom command line arguments.
2019-05-24 17:27:50 +02:00
Jan Stodola
ed8a21a627 Use ansible instead of awscli
Not all parts of the script has been switched from awscli to ansible yet,
because the ansible aws modules do not support importing s3 object as snapshots.
(https://github.com/ansible/ansible/issues/53453)
Workaround using the image_location parameter of the ec2_ami ansible module
would mean adding extra code for generating AMI manifest with pre-signed
URLs.
2019-05-24 12:44:30 +03:00
Jan Stodola
f24877b06e Fix path to generic.prm
Also quote ${extra_boot_args} as in the other templates

Related: rhbz#1712822
2019-05-23 15:28:31 -07:00
Brian C. Lane
be15fa6ea8 Update example fedora-livemedia.ks 2019-05-23 15:27:02 -07:00
Brian C. Lane
708314a1fe Update composer live-iso template 2019-05-23 15:27:02 -07:00
Lars Karlitski
1ac75e98b4 test: Disable test_live_iso test
https://github.com/weldr/lorax/issues/731
2019-05-24 00:23:21 +02:00
Lars Karlitski
1e5a5921e3 tests: Source lib.sh from the right directory
When testing in one of the VMs, tests are in /tests. Source the lib from
a relative directory instead.
2019-05-24 00:23:21 +02:00
Brian C. Lane
6120f74c73 Revert "Add rpmfluff temporarily"
This reverts commit 6ea1c45734.

python-rpmfluff-0.5.7.1-1 is now available.
2019-05-23 14:57:46 -07:00
Brian C. Lane
795cc3400c tests: Update tmux version to 2.9a 2019-05-23 14:57:46 -07:00
Martin Pitt
3b467f1cd7 test: Install beakerlib on non-RHEL images
Commit 4783f6562f introduced the assumption that beakerlib is already
installed in non-RHEL images. As the test script runs without `set -e`,
this hasn't been noticed as the test silently succeeds.

Go back to installing beakerlib everywhere.
2019-05-23 12:00:30 +02:00
Lars Karlitski
c261cec33d tests: Fail immediately when image build fails
We were checking for composer's FINISHED status only, which meant that
when a compose failed, the test ran until it timed out.

Check for failed as well. Also, always time out after 30 minutes.
2019-05-22 08:24:49 +02:00
Lars Karlitski
4783f6562f test: Install beakerlib wehn running on rhel 2019-05-22 08:23:27 +02:00
Lars Karlitski
6ddaa5e0dd test: Generalize fs resizing in vm.install
This should work on RHEL as well.
2019-05-22 08:23:27 +02:00
Lars Karlitski
bdffaa02f4 tests: Re-enable kvm
In 303a69bc, we stopped using kvm, because it doesn't exist on all test
builders. Enable it again on those that have it.
2019-05-21 22:45:32 +02:00
Lars Karlitski
40e9c54cbb test: Fix vm.install to be idempotent
Clean build-results from the VM so that `make vm` can be run without
resetting.

Also be more verbose to catch rpm errors.
2019-05-21 22:38:02 +02:00
Lars Karlitski
303a69bcbd tests: Don't depend on kvm for tar and qcow2 tests
Some test runners don't have nested virtualization enabled. Because
these checks are only checking that a boot works, kvm doesn't give us
that much. Disable for now.

Also remove the check for qemu-kvm. It doesn't abort the test
prematurely anyway.
2019-05-20 20:24:01 +02:00
Lars Karlitski
77acc8972b test_compose_tar: Work around selinux policy change
A compose can change the hosts' policy, which can lead to docker
crashing if the container-selinux policy is not included. Add a
workaround and bug link.
2019-05-20 20:24:01 +02:00
Lars Karlitski
ca171424eb test_compose_tar: Be less verbose
A huge chunk of the output log is tar's file list. Remove that to make
the log more readable.
2019-05-20 20:24:01 +02:00
Lars Karlitski
5f5a2d5337 test_compose_tar: Fix docker test
The docker phase always failed because `-ti` was passed even though the
the output was not a terminal.

Also remove the check for /usr/bin/docker in the setup phase. It didn't
test that the daemon was running. More importantly, it didn't abort the
test anwyay (and there doesn't seem to be a good way to do this in
beakerlib).
2019-05-20 20:24:01 +02:00
Lars Karlitski
1ba2e6fd4b tests: Extract images to /var/tmp, not /tmp
Test VMs don't have enough memory (/tmp is a tmpfs).
2019-05-20 20:24:01 +02:00
Lars Karlitski
250f49f78d Use Cockpit's test images and infrastructure
Allows to run the tests on multiple operating systems and on the
infrastructure that the Cockpit team maintains.

`make vm` downloads one of Cockpit's test images (override which one
with TEST_OS) and installs rpms build from the local checkout of lorax.
The resulting image is placed in `test/images/$TEST_OS`.

TEST_OS can be set to any of Cockpit's supported images (default:
fedora-30).

Run `make check-vm` to run the CLI checks in the VM. The bulk of the
work is done in `test/check-cli`, which uses Cockpit's `bots` library to
start the VM and run the script in it.

Also included is a `test/run` script, which is the entrypoint for
Cockpit's test infrastructure.
2019-05-20 20:24:01 +02:00
Lars Karlitski
aa9c2e13b1 pylint: Remove unused false positive 2019-05-20 20:24:01 +02:00
Brian C. Lane
5d05deae64 Automatic commit of package [lorax] release [31.6-1].
Created by command:

/bin/tito tag
2019-05-16 10:37:59 -07:00
Brian C. Lane
ddb82ef598 Add kernel to ext4-filesystem template
The filesystem was too small because Anaconda always adds the kernel,
but the template uses --nocore so it doesn't take that into account.
Add it to the template so that the filesystem size will be large enough
to hold the extra packages.
2019-05-16 10:26:30 -07:00
Brian C. Lane
2949edf733 Create a lorax-docs package with the html docs
Also include the other top level .rst docs in the main lorax package.

Resolves: rhbz#1695274
2019-05-09 15:19:15 -07:00
Brian C. Lane
c98f9ee40b Add new documentation branches to index.rst
Adds links to the f29 and f30 documentation branches.
2019-05-07 16:21:52 -07:00
Brian C. Lane
947cbb1b84 Automatic commit of package [lorax] release [31.5-1].
Created by command:

/bin/tito tag
2019-05-07 11:51:15 -07:00
Brian C. Lane
afb745552b Add python3-pycdlib to Dockerfile.test 2019-05-07 11:22:02 -07:00
Brian C. Lane
d4d050496b Replace isoinfo with pycdlib
isoinfo is part of genisoimage, which we no longer use, switch to using
a python library to read the label from the iso.
2019-05-07 11:22:02 -07:00
Jiri Kortus
123b5c269d Add test for passing custom option on kernel command line 2019-05-07 12:36:32 +03:00
Jiri Kortus
596cba2255 Use verify_image function as a helper for generic tests 2019-05-07 12:36:32 +03:00
Brian C. Lane
40d3055229 Automatic commit of package [lorax] release [31.4-1].
Created by command:

/bin/tito tag
2019-05-02 17:03:11 -07:00
Brian C. Lane
fa4bda8317 tests: Update openssh-server to v8.* 2019-05-02 17:02:30 -07:00
Brian C. Lane
72ff8e72a9 New lorax documentation - 31.4 2019-05-02 16:21:37 -07:00
Brian C. Lane
3e08389a0f Change customizations.firewall to append items instead of replace
To maintain consistency with the other options this changes firewall to
combine the existing settings from the image template with the settings
from the blueprint.

Also updated the docs, added a new test for it, and sorted the output
for consistency.
2019-05-02 16:21:36 -07:00