Commit Graph

22 Commits

Author SHA1 Message Date
Alexander Todorov
197c326530 test: check the number of tests that ran
Fail if the number of excuted tests != number of dicovered tests.
2019-07-10 09:18:36 +02:00
Jacob Kozol
38f1edd2aa Add test for VALID_BLUEPRINT_NAME check
Blueprint names can only contain alphanumeric characters and the characters: ._-. This test checks that an api request to create a blueprint with invalid characters in its name will return an InvalidChars error.
2019-07-09 22:53:51 +02:00
Jacob Kozol
4174186c14 Add seperate validation for blueprint names
The VALID_API_STRING function allows for characters that should not be
allowed in blueprint names. VALID_BLUEPRINT_NAME allows us to
specifically check if a blueprint contains a valid name.
2019-07-09 22:53:51 +02:00
Lars Karlitski
fce196ad4a test/README.md: Add section explaining GitHub integration 2019-07-03 16:51:37 +02:00
Lars Karlitski
f810924042 test: Output results for cockpit's log.html
Cockpit's log.html has a peculiar format, but parsing logs in html is
much easier, because it shows expandable sections for each test.
2019-06-25 23:50:16 +02:00
Lars Karlitski
bad9e93c3c
Revert "test: Disable test_live_iso test"
Enable the live-iso test again.

This reverts commit 1ac75e98b4.

Fixes: #731
2019-06-25 10:24:51 +02:00
Lars Karlitski
33151372df test: Don't wait on --sit when test succeeds
Python's `filter()` returns a "filter object", which is truthy.
2019-06-04 20:13:18 +02:00
Lars Karlitski
42c7c0691c Monkey-patch beakerlib to fail on first assert
Beakerlib upstream can't do this yet, but might at some point:

https://github.com/beakerlib/beakerlib/issues/42

This is only enabled in combination with the `--sit` option of the
`test/check-*` scripts. It leaves the system in exacly the state it was
in when an assertion failed. Finishing the test run would run cleanup as
well (such as deleting created images). It also takes longer.
2019-06-04 20:12:54 +02:00
Lars Karlitski
4bd03cb8f6 Don't send CORS headers
These are meant for web applications that are accessed by browsers, not
REST APIs.
2019-06-04 20:10:03 +02:00
Lars Karlitski
b50f1967c6 Add API integration test
Test the HTTP API directly from outside the VM by forwarding
/run/weldr/api.socket to a TCP port on the host.

This is only a start to test that the API always returns JSON (see
previous commit).
2019-06-04 11:50:25 +03:00
Alexander Todorov
3587ed3663 Split live-iso and qcow2 and update test scenario execution
this makes it possible to have more granular test execution
reported as separate statuses on GitHub. ATM we will have:

- cockpit/fedora-30
- cockpit/fedora-30/live-iso
- cockpit/fedora-30/qcow2
- cockpit/fedora-30/aws
- cockpit/fedora-30/azure
- cockpit/fedora-30/openstack
- cockpit/fedora-30/vmware
2019-05-31 12:57:23 +02:00
Alexander Todorov
b71e8f74d8 Configure $PACKAGE for beakerlib reports
this makes it more clear that we are testing the installed RPM
instead of sources
2019-05-31 12:57:23 +02:00
Alexander Todorov
6c2b34bf15 Don't execute compose/blueprint sanity tests in Travis CI
instead they will be executed in Cockpit CI
2019-05-31 12:57:23 +02:00
Lars Karlitski
c5ae344b4f test: Add --list option to test/check* scripts
Helps in figuring out which tests are in a file without having to open
it. Use like this:

    $ test/check-cli -l
    TestImages.test_live_iso
    TestImages.test_partitioned_disk
    TestImages.test_qcow2
    TestImages.test_tar
    TestSanity.test_blueprint_sanity
    TestSanity.test_compose_sanity

Names of classes containing multiple tests can be given, just like
normal:

    $ test/check-cli -l TestSanity
    TestSanity.test_blueprint_sanity
    TestSanity.test_compose_sanity
2019-05-29 13:53:02 +02:00
Lars Karlitski
5dda214c39 test: Add --sit argument to check-* scripts
Cockpit has this. It's very useful for debugging a failing test locally.
2019-05-24 17:27:50 +02:00
Lars Karlitski
29cf2c4e8c test: Custom main() function
It's not as full featured as unittest.main(), but this will allow us to
add custom command line arguments.
2019-05-24 17:27:50 +02:00
Lars Karlitski
1ac75e98b4 test: Disable test_live_iso test
https://github.com/weldr/lorax/issues/731
2019-05-24 00:23:21 +02:00
Martin Pitt
3b467f1cd7 test: Install beakerlib on non-RHEL images
Commit 4783f6562f introduced the assumption that beakerlib is already
installed in non-RHEL images. As the test script runs without `set -e`,
this hasn't been noticed as the test silently succeeds.

Go back to installing beakerlib everywhere.
2019-05-23 12:00:30 +02:00
Lars Karlitski
4783f6562f test: Install beakerlib wehn running on rhel 2019-05-22 08:23:27 +02:00
Lars Karlitski
6ddaa5e0dd test: Generalize fs resizing in vm.install
This should work on RHEL as well.
2019-05-22 08:23:27 +02:00
Lars Karlitski
40e9c54cbb test: Fix vm.install to be idempotent
Clean build-results from the VM so that `make vm` can be run without
resetting.

Also be more verbose to catch rpm errors.
2019-05-21 22:38:02 +02:00
Lars Karlitski
250f49f78d Use Cockpit's test images and infrastructure
Allows to run the tests on multiple operating systems and on the
infrastructure that the Cockpit team maintains.

`make vm` downloads one of Cockpit's test images (override which one
with TEST_OS) and installs rpms build from the local checkout of lorax.
The resulting image is placed in `test/images/$TEST_OS`.

TEST_OS can be set to any of Cockpit's supported images (default:
fedora-30).

Run `make check-vm` to run the CLI checks in the VM. The bulk of the
work is done in `test/check-cli`, which uses Cockpit's `bots` library to
start the VM and run the script in it.

Also included is a `test/run` script, which is the entrypoint for
Cockpit's test infrastructure.
2019-05-20 20:24:01 +02:00