Commit Graph

2237 Commits

Author SHA1 Message Date
Lars Karlitski
fd57e1106e tests: set skip_if_unavailable in test repos
dnf seems to have changed the default for skip_if_unavailable. Some
mock repositories are still around in later tests, which then fail
because metadata cannot be synced.

Also expose skip_if_unavailable in dnf_repo_to_file_repo(), so that
tests checking for equality of repo files continue to pass.
2019-07-09 00:23:33 +02:00
Lars Karlitski
fce196ad4a test/README.md: Add section explaining GitHub integration 2019-07-03 16:51:37 +02:00
Brian C. Lane
dc98837667 Automatic commit of package [lorax] release [31.8-1].
Created by command:

/bin/tito tag
2019-06-28 10:53:28 -07:00
Bastien Nocera
eb7f528071 Also search for pxeboot kernel and initrd pairs
As used in the Fedora 30 aarch64 Server DVDs.

Closes: #786
2019-06-28 10:38:38 -07:00
Evan Goode
998d0140be Assert that RuntimeErrors have correct messages 2019-06-28 08:55:36 -07:00
Evan Goode
fef76930bc More descriptive error for a bad ref in repos.git
Fixes #771
2019-06-28 08:55:36 -07:00
Alexander Todorov
e70aae69da Remove unused shell script
the file is not used anywhere, nor is the findtestfiles() function
defined inside of it.
2019-06-28 11:23:31 +03:00
Lars Karlitski
f810924042 test: Output results for cockpit's log.html
Cockpit's log.html has a peculiar format, but parsing logs in html is
much easier, because it shows expandable sections for each test.
2019-06-25 23:50:16 +02:00
Alexander Todorov
e947e01331 Do not generate journal.xml from beakerlib
bacause this requires additional Python modules and we don't
really use it! Fixes
[ WARNING  ] :: cannot create journal.xml due to missing python interpreter
2019-06-25 14:38:59 +02:00
Brian C. Lane
7ff4d7ac9e tests: Add RUN_TESTS to Makefile so you can pass in targets
Also add a test_cli target to run things like the blueprint cli tests.
Run this like so:

RUN_TESTS="ci test_cli" make test-in-docker

If nothing is passed it will run the "ci" target.
2019-06-25 14:13:53 +02:00
Brian C. Lane
f1733369fa tests: Add tests for recipe checking functions 2019-06-25 11:44:58 +02:00
Brian C. Lane
13a7dcf2b1 lorax-composer: Add basic case check to check_recipe_dict
This will detect if an expected field is not all lower case as is
required.
2019-06-25 11:44:58 +02:00
Brian C. Lane
61059a2699 lorax-composer: Add basic recipe checker function
This makes sure that required fields are included, and that sections are
not empty. It does not check for all optional fields.

If there are errors it will gather up all of them and then raise a
RecipeError with a string of all the errors.
2019-06-25 11:44:58 +02:00
Lars Karlitski
bad9e93c3c
Revert "test: Disable test_live_iso test"
Enable the live-iso test again.

This reverts commit 1ac75e98b4.

Fixes: #731
2019-06-25 10:24:51 +02:00
Lars Karlitski
61fc4d2b4e test: Fix test_blueprint_sanity
The new toml library, introduced with abe7df34f, outputs different
whitespace from the old one. Fix the test expectation and strip()
results from toml.dumps(), because it contains superfluous newlines at
the end.
2019-06-24 23:43:08 +02:00
Brian C. Lane
682f8a9295 tests: rpm now returns str, drop decode() call 2019-06-24 20:15:21 +02:00
Brian C. Lane
e7ce7cda72 tests: Drop libgit2 install from koji 2019-06-24 20:15:21 +02:00
Brian C. Lane
c1aa57022b Automatic commit of package [lorax] release [31.7-1].
Created by command:

/bin/tito tag
2019-06-18 15:09:32 -07:00
Brian C. Lane
f0992bc649 New lorax documentation - 31.7 2019-06-18 15:02:04 -07:00
Brian C. Lane
fb89e6f275 Update qemu arguments to work correctly with nographic
Add -monitor none to turn off the qemu monitor multiplexing.
Pass -boot d for -cdrom booting instead of 'c'.

Add 'console=ttyS0,115200n8' to the boot arguments so that kernel output
will show up on the serial port.
2019-06-18 14:28:58 -07:00
Brian C. Lane
abe7df34fc Switch to new toml library
The previous library, pytoml from https://github.com/avakar/pytoml is no
longer supported. So this adds a compatibility layer on top of the
suggested replacement, toml from https://github.com/uiri/toml
2019-06-18 14:01:15 -07:00
Brian C. Lane
932ff5812c composer-cli: Update diff support for customizations and repos.git
composer-cli will now output information about changes to customizations
entries and the repos.git entries.
2019-06-13 14:16:48 -07:00
Brian C. Lane
9011a564e8 Add support for customizations and repos.git to /blueprints/diff/
This also includes extensive tests for each of the currently supported
customizations. It should be generic enough to continue working as long
as the list of dicts includes a 'name' or 'user' field in the dict.
Otherwise support for a new dict key will need to be added to the
customizations_diff function.
2019-06-13 14:16:48 -07:00
Brian C. Lane
316257fbc0 tests: Update custom-base with customizations 2019-06-13 14:16:48 -07:00
Brian C. Lane
ffc3432cf2 Move the v0 API documentation into the functions 2019-06-13 11:28:57 -07:00
Brian C. Lane
6d50a5874e Update the /api/v0/ route handling to use the flask_blueprints Blueprint class
Instead of setting up the routes inside a function we can now use a
BlueprintSkip class, which allows us to register them at different
routes (eg. /api/v0/ and /api/v1/) and override any routes that will be
replaced by the new API version.
2019-06-13 11:28:57 -07:00
Brian C. Lane
7071e62985 Extend Flask Blueprint class to allow skipping routes
When adding a new API we want to use the old code for any routes that
aren't being overridden.

This modifies the Flask Blueprint class so that a skip_rules list can be
passed to server.register_blueprint()
2019-06-13 11:28:57 -07:00
Alexander Todorov
f64fe0f1f3 Remove PR template 2019-06-12 10:05:23 +02:00
Alexander Todorov
f61dfcc2c7 Increase retry count/sleep times when waiting for lorax to start
the biggest slow down is fetching data for many repositories
over a slow network. The previous retry count and sleep times
sometimes are not enough on Fedora.
2019-06-10 13:26:11 +03:00
Alexander Todorov
688f70eefa Revert "remove the check for qemu-kvm"
Partially reverts commit 303a69bcbd.
2019-06-10 13:26:11 +03:00
Alexander Todorov
22b0eb28bc Revert "remove the check for /usr/bin/docker in the setup phase"
this is still helpful when executing these test scripts manually
outside Cockpit CI or in Jenkins (for gating).

Partially reverts commit 5f5a2d5337.
2019-06-10 13:26:11 +03:00
Alexander Todorov
b725eb141a [tests] Define unbound variables in test scripts 2019-06-06 14:56:40 +02:00
Alexander Todorov
649a9e2239 [tests] Handle blueprints in setup_tests/teardown_tests correctly
It's necessary to make sure the blueprints directory doesn't contain
the git/ directory before the tests are run, so that we can just simply
modify the blueprint files without using blueprints push.

Related: rhbz#1714298
2019-06-06 14:56:40 +02:00
Alexander Todorov
16fd1a2340 [tests] grep|cut for IP address in a more robust way
because sometimes these VMs may get an IPv6 address assigned
and we don't want to cut only the first part of it
2019-06-06 14:56:40 +02:00
Alexander Todorov
eb8ca5b5bf Remove quotes around file test in make vm
otherwise make thinks credentials file doesn't exist and
doesn't include it in the VM image causing all cloud tests to
fail.
2019-06-06 14:56:40 +02:00
Lars Karlitski
33151372df test: Don't wait on --sit when test succeeds
Python's `filter()` returns a "filter object", which is truthy.
2019-06-04 20:13:18 +02:00
Lars Karlitski
42c7c0691c Monkey-patch beakerlib to fail on first assert
Beakerlib upstream can't do this yet, but might at some point:

https://github.com/beakerlib/beakerlib/issues/42

This is only enabled in combination with the `--sit` option of the
`test/check-*` scripts. It leaves the system in exacly the state it was
in when an assertion failed. Finishing the test run would run cleanup as
well (such as deleting created images). It also takes longer.
2019-06-04 20:12:54 +02:00
Lars Karlitski
a4dcc34396 test_cli.sh: Return beakerlib's exit code
Read the return status directly from beakerlib's TestResults file. This
is more robust when running the same test multiple times.
2019-06-04 20:12:54 +02:00
Lars Karlitski
4bd03cb8f6 Don't send CORS headers
These are meant for web applications that are accessed by browsers, not
REST APIs.
2019-06-04 20:10:03 +02:00
Lars Karlitski
a164ed3d7d tests: Set BLUEPRINTS_DIR in all cases
`setup_tests()` expected BLUEPRINTS_DIR to be set, but it wasn't when
running in automated mode (with $CLI set).

Fix this and move share and blueprint dirs to function arguments.
2019-06-04 14:07:36 +03:00
Lars Karlitski
ce3a277100 tests: Fail on script errors
Fixes #727
2019-06-04 14:07:36 +03:00
Lars Karlitski
b50f1967c6 Add API integration test
Test the HTTP API directly from outside the VM by forwarding
/run/weldr/api.socket to a TCP port on the host.

This is only a start to test that the API always returns JSON (see
previous commit).
2019-06-04 11:50:25 +03:00
Lars Karlitski
8ed910b29a composer: Set up a custom HTTP error handler
Override flask's default error handler, because that return html. Return
JSON instead with the usual { "status": false, "errors": [ ... ] }
pattern.
2019-06-04 11:50:25 +03:00
Alexander Todorov
3587ed3663 Split live-iso and qcow2 and update test scenario execution
this makes it possible to have more granular test execution
reported as separate statuses on GitHub. ATM we will have:

- cockpit/fedora-30
- cockpit/fedora-30/live-iso
- cockpit/fedora-30/qcow2
- cockpit/fedora-30/aws
- cockpit/fedora-30/azure
- cockpit/fedora-30/openstack
- cockpit/fedora-30/vmware
2019-05-31 12:57:23 +02:00
Alexander Todorov
b71e8f74d8 Configure $PACKAGE for beakerlib reports
this makes it more clear that we are testing the installed RPM
instead of sources
2019-05-31 12:57:23 +02:00
Alexander Todorov
b67ce5379c Use cloud credentials during test if they exist 2019-05-31 12:57:23 +02:00
Alexander Todorov
6c2b34bf15 Don't execute compose/blueprint sanity tests in Travis CI
instead they will be executed in Cockpit CI
2019-05-31 12:57:23 +02:00
Lars Karlitski
c5ae344b4f test: Add --list option to test/check* scripts
Helps in figuring out which tests are in a file without having to open
it. Use like this:

    $ test/check-cli -l
    TestImages.test_live_iso
    TestImages.test_partitioned_disk
    TestImages.test_qcow2
    TestImages.test_tar
    TestSanity.test_blueprint_sanity
    TestSanity.test_compose_sanity

Names of classes containing multiple tests can be given, just like
normal:

    $ test/check-cli -l TestSanity
    TestSanity.test_blueprint_sanity
    TestSanity.test_compose_sanity
2019-05-29 13:53:02 +02:00
Lars Karlitski
5dda214c39 test: Add --sit argument to check-* scripts
Cockpit has this. It's very useful for debugging a failing test locally.
2019-05-24 17:27:50 +02:00
Lars Karlitski
29cf2c4e8c test: Custom main() function
It's not as full featured as unittest.main(), but this will allow us to
add custom command line arguments.
2019-05-24 17:27:50 +02:00