Commit Graph

326 Commits

Author SHA1 Message Date
Brian C. Lane
89539a7bf4 tests: Use journalctl -g to check for failed login
/var/log/audit/audit.log isn't always available (eg. tar liveimg
install), but it is logged to the journal, which can be grepped with
'journalctl -g' so use that instead.
2019-11-01 09:00:57 -07:00
Brian C. Lane
454af0c5d5 tests: Fix check_root_account when used with tar liveimg test
passwd and openssh-clients need to be installed, and there is no need to
use sudo when you are connecting to the system as root.
2019-11-01 09:00:57 -07:00
Alexander Todorov
d67745d755 tests: Use the same asserts as before
after the changes around live-iso and qcow2 test cases the asserts
used have also changed. This commit utilizes the existing test lib.
2019-11-01 09:00:57 -07:00
Alexander Todorov
e1b45958f4 tests: switch to using podman instead of docker
Note: use podman-docker to avoid changing tests too much. This
is also what we have on the RHEL branches.

There's no service to be started/restarted so remove everything
related to docker service.
2019-11-01 09:00:57 -07:00
Brian C. Lane
4705ff3b1e tests: Remove nested vm from tar liveimg kickstart test
Use anaconda to install the root.tar.xz into a disk image.  Copy the
disk image to the host, and use the cockpit vm support to boot it and
test it.
2019-11-01 09:00:57 -07:00
Brian C. Lane
a0ae10c2d7 tests: Use --http0.9 for curl ssh test 2019-11-01 09:00:57 -07:00
Brian C. Lane
e3654e8297 test: Boot the live-iso faster, and login using ssh key
On Fedora 31 passworless root login is no longer working. We already
install a ssh key, may as well use it.

This also reduces the live boot timeout to 2s from 60s, which should
help with timeout problems when booting.
2019-11-01 09:00:57 -07:00
Brian C. Lane
b8bf258a3c tests: Split testing the image into a separate script
Nested virt is not reliable enough, especially on other arches, to rely
on for testing the created images. This moves the test code into
test_boot_* scripts to be run from inside the booted images.

It also adds copying the results of the build into
/var/tmp/test-results/, and includes the generated ssh key so that
whatever boots the image can also log in.

The tests/test_image.sh script has been added to handle running the
test_boot_* scripts without any of the extra lorax-composer specific
setup.
2019-11-01 09:00:57 -07:00
Brian C. Lane
e365e476af Disable some compose types on other architectures
The 'enabled' field in the /compose/types output now reflects whether or
not the type is supported on the current architecture. Disabled types
are not allowed to be built, and will raise an error like:

Compose type 'alibaba' is disabled on this architecture
2019-10-28 16:42:36 -07:00
Alexander Todorov
88162780a7 tests: Document Azure setup 2019-10-21 19:46:37 +02:00
Brian C. Lane
0eb01b84a8 tests: Switch the azure examples used in the lifted tests to use aws 2019-10-16 09:17:54 -07:00
Brian C. Lane
c2620b0c85 lifted: Add support for AWS upload
This uses a new Ansible module, ec2_snapshot_import, which is included
here until it is available from upstream.

It will upload the AMI to s3, convert it to a snapshot, and then
register the snapshot as an AMI. The s3 object is deleted when it has
been successfully uploaded.
2019-10-16 09:17:54 -07:00
Brian C. Lane
b91d742ecb tests: Add tests for deleting unknown upload and profile 2019-10-16 09:17:53 -07:00
Brian C. Lane
d16e8f5bfc Change /compose/uploads/delete to /upload/delete
Remove the requirement to pass the compose uuid and the upload uuid --
only the upload uuid is needed.
2019-10-16 09:17:53 -07:00
Brian C. Lane
577c56ebaa tests: Add test for /compose/uploads/delete
Test deleting an upload from a compose.
2019-10-16 09:17:53 -07:00
Brian C. Lane
b307ff0430 tests: Add tests for /compose/uploads/schedule
Test it with a profile, and with one-time use settings. It does not
actually upload anything, it only tests that the upload is scheduled.
2019-10-16 09:17:53 -07:00
Brian C. Lane
fd2330c4b7 tests: Fix comments about V1 API results including uploads data 2019-10-16 09:17:53 -07:00
Brian C. Lane
acbf63013a lifted: Make sure inputs cannot have path elements
This processes the inputs with os.path.basename to strip off any path
elements and prevent potential path traversal attacks. Also adds a test.
2019-10-16 09:17:53 -07:00
Brian C. Lane
7396c272b2 Use consistent naming for upload uuids
Since we have both compose uuids and upload uuids they need to be
clearly named. This updates the upload naming to use 'upload_uuid' in
the inputs, and 'upload_id' in the output (_id instead of _uuid for
consistency with build_id naming in the status responses).

This also adds 'upload_id' to the /upload/log response.
2019-10-16 09:17:53 -07:00
Brian C. Lane
ade8945da3 tests: Add tests for new upload routes
This tests the routes for saving a profile, listing profiles, deleting
profiles, as well as composing with upload.

The composes run fake composes with upload data, one selects a profile
the other passes in the settings. No actual upload is done, but it tests
that the info, log, and cancel routes work.

This also updates the test setup to copy over the share/lifted directory
so that the providers are available to the tests.
2019-10-16 09:17:53 -07:00
Brian C. Lane
b2fc391677 lifted: Add delete_profile function and tests
Also adds a helper to providers, _get_profile_path, so that the code
doesn't need to be repeated in all the functions.
2019-10-16 09:17:53 -07:00
Brian C. Lane
bc06abceab lifted: Add a function to load the settings for a provider's profile
This returns the settings after loading them and running verify_settings
on them.
2019-10-16 09:17:53 -07:00
Brian C. Lane
de314a0046 tests: Add yamllint of the lifted playbooks 2019-10-16 09:17:53 -07:00
Brian C. Lane
4da569442c tests: Add tests for the new lifted module
These tests cover most of the module's functions, except for the queue
monitor, and the actual execution of the playbooks.
2019-10-16 09:17:53 -07:00
Brian C. Lane
1b84f90963 lifted directories should be under share_dir and lib_dir
Otherwise passing --sharedir pointed to some other path will not use the
correct providers.
2019-10-16 09:17:53 -07:00
Brian C. Lane
1a6bc098d9 tests: Add tests for API v1
Output from some of these are different from API v0. Instead of mixing
the two this moved v1 tests into a new class - ServerAPIV1TestCase to
make them easier to maintain, and removes the v1 tests from
ServerAPIV0TestCase
2019-10-16 09:17:53 -07:00
Brian C. Lane
3a453eaad7 Make sure V0 API doesn't return uploads information
uploads should only be included in the V1 API routes, add `api`
selection to the relevant helper functions and calls to them from v0.py

Add new V1 routes with api=1 to include the uploads information in the
results.

Also add tests to ensure that V0 requests do not include uploads.
2019-10-16 09:17:53 -07:00
Brian C. Lane
697233c14a lorax-composer: Handle RecipeError in commit_recipe_directory
A recipe that is valid TOML can still be an invalid recipe (eg. missing
the 'name' field) so this should also catch RecipeError.

Also added tests for this, as well as making sure commit_recipe_file()
raises the correct errors.

Resolves: rhbz#1755068
2019-10-04 08:07:36 -07:00
Jiri Kortus
5322664432 Add test for running composer with --no-system-repos option 2019-09-24 12:50:19 +03:00
Jiri Kortus
f978d4b01c [tests] Use functions for starting and stopping lorax-composer 2019-09-24 12:50:19 +03:00
Alexander Todorov
8af9723c37 [tests] Collect compose logs after each build
- save compose logs under /var/log/$TEST
- save qemu logs under /var/log/$TEST/qemu.log
- download everything to $TEST_ATTACHMENTS so it can be saved
  in CI results
2019-09-13 12:16:22 +03:00
Jiri Kortus
836f0ddf7a [tests] Use a function to wait for compose to finish
The function is meant to replace the duplicate pieces of code used in various
tests, the polling interval is unified to 20 seconds.
2019-09-13 12:16:22 +03:00
Alexander Todorov
ea98969014 When launching AWS instances wait for the one we just launched
because the ec2.instances variable will return all that are
currently running
2019-09-11 10:46:53 +02:00
Jiri Kortus
421cc9c22f tests: Add kickstart tar installation test 2019-09-09 09:28:52 +02:00
Jiri Kortus
562eb0a15d tests: add option to disable kernel command line parameters check 2019-09-09 09:28:52 +02:00
Brian C. Lane
beff4f6967 tests: Use a loop to wait for VM and sshd to start 2019-09-09 09:28:52 +02:00
Brian C. Lane
3ee07001b1 tests: Drop tito from the Dockerfile.test
It isn't needed to run the tests, and currently a package it depends on
(cheetah) is not working in rawhide.

Also bump glusterfs version to 7.*
2019-09-03 11:02:52 -07:00
Brian C. Lane
1581842132 tests: Drop sort from compose types test
The output from the API is already sorted, and 'sort' changes this
ordering in some situations.
2019-08-29 08:17:29 -07:00
Alexander Todorov
64ac615fa4 Revert "tests: Fix the order of liveimg-tar live-iso"
This reverts commit 3b6c6a35d7.
2019-08-29 08:17:29 -07:00
Alexander Todorov
114c1bbda7 New test: assert toml files in git workspace
tests: export BLUEPRINTS_DIR for use in tests

Depending on the way the tests are run the directory may be a temporary
dir, or it may be the standard /var/lib/lorax/... path.

Related: rhbz#1714103
2019-08-29 08:17:29 -07:00
Brian C. Lane
cda497f0cb tests: Update gpg key to fedora 32 2019-08-20 17:23:08 -07:00
Brian C. Lane
3b6c6a35d7 tests: Fix the order of liveimg-tar live-iso 2019-08-20 17:23:08 -07:00
Brian C. Lane
4da4b665ef tests: Use server-2.repo instead of single.repo
And use the working directory for the graft, not the root filesystem.
2019-08-20 17:23:08 -07:00
Brian C. Lane
6f686ff9d6 lorax-composer: Add support for dnf variables to repo sources
This loads the system dnf vars from /etc/dnf/vars at startup if
system repos are enabled, and it substitutes the values in the sources
when loaded, and when a new source is added.

Also includes tests.
2019-08-20 16:26:03 -07:00
Brian C. Lane
40bb80f10f tests: Expand test coverage of the v0 and v1 sources API 2019-08-09 13:16:12 -07:00
Brian C. Lane
0935a10a0b tests: Temporarily work around rpm and pylint issues
See https://bugzilla.redhat.com/show_bug.cgi?id=1739167

Something in rpm changed and pylint isn't able to find the rpm
constants.
2019-08-09 13:16:03 -07:00
Brian C. Lane
1e88a99443 lorax-composer: Add v1 API for projects/source/
This changes the source 'name' field to match the DNF usage of it as a
descriptive string. 'id' is now used as the short name to refer to the
source. The v0 API remains unchanged.

Tests for v1 behavior have been added.

Now that the v1 API is in use the status message will return api: 1
2019-08-07 13:26:35 -07:00
Brian C. Lane
b9d18216ef lorax-composer: Add liveimg-tar image type
This creates a tar suitable for use with the anaconda kickstart liveimg
command. It adds the kernel, grub2, and grub2-tools packages to the tar
template.
2019-08-05 14:50:32 -07:00
Brian C. Lane
361c6de2de livemedia-creator: Use --compress-arg in mksquashfs
Previously the --compress-arg option was only used for tar, this adds
support for it to the squashfs.img creation used with live isos.
2019-08-05 14:49:25 -07:00
Brian C. Lane
835c912e12 Only use repos with valid urls for test_server.py
libdnf-0.22.5-5 changed something and now the repos with fake urls are
failing when loaded by test_server.py (they still work fine with
test_projects.py) so only use the 'good' repos with the test_server.py
tests -- the others weren't needed for any of its tests anyway.
2019-08-05 14:48:23 -07:00
Brian C. Lane
c46f9f034f tests: Update test_creator.py
make_squashfs has been removed, make_runtime is now used in all paths to
create the install.img

Add a tests for squashfs only and squashfs+ext4 (requires loop so only
runs as root).
2019-07-29 13:21:25 -07:00
Brian C. Lane
0708302464 Use binary mode to tail the file
In python 3 f.seek() on text doesn't work like it does in py2/C because
text is now unicode. So change read_tail to use byte mode and take
unicode into account. Also add tests for it.
2019-07-11 15:16:41 -07:00
Jiri Kortus
9bf8d8a2fc Use passwd --status for locked root account check 2019-07-11 12:47:18 +03:00
Brian C. Lane
3cb007a25b tests: Use liveuser account for live-iso boot check
On rawhide you cannot ssh in as root without changing PermitRootLogin,
and really we should be testing that liveuser can login not root.
2019-07-11 11:47:11 +03:00
Brian C. Lane
d0e947fe3b tests: Add check to make sure the compose actually finished
Previously it was looping, waiting for FINISHED|FAILED but was not
actually failing the test if the compose failed to build.
This adds a function to check the status of the compose and calls it
after each compose.
2019-07-10 08:58:24 -07:00
Lars Karlitski
fd57e1106e tests: set skip_if_unavailable in test repos
dnf seems to have changed the default for skip_if_unavailable. Some
mock repositories are still around in later tests, which then fail
because metadata cannot be synced.

Also expose skip_if_unavailable in dnf_repo_to_file_repo(), so that
tests checking for equality of repo files continue to pass.
2019-07-09 00:23:33 +02:00
Evan Goode
998d0140be Assert that RuntimeErrors have correct messages 2019-06-28 08:55:36 -07:00
Alexander Todorov
e70aae69da Remove unused shell script
the file is not used anywhere, nor is the findtestfiles() function
defined inside of it.
2019-06-28 11:23:31 +03:00
Alexander Todorov
e947e01331 Do not generate journal.xml from beakerlib
bacause this requires additional Python modules and we don't
really use it! Fixes
[ WARNING  ] :: cannot create journal.xml due to missing python interpreter
2019-06-25 14:38:59 +02:00
Brian C. Lane
f1733369fa tests: Add tests for recipe checking functions 2019-06-25 11:44:58 +02:00
Brian C. Lane
61059a2699 lorax-composer: Add basic recipe checker function
This makes sure that required fields are included, and that sections are
not empty. It does not check for all optional fields.

If there are errors it will gather up all of them and then raise a
RecipeError with a string of all the errors.
2019-06-25 11:44:58 +02:00
Lars Karlitski
61fc4d2b4e test: Fix test_blueprint_sanity
The new toml library, introduced with abe7df34f, outputs different
whitespace from the old one. Fix the test expectation and strip()
results from toml.dumps(), because it contains superfluous newlines at
the end.
2019-06-24 23:43:08 +02:00
Brian C. Lane
682f8a9295 tests: rpm now returns str, drop decode() call 2019-06-24 20:15:21 +02:00
Brian C. Lane
fb89e6f275 Update qemu arguments to work correctly with nographic
Add -monitor none to turn off the qemu monitor multiplexing.
Pass -boot d for -cdrom booting instead of 'c'.

Add 'console=ttyS0,115200n8' to the boot arguments so that kernel output
will show up on the serial port.
2019-06-18 14:28:58 -07:00
Brian C. Lane
abe7df34fc Switch to new toml library
The previous library, pytoml from https://github.com/avakar/pytoml is no
longer supported. So this adds a compatibility layer on top of the
suggested replacement, toml from https://github.com/uiri/toml
2019-06-18 14:01:15 -07:00
Brian C. Lane
932ff5812c composer-cli: Update diff support for customizations and repos.git
composer-cli will now output information about changes to customizations
entries and the repos.git entries.
2019-06-13 14:16:48 -07:00
Brian C. Lane
9011a564e8 Add support for customizations and repos.git to /blueprints/diff/
This also includes extensive tests for each of the currently supported
customizations. It should be generic enough to continue working as long
as the list of dicts includes a 'name' or 'user' field in the dict.
Otherwise support for a new dict key will need to be added to the
customizations_diff function.
2019-06-13 14:16:48 -07:00
Brian C. Lane
316257fbc0 tests: Update custom-base with customizations 2019-06-13 14:16:48 -07:00
Alexander Todorov
f61dfcc2c7 Increase retry count/sleep times when waiting for lorax to start
the biggest slow down is fetching data for many repositories
over a slow network. The previous retry count and sleep times
sometimes are not enough on Fedora.
2019-06-10 13:26:11 +03:00
Alexander Todorov
688f70eefa Revert "remove the check for qemu-kvm"
Partially reverts commit 303a69bcbd.
2019-06-10 13:26:11 +03:00
Alexander Todorov
22b0eb28bc Revert "remove the check for /usr/bin/docker in the setup phase"
this is still helpful when executing these test scripts manually
outside Cockpit CI or in Jenkins (for gating).

Partially reverts commit 5f5a2d5337.
2019-06-10 13:26:11 +03:00
Alexander Todorov
b725eb141a [tests] Define unbound variables in test scripts 2019-06-06 14:56:40 +02:00
Alexander Todorov
649a9e2239 [tests] Handle blueprints in setup_tests/teardown_tests correctly
It's necessary to make sure the blueprints directory doesn't contain
the git/ directory before the tests are run, so that we can just simply
modify the blueprint files without using blueprints push.

Related: rhbz#1714298
2019-06-06 14:56:40 +02:00
Alexander Todorov
16fd1a2340 [tests] grep|cut for IP address in a more robust way
because sometimes these VMs may get an IPv6 address assigned
and we don't want to cut only the first part of it
2019-06-06 14:56:40 +02:00
Lars Karlitski
42c7c0691c Monkey-patch beakerlib to fail on first assert
Beakerlib upstream can't do this yet, but might at some point:

https://github.com/beakerlib/beakerlib/issues/42

This is only enabled in combination with the `--sit` option of the
`test/check-*` scripts. It leaves the system in exacly the state it was
in when an assertion failed. Finishing the test run would run cleanup as
well (such as deleting created images). It also takes longer.
2019-06-04 20:12:54 +02:00
Lars Karlitski
a4dcc34396 test_cli.sh: Return beakerlib's exit code
Read the return status directly from beakerlib's TestResults file. This
is more robust when running the same test multiple times.
2019-06-04 20:12:54 +02:00
Lars Karlitski
4bd03cb8f6 Don't send CORS headers
These are meant for web applications that are accessed by browsers, not
REST APIs.
2019-06-04 20:10:03 +02:00
Lars Karlitski
a164ed3d7d tests: Set BLUEPRINTS_DIR in all cases
`setup_tests()` expected BLUEPRINTS_DIR to be set, but it wasn't when
running in automated mode (with $CLI set).

Fix this and move share and blueprint dirs to function arguments.
2019-06-04 14:07:36 +03:00
Lars Karlitski
ce3a277100 tests: Fail on script errors
Fixes #727
2019-06-04 14:07:36 +03:00
Lars Karlitski
8ed910b29a composer: Set up a custom HTTP error handler
Override flask's default error handler, because that return html. Return
JSON instead with the usual { "status": false, "errors": [ ... ] }
pattern.
2019-06-04 11:50:25 +03:00
Alexander Todorov
b71e8f74d8 Configure $PACKAGE for beakerlib reports
this makes it more clear that we are testing the installed RPM
instead of sources
2019-05-31 12:57:23 +02:00
Alexander Todorov
b67ce5379c Use cloud credentials during test if they exist 2019-05-31 12:57:23 +02:00
Jan Stodola
ed8a21a627 Use ansible instead of awscli
Not all parts of the script has been switched from awscli to ansible yet,
because the ansible aws modules do not support importing s3 object as snapshots.
(https://github.com/ansible/ansible/issues/53453)
Workaround using the image_location parameter of the ec2_ami ansible module
would mean adding extra code for generating AMI manifest with pre-signed
URLs.
2019-05-24 12:44:30 +03:00
Lars Karlitski
1e5a5921e3 tests: Source lib.sh from the right directory
When testing in one of the VMs, tests are in /tests. Source the lib from
a relative directory instead.
2019-05-24 00:23:21 +02:00
Brian C. Lane
6120f74c73 Revert "Add rpmfluff temporarily"
This reverts commit 6ea1c45734.

python-rpmfluff-0.5.7.1-1 is now available.
2019-05-23 14:57:46 -07:00
Brian C. Lane
795cc3400c tests: Update tmux version to 2.9a 2019-05-23 14:57:46 -07:00
Lars Karlitski
c261cec33d tests: Fail immediately when image build fails
We were checking for composer's FINISHED status only, which meant that
when a compose failed, the test ran until it timed out.

Check for failed as well. Also, always time out after 30 minutes.
2019-05-22 08:24:49 +02:00
Lars Karlitski
bdffaa02f4 tests: Re-enable kvm
In 303a69bc, we stopped using kvm, because it doesn't exist on all test
builders. Enable it again on those that have it.
2019-05-21 22:45:32 +02:00
Lars Karlitski
303a69bcbd tests: Don't depend on kvm for tar and qcow2 tests
Some test runners don't have nested virtualization enabled. Because
these checks are only checking that a boot works, kvm doesn't give us
that much. Disable for now.

Also remove the check for qemu-kvm. It doesn't abort the test
prematurely anyway.
2019-05-20 20:24:01 +02:00
Lars Karlitski
77acc8972b test_compose_tar: Work around selinux policy change
A compose can change the hosts' policy, which can lead to docker
crashing if the container-selinux policy is not included. Add a
workaround and bug link.
2019-05-20 20:24:01 +02:00
Lars Karlitski
ca171424eb test_compose_tar: Be less verbose
A huge chunk of the output log is tar's file list. Remove that to make
the log more readable.
2019-05-20 20:24:01 +02:00
Lars Karlitski
5f5a2d5337 test_compose_tar: Fix docker test
The docker phase always failed because `-ti` was passed even though the
the output was not a terminal.

Also remove the check for /usr/bin/docker in the setup phase. It didn't
test that the daemon was running. More importantly, it didn't abort the
test anwyay (and there doesn't seem to be a good way to do this in
beakerlib).
2019-05-20 20:24:01 +02:00
Lars Karlitski
1ba2e6fd4b tests: Extract images to /var/tmp, not /tmp
Test VMs don't have enough memory (/tmp is a tmpfs).
2019-05-20 20:24:01 +02:00
Lars Karlitski
250f49f78d Use Cockpit's test images and infrastructure
Allows to run the tests on multiple operating systems and on the
infrastructure that the Cockpit team maintains.

`make vm` downloads one of Cockpit's test images (override which one
with TEST_OS) and installs rpms build from the local checkout of lorax.
The resulting image is placed in `test/images/$TEST_OS`.

TEST_OS can be set to any of Cockpit's supported images (default:
fedora-30).

Run `make check-vm` to run the CLI checks in the VM. The bulk of the
work is done in `test/check-cli`, which uses Cockpit's `bots` library to
start the VM and run the script in it.

Also included is a `test/run` script, which is the entrypoint for
Cockpit's test infrastructure.
2019-05-20 20:24:01 +02:00
Lars Karlitski
aa9c2e13b1 pylint: Remove unused false positive 2019-05-20 20:24:01 +02:00
Jiri Kortus
123b5c269d Add test for passing custom option on kernel command line 2019-05-07 12:36:32 +03:00
Jiri Kortus
596cba2255 Use verify_image function as a helper for generic tests 2019-05-07 12:36:32 +03:00
Brian C. Lane
fa4bda8317 tests: Update openssh-server to v8.* 2019-05-02 17:02:30 -07:00
Brian C. Lane
3e08389a0f Change customizations.firewall to append items instead of replace
To maintain consistency with the other options this changes firewall to
combine the existing settings from the image template with the settings
from the blueprint.

Also updated the docs, added a new test for it, and sorted the output
for consistency.
2019-05-02 16:21:36 -07:00
Brian C. Lane
1111aee92d lorax-composer: Add services support to blueprints
Add support for enabling and disabling systemd services in the
blueprint. It works like this:

    [customizations.services]
    enabled = ["sshd", "cockpit.socket", "httpd"]
    disabled = ["postfix", "telnetd"]

They are *added* to any existing settings in the kickstart templates.
2019-05-02 16:21:36 -07:00