Commit Graph

77 Commits

Author SHA1 Message Date
Brian C. Lane
6f90f18e31 Remove duplicate repositories from the sources list
In some cases when the host has, for whatever reason, multiple copies of
the same repo listed the build may fail with an error about running out
of space.

So this commit removes duplicate entries after the host's repos have been
loaded. It also adjusts some of the test repos to use different
temporary repo names for the tests.

(cherry picked from commit 98482e444d)
2019-01-30 08:43:51 -08:00
Brian C. Lane
9431a85b85 Remove unneeded else from for/else loop. It confuses pylint
(cherry picked from commit 2950f2641b)
2019-01-08 14:13:22 -08:00
David Shea
bc574e2e2f Allow customizations to be specified as a toml list
Support both

  [customizations]
  hostname = "whatever"

and

  [[customizations]]
  hostname = "whatever"

in the blueprint data. The [[ syntax matches the other customization
directives (user, group, sshkey), and as such it's easy to accidentally
use it for the hostname without even realizing it's specifying something
different.

Add some tests for converting customizations to kickstarts.

(cherry picked from commit 35ab6a1336)
2019-01-08 14:12:50 -08:00
Brian C. Lane
ed611933cb lorax-composer: Handle packages with multiple builds
When the repository has multiple arches, eg. i686 and x86_64, it should
add a new entry to the project's builds list, not create a new project
in the list.

This handles that by adding a modified insort_left function and
examining the packages returned from dnf to make sure they aren't
already listed in the results. It also handles adding them in sorted
order so that no further sorting needs to be done on the results.

Resolves: rhbz#1656642
(cherry picked from commit d18934775c)
2018-12-06 16:07:22 -08:00
Brian C. Lane
dfb2b962b3 lorax-composer: Check the queue and results at startup
If the system ran out of space, or was rebooted unexpectedly, the state
of the queue symlinks, or the results STATUS files may be inconsistent.
This checks them and:
 * Removes broken symlinks from queue/new and queue/run
 * Removes symlinks from run and sets the build to FAILED
 * Sets builds w/o a STATUS to FAILED
 * Sets builds with STATUS of RUNNING to FAILED
 * Creates missing queue/new symlinks to results with STATUS of WAITING

So, any builds that were running during the reboot will be FAILED, and
any that were waiting to be started will be started upon rebooting.

Resolves: rhbz#1647985
(cherry picked from commit 4dd9004d13)
2018-12-06 15:26:59 -08:00
Brian C. Lane
c1977b8427 Adjust tmux version in the tests to 2.8 2018-11-29 12:18:41 -08:00
Brian C. Lane
4e46d776d5 Add tests for ltmpl.py
This covers things like installing globbed package names from multiple
repos, pinned package versions, and ltmpl functions

Related: rhbz#1548586
2018-10-29 13:53:59 -07:00
Brian C. Lane
1a7b6c74b4 Update the tests for new make_dnf_dir arguments.
Use the uid and gid that the test is running as instead of hard-coding
0.
2018-10-10 14:25:51 -07:00
Brian C. Lane
d170622357 Change make_dnf_dirs to be run as root
It needs to be root in order to set the ownership and permissions on the
directories that are under /var/lib/lorax/composer/

Refactor the directory creation into a utility function, and use a umask
of 0o006 to ensure that the parent directories created do not have o+rw
set on them (makedirs behavior is different between Python 3.6 and 3.7
so umask of 0 doesn't work consistently).
2018-10-10 14:25:51 -07:00
Brian C. Lane
0037f10205 Add an openstack image type
This is a qcow2 image with cloud-init in the template.
2018-10-09 15:17:45 -07:00
Brian C. Lane
9d73975c9f Update cli tests to use composer-cli name 2018-10-09 15:17:13 -07:00
Brian C. Lane
fb1dfc9488 Report an error if the blueprint doesn't exist
composer-cli uses TOML for 'blueprints save' which was returning an
empty 200 response if the blueprint didn't exist. Change this to return
a standard 400 error response if the blueprint doesn't exist.

composer-cli is already setup to handle receiving json when an error is
returned so just the toml API response for `blueprints/save` needed to
be changed.
2018-10-09 15:16:20 -07:00
Brian C. Lane
c2b0e8a8bb Write a rootpw line if no root customizations in the blueprint
Anaconda requires the root password to be set or locked, so if there
isn't anything setting it we write out 'rootpw --lock'

Also adds tests for this.

Resolves: rhbz#1626122
2018-10-09 15:15:42 -07:00
Brian C. Lane
558fdecde5 Add beakerlib to Dockerfile.test
Also kill the lorax-composer process and remove /run/weldr/api.socket
so that when this is run with podman you don't get an error about
attempting to tar up the socket.
2018-10-09 15:15:23 -07:00
Brian C. Lane
f5679f61b1 Add a test for repo metadata expiration
This tests to make sure that the metadata timer is working (by setting
it to 10s and adding a new package to the repo), and that
DNFLock.lock_check immediately picks up a new package.

This depends on rpmfluff which is available from Fedora or EPEL repos.

Related: rhbz#1631561
2018-10-09 15:12:21 -07:00
Brian C. Lane
8479319621 Add tests for setting root password and ssh key with blueprints
Related: rhbz#1626120
2018-10-09 15:12:09 -07:00
David Shea
3ea07ed44a Add new compose types to compose sanity test 2018-10-09 15:11:36 -07:00
Lars Karlitski
d8cb1a19f8 Fix pylint errors and warnings
Remove `except` block which immediately raises the same exception again (it's
not a subclass of another caught exception, so this is safe).

Remove a false positive, because it is not emitted from the code base.

Disable subprocess-popen-preexec-fn in startProgram, which is not used
internally.
2018-10-09 15:04:43 -07:00
Alexander Todorov
ca1bf01b03 New cli test covering basic compose commands
- need to specify --sharedir so lorax-composer can find its
  kickstart files

- each test script writes results into a separate directory to
  avoid a passing test overwriting the results from a failing one.
  To avoid reporting failures in case of previously failing tests
  (e.g. during development) remove the temporary directories holding
  tets results before execution!
2018-10-09 15:03:24 -07:00
Alexander Todorov
ee62425388 Execute bash tests for composer-cli
these are built on top of beakerlib and we use its internal
protocol to figure out the result without relying on the full
test runner that is tipically used inside of a RHEL environment!

Includes a disabled test snippet for Issue #460
2018-10-09 15:02:57 -07:00
David Shea
de6e1d027e Add a vmdk compose type.
This is similar to the AMI type, but also adds open-vm-tools and does not do
anything special to the partitioning

(cherry picked from commit 1056bfc25b)
2018-10-02 12:56:52 -04:00
David Shea
68c1a7aa96 Add a vhd compose type for Azure images
This does pretty much the same things as the AMI compose type, but also
replaces NetworkManager with the Azure linux agent.

(cherry picked from commit e0c236ff36)
2018-10-02 12:56:47 -04:00
David Shea
f79fd46f1f Add an ami compose type for AWS images
This differs from lmc's --make-ami in that creates a full disk image instead of
an fsimage. Create a raw disk image with a / and /boot partitions, and enable
sshd, chronyd, and cockpit by default.

(cherry picked from commit 18188bf6cf)
2018-10-02 12:56:33 -04:00
Brian C. Lane
8838ee12bc Add tests for limit=0 routes
Passing ?limit=0 to the blueprints/list, blueprints/changes,
projects/list, modules/list should always return the total possible
results, not 0.

Also move the composer-cli test_diff to the end so that it will work
consistently. Do this by naming it test_z_diff.

(cherry picked from commit 972b5c4142)
2018-08-27 12:02:39 -07:00
Brian C. Lane
7773be2bda Fix tests related to blueprint name changes 2018-08-23 13:46:39 -07:00
Brian C. Lane
b198c6e498 Add 'example' to the example blueprint names 2018-08-23 13:45:32 -07:00
David Shea
eedf1a9243 Add tests for /compose/status filter arguments
(cherry picked from commit c52ba4236a)
2018-08-21 16:50:53 -07:00
Chris Lumens
a08c16158d Fix more tests.
These were broken due to me rebasing before sending a PR off.  They
should work now.

(cherry picked from commit 4cb15e0a0f)
2018-08-09 16:33:30 -04:00
Chris Lumens
7c19be3792 Add error IDs to things that can go wrong when running a compose.
Note the exception string checking around compose_type.  I didn't really
want to introduce a new exception type just for this, but also didn't
want to duplicate strings.  I'd be open to other suggestions for how to
do this.

(cherry picked from commit b3bb438254)
2018-08-09 16:32:45 -04:00
Chris Lumens
2a8a9bc506 Add error IDs for when an unknown blueprint is requested.
This adds some fairly redundant code to the beginning of all the
blueprint routes to attempt reading a commit from git for the
blueprint's recipe.  If it succeeds, the blueprint exists and the route
can continue.  Otherwise, return an error.  Hopefully this doesn't slow
things down too much.

(cherry picked from commit a925cc7ddb)
2018-08-09 16:32:22 -04:00
Chris Lumens
44b8b79af7 Add error IDs for when an unknown build UUID is requested.
Note that this also changes the return type of uuid_info to return None
when an unknown ID is given.  The other uuid_* functions are fine
because they are checked ahead of time.

(cherry picked from commit 6497b4fb65)
2018-08-09 16:32:05 -04:00
Brian C. Lane
a034b205aa Drop .decode from UTF8_TEST_STRING 2018-08-07 10:07:39 -07:00
Brian C. Lane
2b0f9c4bd7 Add input string checks to the branch and format arguments
Make sure no UTF8 characters are allowed and return an error if they
are.

Also includes tests to make sure the correct error is returned.

(cherry picked from commit 86d79cd8a6)
2018-08-07 09:45:02 -07:00
Brian C. Lane
f5cdb94c9c Add a test for invalid characters in the API route
Currently the code is not UTF8 safe, so we need to return a clear error
when invalid characters are passed in.

This also adds tests for the routes to confirm that an error is
correctly returned.

(cherry picked from commit 74f5def3d4)
2018-08-07 09:45:02 -07:00
Brian C. Lane
a6c3a82695 Update get_system_repo for dnf
With dnf we iterate the repos and return their names, not the filenames
like we do with yum. Also make the list test more flexible.
2018-08-07 09:20:16 -07:00
Brian C. Lane
e9b16c7a47 Update ConfigParser usage for Py3
SafeConfigParser is now just ConfigParser
2018-08-07 08:41:43 -07:00
Brian C. Lane
4bb7f14e2a Update StringIO use for Py3
It is imported from io now.
2018-08-07 08:41:11 -07:00
Brian C. Lane
619a9a2939 Add a test for the pylorax.api.timestamp functions 2018-08-07 08:40:21 -07:00
Brian C. Lane
5ebcd2cf11 Use the first enabled system repo for the test
(cherry picked from commit 3fd5e50c80)
2018-08-06 17:00:56 -07:00
Brian C. Lane
51c17ae361 Show more details when the system repo delete test fails
Works fine for me locally, but not in Travis for some reason so let's
get more info.

(cherry picked from commit 26df083bd4)
2018-08-06 17:00:56 -07:00
Brian C. Lane
7869c59847 Add composer-cli function tests
These depend on there being a freshly installed lorax-composer API
server running, if there is no /run/weldr/api.socket they will be
skipped.

(cherry picked from commit 7700ae3135)
2018-08-06 17:00:56 -07:00
Brian C. Lane
03e6015f20 Add a test library
This adds empty __init__.py to tests so that a lib.py library of helper
functions can be imported from the tests.

Add captured_output to use with composer-cli tests to capture stdout/err
output from the functions.

(cherry picked from commit eeae331ba0)
2018-08-06 17:00:56 -07:00
Brian C. Lane
74d76789e1 Modify handle_api_result so it can be used in more places
Some results have errors and no status, others have status and errors.
Update the function to return the final rc to exit with, and a bool
indicating whether or not to continue processing the other fields.

Add a bunch of tests for the new function to make sure I have the logic
correct.

(cherry picked from commit 35fa067219)
2018-08-06 17:00:56 -07:00
Brian C. Lane
b0b97adf75 Update rsync version in http-server.toml 2018-07-18 16:50:51 -07:00
Chris Lumens
dbeb653942 Add group-based tests.
(cherry picked from commit ab0655d5a9)
(cherry picked from commit e151c8399d)
2018-07-18 16:35:48 -07:00
Chris Lumens
0259f3564d Include groups in depsolving.
This adds a new argument to projects_depsolve and
projects_depsolve_with_size that contains the group list, unfortunately.
I would have prefered adding a function that just returns a list of all
the contents of a group and then add that to what was being passed into
projects_depsolve.  However, there does not appear to be any good way to
do that in yum aside from a lot of grubbing around in the comps object,
which I am unwilling to do.

(cherry picked from commit 5fe4b47072)
(cherry picked from commit 8c4804eb15)
2018-07-18 16:35:48 -07:00
Chris Lumens
f1f8980c49 Add support for groups to blueprints.
Nothing is currently being done with this information, but it will be
soon.

(cherry picked from commit 0f69d2084c)
(cherry picked from commit 76d376fe18)
2018-07-18 16:35:48 -07:00
Brian C. Lane
d92f2f5b04 Check the compose templates at startup
Depsolve the packages included in the templates and report any errors
using the /api/status 'msgs' field. This should help narrow down
problems with package sources not being setup correctly.
2018-07-13 09:51:36 -07:00
Brian C. Lane
841e65a397 Allow more than 1 bash build in tests
It is perfectly valid to have more than one build of a package, eg. one
in the release repo and one in the updates repo.

(cherry picked from commit 86c4ef5f45)
2018-06-21 16:19:06 -07:00
Brian C. Lane
693dc9a6f6 Fix DNF related issues with source selection
DNF Repo.dump() function cannot be used as a .repo file for dnf due to
it writing baseurl and gpgkey as a list instead of a string. Add a new
function to write this in the correct format, and limited to the fields
we use.

Add a test for the new function.

Fix /projects/source/info to return an error 400 if a nonexistant TOML
source is requested. If JSON is used the error is part of the standard
response.

Update test_server.py to check for the correct error code.

(cherry picked from commit afa89ea657)
2018-06-04 15:47:38 -07:00