Commit Graph

725 Commits

Author SHA1 Message Date
Brian C. Lane
5c878a695f Add --test option to composer-cli
The API server will run a mock compose when a test mode is passed to it.
Passing a 1 will queue a build, pretend to run the build for 10 seconds,
and then fail. Passing a 2 will do the same thing, but it will finish as
if it was successful. All results are available but the output file is
only the string "TEST IMAGE"

This should allow running tests inside docker without calling anaconda
(becuase it will not run in docker, it needs a VM).
2018-06-11 16:54:58 -07:00
Brian C. Lane
1a4a4defcc Make sure lorax-composer tests only use temporary directories
It was using /var/tmp/composer/, now it places everything under a
temporary directory that is removed when it is finished.
2018-06-11 16:54:58 -07:00
Brian C. Lane
d4b99b5ee9 Refactor get_filename so it can be tested
Just pass in the headers instead of the whole response object so that it
can be tested without needing an actual server.
2018-06-11 16:54:58 -07:00
Brian C. Lane
d0676dbf7c Fix bug in prettyDiffEntry output
It was not handling adding or removing modules correctly.
2018-06-11 16:54:58 -07:00
Brian C. Lane
aa076ee53b composer-cli: Handle download errors
The API will return an error 400 with a json error object if the
download fails, catch this and report the error to the user without a
traceback.
2018-06-11 16:54:58 -07:00
Brian C. Lane
de604b37a6 Add a pid file for lorax-composer
This will prevent accidentally running more than 1 instance.
Uses /run/lorax-composer.pid and checks to make sure that the PID
written to it isn't stale.
2018-06-11 16:54:58 -07:00
Brian C. Lane
7ad9939fac Cleanup more /tmp/ files when running with --no-virt
Make extra sure anaconda starts with a clean slate.
2018-06-11 16:54:58 -07:00
Brian C. Lane
e425b7ccca lorax-composer: Update the yum metadata at startup
Otherwise it can become out of date and depsolving will fail.
2018-06-11 16:54:58 -07:00
Brian C. Lane
2e61bc8bf9 Fix the error responses from lorax-composer
They are now all consistent, returning status 400, including a json
object with status=flase, and an error message inside the error object.
2018-06-11 16:54:58 -07:00
Brian C. Lane
60874d0197 Check to make sure image file exists for /compose/image/
Return an error 400 with an error message if it is missing.
2018-06-11 16:54:58 -07:00
Brian C. Lane
dbe9c15ee1 Add UUID prefix to /compose/image/ download filename.
This will make sure it is received as a unique filename, not just
'disk.img' when downloading it.
2018-06-11 16:54:58 -07:00
Brian C. Lane
56766e794f Add support for composer-cli compose commands.
This adds all the commands needed to start, monitor, delete, and
download a compose.
2018-06-11 16:54:58 -07:00
Brian C. Lane
9ba24f305d Add support for modules list, projects list, and projects info 2018-06-11 16:54:58 -07:00
Brian C. Lane
d2f784e5da Add composer-cli utility and implement the recipes commands
composer-cli --help shows the commands.
Output defaults to human readable, but raw json can be displayed by
passing --json
2018-06-11 16:54:58 -07:00
Brian C. Lane
79fa1c957e Add ?format=toml support to /recipes/freeze 2018-06-11 16:54:58 -07:00
Brian C. Lane
0ea4eb5206 Fix epoch to ouput an int instead of a str
Also updates the tests.
2018-06-11 16:54:58 -07:00
Brian C. Lane
413964b92f Add ?format=toml support to /recipes/info/
This will return the recipe in TOML format. Note that this does not
include any extra information about errors. Just the recipes, any
unrecognized recipe names will be skipped.
2018-06-11 16:54:58 -07:00
Brian C. Lane
41fd9ab19c lorax-composer: Add partitioned-disk output support
Also add more documentation on lorax-composer, including how to add
support for new output types.
2018-06-11 16:54:58 -07:00
Brian C. Lane
4ae5944567 Add live-iso output support to lorax-composer
This ended up requiring more intrusive changes, but it should be the
most complex of the output types. After moving the core of
livemedia-creator into a function I added more settings to compose_args,
and more defaults to start_build. It now pulls the release information
from /etc/os-release, and produces a bootable .iso
2018-06-11 16:54:58 -07:00
Brian C. Lane
59fd3934f3 Move core of livemedia-creator to run_creator()
We need to be able to share the output types from livemedia-creator with
lorax-composer, so move the core of the main() function into
run_creatoe(). Pass in the cmdline args or a DataHolder with them set.
2018-06-11 16:54:58 -07:00
Brian C. Lane
85ee784efc Only chown recipe directory if it already exists
Since creation of the directory has been moved after switching users
this now needs to check to make sure it exists first.
2018-06-11 16:54:58 -07:00
Brian C. Lane
74326ba7a3 Fix a problem with diff/NEWEST/WORKSPACE
If there is no WORKSPACE yet it was not falling back to using NEWEST.
Also add a test for this.
2018-06-11 16:54:58 -07:00
Brian C. Lane
4144c3937c Check for a source tree doc install first, not second.
This way a source install and test run will always take precedence over
a system installed version.
2018-06-11 16:54:58 -07:00
Alexander Todorov
1f90455ee6 Add required_methods for decorator
otherwise we can't issue OPTIONS requests to the server, see
first comment at http://flask.pocoo.org/snippets/56/
2018-06-11 16:54:58 -07:00
Alexander Todorov
7f7295e3b0 Convert max_age to int b/c timedelta.total_seconds() is a float 2018-06-11 16:54:58 -07:00
Brian C. Lane
c399447b81 Fix a problem with using a mirror as the primary url
Because of how Anaconda is run it needs to be passed a baseurl (using
--repo on the anaconda cmdline), not a mirrorlist url. This fixes it so
that the first mirror is used if the main repository is using a
mirrorlist.
2018-06-11 16:54:58 -07:00
Brian C. Lane
a1427b1a31 Set the HOME variable to a directory the uid can access
libgit2 will look for $HOME/.gitconfig after dropping root privileges so
this needs to be accessible to the user.
2018-06-11 16:54:58 -07:00
Brian C. Lane
442dc43206 Open the git repo after dropping root privileges
Also make sure the recipe directory and its contents have correct
ownership, and change the default recipe path when using the systemd
service to /var/lib/lorax/composer/recipes/
2018-06-11 16:54:58 -07:00
Brian C. Lane
4ae2e1450d Exit on uid/gid errors before checking directory permissions
If the uid/gid check fails there is no sense in trying to examine the
ownership or permissions.
2018-06-11 16:54:58 -07:00
Brian C. Lane
daa5ed2aaf Add documentation for /compose and /compose/types 2018-06-11 16:54:58 -07:00
Brian C. Lane
eb2a7208c8 Move queue monitor startup into a function
This allows it to be used by the test framework.
2018-06-11 16:54:58 -07:00
Brian C. Lane
63a7897457 Move queue directory creation into a function
So that it can also be used during tests
2018-06-11 16:54:58 -07:00
Brian C. Lane
8da6214e10 Add a test mode to /compose
This will allow testing without having a full system setup with
anaconda, if ?test=1 is passed to the POST /compose command it will wait
10 seconds instead of running Anaconda, and then raise an error to
generate a failed build.

Passing ?test=2 will also wait 10 seconds instead of running Anaconda,
but will finish successfully.
2018-06-11 16:54:58 -07:00
Brian C. Lane
2e1e8c20c7 Cleanup docstrings for queue.py 2018-06-11 16:54:58 -07:00
Brian C. Lane
9603c866af Drop cancel_q from the monitor() function
Not needed, builds are canceled by writing a 'CANCEL' file in the results
directory.
2018-06-11 16:54:58 -07:00
Brian C. Lane
166350fcb6 Fix the jsonify calls to use kwargs
No need to pass it a dict, just set the args.
2018-06-11 16:54:58 -07:00
Brian C. Lane
ff7d36bcbc Add /compose/log/ API to retrieve the end of the build log
This allows the client to request the end of the anaconda.log during and
after a build. The amount of data returned can be set by adding
?size=<kbytes>

Output is raw bytes, starting on the next available line boundry.
2018-06-11 16:54:58 -07:00
Brian C. Lane
caee8e7cdf Return a status of false if the uuid isn't valid 2018-06-11 16:54:58 -07:00
Brian C. Lane
072aa720ff Add /compose/cancel API to cancel a running build
If the build hasn't started yet (state is WAITING) try removing the
symlink to it. If this succeeds, delete the partial results directory.

If the build makes it to RUNNING then it writes a CANCEL file in the
results directory. The callback that is passed to execWithRedirect
catches this, causing a SIGTERM to be sent to anaconda. It then exits
and cleanup happens normally. The partial results directory is then
removed.
2018-06-11 16:54:58 -07:00
Brian C. Lane
8f0bca00c0 Pass the callback_func through novirt_install to execWithRedirect
The default is no callback.
2018-06-11 16:54:58 -07:00
Brian C. Lane
f1d8b10867 Add a callback to execWithRedirect
If the callback returns true send the process a SIGTERM to terminate it.
2018-06-11 16:54:58 -07:00
Alexander Todorov
32db249538 Add tests for yumbase and update how we inspect boolean options
.get_default() returns string so make sure we're actually parsing
the value as boolean and not evaluating a non-empty string in a
boolean context (which will always return True)
2018-06-11 16:54:58 -07:00
Brian C. Lane
6a95a314e9 Add API routes for downloading build results
Also fix a bug with the name of the queue status in the status results
(it is now 'queue_status' not 'status' which is used for error
responses).

This adds the following routes:
 - /compose/metadata/<uuid> to retrieve a .tar of the build metadata
 - /compose/results/<uuid> to retrieve .tar of all of the build results
 - /compose/logs/<uuid> to retrieve a .tar of just the logs from the build
 - /compose/image/<uuid> to retrieve the output image from the build
2018-06-11 16:54:58 -07:00
Brian C. Lane
ed03ac7524 Add /compose/info route to retrieve details about a compose
The results is a JSON string with the following information:

* id - The uuid of the comoposition
* config - containing the configuration settings used to run Anaconda
* recipe - The depsolved recipe used to generate the kickstart
* commit - The (local) git commit hash for the recipe used
* deps - The NEVRA of all of the dependencies used in the composition
* compose_type - The type of output generated (tar, iso, etc.)
* queue_status - The final status of the composition (FINISHED or FAILED)
2018-06-11 16:54:58 -07:00
Brian C. Lane
d3a9ec3002 Return the commit id for the recipe being read
This adds returning the commit id from read_commit, and a new function
read_recipe_and_id() that returns the commit id and the recipe in a
tuple.

If the commit is passed in, it is used as is. If no commit is passed in
it finds the most recent commit for the file on the selected branch and
returns that.

Missing recipes now raise a RecipeError with an informative message.
eg. "No commits for missing-recipe.toml on the master branch."
2018-06-11 16:54:58 -07:00
Brian C. Lane
22c03ceba7 Fix yum config directory creation for projects and server tests 2018-06-11 16:54:58 -07:00
Brian C. Lane
b99a7756de Add DELETE /compose/delete/<uuids> API route
This will delete all of the build's results and cannot be undone.
2018-06-11 16:54:58 -07:00
Brian C. Lane
3fdf3a9819 Turn on o+x permission for the queue and results directories
Otherwise it won't allow a group member to delete things.
2018-06-11 16:54:58 -07:00
Brian C. Lane
fdcf839753 Add /compose/status/<uuids> to retrieve details of a specific build 2018-06-11 16:54:58 -07:00
Brian C. Lane
91c6d1525d Add compose status routes /compose/finished and /compose/failed
These will return a list of details about the finished or failed builds.
2018-06-11 16:54:58 -07:00