Commit Graph

642 Commits

Author SHA1 Message Date
Brian C. Lane
63a7897457 Move queue directory creation into a function
So that it can also be used during tests
2018-06-11 16:54:58 -07:00
Brian C. Lane
8da6214e10 Add a test mode to /compose
This will allow testing without having a full system setup with
anaconda, if ?test=1 is passed to the POST /compose command it will wait
10 seconds instead of running Anaconda, and then raise an error to
generate a failed build.

Passing ?test=2 will also wait 10 seconds instead of running Anaconda,
but will finish successfully.
2018-06-11 16:54:58 -07:00
Brian C. Lane
2e1e8c20c7 Cleanup docstrings for queue.py 2018-06-11 16:54:58 -07:00
Brian C. Lane
9603c866af Drop cancel_q from the monitor() function
Not needed, builds are canceled by writing a 'CANCEL' file in the results
directory.
2018-06-11 16:54:58 -07:00
Brian C. Lane
166350fcb6 Fix the jsonify calls to use kwargs
No need to pass it a dict, just set the args.
2018-06-11 16:54:58 -07:00
Brian C. Lane
ff7d36bcbc Add /compose/log/ API to retrieve the end of the build log
This allows the client to request the end of the anaconda.log during and
after a build. The amount of data returned can be set by adding
?size=<kbytes>

Output is raw bytes, starting on the next available line boundry.
2018-06-11 16:54:58 -07:00
Brian C. Lane
caee8e7cdf Return a status of false if the uuid isn't valid 2018-06-11 16:54:58 -07:00
Brian C. Lane
072aa720ff Add /compose/cancel API to cancel a running build
If the build hasn't started yet (state is WAITING) try removing the
symlink to it. If this succeeds, delete the partial results directory.

If the build makes it to RUNNING then it writes a CANCEL file in the
results directory. The callback that is passed to execWithRedirect
catches this, causing a SIGTERM to be sent to anaconda. It then exits
and cleanup happens normally. The partial results directory is then
removed.
2018-06-11 16:54:58 -07:00
Brian C. Lane
8f0bca00c0 Pass the callback_func through novirt_install to execWithRedirect
The default is no callback.
2018-06-11 16:54:58 -07:00
Brian C. Lane
f1d8b10867 Add a callback to execWithRedirect
If the callback returns true send the process a SIGTERM to terminate it.
2018-06-11 16:54:58 -07:00
Alexander Todorov
32db249538 Add tests for yumbase and update how we inspect boolean options
.get_default() returns string so make sure we're actually parsing
the value as boolean and not evaluating a non-empty string in a
boolean context (which will always return True)
2018-06-11 16:54:58 -07:00
Brian C. Lane
6a95a314e9 Add API routes for downloading build results
Also fix a bug with the name of the queue status in the status results
(it is now 'queue_status' not 'status' which is used for error
responses).

This adds the following routes:
 - /compose/metadata/<uuid> to retrieve a .tar of the build metadata
 - /compose/results/<uuid> to retrieve .tar of all of the build results
 - /compose/logs/<uuid> to retrieve a .tar of just the logs from the build
 - /compose/image/<uuid> to retrieve the output image from the build
2018-06-11 16:54:58 -07:00
Brian C. Lane
ed03ac7524 Add /compose/info route to retrieve details about a compose
The results is a JSON string with the following information:

* id - The uuid of the comoposition
* config - containing the configuration settings used to run Anaconda
* recipe - The depsolved recipe used to generate the kickstart
* commit - The (local) git commit hash for the recipe used
* deps - The NEVRA of all of the dependencies used in the composition
* compose_type - The type of output generated (tar, iso, etc.)
* queue_status - The final status of the composition (FINISHED or FAILED)
2018-06-11 16:54:58 -07:00
Brian C. Lane
d3a9ec3002 Return the commit id for the recipe being read
This adds returning the commit id from read_commit, and a new function
read_recipe_and_id() that returns the commit id and the recipe in a
tuple.

If the commit is passed in, it is used as is. If no commit is passed in
it finds the most recent commit for the file on the selected branch and
returns that.

Missing recipes now raise a RecipeError with an informative message.
eg. "No commits for missing-recipe.toml on the master branch."
2018-06-11 16:54:58 -07:00
Brian C. Lane
22c03ceba7 Fix yum config directory creation for projects and server tests 2018-06-11 16:54:58 -07:00
Brian C. Lane
b99a7756de Add DELETE /compose/delete/<uuids> API route
This will delete all of the build's results and cannot be undone.
2018-06-11 16:54:58 -07:00
Brian C. Lane
fdcf839753 Add /compose/status/<uuids> to retrieve details of a specific build 2018-06-11 16:54:58 -07:00
Brian C. Lane
91c6d1525d Add compose status routes /compose/finished and /compose/failed
These will return a list of details about the finished or failed builds.
2018-06-11 16:54:58 -07:00
Brian C. Lane
520177afca Add /compose/queue to get the status of the build queue
This will return a JSON result that includes the waiting builds,
and the running builds.
2018-06-11 16:54:58 -07:00
Brian C. Lane
47f21af7fb Add reading a recipe directly from a file
Adds helper function recipe_from_file to recipes.py
2018-06-11 16:54:58 -07:00
Brian C. Lane
0065cadf2d Include the recipe in the results of a build
Write original as recipe.toml and the depsolved version as frozen.toml

Also write 'WAITING' to the STATUS file as its first state.
The STATUS states are now WAITING -> RUNNING -> FINISHED|FAILED
2018-06-11 16:54:58 -07:00
Brian C. Lane
9fc973366f Move creating a frozen recipe into recipes.py
Also adds .package_names and .module_names properties. Call
recipe.freeze with a list of NEVRA dependencies and it will return a new
Recipe object with all of the packages and modules set to the depsolved
version.
2018-06-11 16:54:58 -07:00
Brian C. Lane
67da4d6971 Add building an image, and the /compose route to start it
This adds the ability to build a tar output image. The /compose and
/compose/types API routes are now available.

To start a build POST a JSON body to /compose, like this:

{"recipe_name":"glusterfs", "compose_type":"tar", "branch":"master"}

This will return a unique build id:

{
  "build_id": "4d13abb6-aa4e-4c80-a671-0b867e6e77f6",
  "status": true
}

which will be used to keep track of the build status (routes for this
do not exist yet).
2018-06-11 16:54:58 -07:00
Brian C. Lane
de9f5b0456 Remove test configuration and read it from the build directory 2018-06-11 16:54:58 -07:00
Brian C. Lane
7372faa969 Add function to return full NEVRA of a dependency 2018-06-11 16:54:58 -07:00
Brian C. Lane
c92373ecd6 Change config and paths
With the API running as weldr the permissions on the yum directories
needs to be accessable to it, as well as the results and queue/new
directories.
2018-06-11 16:54:58 -07:00
Brian C. Lane
e2f4674fb3 Add basic composer queue handling
The queue is in /var/lib/weldr/queue/new by default. It watches the
directory for new symlinks (to /var/lib/weldr/results/<dirname>) and
handles running anaconda on the kickstart found in final-kickstart.ks
inside the symlinked directory.
2018-06-11 16:54:58 -07:00
Brian C. Lane
0ce4197a1e Change compress to use communicate instead of wait
Also move default_image_name into imgutils so it can be used in other
places.

When running from lorax-composer the wait() call wasn't waiting until
the tar was finished. I think this is due to gevent monkey-patching
something. Using communicate() solves this problem.
2018-06-11 16:54:58 -07:00
Brian C. Lane
57d9d40c29 Add documentation for the API routes.
Use Sphinx on a CentOS7/RHEL7 system to generate the docs with 'make docs'
2018-06-11 16:54:58 -07:00
Brian C. Lane
916a001d73 Add support for other branches to the routes
Passing ?branch=<branch-name> will use the specified branch instead of
master.

The new branch will not exist until a /recipes/new?branch=new-branch
POST is made. At that time the branch will be created based on the
current master branch and the new commit will be added to it.
2018-06-11 16:54:58 -07:00
Alexander Todorov
0af072c77e Silence pocketlint bad-preconf-access warnings 2018-06-11 16:54:58 -07:00
Martin Pitt
176b12dd8c Fix depsolving empty recipes
- Fix `projects_depsolve()` to not consider a successful empty response
   (rc == 0) as an error.
 - Fix recipe_from_dict() to default modules and packages to empty lists
   instead of `None`, to avoid a Python-ism in the API for consumers and
   stay compatible to the bdcs API.

Fixes #290
2018-06-11 16:54:58 -07:00
Martin Pitt
da3616db72 Fix wrong function name in api/v0/recipes/freeze error messages 2018-06-11 16:54:58 -07:00
Brian C. Lane
3e5c0707bb Redirect yum's logging to yum.log 2018-06-11 16:54:58 -07:00
Brian C. Lane
a44e499eee Close the rpmdb after every API operation.
rpm blocks all of the signals until closeRpmDB() is called.
We need to call it after all of the API calls.
2018-06-11 16:54:58 -07:00
Brian C. Lane
cf42e1a480 Fix error string when there is a problem listing projects 2018-06-11 16:54:58 -07:00
Brian C. Lane
efd15aaad7 Add --releasever option to lorax-composer
If it isn't set the releasever value will use yum and the system / to
find the releasever.
2018-06-11 16:54:58 -07:00
Brian C. Lane
bae551b7ab Add filtering and glob support to /modules/list route 2018-06-11 16:54:58 -07:00
Brian C. Lane
6b4bf92e7f Add /recipes/freeze route and tests. 2018-06-11 16:54:58 -07:00
Brian C. Lane
b8212358bc Add /recipes/depsolve route and test 2018-06-11 16:54:58 -07:00
Brian C. Lane
a85d0ced8b Add /projects and /modules API tests 2018-06-11 16:54:58 -07:00
Brian C. Lane
164232ba4d Modify pylorax.api.config.configure so it can also be used for tests. 2018-06-11 16:54:58 -07:00
Brian C. Lane
ad5ab71551 Move ComposerConfig into pylorax.api.config module 2018-06-11 16:54:58 -07:00
Brian C. Lane
8fee86a5db Catch ProjectsError and return an error 400 with a message. 2018-06-11 16:54:58 -07:00
Brian C. Lane
a1e7e4831d Catch Yum errors in the projects functions 2018-06-11 16:54:58 -07:00
Brian C. Lane
23d90ec066 Add /modules/list and /modules/info routes 2018-06-11 16:54:58 -07:00
Brian C. Lane
8635a59ac7 Add modules functions and update function documentation 2018-06-11 16:54:58 -07:00
Brian C. Lane
c3d495fe53 Add /projects/depsolve route 2018-06-11 16:54:58 -07:00
Brian C. Lane
99140b98b9 Add /projects/info route
Elements that have no match in yum (metadata) are set to empty dicts.
2018-06-11 16:54:58 -07:00
Brian C. Lane
0819e97646 Add /projects/list route 2018-06-11 16:54:58 -07:00
Brian C. Lane
105679d79d Add /api/v0/test route
Returns a simple string to indicate that the API server is running.

/api/v0/status should be used instead, it provides more detailed info in
JSON format.
2018-06-11 16:54:58 -07:00
Brian C. Lane
569ce1d3dc Add support for yum to lorax-composer
This includes a new configuration file at /etc/lorax/composer.conf with
built-in defaults. It also adds a YUMLOCK server config object so that
request handlers can access the yum base object without interfering
with each other.
2018-06-11 16:54:58 -07:00
Brian C. Lane
5d6c236af6 Add /api/docs to serve up the documentation
This requires that the docs be at /usr/share/doc/lorax-*/html/ or if
running from the source tree, at ./docs/html/

They can be re-created by running 'make docs'
2018-06-11 16:54:58 -07:00
Brian C. Lane
1d53d00532 Add limit/offset to recipes/list 2018-06-11 16:54:58 -07:00
Brian C. Lane
2cd4fcfd88 Add error message for offset/limit type errors 2018-06-11 16:54:58 -07:00
Brian C. Lane
9ca9bfe46f Add error logging to api/v0.py
These will end up in the /var/log/lorax-composer/composer.log
2018-06-11 16:54:58 -07:00
Brian C. Lane
d701120d90 Add /recipes/diff route and tests 2018-06-11 16:54:57 -07:00
Brian C. Lane
052a8ba991 Add recipe_diff function and helpers.
This takes a pair of Recipe objects and returns a list of diff dicts
that include what was changed between the two recipes.
2018-06-11 16:54:57 -07:00
Brian C. Lane
a76e95dcb5 Add POST /recipes/tag/ route and tests 2018-06-11 16:54:57 -07:00
Brian C. Lane
7f1adf120c Add tag_recipe_commit helper function
And change tests to use it so that both it and tag_file_commit are used.
2018-06-11 16:54:57 -07:00
Brian C. Lane
3f4140d5d3 Add POST /recipes/undo route and tests 2018-06-11 16:54:57 -07:00
Brian C. Lane
639d325a0c Change read_recipe_commit to use the recipe name
Callers really shouldn't need to know the details of the filenames, so
change it to convert it internally.
2018-06-11 16:54:57 -07:00
Brian C. Lane
ab6decec0b Add revert_recipe function
And adjust the tests to use it so that it covers both revert_recipe and
revert_file.
2018-06-11 16:54:57 -07:00
Brian C. Lane
c294b7747d Add DELETE /recipes/delete/<recipe_name> route and tests
Also add numbering to the tests so that they are executed in the
expected order.
2018-06-11 16:54:57 -07:00
Brian C. Lane
c937e69d3b Add delete_recipe helper function and test
Use delete_recipe in the test so that both it and delete_file are
covered.
2018-06-11 16:54:57 -07:00
Brian C. Lane
6e453e70dc Add DELETE /recipes/workspace/<recipe_name> route and tests
This will delete the workspace copy of the recipe.
2018-06-11 16:54:57 -07:00
Brian C. Lane
72b4c4fe28 Add POST /recipes/workspace route
Also fix use of workspace in /recipes/info (was using filename instead
of recipe name and logic for changed was backwards).
2018-06-11 16:54:57 -07:00
Brian C. Lane
711cdf53b3 Add /recipes/new route and tests
This handles json data when the mime type is "application/json" and toml
recipes when it is "text/x-toml"
2018-06-11 16:54:57 -07:00
Brian C. Lane
7b5115d19c Split recipe_from_toml into recipe_from_dict helper.
This will be useful for creating Recipe objects from json created dicts.
2018-06-11 16:54:57 -07:00
Brian C. Lane
3c75711b30 Fix the recipe version bumping
Recipe should have its version bumped based on the version from the
previous commit, and not be bumped on the first commit. Fix the code and
the tests.
2018-06-11 16:54:57 -07:00
Brian C. Lane
02db2afb2b Add /recipes/changes route with tests. 2018-06-11 16:54:57 -07:00
Brian C. Lane
1e7335a506 Add /recipes/info route and tests 2018-06-11 16:54:57 -07:00
Brian C. Lane
b7f651a631 Add workspace module and tests
This handles the recipe workspace, which is a temporary location to
store recipes before committing them to git. It will be used by the
/recipes/ API.
2018-06-11 16:54:57 -07:00
Brian C. Lane
b6fb22133c Add /recipes/list route and tests
Includes adding a lock for access to the git repo from the API.
2018-06-11 16:54:57 -07:00
Brian C. Lane
90a8798f4c Move the git repo into a subdirectory
And fix the import of recipes (needs to have full path passed in).
2018-06-11 16:54:57 -07:00
Brian C. Lane
596f06d5c0 Fix list_commits sort order.
It appears that with libgit2 v0.24.6 reverse causes it to list them
newest first. In 0.25.1 it lists them oldest first. On both versions
just using SortMode.TIME gives the desired result of oldest first.
2018-06-11 16:54:57 -07:00
Brian C. Lane
bfc9c6913f Add pylorax.api.recipes code for handling the Recipe's Git repository 2018-06-11 16:54:57 -07:00
Alexander Todorov
2cd4f73177 Don't do wildcard imports 2018-06-11 16:54:57 -07:00
Alexander Todorov
e9e6e5701e Misc pylint fixes that are reported usually once 2018-06-11 16:54:57 -07:00
Alexander Todorov
bf9ed50d51 Fix dangerous-default value warnings
when default value is list or dict the default arguments are
instantiated as objects at the time of definition. This is significant
(exposing visible semantics) when the object is mutable. There’s no way
of re-binding that default argument name in the function’s closure. When
function is executed multiple times with its default value the value
will change between executions, possibly leading to strange side effects.

For more information see:
http://satran.in/2012/01/12/python-dangerous-default-value-as-argument.html
2018-06-11 16:54:57 -07:00
Alexander Todorov
a0766b1d70 Don't redefine variables from outer scope 2018-06-11 16:54:57 -07:00
Alexander Todorov
4d8e2b5356 Define all class attributes inside __init__ 2018-06-11 16:54:57 -07:00
Alexander Todorov
663b0f8378 Fix logging formatting 2018-06-11 16:54:57 -07:00
Alexander Todorov
18528b4f47 Don't redefine builtins 2018-06-11 16:54:57 -07:00
Alexander Todorov
b15df5a0af Silence relative import warnings 2018-06-11 16:54:57 -07:00
Alexander Todorov
7f5b492198 pylint fix: unused variable warning 2018-06-11 16:54:57 -07:00
Alexander Todorov
15de3e1687 pylint fix: remove unused imports 2018-06-11 16:54:57 -07:00
Brian C. Lane
2008a413b6 lorax-composer initial commit
The lorax-composer program will launch a BDCS compatible API server
using Flask and Gevent. Currently this is a skeleton application with
only one active route (/api/v0/status).

The API code lives in ./src/pylorax/api/v0.py with related code in other
pylorax/api/* modules.
2018-06-11 16:54:57 -07:00
Brian C. Lane
bf8be43c90 livemedia-creator: Move core functions into pylorax modules
This reduces the amount of code in livemedia-creator to the cmdline
parsing and calling of the installer functions. Moving them into other
modules will allow them to be used by other projects, like the
lorax-composer API server.
2018-06-11 16:54:57 -07:00
Brian C. Lane
c746e8b0c3 Retry losetup if loop_attach fails
It appears that sometimes the loop device doesn't get setup properly,
this may be a race with other users of loop devices on the system, or
some other mechanism that isn't understood.

To try and prevent total failure when this happens this patch retries
the loop setup 3 times before giving up. Previously it would wait for
the loop device to appear (checking 5 times), that operation is now
executed 3 times with a new losetup attempt each time.

Resolves: rhbz#1589084
2018-06-11 13:56:53 -07:00
Brian C. Lane
22392b64fc Add --dracut-arg support to lorax
Use it to override the default dracut arguments (displayed as part of
the --help output). If you want to extend the default arguments they
all need to be passed in on the cmdline as well. eg.

--dracut-arg='--xz' --dracut-arg='--install /.buildstamp' ...

Resolves: rhbz#1452220
2018-06-06 17:38:55 -07:00
Brian C. Lane
54d973e059 Write a list of installed packages to /root/lorax-packages.log (#1416155)
This can't be done the same way as on master because there is no rpm
database inside the installroot to run rpm -qa against. Do it at the end
of the yum transaction.

Resolves: rhbz#1416155
2017-09-27 08:30:04 -07:00
Brian C. Lane
5238b9cd18 Fix systemctl command (#1478247)
When multiple units are passed to systemctl and one fails it doesn't
finish the others. Change the template command to call systemctl for
each unit individually.

This also removes the lvm2-activation-generator in runtime-cleanup.tmpl

Resolves: rhbz#1478247
2017-08-11 15:49:10 -07:00
Brian C. Lane
1b29041089 Add the version to the log (#1335456)
Log the version number as INFO and add -V version display to
livemedia-creator.

Resolves: rhbz#1335456
2017-08-11 15:42:16 -07:00
Brian C. Lane
7aa71188b9 Include the dracut fips module in the initrd (#1341280)
This will allow anaconda to fetch kickstarts using https when installing
with fips=1

Leave vmlinuz and .vmlinuz.hmac in /boot

dracut-fips module needs the vmlinuz.hmac file in order to boot.

Resolves: rhbz#1341280
2017-08-11 15:22:50 -07:00
Brian C. Lane
a883412ccc Fix loop_wait (#1462150)
The previous code used losetup --list -O to return the backing store
associated with the loop device. This can fail due to losetup truncating
the output filename if sysfs isn't setup. Instead of printing the full
path it will truncate it to 64 characters with a * at the end.

See util-linux lib/loopdev.c for the code that does this.

This commit changes it to use the existing get_loop_name function, which
uses losetup -j to lookup the loop device associated with the backing
store which avoids the truncation problem.

Resolves: rhbz#1462150
2017-08-11 15:18:00 -07:00
Brian C. Lane
aaf1cc3894 Fix waiting for loop devices (#1462150)
Missing change for the previous commit.

Resolves: rhbz#1462150
2017-06-23 08:28:32 -07:00
Brian C. Lane
e0d9e309ae Make sure loop device is setup (#1462150)
It seems that on rare occasions losetup can return before the /dev/loopX
is ready for use, causing problems with mkfs. This tries to make sure
that the loop device really is associated with the backing file before
continuing.

Resolves: rhbz#1462150
2017-06-21 14:31:35 -07:00
Brian C. Lane
391f1a7b57 Copying same file shouldn't crash (#1269213)
When using the template install command copying the same file to itself
shouldn't crash. Just log the error and continue.

Also copy the s390 configuration files for use with livemedia-creator

Resolves: rhbz#1269213
(cherry picked from commit 701ab02619)
2016-04-06 17:02:41 -07:00
Colin Walters
7c4d7b0c42 Add --installpkgs argument (#1272222)
Recently, Fedora has been trying to do a 3 product split.  As part of
that, lorax was changed to do "installpkg lorax-product-*" via
provides.

I think that approach is awkward; a much simpler approach is to simply
specify the product package as input to lorax on the command line, via
external rel-eng scripts.

This patch therefore adds --installpkgs (and we should probably add an
option to remove the implicit lorax-product-* glob).

(cherry picked from commit 52d962d613)

Resolves: rhbz#1272222
2016-02-19 15:58:50 -08:00