Commit Graph

1468 Commits

Author SHA1 Message Date
Brian C. Lane
fd16f70bd1 Add documentation for /compose and /compose/types 2018-02-13 17:22:12 -08:00
Brian C. Lane
0c15e70010 Move queue monitor startup into a function
This allows it to be used by the test framework.
2018-02-13 17:22:12 -08:00
Brian C. Lane
829628192a Move queue directory creation into a function
So that it can also be used during tests
2018-02-13 17:22:12 -08:00
Brian C. Lane
30b1ea257b Add a test mode to /compose
This will allow testing without having a full system setup with
anaconda, if ?test=1 is passed to the POST /compose command it will wait
10 seconds instead of running Anaconda, and then raise an error to
generate a failed build.

Passing ?test=2 will also wait 10 seconds instead of running Anaconda,
but will finish successfully.
2018-02-13 17:22:12 -08:00
Brian C. Lane
793318d4cc Cleanup docstrings for queue.py 2018-02-13 17:22:12 -08:00
Brian C. Lane
3e4e339114 Drop cancel_q from the monitor() function
Not needed, builds are canceled by writing a 'CANCEL' file in the results
directory.
2018-02-13 17:22:12 -08:00
Brian C. Lane
86b9074d3f Fix the jsonify calls to use kwargs
No need to pass it a dict, just set the args.
2018-02-13 17:22:12 -08:00
Brian C. Lane
d1114086db Add /compose/log/ API to retrieve the end of the build log
This allows the client to request the end of the anaconda.log during and
after a build. The amount of data returned can be set by adding
?size=<kbytes>

Output is raw bytes, starting on the next available line boundry.
2018-02-13 17:22:12 -08:00
Brian C. Lane
0acbb0f061 Return a status of false if the uuid isn't valid 2018-02-13 17:22:12 -08:00
Brian C. Lane
dfcdc08844 Add /compose/cancel API to cancel a running build
If the build hasn't started yet (state is WAITING) try removing the
symlink to it. If this succeeds, delete the partial results directory.

If the build makes it to RUNNING then it writes a CANCEL file in the
results directory. The callback that is passed to execWithRedirect
catches this, causing a SIGTERM to be sent to anaconda. It then exits
and cleanup happens normally. The partial results directory is then
removed.
2018-02-13 17:22:12 -08:00
Brian C. Lane
ddcdc5df72 Pass the callback_func through novirt_install to execWithRedirect
The default is no callback.
2018-02-13 17:22:12 -08:00
Brian C. Lane
39c1dab256 Add a callback to execWithRedirect
If the callback returns true send the process a SIGTERM to terminate it.
2018-02-13 17:22:12 -08:00
Alexander Todorov
12d1c751a2 Update how we pass the source to docker so it includes docs/ dir
since the last commit the tests require to have the docs built
before hand but the docs/ dir was missing from the docker image.
2018-02-13 10:52:16 +01:00
Alexander Todorov
e62c1e8932 Add tests for functions in api/projects 2018-02-13 10:52:16 +01:00
Alexander Todorov
4a756deb18 Add tests for api/server.py
some of the new tests require that we first build the documentation
2018-02-13 10:52:16 +01:00
Alexander Todorov
a3c934214b Add tests for yumbase and update how we inspect boolean options
.get_default() returns string so make sure we're actually parsing
the value as boolean and not evaluating a non-empty string in a
boolean context (which will always return True)
2018-02-13 10:52:16 +01:00
Alexander Todorov
28e936d990 Add new tests for workspace_read() and workspace_delete() 2018-02-13 10:52:16 +01:00
Alexander Todorov
1045d78640 Add new tests for configure() 2018-02-13 10:52:16 +01:00
Alexander Todorov
c427b772a5 Add more tests for api.recipes
- test against already existing git repository
- test commit_recipe_file() handling of IOError
- update tests for commit_recipe_directory()
  - add asserts on the existing test. Not raising an exception
    isn't enough!
  - add test which exercises the method under test while it handles
    exceptions raised by other methods
- test for list_commits() when the underlying calls fail to convert
  timestamp
- test for find_name() when `name' is not on the list
- tests for get_revision_from_tag()
2018-02-13 10:52:16 +01:00
Brian C. Lane
9154044408 Add API routes for downloading build results
Also fix a bug with the name of the queue status in the status results
(it is now 'queue_status' not 'status' which is used for error
responses).

This adds the following routes:
 - /compose/metadata/<uuid> to retrieve a .tar of the build metadata
 - /compose/results/<uuid> to retrieve .tar of all of the build results
 - /compose/logs/<uuid> to retrieve a .tar of just the logs from the build
 - /compose/image/<uuid> to retrieve the output image from the build
2018-02-06 16:56:39 -08:00
Brian C. Lane
f063651e2e Add /compose/info route to retrieve details about a compose
The results is a JSON string with the following information:

* id - The uuid of the comoposition
* config - containing the configuration settings used to run Anaconda
* recipe - The depsolved recipe used to generate the kickstart
* commit - The (local) git commit hash for the recipe used
* deps - The NEVRA of all of the dependencies used in the composition
* compose_type - The type of output generated (tar, iso, etc.)
* queue_status - The final status of the composition (FINISHED or FAILED)
2018-02-06 16:55:14 -08:00
Brian C. Lane
fe6c769349 Return the commit id for the recipe being read
This adds returning the commit id from read_commit, and a new function
read_recipe_and_id() that returns the commit id and the recipe in a
tuple.

If the commit is passed in, it is used as is. If no commit is passed in
it finds the most recent commit for the file on the selected branch and
returns that.

Missing recipes now raise a RecipeError with an informative message.
eg. "No commits for missing-recipe.toml on the master branch."
2018-02-06 16:55:14 -08:00
Brian C. Lane
29194aef62 Fix yum config directory creation for projects and server tests 2018-02-06 16:55:14 -08:00
Brian C. Lane
1e73041e78 Add DELETE /compose/delete/<uuids> API route
This will delete all of the build's results and cannot be undone.
2018-02-06 16:55:14 -08:00
Brian C. Lane
3ce6d9f11e Turn on o+x permission for the queue and results directories
Otherwise it won't allow a group member to delete things.
2018-02-06 16:55:14 -08:00
Brian C. Lane
da62b74d83 Add /compose/status/<uuids> to retrieve details of a specific build 2018-02-06 16:55:14 -08:00
Brian C. Lane
c5a82ffd9e Add compose status routes /compose/finished and /compose/failed
These will return a list of details about the finished or failed builds.
2018-02-06 16:55:14 -08:00
Brian C. Lane
b191344113 Automatic commit of package [lorax] release [19.7.7-1].
Created by command:

/usr/bin/tito tag
2018-02-01 15:15:01 -08:00
Brian C. Lane
b098e2916e Add /compose/queue to get the status of the build queue
This will return a JSON result that includes the waiting builds,
and the running builds.
2018-02-01 10:53:32 -08:00
Brian C. Lane
888f319b60 Add reading a recipe directly from a file
Adds helper function recipe_from_file to recipes.py
2018-02-01 10:46:31 -08:00
Brian C. Lane
8871dd2646 Include the recipe in the results of a build
Write original as recipe.toml and the depsolved version as frozen.toml

Also write 'WAITING' to the STATUS file as its first state.
The STATUS states are now WAITING -> RUNNING -> FINISHED|FAILED
2018-02-01 10:43:40 -08:00
Brian C. Lane
6087ef2932 Move creating a frozen recipe into recipes.py
Also adds .package_names and .module_names properties. Call
recipe.freeze with a list of NEVRA dependencies and it will return a new
Recipe object with all of the packages and modules set to the depsolved
version.
2018-01-30 13:52:05 -08:00
Brian C. Lane
0346a04dad Add building an image, and the /compose route to start it
This adds the ability to build a tar output image. The /compose and
/compose/types API routes are now available.

To start a build POST a JSON body to /compose, like this:

{"recipe_name":"glusterfs", "compose_type":"tar", "branch":"master"}

This will return a unique build id:

{
  "build_id": "4d13abb6-aa4e-4c80-a671-0b867e6e77f6",
  "status": true
}

which will be used to keep track of the build status (routes for this
do not exist yet).
2018-01-29 17:14:36 -08:00
Brian C. Lane
97fe514ceb Remove test configuration and read it from the build directory 2018-01-26 16:55:56 -08:00
Brian C. Lane
42c1b0f0e2 Add function to return full NEVRA of a dependency 2018-01-26 16:54:47 -08:00
Brian C. Lane
bb3d6b1003 Change config and paths
With the API running as weldr the permissions on the yum directories
needs to be accessable to it, as well as the results and queue/new
directories.
2018-01-26 16:53:12 -08:00
Brian C. Lane
8b725a1cf5 Add basic composer queue handling
The queue is in /var/lib/weldr/queue/new by default. It watches the
directory for new symlinks (to /var/lib/weldr/results/<dirname>) and
handles running anaconda on the kickstart found in final-kickstart.ks
inside the symlinked directory.
2018-01-24 12:11:03 -08:00
Brian C. Lane
c74582a647 Change compress to use communicate instead of wait
Also move default_image_name into imgutils so it can be used in other
places.

When running from lorax-composer the wait() call wasn't waiting until
the tar was finished. I think this is due to gevent monkey-patching
something. Using communicate() solves this problem.
2018-01-23 13:34:28 -08:00
Brian C. Lane
8ff851c6f5 Automatic commit of package [lorax] release [19.7.6-1].
Created by command:

/usr/bin/tito tag
2018-01-15 14:19:29 -08:00
Brian C. Lane
6a5ff22e61 Add documentation for the API routes.
Use Sphinx on a CentOS7/RHEL7 system to generate the docs with 'make docs'
2018-01-15 14:14:35 -08:00
Brian C. Lane
33c5d0ce4a Switch the API to use a Unix Domain Socket
This drops support for the TCP port and switches to using a socket at
/var/run/weldr/api.socket

Also add the start of some docs for lorax-composer.

--host and --port argument have been removed.

--group sets the group name to use for access to the socket and its
parent directory. Defaults to 'weldr'

--socket sets the full path to the socket to create. Defaults to
'/var/run/weldr/api.socket'
2018-01-02 14:01:56 -08:00
Brian C. Lane
3c18a63f76 Add support for other branches to the routes
Passing ?branch=<branch-name> will use the specified branch instead of
master.

The new branch will not exist until a /recipes/new?branch=new-branch
POST is made. At that time the branch will be created based on the
current master branch and the new commit will be added to it.
2018-01-02 13:49:09 -08:00
Alexander Todorov
89251d0906 Silence pocketlint bad-preconf-access warnings 2017-12-21 22:20:46 +02:00
Alexander Todorov
70f6bc3bcf Properly report coverage
- use same version of coverage.py as is inside the container b/c
  data format has changed between versions
- use `coverage combine' with a .coveragerc mapping to adjust for
  differences in file paths between the container and Travis
2017-12-19 22:48:14 +02:00
Alexander Todorov
e9aa137218 Enable testing in Travis CI using Docker container 2017-12-18 10:49:58 +02:00
Martin Pitt
c58b367b6a Fix depsolving empty recipes
- Fix `projects_depsolve()` to not consider a successful empty response
   (rc == 0) as an error.
 - Fix recipe_from_dict() to default modules and packages to empty lists
   instead of `None`, to avoid a Python-ism in the API for consumers and
   stay compatible to the bdcs API.

Fixes #290
2017-12-12 09:23:06 -08:00
Martin Pitt
32c00c75b3 Fix wrong function name in api/v0/recipes/freeze error messages 2017-12-12 09:23:06 -08:00
Martin Pitt
a6b96c2419 Fix project tests for non-Central time zones
When running tests somewhere else than Central Time, `test_api_time()`
fails like

    AssertionError: '1985-10-27T02:00:00' != '1985-10-26T21:00:00'

`test_yaps_to_project_info()` fails in a similar way.

Pin the time zone to UTC for this test and adjust timestamps to UTC.
2017-12-12 09:23:06 -08:00
Brian C. Lane
b21511c1cb Automatic commit of package [lorax] release [19.7.5-1].
Created by command:

/usr/bin/tito tag
2017-11-28 16:38:50 -08:00
Brian C. Lane
5cec7e02c4 Redirect yum's logging to yum.log 2017-11-28 16:35:49 -08:00