Commit Graph

847 Commits

Author SHA1 Message Date
Brian C. Lane
3a9d39f397 Add support for DNF 3.2 module_platform_id config value
This borrows simpleconfig.py from Anaconda to make parsing os-release
easier.

It defaults to platform:el8
2018-08-29 10:48:02 -07:00
Brian C. Lane
a2375e870f Fix /compose/cancel API documentation
It said /blueprints/cancel which is incorrect.
2018-08-29 10:08:37 -07:00
Brian C. Lane
55d2911c7d Fix composer-cli blueprints changes to get correct total
blueprints/changes is different, each blueprint has it's own total,
limited by the call's limit. So it needs to find the max total of all
the requested blueprints.

(cherry picked from commit 57674c9a1a)
2018-08-27 12:09:27 -07:00
Brian C. Lane
d86c53527b Fix blueprints/list and blueprints/changes to return the correct total
(cherry picked from commit ec908fcd2a)
2018-08-27 12:09:27 -07:00
Brian C. Lane
8e3d2b0252 Add a function to get_url_json_unlimited to retrieve the total
The blueprints/changes API is a bit different from the others, the total
that it includes is for each blueprint, not one total for all of them,
since there will be a different number of commits for each.

The function is passed the dict, and it can be used to select the total
to use for retrieving all of the results. If it isn't included it will
use data["total"] which works fine in most cases.

(cherry picked from commit 0a76d635ca)
2018-08-27 12:09:27 -07:00
Brian C. Lane
a2b83bd771 Use urllib.parse instead of urlparse
python3 moved this to a new module.

(cherry picked from commit 86d556e87c)
2018-08-21 17:05:24 -07:00
David Shea
d6418246ad In composer-cli, request all results
Add a limit argument to all potentially paginated results, equal to
whatever the composer backend is the total number of results. This still
has the potential to provide truncated data if the number of results
increases between the two HTTP requests.

Resolves: #404
(cherry picked from commit ee98d87cea)
2018-08-21 17:05:24 -07:00
David Shea
5057fef248 Allow '*' as a uuid in /compose/status/<uuid>
This will display all UUIDs that match the filter arguments

(cherry picked from commit deff4d325f)
2018-08-21 17:05:24 -07:00
David Shea
1938e8ef92 Add filter arguments to /compose/status
This adds the following optional arguments to the /compose/status route:

  - type, matches the compose_type field
  - status, matches the queue_status field
  - blueprint, matches the blueprint field

(cherry picked from commit 40f23f093d)
2018-08-21 17:05:24 -07:00
Brian C. Lane
d8750db166 composer-cli should not log to a file by default
The user can pass --log /path/to/logfile.log if they want logging
enabled.

(cherry picked from commit 844ff9998b)
2018-08-21 12:09:38 -07:00
Lars Karlitski
39837fb7f8 Set TCP listen backlog for API socket to SOMAXCONN
A value of 1 is too low for heavy users of the API, such as the weldr-web
interface.

This is also systemd's default for sockets it opens. Using lorax-composer with
socket activation already results in a backlog of SOMAXCONN connections.

(cherry picked from commit be5d50e6f3)

Related: rhbz#1613058
2018-08-20 15:19:17 -07:00
Chris Lumens
ff5ac5045b Fix a little bug in running "modules list".
(cherry picked from commit 5dddef6f49)
2018-08-10 12:01:10 -04:00
Brian C. Lane
22a6852b44 Move disklabel and UEFI support to compose.py
Currently we are making MBR disk images for qcow2 and partitioned disk,
so the UEFI packages aren't required at this point.

Move the clearpart command into compose.py so that in the futute it can
use clearpart --disklabel to create a GPT image, and add the required
packages to the package set.
2018-08-09 15:35:57 -07:00
Chris Lumens
e433c66f2a Change INVALID_NAME to INVALID_CHARS.
It's getting used in a whole lot more places now, so make it a bit more
generic sounding.

(cherry picked from commit bf0318eab6)
2018-08-09 16:50:53 -04:00
Chris Lumens
6b36d2a188 Update composer-cli for the new error return types.
(cherry picked from commit bc96f75992)
2018-08-09 16:50:46 -04:00
Chris Lumens
90aa81e964 Add default error IDs everywhere else.
The idea here is to make sure all return points have the same type for
the error cases.  There's not really all that many, so they just go in
one patch.  Some of these could potentially turn into more specialized
errors later.

(cherry picked from commit fd901c5e3f)
2018-08-09 16:50:37 -04:00
Chris Lumens
7b977745c7 Add error IDs to things that can go wrong when running a compose.
Note the exception string checking around compose_type.  I didn't really
want to introduce a new exception type just for this, but also didn't
want to duplicate strings.  I'd be open to other suggestions for how to
do this.

(cherry picked from commit b3bb438254)
2018-08-09 16:50:11 -04:00
Chris Lumens
93d084b30a Add error IDs for common source-related errors.
(cherry picked from commit e43adfc7af)
2018-08-09 16:50:07 -04:00
Chris Lumens
a1803c218f Add error IDs for unknown modules and unknown projects.
(cherry picked from commit 2adcfc9563)
2018-08-09 16:50:03 -04:00
Chris Lumens
56da204f13 Add error IDs for when an unknown commit is requested.
(cherry picked from commit 07528a431e)
2018-08-09 16:49:54 -04:00
Chris Lumens
22d75526ef Add error IDs for when an unknown blueprint is requested.
This adds some fairly redundant code to the beginning of all the
blueprint routes to attempt reading a commit from git for the
blueprint's recipe.  If it succeeds, the blueprint exists and the route
can continue.  Otherwise, return an error.  Hopefully this doesn't slow
things down too much.

(cherry picked from commit a925cc7ddb)
2018-08-09 16:49:48 -04:00
Chris Lumens
36b6444050 Add error IDs for when an unknown build UUID is requested.
Note that this also changes the return type of uuid_info to return None
when an unknown ID is given.  The other uuid_* functions are fine
because they are checked ahead of time.

(cherry picked from commit 6497b4fb65)
2018-08-09 16:49:38 -04:00
Chris Lumens
ea10339ef4 Add error IDs for bad state conditions.
(cherry picked from commit d76e24053a)
2018-08-09 16:49:16 -04:00
Chris Lumens
672708e68d Change the error return type for bad limit= and offset=.
Each element in the errors value is now a dict, with a msg field and an
id field.  The id field contains a value out of errors.py that can be
used by the front end to key on.  The msg field is the same as what's
been there.

The idea is to keep the number of IDs somewhat limited so there's not a
huge number of things for the front end to know.

(cherry picked from commit 9677b012da)
2018-08-09 16:49:07 -04:00
Chris Lumens
78ae380eb3 Don't sort error messages.
This should make it easier to return more complex error structures.  It
also doesn't appear to matter - tests still pass without changes.

(cherry picked from commit 4c3f93e329)
2018-08-09 16:48:57 -04:00
Chris Lumens
dbc4d08de4 Run as root/weldr by default.
We need to be root to read the certificates that give access to the
package repos.  Right now, the alternative seems to be changing
permissions on the certs themselves, which seems less good.  We're
running anaconda as root anyway.
2018-08-08 17:07:59 -07:00
Brian C. Lane
8b0c4a8465 Add + to the allowed API string character set 2018-08-07 16:48:25 -07:00
Brian C. Lane
f7c4c3f5d5 Add job_* timestamp support to compose status 2018-08-07 15:34:35 -07:00
Will Woods
e673fd5927 composer-cli: clean up "list" commands
This patch does two things:

1) Add "compose list", which lists compose UUIDs and other basic info,
2) Fix up "blueprints list", "modules list", "sources list", and
   "compose types" so their output is just a plain list of identifiers
2018-08-07 12:12:17 -07:00
Brian C. Lane
991091c287 Add input string checks to the branch and format arguments
Make sure no UTF8 characters are allowed and return an error if they
are.

Also includes tests to make sure the correct error is returned.

(cherry picked from commit 86d79cd8a6)
2018-08-07 12:12:17 -07:00
Brian C. Lane
8786c9764c Add a test for invalid characters in the API route
Currently the code is not UTF8 safe, so we need to return a clear error
when invalid characters are passed in.

This also adds tests for the routes to confirm that an error is
correctly returned.

(cherry picked from commit 74f5def3d4)
2018-08-07 12:12:17 -07:00
Brian C. Lane
03e339cbe5 Fix logging argument 2018-08-07 12:12:17 -07:00
Brian C. Lane
2b0efff9e4 Fix write_timestamp for py3
toml.dumps() returns a string for write() so .encode is not needed.
2018-08-07 12:12:17 -07:00
Chris Lumens
95462600fa Return a JSON error instead of a 404 on certain malformed URLs.
This handles the case where a route is requested, but without a required
parameter.  So, /blueprints/info is requested instead of
/blueprints/info/http-server.  It accomplishes this via a decorator, so
a lot of these route-related functions now have quite a few decorators
attached to them.

Typo'd URLs (/blueprints/nfo for instance) will still return a 404.  I
think this is a reasonable thing to do.

(cherry picked from commit 5daf2d416a)
2018-08-07 12:12:17 -07:00
Chris Lumens
191897d40e Return an error if /modules/info doesn't return anything.
Unfortunately, this isn't very useful if /modules/info is provided with
multiple modules.  yum doesn't traceback when doPackageLists is given
something that doesn't exist.  It just returns an empty list.  If
/modules/info is given just one module and yum gives us an empty list,
it's easy to say what happened.  If /modules/info is given several
modules and just one does not exist, we will not be able to detect that.

Fixing this would require doing more yum operations, which is likely to
slow things down and isn't the direction I want to be going.

(cherry picked from commit 8e948e4a4d)
2018-08-07 12:12:17 -07:00
Chris Lumens
3e3598cd95 Update documentation (#409).
(cherry picked from commit 6193a7c9d8)
2018-08-07 12:12:17 -07:00
Chris Lumens
e4d06b587d Use constants instead of strings (#409).
(cherry picked from commit fdf4d63f3b)
2018-08-07 12:12:17 -07:00
Chris Lumens
83196b5f6f Write timestamps when important events happen during the compose (#409).
Right now, this is when the compose is queued up, when it is started by
anaconda, and when it is finished (whether that's success or not).

(cherry picked from commit 3ba9d53b8b)
2018-08-07 12:12:17 -07:00
Chris Lumens
40ff8faad8 Return multiple timestamps in API results (#409).
If one of the timestamps isn't present (for instance, the finished
timestamp for a job that is still running), null is returned.

(cherry picked from commit 17c40ef271)
2018-08-07 12:12:17 -07:00
Chris Lumens
099cdfcd03 Add a new timestamp.py file to the API directory (#409).
This is responsible for writing out a new times.toml file, containing
important timestamps in the life of a compose.  This seems a little more
reliable than attempting to infer things from the filesystem, especially
in light of the fact that we can't ever really know when a file was
created.

(cherry picked from commit b59d59b124)
2018-08-07 12:12:17 -07:00
Brian C. Lane
f791d5aff6 composer-cli: Add support for Group to blueprints diff
(cherry picked from commit 2edd7a995b)
2018-08-07 12:12:17 -07:00
Brian C. Lane
5af7ad6600 Update status.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.

(cherry picked from commit 642f117d54)
2018-08-07 12:12:17 -07:00
Brian C. Lane
926d88a3bd Update sources.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.

(cherry picked from commit 3205e47a13)
2018-08-07 12:12:17 -07:00
Brian C. Lane
7e999cc9c6 Update projects.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.

(cherry picked from commit 77767cd93a)
2018-08-07 12:12:17 -07:00
Brian C. Lane
ecefa18110 Update modules.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.

(cherry picked from commit b55b86d0b3)
2018-08-07 12:12:17 -07:00
Brian C. Lane
92f5860182 Update compose.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.

(cherry picked from commit 760aeb61a7)
2018-08-07 12:12:17 -07:00
Brian C. Lane
e21c53f507 Update blueprints.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.

(cherry picked from commit c1c5536ec8)
2018-08-07 12:12:17 -07:00
Brian C. Lane
85b2807a15 Modify handle_api_result so it can be used in more places
Some results have errors and no status, others have status and errors.
Update the function to return the final rc to exit with, and a bool
indicating whether or not to continue processing the other fields.

Add a bunch of tests for the new function to make sure I have the logic
correct.

(cherry picked from commit 35fa067219)
2018-08-07 12:12:17 -07:00
Brian C. Lane
619026ff3d Update to use only qemu-kvm
We only have qemu-kvm available, so use that. This also means that there
will not me any support for using qemu with arches that are different
from the host.
2018-07-27 15:10:22 -07:00
Chris Lumens
c3987cc4cb Fix help output on the compose subcommand.
(cherry picked from commit a3572c9fdd)
2018-07-26 14:53:57 -04:00