Commit Graph

821 Commits

Author SHA1 Message Date
Brian C. Lane
8b0c4a8465 Add + to the allowed API string character set 2018-08-07 16:48:25 -07:00
Brian C. Lane
f7c4c3f5d5 Add job_* timestamp support to compose status 2018-08-07 15:34:35 -07:00
Will Woods
e673fd5927 composer-cli: clean up "list" commands
This patch does two things:

1) Add "compose list", which lists compose UUIDs and other basic info,
2) Fix up "blueprints list", "modules list", "sources list", and
   "compose types" so their output is just a plain list of identifiers
2018-08-07 12:12:17 -07:00
Brian C. Lane
991091c287 Add input string checks to the branch and format arguments
Make sure no UTF8 characters are allowed and return an error if they
are.

Also includes tests to make sure the correct error is returned.

(cherry picked from commit 86d79cd8a6)
2018-08-07 12:12:17 -07:00
Brian C. Lane
8786c9764c Add a test for invalid characters in the API route
Currently the code is not UTF8 safe, so we need to return a clear error
when invalid characters are passed in.

This also adds tests for the routes to confirm that an error is
correctly returned.

(cherry picked from commit 74f5def3d4)
2018-08-07 12:12:17 -07:00
Brian C. Lane
03e339cbe5 Fix logging argument 2018-08-07 12:12:17 -07:00
Brian C. Lane
2b0efff9e4 Fix write_timestamp for py3
toml.dumps() returns a string for write() so .encode is not needed.
2018-08-07 12:12:17 -07:00
Chris Lumens
95462600fa Return a JSON error instead of a 404 on certain malformed URLs.
This handles the case where a route is requested, but without a required
parameter.  So, /blueprints/info is requested instead of
/blueprints/info/http-server.  It accomplishes this via a decorator, so
a lot of these route-related functions now have quite a few decorators
attached to them.

Typo'd URLs (/blueprints/nfo for instance) will still return a 404.  I
think this is a reasonable thing to do.

(cherry picked from commit 5daf2d416a)
2018-08-07 12:12:17 -07:00
Chris Lumens
191897d40e Return an error if /modules/info doesn't return anything.
Unfortunately, this isn't very useful if /modules/info is provided with
multiple modules.  yum doesn't traceback when doPackageLists is given
something that doesn't exist.  It just returns an empty list.  If
/modules/info is given just one module and yum gives us an empty list,
it's easy to say what happened.  If /modules/info is given several
modules and just one does not exist, we will not be able to detect that.

Fixing this would require doing more yum operations, which is likely to
slow things down and isn't the direction I want to be going.

(cherry picked from commit 8e948e4a4d)
2018-08-07 12:12:17 -07:00
Chris Lumens
3e3598cd95 Update documentation (#409).
(cherry picked from commit 6193a7c9d8)
2018-08-07 12:12:17 -07:00
Chris Lumens
e4d06b587d Use constants instead of strings (#409).
(cherry picked from commit fdf4d63f3b)
2018-08-07 12:12:17 -07:00
Chris Lumens
83196b5f6f Write timestamps when important events happen during the compose (#409).
Right now, this is when the compose is queued up, when it is started by
anaconda, and when it is finished (whether that's success or not).

(cherry picked from commit 3ba9d53b8b)
2018-08-07 12:12:17 -07:00
Chris Lumens
40ff8faad8 Return multiple timestamps in API results (#409).
If one of the timestamps isn't present (for instance, the finished
timestamp for a job that is still running), null is returned.

(cherry picked from commit 17c40ef271)
2018-08-07 12:12:17 -07:00
Chris Lumens
099cdfcd03 Add a new timestamp.py file to the API directory (#409).
This is responsible for writing out a new times.toml file, containing
important timestamps in the life of a compose.  This seems a little more
reliable than attempting to infer things from the filesystem, especially
in light of the fact that we can't ever really know when a file was
created.

(cherry picked from commit b59d59b124)
2018-08-07 12:12:17 -07:00
Brian C. Lane
f791d5aff6 composer-cli: Add support for Group to blueprints diff
(cherry picked from commit 2edd7a995b)
2018-08-07 12:12:17 -07:00
Brian C. Lane
5af7ad6600 Update status.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.

(cherry picked from commit 642f117d54)
2018-08-07 12:12:17 -07:00
Brian C. Lane
926d88a3bd Update sources.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.

(cherry picked from commit 3205e47a13)
2018-08-07 12:12:17 -07:00
Brian C. Lane
7e999cc9c6 Update projects.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.

(cherry picked from commit 77767cd93a)
2018-08-07 12:12:17 -07:00
Brian C. Lane
ecefa18110 Update modules.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.

(cherry picked from commit b55b86d0b3)
2018-08-07 12:12:17 -07:00
Brian C. Lane
92f5860182 Update compose.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.

(cherry picked from commit 760aeb61a7)
2018-08-07 12:12:17 -07:00
Brian C. Lane
e21c53f507 Update blueprints.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.

(cherry picked from commit c1c5536ec8)
2018-08-07 12:12:17 -07:00
Brian C. Lane
85b2807a15 Modify handle_api_result so it can be used in more places
Some results have errors and no status, others have status and errors.
Update the function to return the final rc to exit with, and a bool
indicating whether or not to continue processing the other fields.

Add a bunch of tests for the new function to make sure I have the logic
correct.

(cherry picked from commit 35fa067219)
2018-08-07 12:12:17 -07:00
Brian C. Lane
619026ff3d Update to use only qemu-kvm
We only have qemu-kvm available, so use that. This also means that there
will not me any support for using qemu with arches that are different
from the host.
2018-07-27 15:10:22 -07:00
Chris Lumens
c3987cc4cb Fix help output on the compose subcommand.
(cherry picked from commit a3572c9fdd)
2018-07-26 14:53:57 -04:00
Chris Lumens
db98df8373 Add timestamps to "compose-cli compose status" output.
(cherry picked from commit 7bcb61849d)
2018-07-26 14:53:53 -04:00
Chris Lumens
5ff96bc627 And then add real output to the status command.
(cherry picked from commit eb0939d967)
2018-07-26 14:53:50 -04:00
Chris Lumens
c6efcd1fc2 Add the beginnings of a new status subcommand.
(cherry picked from commit 9eafc60fa0)
2018-07-26 14:53:47 -04:00
Brian C. Lane
4b8211a471 composer-cli: Fix non-zero epoch in projets info
The json value is an int, so convert it to a str.

(cherry picked from commit 18521cec1a)
2018-07-25 11:39:55 -07:00
Brian C. Lane
d50321239c Log and exit on metadata update errors at startup
A bad system repo can cause lorax-composer to fail to start. Instead of
a traceback log the error and exit.

(note that the exit still results in an OSError traceback due to part of
it running as root, this needs to be addressed in another commit).

(cherry picked from commit 49380b4b49)
2018-07-19 11:01:08 -07:00
Brian C. Lane
ea5e34e9c6 Check /projects responses for null values.
Some values can be null/None so check for that instead of crashing.

(cherry picked from commit de0e5d2b59)
2018-07-19 11:00:58 -07:00
Brian C. Lane
9e3469b354 Clarify error message from /source/new
(cherry picked from commit 5993802bc6)
2018-07-19 11:00:54 -07:00
Chris Lumens
67b4a6c1eb Support loading groups from the kickstart template files.
(cherry picked from commit 1de67f4f63)
2018-07-19 11:00:46 -07:00
Chris Lumens
22ddaa667e Include groups in depsolving.
This adds a new argument to projects_depsolve and
projects_depsolve_with_size that contains the group list, unfortunately.
I would have prefered adding a function that just returns a list of all
the contents of a group and then add that to what was being passed into
projects_depsolve.  However, there does not appear to be any good way to
do that in yum aside from a lot of grubbing around in the comps object,
which I am unwilling to do.

(cherry picked from commit 0259f3564d)
2018-07-19 11:00:16 -07:00
Chris Lumens
76c811b7b8 Add help output to each subcommand.
This is the same as the output at the top level, just trimmed down to
only the options for a single subcommand.  It's trigged by providing
"help" or "--help" as a subcommand option.

(cherry picked from commit f5115291bd)
2018-07-19 10:59:45 -07:00
Chris Lumens
cd5c9385f6 Split the help output into its own module.
This means I can reuse it for help output for individual subcommands.

(cherry picked from commit 0179a976ba)
2018-07-19 10:59:38 -07:00
Chris Lumens
eca6a01008 If the help subcommand is given, print the help output.
This isn't a real subcommand like the others.  The option processing
just intercepts it and prints the output.  Given that we're subcommand
based, it makes sense to support this in addition to --help.

(cherry picked from commit 18620700fd)
2018-07-19 10:59:31 -07:00
Brian C. Lane
3f054d6a3a Check the compose templates at startup
Depsolve the packages included in the templates and report any errors
using the /api/status 'msgs' field. This should help narrow down
problems with package sources not being setup correctly.

(cherry picked from commit d92f2f5b04)
2018-07-13 10:17:08 -07:00
Adam Williamson
6b8009ce8f DNF 3: progress callback constants moved to dnf.transaction
Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit 4a3f9e12aa)
2018-06-28 16:20:26 -07:00
Brian C. Lane
a1905c3b4b Add --dracut-arg support to lorax
Use it to override the default dracut arguments (displayed as part of
the --help output). If you want to extend the default arguments they
all need to be passed in on the cmdline as well. eg.

--dracut-arg='--xz' --dracut-arg='--install /.buildstamp' ...

Resolves: rhbz#1452220
(cherry picked from commit d8ce013a2b)
2018-06-07 09:01:42 -07:00
Brian C. Lane
b9fe90000e gevent has deprecated .wsgi, should use .pywsgi instead
https://github.com/gevent/gevent/blob/master/doc/api/gevent.wsgi.rst
(cherry picked from commit c9ca451568)
2018-06-05 16:58:25 -07:00
Brian C. Lane
0314cbd0e8 Add support for sources to composer-cli
This adds the sources command which can be used to list, add, change,
and delete sources using the TOML formatted source file.

(cherry picked from commit 6f6ce410c0)
2018-06-04 15:48:41 -07:00
Brian C. Lane
693dc9a6f6 Fix DNF related issues with source selection
DNF Repo.dump() function cannot be used as a .repo file for dnf due to
it writing baseurl and gpgkey as a list instead of a string. Add a new
function to write this in the correct format, and limited to the fields
we use.

Add a test for the new function.

Fix /projects/source/info to return an error 400 if a nonexistant TOML
source is requested. If JSON is used the error is part of the standard
response.

Update test_server.py to check for the correct error code.

(cherry picked from commit afa89ea657)
2018-06-04 15:47:38 -07:00
Brian C. Lane
1ab33b6416 Fix handling bad source repos and add a test
When adding a source failed it wasn't being removed from the dnf object.
This fixes that, and returns an error when setting up the source fails.
Also adds a test for it.
This also includes detecting rawhide vs. non-rawhide releases and
adjusting the tests accordingly (some of the source names change).

(cherry picked from commit dd8e4d9e99)
2018-06-04 15:47:32 -07:00
Brian C. Lane
a283ec9582 Fix make_dnf_dirs
It was chopping off an extra directory level due to realpath removing
the trailing / from the paths when they are setup.

(cherry picked from commit 23f4b2a3ec)
2018-06-04 15:47:15 -07:00
Brian C. Lane
8ac04a1521 Add support for user defined package sources API
This lives under /api/v0/projects/source/*

See the documentation for details

(cherry picked from commit 6d677b2207)
2018-06-04 15:47:00 -07:00
Colin Walters
e5ef195a3b templates: Stop using gconfset
We had only been indirectly pulling in GConf, and anyways
nothing was listening to these keys.

<kalev> I still think it's a fallout from 27a90d973f

Really in general, if we wanted to make changes like this
it'd probably be a lot simpler to do them on boot or so.

https://bugzilla.redhat.com/show_bug.cgi?id=1581838
(cherry picked from commit bb3d8edd06)
2018-05-24 09:13:05 -07:00
Brian C. Lane
1af6040143 Add support for version globs to blueprints
This uses dnf's version__glob filter to implement it. It amounts to '*'
wildcards and '?' for single character matching.

(cherry picked from commit 095829171a)
2018-05-21 15:41:45 -07:00
Brian C. Lane
55b89d1aeb Add support for systemd socket activation
Instead of enabling lorax-composer.service enable lorax-composer.socket
and it will start lorax-composer on first access to
/run/weldr/api.socket
2018-05-17 10:34:35 -07:00
Brian C. Lane
e7f9028fca Move lorax-composer and composer-cli argument parsing into modules
This allows sphinx-argparse to document them automatically.
2018-05-17 10:34:35 -07:00
Brian C. Lane
cd1d414e0e Add new cmdline args to compose_args settings 2018-05-17 10:34:35 -07:00