Commit Graph

806 Commits

Author SHA1 Message Date
Brian C. Lane
97ca285cf2 Add job_* timestamp support to compose status 2018-08-07 14:05:53 -07:00
Will Woods
80b5a37036 composer-cli: clean up "list" commands
This patch does two things:

1) Add "compose list", which lists compose UUIDs and other basic info,
2) Fix up "blueprints list", "modules list", "sources list", and
   "compose types" so their output is just a plain list of identifiers
2018-08-07 12:57:34 -04:00
Brian C. Lane
86d79cd8a6 Add input string checks to the branch and format arguments
Make sure no UTF8 characters are allowed and return an error if they
are.

Also includes tests to make sure the correct error is returned.
2018-08-06 14:21:24 -07:00
Brian C. Lane
74f5def3d4 Add a test for invalid characters in the API route
Currently the code is not UTF8 safe, so we need to return a clear error
when invalid characters are passed in.

This also adds tests for the routes to confirm that an error is
correctly returned.
2018-08-06 13:24:24 -07:00
Chris Lumens
5daf2d416a Return a JSON error instead of a 404 on certain malformed URLs.
This handles the case where a route is requested, but without a required
parameter.  So, /blueprints/info is requested instead of
/blueprints/info/http-server.  It accomplishes this via a decorator, so
a lot of these route-related functions now have quite a few decorators
attached to them.

Typo'd URLs (/blueprints/nfo for instance) will still return a 404.  I
think this is a reasonable thing to do.
2018-08-06 13:38:14 -04:00
Chris Lumens
8e948e4a4d Return an error if /modules/info doesn't return anything.
Unfortunately, this isn't very useful if /modules/info is provided with
multiple modules.  yum doesn't traceback when doPackageLists is given
something that doesn't exist.  It just returns an empty list.  If
/modules/info is given just one module and yum gives us an empty list,
it's easy to say what happened.  If /modules/info is given several
modules and just one does not exist, we will not be able to detect that.

Fixing this would require doing more yum operations, which is likely to
slow things down and isn't the direction I want to be going.
2018-08-06 12:37:58 -04:00
Chris Lumens
6193a7c9d8 Update documentation (#409). 2018-08-06 12:04:00 -04:00
Chris Lumens
fdf4d63f3b Use constants instead of strings (#409). 2018-08-06 12:04:00 -04:00
Chris Lumens
3ba9d53b8b Write timestamps when important events happen during the compose (#409).
Right now, this is when the compose is queued up, when it is started by
anaconda, and when it is finished (whether that's success or not).
2018-08-06 12:04:00 -04:00
Chris Lumens
17c40ef271 Return multiple timestamps in API results (#409).
If one of the timestamps isn't present (for instance, the finished
timestamp for a job that is still running), null is returned.
2018-08-06 12:04:00 -04:00
Chris Lumens
b59d59b124 Add a new timestamp.py file to the API directory (#409).
This is responsible for writing out a new times.toml file, containing
important timestamps in the life of a compose.  This seems a little more
reliable than attempting to infer things from the filesystem, especially
in light of the fact that we can't ever really know when a file was
created.
2018-08-06 12:04:00 -04:00
Chris Lumens
01a3d365ef Run as root/weldr by default.
We need to be root to read the certificates that give access to the
package repos.  Right now, the alternative seems to be changing
permissions on the certs themselves, which seems less good.  We're
running anaconda as root anyway.
2018-08-02 08:55:17 -04:00
Brian C. Lane
2edd7a995b composer-cli: Add support for Group to blueprints diff 2018-07-26 18:12:02 -07:00
Brian C. Lane
642f117d54 Update status.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.
2018-07-26 11:22:22 -07:00
Brian C. Lane
3205e47a13 Update sources.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.
2018-07-26 10:44:36 -07:00
Brian C. Lane
77767cd93a Update projects.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.
2018-07-26 10:33:32 -07:00
Brian C. Lane
b55b86d0b3 Update modules.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.
2018-07-26 10:28:54 -07:00
Brian C. Lane
760aeb61a7 Update compose.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.
2018-07-26 10:23:39 -07:00
Brian C. Lane
c1c5536ec8 Update blueprints.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.
2018-07-25 17:32:48 -07:00
Brian C. Lane
35fa067219 Modify handle_api_result so it can be used in more places
Some results have errors and no status, others have status and errors.
Update the function to return the final rc to exit with, and a bool
indicating whether or not to continue processing the other fields.

Add a bunch of tests for the new function to make sure I have the logic
correct.
2018-07-25 17:30:50 -07:00
Brian C. Lane
e974ec2477 composer-cli: Fix non-zero epoch in projets info
The json value is an int, so convert it to a str.

(cherry picked from commit 18521cec1a)
2018-07-25 11:40:16 -07:00
Chris Lumens
a3572c9fdd Fix help output on the compose subcommand. 2018-07-25 14:29:39 -04:00
Chris Lumens
7bcb61849d Add timestamps to "compose-cli compose status" output. 2018-07-25 12:47:17 -04:00
Chris Lumens
eb0939d967 And then add real output to the status command. 2018-07-23 17:08:08 -04:00
Chris Lumens
9eafc60fa0 Add the beginnings of a new status subcommand. 2018-07-23 17:08:08 -04:00
Brian C. Lane
8080abda67 Log and exit on metadata update errors at startup
A bad system repo can cause lorax-composer to fail to start. Instead of
a traceback log the error and exit.

(note that the exit still results in an OSError traceback due to part of
it running as root, this needs to be addressed in another commit).
2018-07-17 13:47:26 -07:00
Brian C. Lane
f916e41d00 Check /projects responses for null values.
Some values can be null/None so check for that instead of crashing.
2018-07-17 11:51:04 -07:00
Brian C. Lane
b20aa4609c Clarify error message from /source/new 2018-07-17 10:17:06 -07:00
Brian C. Lane
7e8245abcc Download metadata when updating or adding new repos
cacheonly=1 was skipping downloading new metadata when a new source was
added. Set it to 0 so that the update is immediate.
2018-07-17 09:59:26 -07:00
Chris Lumens
c8d2045f89 Support loading groups from the kickstart template files. 2018-07-12 17:31:56 -04:00
Chris Lumens
5fe4b47072 Include groups in depsolving.
This adds a new argument to projects_depsolve and
projects_depsolve_with_size that contains the group list, unfortunately.
I would have prefered adding a function that just returns a list of all
the contents of a group and then add that to what was being passed into
projects_depsolve.  However, there does not appear to be any good way to
do that in yum aside from a lot of grubbing around in the comps object,
which I am unwilling to do.
2018-07-12 17:31:44 -04:00
Chris Lumens
0f69d2084c Add support for groups to blueprints.
Nothing is currently being done with this information, but it will be
soon.
2018-07-12 17:31:44 -04:00
Brian C. Lane
d692a7dddd Check the compose templates at startup
Depsolve the packages included in the templates and report any errors
using the /api/status 'msgs' field. This should help narrow down
problems with package sources not being setup correctly.
2018-07-12 09:25:22 -07:00
Brian C. Lane
f1814ce35f List individual package install failures
Previously it was impossible to know which package in a blueprint caused
a failure, if it was just one of them, or all of them, etc. This catches
the error when calling yb.install and lists all the failures in the
error message that is raised.
2018-07-11 15:35:04 -07:00
Chris Lumens
954f330ace Add help output to each subcommand.
This is the same as the output at the top level, just trimmed down to
only the options for a single subcommand.  It's trigged by providing
"help" or "--help" as a subcommand option.
2018-06-28 14:38:05 -04:00
Chris Lumens
70e4211ad1 Split the help output into its own module.
This means I can reuse it for help output for individual subcommands.
2018-06-28 14:38:05 -04:00
Chris Lumens
3743d6d208 If the help subcommand is given, print the help output.
This isn't a real subcommand like the others.  The option processing
just intercepts it and prints the output.  Given that we're subcommand
based, it makes sense to support this in addition to --help.
2018-06-28 14:38:05 -04:00
Brian C. Lane
bb8c7851d2 Fix composer-cli command help text
sources shouldn't be indented
details for a compose are 'details' not 'info'
2018-06-18 10:45:36 -07:00
Brian C. Lane
4bfcd8fcd2 Remove SortMode from list_commits
The current version of libgit2 available (0.26.3) has different behavior
with SortMode.TIME. It works correctly when left at the default (which
is also how the rawhide version works).
2018-06-11 16:54:59 -07:00
Brian C. Lane
3cfb895054 Use yum.add_enable_repo() for new repos
Yum needs to have some other attrs setup on the YumRepository object, so
use the function provided to ensure that everything is correct. Also
switch the related functions to use a dict instead of a YumRepository
object.
2018-06-11 16:54:59 -07:00
Brian C. Lane
957def1264 Add support for sources to composer-cli
This adds the sources command which can be used to list, add, change,
and delete sources using the TOML formatted source file.
2018-06-11 16:54:59 -07:00
Brian C. Lane
a5eaebeefc yum repos has a delete() function. But it doesn't clear the cache.
yum also has a cache it uses for listEnabled(), but the cache isn't
invalidated when a repo is deleted it any following metadata update
will fail because it is still using the deleted repo.

We are forced to use the heavy hammer on a yum private variable yet
again to force the cache to be cleared so that it won't crash.
2018-06-11 16:54:59 -07:00
Brian C. Lane
2e95b56508 Fix DNF related issues with source selection
yum TumRepository.dump() function cannot be used as a .repo file Add a
new function to write this in the correct format, and limited to the
fields we use.

Add a test for the new function.

Fix /projects/source/info to return an error 400 if a nonexistant TOML
source is requested. If JSON is used the error is part of the standard
response.

Update test_server.py to check for the correct error code.
2018-06-11 16:54:59 -07:00
Brian C. Lane
3f7997d7ae Fix handling bad source repos and add a test
When adding a source failed it wasn't being removed from the dnf object.
This fixes that, and returns an error when setting up the source fails.
Also adds a test for it.
2018-06-11 16:54:59 -07:00
Brian C. Lane
25bae61520 Fix make_dnf_dirs
It was chopping off an extra directory level due to realpath removing
the trailing / from the paths when they are setup.
2018-06-11 16:54:59 -07:00
Brian C. Lane
33c84331fe Add support for user defined package sources API
This lives under /api/v0/projects/source/*

See the documentation for details
2018-06-11 16:54:59 -07:00
Brian C. Lane
82c8c3a491 gevent has deprecated .wsgi, should use .pywsgi instead
https://github.com/gevent/gevent/blob/master/doc/api/gevent.wsgi.rst
(cherry picked from commit c9ca451568)
2018-06-11 16:54:59 -07:00
Brian C. Lane
b99d8d7f6b Add support for version globs to blueprints
You can use '*' wildcards and '?' for single character matching.
2018-06-11 16:54:59 -07:00
Brian C. Lane
ce715ad15f Check to make sure blueprints directory exists 2018-06-11 16:54:59 -07:00
Brian C. Lane
3c60f07d98 Remove temporary files after run_compose
A crash can also leave temporary lmc-* files, remove them as well.
2018-06-11 16:54:59 -07:00