Commit Graph

803 Commits

Author SHA1 Message Date
Brian C. Lane
7e5590d459 Update blueprints.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.

(cherry picked from commit c1c5536ec8)
2018-08-06 17:00:56 -07:00
Brian C. Lane
74d76789e1 Modify handle_api_result so it can be used in more places
Some results have errors and no status, others have status and errors.
Update the function to return the final rc to exit with, and a bool
indicating whether or not to continue processing the other fields.

Add a bunch of tests for the new function to make sure I have the logic
correct.

(cherry picked from commit 35fa067219)
2018-08-06 17:00:56 -07:00
Chris Lumens
d598b51294 Fix help output on the compose subcommand.
(cherry picked from commit a3572c9fdd)
2018-07-26 14:25:55 -04:00
Chris Lumens
4aa6768e03 Add timestamps to "compose-cli compose status" output.
(cherry picked from commit 7bcb61849d)
2018-07-26 14:25:52 -04:00
Chris Lumens
20fb27ec41 And then add real output to the status command.
(cherry picked from commit eb0939d967)
2018-07-26 14:25:48 -04:00
Chris Lumens
23892f19eb Add the beginnings of a new status subcommand.
(cherry picked from commit 9eafc60fa0)
2018-07-26 14:25:38 -04:00
Brian C. Lane
4eae24de40 composer-cli: Fix non-zero epoch in projets info
The json value is an int, so convert it to a str.

(cherry picked from commit 18521cec1a)
2018-07-25 11:39:34 -07:00
Brian C. Lane
49380b4b49 Log and exit on metadata update errors at startup
A bad system repo can cause lorax-composer to fail to start. Instead of
a traceback log the error and exit.

(note that the exit still results in an OSError traceback due to part of
it running as root, this needs to be addressed in another commit).

(cherry picked from commit b1dd22afa6)
2018-07-18 16:44:44 -07:00
Brian C. Lane
de0e5d2b59 Check /projects responses for null values.
Some values can be null/None so check for that instead of crashing.

(cherry picked from commit f916e41d00)
(cherry picked from commit 83680d92b0)
2018-07-18 16:44:44 -07:00
Brian C. Lane
5993802bc6 Clarify error message from /source/new
(cherry picked from commit b20aa4609c)
(cherry picked from commit 816f1658db)
2018-07-18 16:44:44 -07:00
Chris Lumens
1de67f4f63 Support loading groups from the kickstart template files.
(cherry picked from commit c8d2045f89)
(cherry picked from commit fbda20b8b7)
2018-07-18 16:35:48 -07:00
Chris Lumens
0259f3564d Include groups in depsolving.
This adds a new argument to projects_depsolve and
projects_depsolve_with_size that contains the group list, unfortunately.
I would have prefered adding a function that just returns a list of all
the contents of a group and then add that to what was being passed into
projects_depsolve.  However, there does not appear to be any good way to
do that in yum aside from a lot of grubbing around in the comps object,
which I am unwilling to do.

(cherry picked from commit 5fe4b47072)
(cherry picked from commit 8c4804eb15)
2018-07-18 16:35:48 -07:00
Chris Lumens
f1f8980c49 Add support for groups to blueprints.
Nothing is currently being done with this information, but it will be
soon.

(cherry picked from commit 0f69d2084c)
(cherry picked from commit 76d376fe18)
2018-07-18 16:35:48 -07:00
Chris Lumens
f5115291bd Add help output to each subcommand.
This is the same as the output at the top level, just trimmed down to
only the options for a single subcommand.  It's trigged by providing
"help" or "--help" as a subcommand option.

(cherry picked from commit 954f330ace)
(cherry picked from commit bb8fdcb854)
2018-07-18 16:35:48 -07:00
Chris Lumens
0179a976ba Split the help output into its own module.
This means I can reuse it for help output for individual subcommands.

(cherry picked from commit 70e4211ad1)
(cherry picked from commit eba5658a71)
2018-07-18 16:35:48 -07:00
Chris Lumens
18620700fd If the help subcommand is given, print the help output.
This isn't a real subcommand like the others.  The option processing
just intercepts it and prints the output.  Given that we're subcommand
based, it makes sense to support this in addition to --help.

(cherry picked from commit 3743d6d208)
(cherry picked from commit 3c6478e179)
2018-07-18 16:35:48 -07:00
Brian C. Lane
d92f2f5b04 Check the compose templates at startup
Depsolve the packages included in the templates and report any errors
using the /api/status 'msgs' field. This should help narrow down
problems with package sources not being setup correctly.
2018-07-13 09:51:36 -07:00
Brian C. Lane
590341f735 Retry losetup if loop_attach fails
It appears that sometimes the loop device doesn't get setup properly,
this may be a race with other users of loop devices on the system, or
some other mechanism that isn't understood.

To try and prevent total failure when this happens this patch retries
the loop setup 3 times before giving up. Previously it would wait for
the loop device to appear (checking 5 times), that operation is now
executed 3 times with a new losetup attempt each time.

Resolves: rhbz#1589084
(cherry picked from commit c746e8b0c3)
2018-06-11 14:00:42 -07:00
Brian C. Lane
ec8ecba9d1 Increase default ram used with lmc and virt to 2048
1024 is a bit small in some situations, double it.

Resolves: rhbz#1538747
(cherry picked from commit 594e762d63)
2018-06-11 13:47:59 -07:00
Martin Kolman
68e850d57a Make LogRequestHandler configurable
Make it possible to manipulate the simple and regexp
tests the LogRequestHandler class uses to check error
messages for potential error states.

This is accomplished by moving the simple and regexp test
strings to class members, where they can be easily
manipulated by users of the pylorax module.

It's also now possible to set the log request handler class
for a LogMonitor.

This functionality can then be used for example like this:

customized_log_request_handler = monitor.LogRequestHandler
customized_log_request_handler.simple_tests.remove("Call Trace:")
log_monitor = monitor.LogMonitor(install_log,
                                 timeout=opts.timeout,
                                 log_request_handler_class = customized_log_request_handler)

This way installation will continue even if there was a call
trace in the logs. In a similar way additional tests and regexps can be
also added.

(cherry picked from commit d5d3dd3be3)
2018-06-11 11:04:00 -07:00
Brian C. Lane
a1905c3b4b Add --dracut-arg support to lorax
Use it to override the default dracut arguments (displayed as part of
the --help output). If you want to extend the default arguments they
all need to be passed in on the cmdline as well. eg.

--dracut-arg='--xz' --dracut-arg='--install /.buildstamp' ...

Resolves: rhbz#1452220
(cherry picked from commit d8ce013a2b)
2018-06-07 09:01:42 -07:00
Brian C. Lane
b9fe90000e gevent has deprecated .wsgi, should use .pywsgi instead
https://github.com/gevent/gevent/blob/master/doc/api/gevent.wsgi.rst
(cherry picked from commit c9ca451568)
2018-06-05 16:58:25 -07:00
Brian C. Lane
0314cbd0e8 Add support for sources to composer-cli
This adds the sources command which can be used to list, add, change,
and delete sources using the TOML formatted source file.

(cherry picked from commit 6f6ce410c0)
2018-06-04 15:48:41 -07:00
Brian C. Lane
693dc9a6f6 Fix DNF related issues with source selection
DNF Repo.dump() function cannot be used as a .repo file for dnf due to
it writing baseurl and gpgkey as a list instead of a string. Add a new
function to write this in the correct format, and limited to the fields
we use.

Add a test for the new function.

Fix /projects/source/info to return an error 400 if a nonexistant TOML
source is requested. If JSON is used the error is part of the standard
response.

Update test_server.py to check for the correct error code.

(cherry picked from commit afa89ea657)
2018-06-04 15:47:38 -07:00
Brian C. Lane
1ab33b6416 Fix handling bad source repos and add a test
When adding a source failed it wasn't being removed from the dnf object.
This fixes that, and returns an error when setting up the source fails.
Also adds a test for it.
This also includes detecting rawhide vs. non-rawhide releases and
adjusting the tests accordingly (some of the source names change).

(cherry picked from commit dd8e4d9e99)
2018-06-04 15:47:32 -07:00
Brian C. Lane
a283ec9582 Fix make_dnf_dirs
It was chopping off an extra directory level due to realpath removing
the trailing / from the paths when they are setup.

(cherry picked from commit 23f4b2a3ec)
2018-06-04 15:47:15 -07:00
Brian C. Lane
8ac04a1521 Add support for user defined package sources API
This lives under /api/v0/projects/source/*

See the documentation for details

(cherry picked from commit 6d677b2207)
2018-06-04 15:47:00 -07:00
Colin Walters
e5ef195a3b templates: Stop using gconfset
We had only been indirectly pulling in GConf, and anyways
nothing was listening to these keys.

<kalev> I still think it's a fallout from 27a90d973f

Really in general, if we wanted to make changes like this
it'd probably be a lot simpler to do them on boot or so.

https://bugzilla.redhat.com/show_bug.cgi?id=1581838
(cherry picked from commit bb3d8edd06)
2018-05-24 09:13:05 -07:00
Brian C. Lane
1af6040143 Add support for version globs to blueprints
This uses dnf's version__glob filter to implement it. It amounts to '*'
wildcards and '?' for single character matching.

(cherry picked from commit 095829171a)
2018-05-21 15:41:45 -07:00
Brian C. Lane
55b89d1aeb Add support for systemd socket activation
Instead of enabling lorax-composer.service enable lorax-composer.socket
and it will start lorax-composer on first access to
/run/weldr/api.socket
2018-05-17 10:34:35 -07:00
Brian C. Lane
e7f9028fca Move lorax-composer and composer-cli argument parsing into modules
This allows sphinx-argparse to document them automatically.
2018-05-17 10:34:35 -07:00
Brian C. Lane
cd1d414e0e Add new cmdline args to compose_args settings 2018-05-17 10:34:35 -07:00
Brian C. Lane
e91aed9b10 Remove temporary files after run_compose
A crash can also leave temporary lmc-* files, remove them as well.
2018-05-17 10:34:35 -07:00
Brian C. Lane
611f5e62f3 Add --proxy to lorax-composer cmdline
Overrides the [dnf] proxy setting in the config file.
2018-05-17 10:34:35 -07:00
Brian C. Lane
d9c4a6c951 Pass the --tmp value into run_creator and cleanup after a crash
Crashing can sometimes leave directories in /var/tmp/lmc-* so clean
those up after run_creator is finished.
2018-05-17 10:34:35 -07:00
Brian C. Lane
559b66b7e3 Add --tmp to lorax-composer and set default tempdir
It was using /tmp/ which can fill up quickly when building images.
Default to /var/tmp unless the user passes --tmp /tmp/
2018-05-17 10:34:35 -07:00
Brian C. Lane
bb652368dc Set lorax_templates to the correct directory
Templates are under the share dir, so use find_templates() to find them.
2018-05-17 10:34:35 -07:00
Brian C. Lane
d6283f9540 Adjust the disk size estimates to match Anaconda
First is Anaconda uses 6k blocks per file for its estimate, and it
fudges by 10% so adjust for those with an extra 10% of headroom just in
case.

Second is an Anaconda bug that won't allow it to do a kickstart install
to a disk smaller than 3000 MB. There is a PR to fix it upstream, but
for now the minimum size has to be 3000e9
2018-05-17 10:34:35 -07:00
Brian C. Lane
5443c1cfea Skip creating groups with the same name as a user
Otherwise the user creation fails when anaconda sees there is already a
group with that name. Log a warning and continue on.
2018-05-17 10:34:35 -07:00
Brian C. Lane
cb7e4e55ba Add user and group creation to blueprint
[[customizations.user]] and [[customizations.group]]
2018-05-17 10:34:34 -07:00
Brian C. Lane
8839182f43 Add blueprint customization support for hostname and ssh key
This adds support for the optional blueprint section [customizations].

Use it like this:

[customizations]
hostname = yourhostnamehere

[[customizations.sshkey]]
user = root
key = root user key
2018-05-17 10:34:34 -07:00
Brian C. Lane
fb58277028 Add composer-cli and tests 2018-05-17 10:34:34 -07:00
Brian C. Lane
b67c821706 Fix the compose arguments for the Fedora version of Anaconda
This adds make_oci, and moves qcow2 to the more generic image_type
(which just uses qemu-img disk names).
2018-05-17 10:34:34 -07:00
Brian C. Lane
b6e0ba2e45 Add selinux check to lorax-composer
anaconda needs to have SELinux set to disabled or permissive in order to
run correctly. Check at startup and exit with an error.
2018-05-17 10:34:34 -07:00
Brian C. Lane
e6145faf86 Convert Yum usage to DNF
The DNF api is similar, but not the same, as Yum. Make the needed
changes, and rename yum references to dnf to avoid confustion later.
2018-05-17 10:34:34 -07:00
Brian C. Lane
c4a09c42cc workspace read and write needs UTF-8 conversion
reading needs to decode the bytes to str and writing needs to encode the
str to bytes.
2018-05-17 10:34:34 -07:00
Brian C. Lane
99cdd573a3 Return an empty list if depsolve results are empty 2018-05-17 10:34:34 -07:00
Brian C. Lane
6aa52b0b53 The git blob needs to be bytes
In py3 str is unicode so it needs to be encoded as UTF-8 when writing to
the blob.
2018-05-17 10:34:34 -07:00
Brian C. Lane
0f59710b36 Use default commit sort order instead of TIME
Different versions of libgit2 act differently. Using TIME results in
some commits (like a revert) being listed correctly, but the rest being
listed in reverse order. Leaving it at the default works for
libgit2-0.26.3
2018-05-17 10:34:34 -07:00
Brian C. Lane
cb0158ca22 Add lorax-composer and the composer kickstart templates 2018-05-17 10:34:34 -07:00