Commit Graph

956 Commits

Author SHA1 Message Date
Lars Karlitski
c9be91c879 Rename composer-cli to composer 2018-09-20 12:02:03 +03:00
David Shea
c00036251e Fix a DeprecationWarning
SafeConfigParser is just a deprecated version of ConfigParser in
python3, so use ConfigParser.
2018-09-07 13:34:30 -04:00
Brian C. Lane
a0aab15311 Ignore a pylint warning about UnquotingConfigParser get args
The args differ, but we are accepting and passing through all args so
it's ok.

Related: rhbz#1613058
2018-09-04 11:16:39 -07:00
Adam Williamson
5c0dfb74e6 Ditch all use of pyanaconda's simpleconfig
lorax uses pyanaconda's SimpleConfigParser in three different
places (twice with a copy that's been dumped into pylorax, once
by importing it), just to do a fairly simple job: read some
values out of /etc/os-release. The only value SimpleConfigParser
is adding over Python's own ConfigParser here is to read a file
with no section headers, and to unquote the values. The cost is
either a dependency on pyanaconda, or needing to copy the whole
of simpleparser plus some other utility bits from pyanaconda
into lorax. This seems like a bad trade-off.

This changes the approach: we copy one very simple utility
function from pyanaconda (`unquote`), and do some very simple
wrapping of ConfigParser to handle reading a file without any
section headers, and returning unquoted values. This way we can
read what we need out of os-release without needing a dep on
pyanaconda or to copy lots of things from it into pylorax.

Resolves: #449
Resolves: #450

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-08-30 16:21:09 -07:00
Brian C. Lane
4d9ef60460 Add support for DNF 3.2 module_platform_id config value
This borrows simpleconfig.py from Anaconda to make parsing os-release
easier.

It defaults to platform:el8
2018-08-29 11:04:50 -07:00
Brian C. Lane
07acd2e780 lorax: Only run depmod on the installed kernels
In the near-future there may be /lib/modules/ directories for older
kernels with weak dependencies listed. These may not match the installed
kernel(s) so we cannot depend on them to drive generate_module_data.

Instead use the existing findkernels() function to get the list of
installed kernels and iterate those, running depmod on them.

Resolves: rhbz#1622213
2018-08-28 16:56:50 -07:00
Brian C. Lane
57674c9a1a Fix composer-cli blueprints changes to get correct total
blueprints/changes is different, each blueprint has it's own total,
limited by the call's limit. So it needs to find the max total of all
the requested blueprints.
2018-08-27 11:08:04 -07:00
Brian C. Lane
ec908fcd2a Fix blueprints/list and blueprints/changes to return the correct total 2018-08-27 11:08:01 -07:00
Brian C. Lane
0a76d635ca Add a function to get_url_json_unlimited to retrieve the total
The blueprints/changes API is a bit different from the others, the total
that it includes is for each blueprint, not one total for all of them,
since there will be a different number of commits for each.

The function is passed the dict, and it can be used to select the total
to use for retrieving all of the results. If it isn't included it will
use data["total"] which works fine in most cases.
2018-08-27 11:03:59 -07:00
Brian C. Lane
86d556e87c Use urllib.parse instead of urlparse
python3 moved this to a new module.
2018-08-21 16:49:41 -07:00
David Shea
ee98d87cea In composer-cli, request all results
Add a limit argument to all potentially paginated results, equal to
whatever the composer backend is the total number of results. This still
has the potential to provide truncated data if the number of results
increases between the two HTTP requests.

Resolves: #404
2018-08-21 16:43:44 -07:00
David Shea
deff4d325f Allow '*' as a uuid in /compose/status/<uuid>
This will display all UUIDs that match the filter arguments
2018-08-21 16:42:18 -07:00
David Shea
40f23f093d Add filter arguments to /compose/status
This adds the following optional arguments to the /compose/status route:

  - type, matches the compose_type field
  - status, matches the queue_status field
  - blueprint, matches the blueprint field
2018-08-21 16:42:07 -07:00
Brian C. Lane
844ff9998b composer-cli should not log to a file by default
The user can pass --log /path/to/logfile.log if they want logging
enabled.
2018-08-21 11:49:28 -07:00
Lars Karlitski
be5d50e6f3 Set TCP listen backlog for API socket to SOMAXCONN
A value of 1 is too low for heavy users of the API, such as the weldr-web
interface.

This is also systemd's default for sockets it opens. Using lorax-composer with
socket activation already results in a backlog of SOMAXCONN connections.
2018-08-20 14:54:07 -07:00
Chris Lumens
47d3a72014 Fix a little bug in running "modules list".
(cherry picked from commit 5dddef6f49)
2018-08-10 12:00:23 -04:00
Brian C. Lane
c8283adc17 Move disklabel and UEFI support to compose.py
Currently we are making MBR disk images for qcow2 and partitioned disk,
so the UEFI packages aren't required at this point.

Move the clearpart command into compose.py so that in the futute it can
use clearpart --disklabel to create a GPT image, and add the required
packages to the package set.
2018-08-09 15:17:27 -07:00
Chris Lumens
faba254be5 Change INVALID_NAME to INVALID_CHARS.
It's getting used in a whole lot more places now, so make it a bit more
generic sounding.

(cherry picked from commit bf0318eab6)
2018-08-09 16:20:23 -04:00
Chris Lumens
473ee99eb9 Update composer-cli for the new error return types.
(cherry picked from commit bc96f75992)
2018-08-09 16:20:17 -04:00
Chris Lumens
d03a198a8c Add default error IDs everywhere else.
The idea here is to make sure all return points have the same type for
the error cases.  There's not really all that many, so they just go in
one patch.  Some of these could potentially turn into more specialized
errors later.

(cherry picked from commit fd901c5e3f)
2018-08-09 16:20:01 -04:00
Chris Lumens
6a3ccd6ade Add error IDs to things that can go wrong when running a compose.
Note the exception string checking around compose_type.  I didn't really
want to introduce a new exception type just for this, but also didn't
want to duplicate strings.  I'd be open to other suggestions for how to
do this.

(cherry picked from commit b3bb438254)
2018-08-09 16:19:19 -04:00
Chris Lumens
c1af7dd69d Add error IDs for common source-related errors.
(cherry picked from commit e43adfc7af)
2018-08-09 16:19:12 -04:00
Chris Lumens
f67e2c5d52 Add error IDs for unknown modules and unknown projects.
(cherry picked from commit 2adcfc9563)
2018-08-09 16:19:08 -04:00
Chris Lumens
785ee7bcc3 Add error IDs for when an unknown commit is requested.
(cherry picked from commit 07528a431e)
2018-08-09 16:19:03 -04:00
Chris Lumens
2f8f076b1f Add error IDs for when an unknown blueprint is requested.
This adds some fairly redundant code to the beginning of all the
blueprint routes to attempt reading a commit from git for the
blueprint's recipe.  If it succeeds, the blueprint exists and the route
can continue.  Otherwise, return an error.  Hopefully this doesn't slow
things down too much.

(cherry picked from commit a925cc7ddb)
2018-08-09 16:18:49 -04:00
Chris Lumens
a2ce0686ca Add error IDs for when an unknown build UUID is requested.
Note that this also changes the return type of uuid_info to return None
when an unknown ID is given.  The other uuid_* functions are fine
because they are checked ahead of time.

(cherry picked from commit 6497b4fb65)
2018-08-09 16:18:40 -04:00
Chris Lumens
b26d12e457 Add error IDs for bad state conditions.
(cherry picked from commit d76e24053a)
2018-08-09 16:18:23 -04:00
Chris Lumens
2efbd7cae5 Change the error return type for bad limit= and offset=.
Each element in the errors value is now a dict, with a msg field and an
id field.  The id field contains a value out of errors.py that can be
used by the front end to key on.  The msg field is the same as what's
been there.

The idea is to keep the number of IDs somewhat limited so there's not a
huge number of things for the front end to know.

(cherry picked from commit 9677b012da)
2018-08-09 16:18:17 -04:00
Chris Lumens
6c3b0b3cd2 Don't sort error messages.
This should make it easier to return more complex error structures.  It
also doesn't appear to matter - tests still pass without changes.

(cherry picked from commit 4c3f93e329)
2018-08-09 16:18:11 -04:00
Brian C. Lane
0e97afdbd6 Add + to the allowed API string character set 2018-08-07 16:47:54 -07:00
Brian C. Lane
316a829ff7 Add job_* timestamp support to compose status 2018-08-07 15:21:10 -07:00
Brian C. Lane
ec4c555174 Add input string checks to the branch and format arguments
Make sure no UTF8 characters are allowed and return an error if they
are.

Also includes tests to make sure the correct error is returned.

(cherry picked from commit 86d79cd8a6)
2018-08-07 10:12:02 -07:00
Brian C. Lane
f93f64a061 Add a test for invalid characters in the API route
Currently the code is not UTF8 safe, so we need to return a clear error
when invalid characters are passed in.

This also adds tests for the routes to confirm that an error is
correctly returned.

(cherry picked from commit 74f5def3d4)
2018-08-07 10:12:02 -07:00
Will Woods
439a894947 composer-cli: clean up "list" commands
This patch does two things:

1) Add "compose list", which lists compose UUIDs and other basic info,
2) Fix up "blueprints list", "modules list", "sources list", and
   "compose types" so their output is just a plain list of identifiers
2018-08-07 13:00:56 -04:00
Brian C. Lane
741e73265c Fix logging argument 2018-08-07 09:24:11 -07:00
Brian C. Lane
b4f9fb5c1f Fix write_timestamp for py3
toml.dumps() returns a string for write() so .encode is not needed.
2018-08-07 09:22:18 -07:00
Chris Lumens
8a2a43be99 Return a JSON error instead of a 404 on certain malformed URLs.
This handles the case where a route is requested, but without a required
parameter.  So, /blueprints/info is requested instead of
/blueprints/info/http-server.  It accomplishes this via a decorator, so
a lot of these route-related functions now have quite a few decorators
attached to them.

Typo'd URLs (/blueprints/nfo for instance) will still return a 404.  I
think this is a reasonable thing to do.

(cherry picked from commit 5daf2d416a)
2018-08-06 16:57:49 -07:00
Chris Lumens
76f714eab5 Return an error if /modules/info doesn't return anything.
Unfortunately, this isn't very useful if /modules/info is provided with
multiple modules.  yum doesn't traceback when doPackageLists is given
something that doesn't exist.  It just returns an empty list.  If
/modules/info is given just one module and yum gives us an empty list,
it's easy to say what happened.  If /modules/info is given several
modules and just one does not exist, we will not be able to detect that.

Fixing this would require doing more yum operations, which is likely to
slow things down and isn't the direction I want to be going.

(cherry picked from commit 8e948e4a4d)
2018-08-06 16:57:40 -07:00
Chris Lumens
ea7b22a8f5 Update documentation (#409).
(cherry picked from commit 6193a7c9d8)
2018-08-06 16:57:34 -07:00
Chris Lumens
022734ab01 Use constants instead of strings (#409).
(cherry picked from commit fdf4d63f3b)
2018-08-06 16:57:26 -07:00
Chris Lumens
495dcd13ec Write timestamps when important events happen during the compose (#409).
Right now, this is when the compose is queued up, when it is started by
anaconda, and when it is finished (whether that's success or not).

(cherry picked from commit 3ba9d53b8b)
2018-08-06 16:57:19 -07:00
Chris Lumens
e42b797295 Return multiple timestamps in API results (#409).
If one of the timestamps isn't present (for instance, the finished
timestamp for a job that is still running), null is returned.

(cherry picked from commit 17c40ef271)
2018-08-06 16:57:13 -07:00
Chris Lumens
ab76c6170e Add a new timestamp.py file to the API directory (#409).
This is responsible for writing out a new times.toml file, containing
important timestamps in the life of a compose.  This seems a little more
reliable than attempting to infer things from the filesystem, especially
in light of the fact that we can't ever really know when a file was
created.

(cherry picked from commit b59d59b124)
2018-08-06 16:57:07 -07:00
Brian C. Lane
0c19c3304e composer-cli: Add support for Group to blueprints diff
(cherry picked from commit 2edd7a995b)
2018-08-06 16:53:39 -07:00
Brian C. Lane
08fb780bb9 Update status.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.

(cherry picked from commit 642f117d54)
2018-08-06 16:53:07 -07:00
Brian C. Lane
db587afb16 Update sources.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.

(cherry picked from commit 3205e47a13)
2018-08-06 16:53:02 -07:00
Brian C. Lane
f0dd50fbae Update projects.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.

(cherry picked from commit 77767cd93a)
2018-08-06 16:52:56 -07:00
Brian C. Lane
42ad6ad0cc Update modules.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.

(cherry picked from commit b55b86d0b3)
2018-08-06 16:52:51 -07:00
Brian C. Lane
4e3044de20 Update compose.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.

(cherry picked from commit 760aeb61a7)
2018-08-06 16:52:43 -07:00
Brian C. Lane
b40c3b2ce3 Update blueprints.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.

(cherry picked from commit c1c5536ec8)
2018-08-06 16:52:37 -07:00
Brian C. Lane
08c2965fab Modify handle_api_result so it can be used in more places
Some results have errors and no status, others have status and errors.
Update the function to return the final rc to exit with, and a bool
indicating whether or not to continue processing the other fields.

Add a bunch of tests for the new function to make sure I have the logic
correct.

(cherry picked from commit 35fa067219)
2018-08-06 16:52:28 -07:00
Chris Lumens
2d5f434b12 Fix help output on the compose subcommand.
(cherry picked from commit a3572c9fdd)
2018-07-26 10:43:01 -04:00
Chris Lumens
2251c540c8 Add timestamps to "compose-cli compose status" output.
(cherry picked from commit 7bcb61849d)
2018-07-26 10:42:57 -04:00
Chris Lumens
01582e227d And then add real output to the status command.
(cherry picked from commit eb0939d967)
2018-07-26 10:42:08 -04:00
Chris Lumens
80e2e8e6db Add the beginnings of a new status subcommand.
(cherry picked from commit 9eafc60fa0)
2018-07-26 10:42:04 -04:00
Brian C. Lane
18521cec1a composer-cli: Fix non-zero epoch in projets info
The json value is an int, so convert it to a str.
2018-07-25 11:35:03 -07:00
Brian C. Lane
b1dd22afa6 Log and exit on metadata update errors at startup
A bad system repo can cause lorax-composer to fail to start. Instead of
a traceback log the error and exit.

(note that the exit still results in an OSError traceback due to part of
it running as root, this needs to be addressed in another commit).
2018-07-18 16:09:03 -07:00
Brian C. Lane
83680d92b0 Check /projects responses for null values.
Some values can be null/None so check for that instead of crashing.

(cherry picked from commit f916e41d00)
2018-07-18 16:01:18 -07:00
Brian C. Lane
816f1658db Clarify error message from /source/new
(cherry picked from commit b20aa4609c)
2018-07-18 16:01:06 -07:00
Chris Lumens
fbda20b8b7 Support loading groups from the kickstart template files.
(cherry picked from commit c8d2045f89)
2018-07-18 15:38:40 -07:00
Chris Lumens
8c4804eb15 Include groups in depsolving.
This adds a new argument to projects_depsolve and
projects_depsolve_with_size that contains the group list, unfortunately.
I would have prefered adding a function that just returns a list of all
the contents of a group and then add that to what was being passed into
projects_depsolve.  However, there does not appear to be any good way to
do that in yum aside from a lot of grubbing around in the comps object,
which I am unwilling to do.

(cherry picked from commit 5fe4b47072)
2018-07-18 14:36:28 -07:00
Chris Lumens
76d376fe18 Add support for groups to blueprints.
Nothing is currently being done with this information, but it will be
soon.

(cherry picked from commit 0f69d2084c)
2018-07-18 13:48:25 -07:00
Chris Lumens
bb8fdcb854 Add help output to each subcommand.
This is the same as the output at the top level, just trimmed down to
only the options for a single subcommand.  It's trigged by providing
"help" or "--help" as a subcommand option.

(cherry picked from commit 954f330ace)
2018-07-18 11:58:56 -07:00
Chris Lumens
eba5658a71 Split the help output into its own module.
This means I can reuse it for help output for individual subcommands.

(cherry picked from commit 70e4211ad1)
2018-07-18 11:57:45 -07:00
Chris Lumens
3c6478e179 If the help subcommand is given, print the help output.
This isn't a real subcommand like the others.  The option processing
just intercepts it and prints the output.  Given that we're subcommand
based, it makes sense to support this in addition to --help.

(cherry picked from commit 3743d6d208)
2018-07-18 11:49:16 -07:00
Brian C. Lane
460a277d4e Check the compose templates at startup
Depsolve the packages included in the templates and report any errors
using the /api/status 'msgs' field. This should help narrow down
problems with package sources not being setup correctly.
2018-07-13 10:18:07 -07:00
Adam Williamson
4a3f9e12aa DNF 3: progress callback constants moved to dnf.transaction
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-27 16:12:25 -07:00
Brian C. Lane
56926baaec Add the ability to enable DNF plugins for lorax
Pass --dnfplugin='*' to enable all of them.
Pass --dnfplugin='plugin-name' to enable one fo them. You can use it
multiple times to enable multiple plugins. Globs work as well.
2018-06-22 09:07:01 -07:00
Chris Lumens
e8894b7266 It's /compose/cancel, not /blueprints/cancel. 2018-06-18 10:51:12 -04:00
Brian C. Lane
936d193413 Retry losetup if loop_attach fails
It appears that sometimes the loop device doesn't get setup properly,
this may be a race with other users of loop devices on the system, or
some other mechanism that isn't understood.

To try and prevent total failure when this happens this patch retries
the loop setup 3 times before giving up. Previously it would wait for
the loop device to appear (checking 5 times), that operation is now
executed 3 times with a new losetup attempt each time.

Resolves: rhbz#1589084
(cherry picked from commit c746e8b0c3)
2018-06-11 14:00:05 -07:00
Brian C. Lane
594e762d63 Increase default ram used with lmc and virt to 2048
1024 is a bit small in some situations, double it.

Resolves: rhbz#1538747
2018-06-08 11:44:09 -07:00
Brian C. Lane
d8ce013a2b Add --dracut-arg support to lorax
Use it to override the default dracut arguments (displayed as part of
the --help output). If you want to extend the default arguments they
all need to be passed in on the cmdline as well. eg.

--dracut-arg='--xz' --dracut-arg='--install /.buildstamp' ...

Resolves: rhbz#1452220
2018-06-06 17:10:35 -07:00
Martin Kolman
d5d3dd3be3 Make LogRequestHandler configurable
Make it possible to manipulate the simple and regexp
tests the LogRequestHandler class uses to check error
messages for potential error states.

This is accomplished by moving the simple and regexp test
strings to class members, where they can be easily
manipulated by users of the pylorax module.

It's also now possible to set the log request handler class
for a LogMonitor.

This functionality can then be used for example like this:

customized_log_request_handler = monitor.LogRequestHandler
customized_log_request_handler.simple_tests.remove("Call Trace:")
log_monitor = monitor.LogMonitor(install_log,
                                 timeout=opts.timeout,
                                 log_request_handler_class = customized_log_request_handler)

This way installation will continue even if there was a call
trace in the logs. In a similar way additional tests and regexps can be
also added.
2018-06-06 14:30:28 -07:00
Brian C. Lane
c9ca451568 gevent has deprecated .wsgi, should use .pywsgi instead
https://github.com/gevent/gevent/blob/master/doc/api/gevent.wsgi.rst
2018-06-05 14:57:21 -07:00
Brian C. Lane
6f6ce410c0 Add support for sources to composer-cli
This adds the sources command which can be used to list, add, change,
and delete sources using the TOML formatted source file.
2018-06-04 15:45:36 -07:00
Brian C. Lane
afa89ea657 Fix DNF related issues with source selection
DNF Repo.dump() function cannot be used as a .repo file for dnf due to
it writing baseurl and gpgkey as a list instead of a string. Add a new
function to write this in the correct format, and limited to the fields
we use.

Add a test for the new function.

Fix /projects/source/info to return an error 400 if a nonexistant TOML
source is requested. If JSON is used the error is part of the standard
response.

Update test_server.py to check for the correct error code.
2018-06-04 15:45:36 -07:00
Brian C. Lane
dd8e4d9e99 Fix handling bad source repos and add a test
When adding a source failed it wasn't being removed from the dnf object.
This fixes that, and returns an error when setting up the source fails.
Also adds a test for it.
This also includes detecting rawhide vs. non-rawhide releases and
adjusting the tests accordingly (some of the source names change).
2018-06-04 15:45:36 -07:00
Brian C. Lane
23f4b2a3ec Fix make_dnf_dirs
It was chopping off an extra directory level due to realpath removing
the trailing / from the paths when they are setup.
2018-06-04 15:45:36 -07:00
Brian C. Lane
6d677b2207 Add support for user defined package sources API
This lives under /api/v0/projects/source/*

See the documentation for details
2018-06-04 15:45:36 -07:00
Colin Walters
bb3d8edd06 templates: Stop using gconfset
We had only been indirectly pulling in GConf, and anyways
nothing was listening to these keys.

<kalev> I still think it's a fallout from 27a90d973f

Really in general, if we wanted to make changes like this
it'd probably be a lot simpler to do them on boot or so.

https://bugzilla.redhat.com/show_bug.cgi?id=1581838
2018-05-23 14:20:44 -07:00
Brian C. Lane
095829171a Add support for version globs to blueprints
This uses dnf's version__glob filter to implement it. It amounts to '*'
wildcards and '?' for single character matching.
2018-05-18 12:03:26 -07:00
Brian C. Lane
48e318b391 Add support for systemd socket activation
Instead of enabling lorax-composer.service enable lorax-composer.socket
and it will start lorax-composer on first access to
/run/weldr/api.socket
2018-05-14 13:57:44 -07:00
Brian C. Lane
c726f19434 Update the generated html docs 2018-05-14 13:00:15 -07:00
Brian C. Lane
db67c1267a Move lorax-composer and composer-cli argument parsing into modules
This allows sphinx-argparse to document them automatically.
2018-05-14 13:00:15 -07:00
Brian C. Lane
6e5ca1f062 Add new cmdline args to compose_args settings 2018-05-14 13:00:15 -07:00
Brian C. Lane
b1319c7bc1 Remove temporary files after run_compose
A crash can also leave temporary lmc-* files, remove them as well.
2018-05-14 13:00:15 -07:00
Brian C. Lane
0455a5d740 Add --proxy to lorax-composer cmdline
Overrides the [dnf] proxy setting in the config file.
2018-05-14 13:00:15 -07:00
Brian C. Lane
12cb2736ac Pass the --tmp value into run_creator and cleanup after a crash
Crashing can sometimes leave directories in /var/tmp/lmc-* so clean
those up after run_creator is finished.
2018-05-14 13:00:15 -07:00
Brian C. Lane
3c31e52b3c Add --tmp to lorax-composer and set default tempdir
It was using /tmp/ which can fill up quickly when building images.
Default to /var/tmp unless the user passes --tmp /tmp/
2018-05-14 13:00:15 -07:00
Brian C. Lane
9a1dce5619 Set lorax_templates to the correct directory
Templates are under the share dir, so use find_templates() to find them.
2018-05-14 13:00:15 -07:00
Brian C. Lane
3974a4e479 Adjust the disk size estimates to match Anaconda
First is Anaconda uses 6k blocks per file for its estimate, and it
fudges by 10% so adjust for those with an extra 10% of headroom just in
case.

Second is an Anaconda bug that won't allow it to do a kickstart install
to a disk smaller than 3000 MB. There is a PR to fix it upstream, but
for now the minimum size has to be 3000e9
2018-05-14 13:00:15 -07:00
Brian C. Lane
a363aee971 Skip creating groups with the same name as a user
Otherwise the user creation fails when anaconda sees there is already a
group with that name. Log a warning and continue on.
2018-05-14 13:00:15 -07:00
Brian C. Lane
07ea61be77 Add user and group creation to blueprint
[[customize.user]] and [[customize.group]]
2018-05-14 13:00:15 -07:00
Brian C. Lane
25bec0b50d Add blueprint customization support for hostname and ssh key
This adds support for the optional blueprint section [customizations].

Use it like this:

[customizations]
hostname = yourhostnamehere

[[customiations.sshkey]]
user = root
key = root user key
2018-05-14 13:00:15 -07:00
Brian C. Lane
825d1931e4 Add composer-cli and tests 2018-05-14 13:00:15 -07:00
Brian C. Lane
a9b626a706 Fix the compose arguments for the Fedora version of Anaconda
This adds make_oci, and moves qcow2 to the more generic image_type
(which just uses qemu-img disk names).
2018-05-14 13:00:15 -07:00
Brian C. Lane
5e0246d4a9 Add selinux check to lorax-composer
anaconda needs to have SELinux set to disabled or permissive in order to
run correctly. Check at startup and exit with an error.
2018-05-14 13:00:15 -07:00
Brian C. Lane
c9187ce4ca Convert Yum usage to DNF
The DNF api is similar, but not the same, as Yum. Make the needed
changes, and rename yum references to dnf to avoid confustion later.
2018-05-14 13:00:15 -07:00
Brian C. Lane
48671b682a workspace read and write needs UTF-8 conversion
reading needs to decode the bytes to str and writing needs to encode the
str to bytes.
2018-05-14 13:00:15 -07:00
Brian C. Lane
d478864fb3 Return an empty list if depsolve results are empty 2018-05-14 13:00:15 -07:00
Brian C. Lane
8ed039107b The git blob needs to be bytes
In py3 str is unicode so it needs to be encoded as UTF-8 when writing to
the blob.
2018-05-14 13:00:15 -07:00
Brian C. Lane
719aceac82 Use default commit sort order instead of TIME
Different versions of libgit2 act differently. Using TIME results in
some commits (like a revert) being listed correctly, but the rest being
listed in reverse order. Leaving it at the default works for
libgit2-0.26.3
2018-05-14 13:00:15 -07:00
Brian C. Lane
2fa1da0f5e Add lorax-composer and the composer kickstart templates 2018-05-14 13:00:15 -07:00
Brian C. Lane
9b9b627fe5 Update pylorax.api.projects for DNF usage
And adjust tests to match.
2018-05-14 13:00:14 -07:00
Brian C. Lane
e86f4f9a36 Update dnfbase (formerly yumbase) for DNF support
And adjust tests to match.
2018-05-14 13:00:14 -07:00
Brian C. Lane
06c227598c Move core of livemedia-creator into pylorax.creator
This moves everything except the cmdline checking into run_creator in
pylorax.creator

It also rearranges some functions to prevent import loops, and adds a
utility function to imgutils (mkfsimage_from_disk for copying a
partition into a filesystem image).
2018-05-14 13:00:14 -07:00
Brian C. Lane
63b473d4ba Convert lorax-composer yum base object to DNF
This no longer uses the enabled configuration setting to select repos to
use. It uses everything in the repo_dir, and if system repos have not
been disabled it copies them into the repo_dir at startup, overwriting
the previous copy.
2018-05-14 13:00:14 -07:00
Brian C. Lane
d31bea7bd3 Use 2to3 to convert the python2 lorax-composer code to python3 2018-05-14 13:00:14 -07:00
Brian C. Lane
89050f068d livemedia-creator: Move core functions into pylorax modules
This reduces the amount of code in livemedia-creator to the cmdline
parsing and calling of the installer functions. Moving them into other
modules will allow them to be used by other projects, like the
lorax-composer API server.
2018-05-14 13:00:14 -07:00
Brian C. Lane
dc348fac25 Check selinux state before creating output directory
This closes issue #163
2018-04-26 11:12:24 -07:00
Jiri Konecny
2f3de6060b Update default releasever to Fedora 29 (rawhide)
There was Fedora 25 which is now unsupported.
2018-03-15 12:07:34 -07:00
Brian C. Lane
e1c839de1c Fix _install_branding
filter(provides=...) doesn't work with paths. The release packages
provide system-release so just look for that instead of a file.

Now it finds the release package and selects it along with the
corresponding logos package.

Note, this has been broken since commit 431ca6ce
2018-02-23 12:50:07 -08:00
Yuval Turgeman
68d1f4208d LMC: Make the QEMU RNG device optional
Added a new QEMU argument '--with-rng' to specify a RNG device other
than /dev/random or "none".

Signed-off-by: Yuval Turgeman <yturgema@redhat.com>
2018-01-23 08:22:38 -08:00
Brian C. Lane
dc635a97f6 Write the --variant string to .buildstamp as 'Variant=' 2018-01-17 09:38:16 -08:00
Brian C. Lane
629d04dc68 Fix installpkg exclude operation
Commit 8edaefd4d1 added the ability to install specific NVR's of
packages, but it did not adjust the exclude operation to account for
this.

This patch fixes that, applying the exclude only to the name part of the
package NVR, and changes some variable names to pkgnvr/pkgnvrs to make
it more clear that the content has changed to <name>-<version>-<release>
2018-01-15 08:38:43 -08:00
Brian C. Lane
73628a6e07 Log a more descriptive error when setfiles fails (#1499771)
Some lorax users run it from inside mock, which isn't able to detect
whether the host is in Permissive mode. This can lead to confusing
error messages, so this points them in the right direction.
2017-11-27 12:03:45 -08:00
Claudio Zumbo
8edaefd4d1 Allow installpkgs to do version pinning through globbing 2017-11-10 09:56:27 -08:00
Brian C. Lane
6147d58ccf Use bytes when writing strings in mk-s390-cdboot (#1504026)
python3 needs bytes when calling .write()
2017-10-19 14:37:12 -07:00
Brian C. Lane
aa0067e32c Convert mk-s390-cdboot to python3 (#1497141)
It was pulling in py2 along with py3. Fix #! and make sure code works
with py3.
2017-10-11 10:27:52 -07:00
Alexander Todorov
fdb7593cb0 Rename parameters to match names that dnf uses
avoids 'Parameters differ from overridden method' warnings
2017-10-09 13:38:56 -07:00
Alexander Todorov
ac1623d9cd Don't override 'line' from outer scope 2017-10-09 13:38:56 -07:00
Brian C. Lane
9181d69098 Add mk-s360-cdboot utility (#1478448)
Add a utility to concatenate the kernel and initrd, and edit the kernel.

This is based on:
https://github.com/rhinstaller/anaconda/blob/rhel6-branch/utils/mk-s390-cdboot.c

It takes into account byteorder and it wipes the cmdline area before
writing the parmfile into the kernel.

Related: rhbz#1478448
(cherry picked from commit 6007b35096)
2017-08-14 14:29:47 -07:00
Brian C. Lane
3382a24498 Fix systemctl command (#1478247)
When multiple units are passed to systemctl and one fails it doesn't
finish the others. Change the template command to call systemctl for
each unit individually.

This also removes the lvm2-activation-generator in runtime-cleanup.tmpl
2017-08-14 14:29:47 -07:00
Brian C. Lane
e3dcb1e329 Add version output (#1335456)
Add -V to livemedia-creator, and log the running version when lorax or
lmc is started.
2017-08-14 13:54:02 -07:00
Brian C. Lane
8154b3f7a3 Include the dracut fips module in the initrd (#1341280)
This will allow anaconda to fetch kickstarts using https when installing
with fips=1

Leave vmlinuz and .vmlinuz.hmac in /boot

dracut-fips module needs the vmlinuz.hmac file in order to boot.
2017-08-14 13:20:27 -07:00
Brian C. Lane
7bc818507c Make sure loop device is setup (#1462150)
It seems that on rare occasions losetup can return before the /dev/loopX
is ready for use, causing problems with mkfs. This tries to make sure
that the loop device really is associated with the backing file before
continuing.

NOTE that using losetup --list -O to return the backing store
associated with the loop device can fail due to losetup truncating
the output filename if sysfs isn't setup. Instead of printing the full
path it will truncate it to 64 characters with a * at the end.

See util-linux lib/loopdev.c for the code that does this.

Use the existing get_loop_name function, which uses losetup -j, to lookup
the loop device associated with the backing store which should work the
same, just in the opposite direction.
2017-08-14 09:20:42 -07:00
Stephen Gallagher
2c3007beb8 Use anaconda-core to detect buildarch
For historical reasons, lorax used the 'anaconda' package as a
touchstone to determine the architecture for the build. At some
point, this package became a metapackage that pulls in both the
GUI and headless installers.

In the modular world, it's possible that only the core and TUI bits
may be available for use. The only subpackage of anaconda that is
guaranteed to be on any viable system is anaconda-core, so let's
switch to using that for the touchstone instead of the metapackage.

Signed-off-by: Stephen Gallagher <sgallagh@redhat.com>
2017-05-31 10:07:41 -07:00
Brian C. Lane
1c6b083260 Try all packages when installpkg --optional is used.
Also sort the expanded list of packages so that any failures will
be consistent instead of depending on the randomness of a set().
And add better logging when things fail.

The core issue is that repodata may have packages that match globs, but
they cannot actually be installed (eg. sigrok-firmware). This can cause
*some* of the globbed packages to be installed before hitting the
failure package.

With this change it will log the expanded list of packages if a glob is
used. It will skip any packages that fail to install when using
--optional with the glob, and continue to install the rest.

Related: rhbz#1440417
2017-05-30 09:47:37 -07:00
Brian C. Lane
a92aa63491 pylint: Ignore different argument lengths for dnf callback.
It may or may not be used with a newer version of dnf. The callback
doesn't use the argument so just ignore it.
2017-05-24 09:13:42 -07:00
Jaroslav Mracek
10bbeac6d1 Adds additional callbacks keyword for start()
It reflects latest changes in dnf-2.5.0-1
2017-05-24 08:05:15 -07:00
Brian C. Lane
cc56a5b5c9 Add --noverifyssl to lorax (#1430483)
Previously lorax had no way to use repos with self-signed certificates.
This adds the --noverifyssl cmdline option which will ignore certificate
errors.

Resolves: rhbz#1430483
2017-03-08 13:27:22 -08:00
Brian C. Lane
f8c9e8c120 Print the full NEVRA when installing packages.
(cherry picked from commit 92e60d7dd1)
2017-01-30 09:43:31 -08:00
Yuval Turgeman
46277f36db Fixed NameError on result_dir when calling with --image-only
Signed-off-by: Brian C. Lane <bcl@redhat.com>
2017-01-13 13:38:33 -08:00
Colin Walters
3fa3d20489 imgutils: Don't relabel /ostree
OSTree is a deduplicating hardlink store using a new file path
`/ostree`, which SELinux policy doesn't know about.  However, OSTree
has SELinux support built in, and rpm-ostree (for example) uses this
to ensure the attributes on files stored there are simply always
correct.  Relabeling it will corrupt it.

Hence, let's skip it.
2016-11-16 16:39:53 -05:00
Pat Riehecky
7ca356845b Added option to remove packages (parallel to installpkgs) 2016-11-14 09:06:10 -08:00
Colin Walters
5aa6fea8c3 templates: When a subprocess fatally errors, output its stderr directly
Right now we dump all subprocess output to `program.log`.  Unfortunately,
The pungi/koji stack doesn't know how to scrape out the lorax logs.

And even when running interactively, it's annoying that *some* fatal
errors show up on stderr, but if it's from a subprocess, I need to go
over and `tail program.log`.

Let's output the subprocess stderr directly, since the user is
going to want it prominently anyways.
2016-11-14 09:04:08 -08:00
Igor Gnatenko
ef5475e552 adapt to DNF 2.0 API changes
References: https://bugzilla.redhat.com/show_bug.cgi?id=1380566
Signed-off-by: Igor Gnatenko <i.gnatenko.brain@gmail.com>
2016-09-30 09:09:41 -07:00
Brian C. Lane
492ab9e4a5 livemedia-creator: Check for packaging failures in the logs (#1374809)
anaconda-26.1 changed how package scriptlet failures are handled. They
are now fatal, and anaconda hangs after logging an Installation failure.

ERR packaging: Installation failed: PayloadInstallError('DNF error:
Non-fatal POSTIN scriptlet failure in rpm package mlocate',)

Catch this (the 'packaging: Installation failed' part) and terminate the
image creation.
2016-09-21 09:00:49 -07:00
Manjunath A Kumatagi
f4a1302358 Add ppc64le kernel path
(cherry picked from commit b76fba85ea)
2016-09-08 12:12:07 -07:00
Brian C. Lane
37d36daa9c lorax: Add --rootfs-size (#1368743)
This controls how big the root filesystem is for the squashfs used in
the boot.iso, the default is 2GiB.

Note that larger rootfs sizes will require more memory and may cause the
build to fail.
2016-08-22 17:20:48 -07:00
Brian C. Lane
442d0a0140 Revert "Use size=10 by default"
This reverts commit e61223d486.

On systems with less RAM (eg. ARM with 4G) this size change causes the
build to get stuck.
2016-08-22 16:17:35 -07:00
Friedrich Schöller
c60338464b Allow supplying a disk image for PXE live systems
Fixes that the combination of command line arguments
"--make-ostree-live" and "--disk-image" lead to a program error.
2016-08-12 13:12:54 -07:00
Colin Walters
e61223d486 Use size=10 by default
I'm working on
https://fedoraproject.org/wiki/Changes/WorkstationOstree and when
using lorax to make an installer ISO with content embedded, I run out
of disk space since the desktop+various apps is large.

Since this ends up being compressed anyways, let's just bump the
currently arbitrary `2` to `10` - the only real cost I can think of is
going to be a few more superblock entries.
2016-08-12 13:05:40 -07:00
Brian C. Lane
658df4f3f9 Fix installpkg error handling
If the query filter doesn't return anything it would just ignore the
install request instead of logging and raising an error when
required=True.

This checks for no packages matching, and if required is True raises an
error after all of the requested packages have been processed, instead
of after the first one to fail.
2016-06-24 13:40:58 -07:00
Brian C. Lane
fe17f970c9 Switch installpkg default to --required
Previous versions of lorax assumed that installpkg was optional, and
would continue on if the PKGGLOB didn't match anything. But the majority
of the packages are required so this allows the boot.iso to be built
with missing packages that are hard to track down.

It makes more sense to make the PKGGLOB required and to flag the
few exceptions to this with --optional.
2016-06-24 13:40:58 -07:00
Brian C. Lane
748259619b Make sure cmdline config file exists (#1348304)
If the --config file doesn't exist lorax will fall back to its defaults,
which are probably not what you want. Now it will exit if it doesn't
exist.
2016-06-20 16:13:45 -07:00
Brian C. Lane
5ef7c093b1 Stop using undocumented DNF logging API
DNF doesn't want users to access base.logging anymore.

Lorax already takes over the "dnf" logger and directs it to ./dnf.log,
so it wasn't really being used.

This raises the debug level to DNF's custom DDEBUG, and sets it up so
that dnf.librepo.log and hawkey.log are next to dnf.log
2016-06-03 16:44:30 -07:00
Brian C. Lane
3485fe462c livemedia-creator: Always copy novirt logs before cleanup
Before attempting to cleanup any dangling anaconda mounts copy the
anaconda logs to their final location.

Also, catch failures to cleanup the mounts, log it, and continue trying
the other mountpoints. A cleanup failure will result in an InstallError
instead of a CalledProcessError.
2016-06-01 11:48:41 -07:00
Brian C. Lane
551b27a8fa Update lmc UEFI support to use the edk2-ovmf package
Fedora now has a edk2 package so use the OVMF code from there. This also
adds using a copy of OVMF_VARS for each boot instead of reusing the one
provided by the package.
2016-05-17 16:44:22 -07:00
Brian C. Lane
2f66212c0f livemedia-creator: Update make-pxe-live to support missing initramfs
In some cases the initramfs may not be present in /boot to save space.
Use it if present, otherwise use the kernel version to recreate the name
of it.

This also fixes problems with dracut running out of space when not using
--live-rootfs-keep-size
2016-05-05 08:22:59 -07:00