Commit Graph

620 Commits

Author SHA1 Message Date
Chris Lumens
e42b797295 Return multiple timestamps in API results (#409).
If one of the timestamps isn't present (for instance, the finished
timestamp for a job that is still running), null is returned.

(cherry picked from commit 17c40ef271)
2018-08-06 16:57:13 -07:00
Chris Lumens
ab76c6170e Add a new timestamp.py file to the API directory (#409).
This is responsible for writing out a new times.toml file, containing
important timestamps in the life of a compose.  This seems a little more
reliable than attempting to infer things from the filesystem, especially
in light of the fact that we can't ever really know when a file was
created.

(cherry picked from commit b59d59b124)
2018-08-06 16:57:07 -07:00
Brian C. Lane
b1dd22afa6 Log and exit on metadata update errors at startup
A bad system repo can cause lorax-composer to fail to start. Instead of
a traceback log the error and exit.

(note that the exit still results in an OSError traceback due to part of
it running as root, this needs to be addressed in another commit).
2018-07-18 16:09:03 -07:00
Brian C. Lane
816f1658db Clarify error message from /source/new
(cherry picked from commit b20aa4609c)
2018-07-18 16:01:06 -07:00
Chris Lumens
fbda20b8b7 Support loading groups from the kickstart template files.
(cherry picked from commit c8d2045f89)
2018-07-18 15:38:40 -07:00
Chris Lumens
8c4804eb15 Include groups in depsolving.
This adds a new argument to projects_depsolve and
projects_depsolve_with_size that contains the group list, unfortunately.
I would have prefered adding a function that just returns a list of all
the contents of a group and then add that to what was being passed into
projects_depsolve.  However, there does not appear to be any good way to
do that in yum aside from a lot of grubbing around in the comps object,
which I am unwilling to do.

(cherry picked from commit 5fe4b47072)
2018-07-18 14:36:28 -07:00
Chris Lumens
76d376fe18 Add support for groups to blueprints.
Nothing is currently being done with this information, but it will be
soon.

(cherry picked from commit 0f69d2084c)
2018-07-18 13:48:25 -07:00
Brian C. Lane
460a277d4e Check the compose templates at startup
Depsolve the packages included in the templates and report any errors
using the /api/status 'msgs' field. This should help narrow down
problems with package sources not being setup correctly.
2018-07-13 10:18:07 -07:00
Adam Williamson
4a3f9e12aa DNF 3: progress callback constants moved to dnf.transaction
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-27 16:12:25 -07:00
Brian C. Lane
56926baaec Add the ability to enable DNF plugins for lorax
Pass --dnfplugin='*' to enable all of them.
Pass --dnfplugin='plugin-name' to enable one fo them. You can use it
multiple times to enable multiple plugins. Globs work as well.
2018-06-22 09:07:01 -07:00
Chris Lumens
e8894b7266 It's /compose/cancel, not /blueprints/cancel. 2018-06-18 10:51:12 -04:00
Brian C. Lane
936d193413 Retry losetup if loop_attach fails
It appears that sometimes the loop device doesn't get setup properly,
this may be a race with other users of loop devices on the system, or
some other mechanism that isn't understood.

To try and prevent total failure when this happens this patch retries
the loop setup 3 times before giving up. Previously it would wait for
the loop device to appear (checking 5 times), that operation is now
executed 3 times with a new losetup attempt each time.

Resolves: rhbz#1589084
(cherry picked from commit c746e8b0c3)
2018-06-11 14:00:05 -07:00
Brian C. Lane
594e762d63 Increase default ram used with lmc and virt to 2048
1024 is a bit small in some situations, double it.

Resolves: rhbz#1538747
2018-06-08 11:44:09 -07:00
Brian C. Lane
d8ce013a2b Add --dracut-arg support to lorax
Use it to override the default dracut arguments (displayed as part of
the --help output). If you want to extend the default arguments they
all need to be passed in on the cmdline as well. eg.

--dracut-arg='--xz' --dracut-arg='--install /.buildstamp' ...

Resolves: rhbz#1452220
2018-06-06 17:10:35 -07:00
Martin Kolman
d5d3dd3be3 Make LogRequestHandler configurable
Make it possible to manipulate the simple and regexp
tests the LogRequestHandler class uses to check error
messages for potential error states.

This is accomplished by moving the simple and regexp test
strings to class members, where they can be easily
manipulated by users of the pylorax module.

It's also now possible to set the log request handler class
for a LogMonitor.

This functionality can then be used for example like this:

customized_log_request_handler = monitor.LogRequestHandler
customized_log_request_handler.simple_tests.remove("Call Trace:")
log_monitor = monitor.LogMonitor(install_log,
                                 timeout=opts.timeout,
                                 log_request_handler_class = customized_log_request_handler)

This way installation will continue even if there was a call
trace in the logs. In a similar way additional tests and regexps can be
also added.
2018-06-06 14:30:28 -07:00
Brian C. Lane
afa89ea657 Fix DNF related issues with source selection
DNF Repo.dump() function cannot be used as a .repo file for dnf due to
it writing baseurl and gpgkey as a list instead of a string. Add a new
function to write this in the correct format, and limited to the fields
we use.

Add a test for the new function.

Fix /projects/source/info to return an error 400 if a nonexistant TOML
source is requested. If JSON is used the error is part of the standard
response.

Update test_server.py to check for the correct error code.
2018-06-04 15:45:36 -07:00
Brian C. Lane
dd8e4d9e99 Fix handling bad source repos and add a test
When adding a source failed it wasn't being removed from the dnf object.
This fixes that, and returns an error when setting up the source fails.
Also adds a test for it.
This also includes detecting rawhide vs. non-rawhide releases and
adjusting the tests accordingly (some of the source names change).
2018-06-04 15:45:36 -07:00
Brian C. Lane
23f4b2a3ec Fix make_dnf_dirs
It was chopping off an extra directory level due to realpath removing
the trailing / from the paths when they are setup.
2018-06-04 15:45:36 -07:00
Brian C. Lane
6d677b2207 Add support for user defined package sources API
This lives under /api/v0/projects/source/*

See the documentation for details
2018-06-04 15:45:36 -07:00
Colin Walters
bb3d8edd06 templates: Stop using gconfset
We had only been indirectly pulling in GConf, and anyways
nothing was listening to these keys.

<kalev> I still think it's a fallout from 27a90d973f

Really in general, if we wanted to make changes like this
it'd probably be a lot simpler to do them on boot or so.

https://bugzilla.redhat.com/show_bug.cgi?id=1581838
2018-05-23 14:20:44 -07:00
Brian C. Lane
095829171a Add support for version globs to blueprints
This uses dnf's version__glob filter to implement it. It amounts to '*'
wildcards and '?' for single character matching.
2018-05-18 12:03:26 -07:00
Brian C. Lane
db67c1267a Move lorax-composer and composer-cli argument parsing into modules
This allows sphinx-argparse to document them automatically.
2018-05-14 13:00:15 -07:00
Brian C. Lane
6e5ca1f062 Add new cmdline args to compose_args settings 2018-05-14 13:00:15 -07:00
Brian C. Lane
b1319c7bc1 Remove temporary files after run_compose
A crash can also leave temporary lmc-* files, remove them as well.
2018-05-14 13:00:15 -07:00
Brian C. Lane
0455a5d740 Add --proxy to lorax-composer cmdline
Overrides the [dnf] proxy setting in the config file.
2018-05-14 13:00:15 -07:00
Brian C. Lane
12cb2736ac Pass the --tmp value into run_creator and cleanup after a crash
Crashing can sometimes leave directories in /var/tmp/lmc-* so clean
those up after run_creator is finished.
2018-05-14 13:00:15 -07:00
Brian C. Lane
9a1dce5619 Set lorax_templates to the correct directory
Templates are under the share dir, so use find_templates() to find them.
2018-05-14 13:00:15 -07:00
Brian C. Lane
3974a4e479 Adjust the disk size estimates to match Anaconda
First is Anaconda uses 6k blocks per file for its estimate, and it
fudges by 10% so adjust for those with an extra 10% of headroom just in
case.

Second is an Anaconda bug that won't allow it to do a kickstart install
to a disk smaller than 3000 MB. There is a PR to fix it upstream, but
for now the minimum size has to be 3000e9
2018-05-14 13:00:15 -07:00
Brian C. Lane
a363aee971 Skip creating groups with the same name as a user
Otherwise the user creation fails when anaconda sees there is already a
group with that name. Log a warning and continue on.
2018-05-14 13:00:15 -07:00
Brian C. Lane
07ea61be77 Add user and group creation to blueprint
[[customize.user]] and [[customize.group]]
2018-05-14 13:00:15 -07:00
Brian C. Lane
25bec0b50d Add blueprint customization support for hostname and ssh key
This adds support for the optional blueprint section [customizations].

Use it like this:

[customizations]
hostname = yourhostnamehere

[[customiations.sshkey]]
user = root
key = root user key
2018-05-14 13:00:15 -07:00
Brian C. Lane
a9b626a706 Fix the compose arguments for the Fedora version of Anaconda
This adds make_oci, and moves qcow2 to the more generic image_type
(which just uses qemu-img disk names).
2018-05-14 13:00:15 -07:00
Brian C. Lane
c9187ce4ca Convert Yum usage to DNF
The DNF api is similar, but not the same, as Yum. Make the needed
changes, and rename yum references to dnf to avoid confustion later.
2018-05-14 13:00:15 -07:00
Brian C. Lane
48671b682a workspace read and write needs UTF-8 conversion
reading needs to decode the bytes to str and writing needs to encode the
str to bytes.
2018-05-14 13:00:15 -07:00
Brian C. Lane
d478864fb3 Return an empty list if depsolve results are empty 2018-05-14 13:00:15 -07:00
Brian C. Lane
8ed039107b The git blob needs to be bytes
In py3 str is unicode so it needs to be encoded as UTF-8 when writing to
the blob.
2018-05-14 13:00:15 -07:00
Brian C. Lane
719aceac82 Use default commit sort order instead of TIME
Different versions of libgit2 act differently. Using TIME results in
some commits (like a revert) being listed correctly, but the rest being
listed in reverse order. Leaving it at the default works for
libgit2-0.26.3
2018-05-14 13:00:15 -07:00
Brian C. Lane
9b9b627fe5 Update pylorax.api.projects for DNF usage
And adjust tests to match.
2018-05-14 13:00:14 -07:00
Brian C. Lane
e86f4f9a36 Update dnfbase (formerly yumbase) for DNF support
And adjust tests to match.
2018-05-14 13:00:14 -07:00
Brian C. Lane
06c227598c Move core of livemedia-creator into pylorax.creator
This moves everything except the cmdline checking into run_creator in
pylorax.creator

It also rearranges some functions to prevent import loops, and adds a
utility function to imgutils (mkfsimage_from_disk for copying a
partition into a filesystem image).
2018-05-14 13:00:14 -07:00
Brian C. Lane
63b473d4ba Convert lorax-composer yum base object to DNF
This no longer uses the enabled configuration setting to select repos to
use. It uses everything in the repo_dir, and if system repos have not
been disabled it copies them into the repo_dir at startup, overwriting
the previous copy.
2018-05-14 13:00:14 -07:00
Brian C. Lane
d31bea7bd3 Use 2to3 to convert the python2 lorax-composer code to python3 2018-05-14 13:00:14 -07:00
Brian C. Lane
89050f068d livemedia-creator: Move core functions into pylorax modules
This reduces the amount of code in livemedia-creator to the cmdline
parsing and calling of the installer functions. Moving them into other
modules will allow them to be used by other projects, like the
lorax-composer API server.
2018-05-14 13:00:14 -07:00
Jiri Konecny
2f3de6060b Update default releasever to Fedora 29 (rawhide)
There was Fedora 25 which is now unsupported.
2018-03-15 12:07:34 -07:00
Brian C. Lane
e1c839de1c Fix _install_branding
filter(provides=...) doesn't work with paths. The release packages
provide system-release so just look for that instead of a file.

Now it finds the release package and selects it along with the
corresponding logos package.

Note, this has been broken since commit 431ca6ce
2018-02-23 12:50:07 -08:00
Yuval Turgeman
68d1f4208d LMC: Make the QEMU RNG device optional
Added a new QEMU argument '--with-rng' to specify a RNG device other
than /dev/random or "none".

Signed-off-by: Yuval Turgeman <yturgema@redhat.com>
2018-01-23 08:22:38 -08:00
Brian C. Lane
dc635a97f6 Write the --variant string to .buildstamp as 'Variant=' 2018-01-17 09:38:16 -08:00
Brian C. Lane
629d04dc68 Fix installpkg exclude operation
Commit 8edaefd4d1 added the ability to install specific NVR's of
packages, but it did not adjust the exclude operation to account for
this.

This patch fixes that, applying the exclude only to the name part of the
package NVR, and changes some variable names to pkgnvr/pkgnvrs to make
it more clear that the content has changed to <name>-<version>-<release>
2018-01-15 08:38:43 -08:00
Brian C. Lane
73628a6e07 Log a more descriptive error when setfiles fails (#1499771)
Some lorax users run it from inside mock, which isn't able to detect
whether the host is in Permissive mode. This can lead to confusing
error messages, so this points them in the right direction.
2017-11-27 12:03:45 -08:00
Claudio Zumbo
8edaefd4d1 Allow installpkgs to do version pinning through globbing 2017-11-10 09:56:27 -08:00