This also cleans up the runroot method detection code to not rely on the
now removed option.
JIRA: COMPOSE-2634
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The directory with logs should have updated owner and permission as
well as the actual output. This patch lays foundation for that by
allowing multiple paths to be specified.
JIRA: COMPOSE-3545
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This adds new `Runroot` class and new `runroot_method` option which makes
it possible to choose between two currently available runroot methods:
- Local
- Koji
The main goal of this commit is to make it possible to add new runroot
methods in the future and this is the first step in that direction.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
This should tell lorax what arch to use and avoids fragile detection
based on contents of source repo.
It uses the same logic buildinstall phase uses to get the buildarch.
Related: https://pagure.io/teamsilverblue/issue/67
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It does not abort the compose, but it's still an error. It should be
marked as such in the log. This will allow easier searching for the
failures.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the compose is not using comps, we can't pass the comps to lorax,
since it doesn't exist and it would cause a crash.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The commands in runroot run as root every time. If they create files
that are not readable to other users, the reset of compose could have
problems with it if it does not run as root too. Particularly updates
composes in Bodhi run under apache user.
Relates: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should give us better error reporting. The `copy_all` function
should preserve permissions on all files.
Relates: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The files created in koji runroot will be owned by root. If the compose
is done under different user, there could be a problem with copying the
files preserving the owner. Let's just copy them without that.
Fixes: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The comps could potentially be different in different variants, so
instead we can create the comps repo for every variant separately and
use two repos instead of one (packages in one repository, comps in
another one).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
OstreeInstaller phase will be moved to a different timeslot
and therefore needs different repo not to depend on Gather
phase which runs at the same time.
Related: https://pagure.io/pungi/issue/778
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
The Koji task can be restarted and lorax will fail if the output
directory already exists. Let's start the work in runroot by removing
the output directory.
Relates: #641
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The config now uses similar logic what previous commit did for OSTree.
Also we should report error when an unknown generator is used.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Config option 'repo' and 'repo_from' are used in several phases, merge
them with one option 'repo'. 'append' in schema is used for appending
the values from deprecated options to 'repo', so it won't break on any
existing config files that have the old options of 'repo_from' and
'source_repo_from' (which is an alias of 'repo_from').
And 'repo' schema is updated to support repo dict as the value or an
item in the values, a repo dict is just a dict contains repo options,
'baseurl' is required in the dict, like:
{"baseurl": "http://example.com/url/to/repo"}
or:
{"baseurl": "Serer"}
currently this is used in ostree phase to support extra repo options
like:
{"baseurl": "Server", "exclude": "systemd-container"}
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Currently if there are multiple ostree installers for the same
architecture, the logfiles all end up in the same location and overwrite
each other.
This patch moves the logs to logs/<arch>/<variant>/ostree_installer-X/
for a unique value of X so that there can be multiple runs even for the
same tree.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
live_images: additional_repos -> repo
ostree: source_repo_from -> repo_from
extra_source_repos -> repo
ostree_installer: source_repo_from -> repo_from
With the change, the phases have consolidate option names for variant
repos and external repos.
Old option names will continue to work, old names will be converted
to new names after validation automatically if new options are not
specified in config.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
For different cases where runroot is used it's now possible to set
custom weight. The usecase for this is to avoid one builder taking too
many tasks. Especially buildinstall is quite resource intensive, so one
builder taking multiple tasks at the same time leads to very slow
compose time.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Add new key 'repo' to allow specifying multiple repos as the source
repositories. And change 'source_repo_from' to allow specifying multiple
vairant names to use variant repos as source repositories.
Doc of 'source_repo_from' is updated to not mention URL is supported,
though we still support that in code. User should add url of repos in
'repo' key instead of 'source_repo_from'.
Fixes: #508
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Phases createiso, liveimages, image_build, ostree_installer and osbs are
done in parallel, logs from these phases are mixed and and it's not
obvious which log message belongs to which phase. This change adds phase
name in log message for these phases.
The new mixin 'PhaseLoggerMixin' is added to extend a Pungi phase with a
logging logger which copy handlers from compose's logger but with
formatter changed.
Fixes: #58
Signed-off-by: Qixiang Wan <qwan@redhat.com>
It really is just a group of independent functions, so we can simplify
it by removing the unused wrapper class. Instead of importing the
wrapper, instantiating it and calling its methods we can import the
module and call its functions directly.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The assertions were duplicated across multiple test cases. This patch
moves them into shared methods so that they are defined only once.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of overloading the `dvd` value, use new value `ostree` that can
be overridden by a config change.
Fixes: https://pagure.io/pungi/issue/418
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The schema is written in Python to reduce duplication. When
configuration is loaded, the validation checks if it's correct and fills
in default values.
There is a custom extension to the schema to report deprecated options.
The config dependencies are implemented as a separate pass. While it's
technically possible to express the dependencies in the schema itself,
the error messages are not very helpful and it makes the schema much
harder to read.
Phases no longer define `config_options`. New options should be added to
the schema. Since the default values are populated automatically during
validation, there is no need to duplicate them into the code.
The `pungi-config-validate` script is updated to use the schema and
report errors even for deeply nested fields.
The dependencies are updated: pungi now depends on `python-jsonschema`
(which is already available in Fedora).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is a breaking change as big part of current failable_deliverables
options will be ignored.
There is no change for buildinstall and creatiso phase.
Failability for artifacts in other phases is now configured per
artifact. It already works correctly for ostree and ostree_installer
phases (even per-arch). For OSBS phase there is currently only a binary
switch as it does not handle multiple arches yet. When it gains that
support, the option should contain list of non-blocking architectures.
For live images, live media and image build phases each config block can
configure list of failable arches. If the list is not empty, it can
fail. Once we have a way to fail only some arches, the config will not
need to change.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of putting everything to <variant>/<arch>/iso, move images/,
EFI/ and isolinux/ subdirs to <variant>/<arch>/os. The nicely named ISO
image is still in the original location.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
On Python 2.6, it requires the format placeholder to have explicit index
of argument, so using % formatting is easier.
There are a couple places where the method is still used because the
same argument is used twice.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We can't assume the templates will just be available. This patch adds a
configuration option to point to the git repo with them. It will be
cloned at compose box and relative paths to templates will be made
absolute respective to this clone.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
