Instead use the definition from python-six. Once we drop Py 2 support
completely, we'll just swap underscores with dots.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of the old default value and custom handling introduce new
option `live_images_target` (for consistency with other phases) and use
the usual inheritance rules.
Fixes: https://pagure.io/pungi/issue/749
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It's possible a variant is excluded via tree_variants option and the
section does not match anything. It can be confusing to users why
nothing is happening. This patch lets Pungi log all unmatched patterns.
Fixes: https://pagure.io/pungi/issue/692
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Config option 'repo' and 'repo_from' are used in several phases, merge
them with one option 'repo'. 'append' in schema is used for appending
the values from deprecated options to 'repo', so it won't break on any
existing config files that have the old options of 'repo_from' and
'source_repo_from' (which is an alias of 'repo_from').
And 'repo' schema is updated to support repo dict as the value or an
item in the values, a repo dict is just a dict contains repo options,
'baseurl' is required in the dict, like:
{"baseurl": "http://example.com/url/to/repo"}
or:
{"baseurl": "Serer"}
currently this is used in ostree phase to support extra repo options
like:
{"baseurl": "Server", "exclude": "systemd-container"}
Signed-off-by: Qixiang Wan <qwan@redhat.com>
live_images: additional_repos -> repo
ostree: source_repo_from -> repo_from
extra_source_repos -> repo
ostree_installer: source_repo_from -> repo_from
With the change, the phases have consolidate option names for variant
repos and external repos.
Old option names will continue to work, old names will be converted
to new names after validation automatically if new options are not
specified in config.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Also remove the TODO comment from live images phase: the appliances are
already submitted one task per single arch, so this change is not
necessary.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Phases createiso, liveimages, image_build, ostree_installer and osbs are
done in parallel, logs from these phases are mixed and and it's not
obvious which log message belongs to which phase. This change adds phase
name in log message for these phases.
The new mixin 'PhaseLoggerMixin' is added to extend a Pungi phase with a
logging logger which copy handlers from compose's logger but with
formatter changed.
Fixes: #58
Signed-off-by: Qixiang Wan <qwan@redhat.com>
The code to search for install tree and repo for image-build and
live-media was only looking at top-level variants. Addons, optional or
integrated layered products could not have been found. This would lead
to error messages such as "There is no variant Server-optional to get
repo from when building live image for Client" even though the variant
exists.
Various tests are updated to exercise this edge case.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It really is just a group of independent functions, so we can simplify
it by removing the unused wrapper class. Instead of importing the
wrapper, instantiating it and calling its methods we can import the
module and call its functions directly.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The address is no longer correct. We can just as well simply point to
the web page describing the license.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the configuration does not specify version for images or live media,
Pungi will create a default value based on `release_version`. If label
is used for the compose, the milestone from it will be appended to the
version (unless it's RC).
This change is backwards compatible: nothing changes when version is set
in configuration. If the version was missing before, building the
artifacts would fail. With this patch, default values will be supplied.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The schema is written in Python to reduce duplication. When
configuration is loaded, the validation checks if it's correct and fills
in default values.
There is a custom extension to the schema to report deprecated options.
The config dependencies are implemented as a separate pass. While it's
technically possible to express the dependencies in the schema itself,
the error messages are not very helpful and it makes the schema much
harder to read.
Phases no longer define `config_options`. New options should be added to
the schema. Since the default values are populated automatically during
validation, there is no need to duplicate them into the code.
The `pungi-config-validate` script is updated to use the schema and
report errors even for deeply nested fields.
The dependencies are updated: pungi now depends on `python-jsonschema`
(which is already available in Fedora).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is a breaking change as big part of current failable_deliverables
options will be ignored.
There is no change for buildinstall and creatiso phase.
Failability for artifacts in other phases is now configured per
artifact. It already works correctly for ostree and ostree_installer
phases (even per-arch). For OSBS phase there is currently only a binary
switch as it does not handle multiple arches yet. When it gains that
support, the option should contain list of non-blocking architectures.
For live images, live media and image build phases each config block can
configure list of failable arches. If the list is not empty, it can
fail. Once we have a way to fail only some arches, the config will not
need to change.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The performed checks:
* If format is ISO, the file must have correct magic string
* If it's bootable, there must be MBR or GPT
When a check fails on any failable deliverable, it will be logged and
the file removed from metadata (it will still remain on the disk). This
required a change to write the images.json file later (after test
phase).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The live_images, live_media and image_build phases have same options
that need to be specified on each image. This leads to a lot of
duplication in the config file. This patch adds global settings and
phase-level settings that allow to significantly reduce duplication.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The pungi.global.log should show subvariant for each failed deliverable
(if available). When the compose finishes, there is a new log file in
logs/global/deliverables.json containing details about all deliverables
as triples of (variant, arch, subvariant).
* `required` lists all deliverables that can not fail
* `attempted` lists all failable deliverables that were started
* `failed` is a subset of `attempted` and only contains deliverables
that failed
If the compose fails, the lists may be incomplete.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This uses a new common function that will ensure consistent logging
without duplicating the functionality all over the place.
A couple tests are updated to verify that correct data is logged.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Move getting of the precomputed value to a single place in Compose
class. The value now has format `date[.type_suffix].respin`.
Resolves: rhbz#1319924
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This adds the new 'subvariant' property to all images produced by Pungi.
For most image types this is the same as the variant, for now. For
live_images and livemedia_phase, the image config may specify
'subvariant' and the value will be used if present, otherwise the
variant name will be used.
We also now synthesize the Koji image 'name' for live and appliance
images if the image config doesn't specify it, and use a name that
complies better with the naming policy for ARM disk images (partially
resolving https://pagure.io/pungi-fedora/issue/7 ).
This is completely untested as yet, just bashing stuff out on a Sunday,
will work more on it tomorrow.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The log file started with createiso-, which is utterly confusing. With
this patch, the logs are named with liveimage- prefix.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Extracting extensions from the file name is not reliable as there is no
way to determine where extensions start. There can very well be a
version.release separated by dot. To bypass this, just use hardcoded
list of possible formats.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
* Add option to keep filenames generated by Koji.
* Put results of spin-appliance into image dir
* On failure images are no longer deleted.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
With this patch, you can specify a command for
signing of koji builds. For example:
signing_key_password_file = '~/file_with_password_for_key_fedora-24'
signing_key_id = '81b46521'
signing_command = '~/git/releng/scripts/sigulsign_unsigned.py -vv --password=%(signing_key_password)s fedora-24'
'signing_key_password_file' is a path to a file which contains
a password that will be formatted into 'signing_command' string
via '%(signing_key_password)s' string format syntax (if used).
Because pungi config is usualy stored in git and part of compose
logs we don't want password to be included directly in the config.
Note: If '-' is used instead of a filename, then you will be asked
for the password interactivelly right after pungi starts.
'signing_key_id' is ID of the key that will be used for the signing.
This ID will be used when crafting koji paths to signed files
(kojipkgs.fedoraproject.org/packages/NAME/VER/REL/data/signed/KEYID/..).
'signing_command' a command that will be run with a build as a single
argument. This command mustn't require any user interaction.
If you need to pass a password for a signing key to the command,
do this via command line option of the command with use of string
formatting syntax '%(signing_key_password)s' (see details
about 'signing_key_password_file').
Signed-off-by: Tomáš Mlčoch <tmlcoch@redhat.com>
The methods mentioning image build are generic and can work for other
task types.
get_image_build_paths -> get_image_paths
run_create_image_cmd -> run_blocking_cmd
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch removes some duplicated variables that get passed on to the
build thread. It also moves creation of the command for generating image
manifest closer to where it is used. Finally it adds tests for the
thread.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch adds option for adding repositories from other variants to
the koji task for building images.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is a new configuration option that allows listing what can fail
without aborting the whole compose. So far, only buildinstall, createiso
and liveimages phases react to this option.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Haibo Lin
Lubomír Sedlář
Qixiang Wan
Dennis Gilmore
Adam Williamson
Tomas Mlcoch