Commit Graph

40 Commits

Author SHA1 Message Date
Lubomír Sedlář
ed8fffb6d1 koji-wrapper: Log failed subtasks
If the parent task is successful, there can still be failed child tasks
for failable arches. We need to log those and potentially mark the
compose as incomplete.

Fixes: https://pagure.io/pungi/issue/874
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-02 08:27:44 +02:00
Ondrej Nosek
91ee1fb854 Remove deprecated options
Options that are currently marked as deprecated do not have any effect
anymore (other than printing warning). We should remove them and update
the message so that we can mark options as deprecated even when they
still work.

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2017-11-16 11:12:34 +01:00
Lubomír Sedlář
6d6cf6e233 tests: Fix remaining missing assertions
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 09:29:28 +01:00
Lubomír Sedlář
ed22e07ef9 Port to Python 3
This should make all tests pass on both Python 2 and Python 3.

Unittest2 is required on Py 2.6 and Py 3.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-24 15:15:31 +02:00
Qixiang Wan
2f5d6d7dcd unify repo and repo_from options
Config option 'repo' and 'repo_from' are used in several phases, merge
them with one option 'repo'. 'append' in schema is used for appending
the values from deprecated options to 'repo', so it won't break on any
existing config files that have the old options of 'repo_from' and
'source_repo_from' (which is an alias of 'repo_from').

And 'repo' schema is updated to support repo dict as the value or an
item in the values, a repo dict is just a dict contains repo options,
'baseurl' is required in the dict, like:

{"baseurl": "http://example.com/url/to/repo"}

or:

{"baseurl": "Serer"}

currently this is used in ostree phase to support extra repo options
like:

{"baseurl": "Server", "exclude": "systemd-container"}

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-29 10:12:32 +08:00
Lubomír Sedlář
9533fca96c koji_wrapper: Always use --profile option with koji
This patch adds this option to all invoked koji commands.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-30 13:00:25 +01:00
Dennis Gilmore
42cdf047cc add missing koji_profile from test compose setting
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2017-01-30 05:06:46 -06:00
Lubomír Sedlář
7f56b978ce live-media: Allow some arches to fail
This patch uses the `--can-fail` option of koji command line. If only
optional arches fail, the task will report as success. Failures on
compose box side are ignored if and only if all architectures are
optional.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-14 10:14:30 +01:00
Lubomír Sedlář
c338219ef0 config: Report unknown options as warnings
Given the way config files can include other files, it is entirely
possible to use the same config file for both Pungi and Distill. Pungi
will however complain about unknown options.

This patch reverts part of c38bb480 and moves deprecation handling back
into schema validation. The validation method then returns a list of
errors and a list of warnings.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-13 11:58:08 +01:00
Qixiang Wan
80fa723b1d Include phase name in log for some phases
Phases createiso, liveimages, image_build, ostree_installer and osbs are
done in parallel, logs from these phases are mixed and and it's not
obvious which log message belongs to which phase. This change adds phase
name in log message for these phases.

The new mixin 'PhaseLoggerMixin' is added to extend a Pungi phase with a
logging logger which copy handlers from compose's logger but with
formatter changed.

Fixes: #58
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-11-23 16:47:22 +08:00
Lubomír Sedlář
13871b64fb compose: Drop unused argument
The `get_variants()` method had a `recursive` argument with default
value of `False. However, this argument had no effect and the method
always returned all variants recursively.

We can just drop the argument. All callers are updated to not supply the
argument. Should any need for getting the top-level variants only arise,
they can be accessed as the `variants` attribute directly on the Compose
object.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 13:39:01 +01:00
Lubomír Sedlář
bd00920c62 compose: Search all nested variants
The code to search for install tree and repo for image-build and
live-media was only looking at top-level variants. Addons, optional or
integrated layered products could not have been found. This would lead
to error messages such as "There is no variant Server-optional to get
repo from when building live image for Client" even though the variant
exists.

Various tests are updated to exercise this edge case.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 13:18:34 +01:00
Patrick Uiterwijk
59dbd9d35f Remove executable permissions on test scripts
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2016-10-25 10:42:16 +00:00
Dennis Gilmore
6aeab9ee9d Merge #381 Automatically generate missing image version 2016-09-06 15:19:55 +00:00
Lubomír Sedlář
223a015898 Automatically generate missing image version
If the configuration does not specify version for images or live media,
Pungi will create a default value based on `release_version`. If label
is used for the compose, the milestone from it will be appended to the
version (unless it's RC).

This change is backwards compatible: nothing changes when version is set
in configuration. If the version was missing before, building the
artifacts would fail. With this patch, default values will be supplied.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-01 15:13:04 +02:00
Lubomír Sedlář
f9a6c8418f Add JSON Schema for configuration
The schema is written in Python to reduce duplication. When
configuration is loaded, the validation checks if it's correct and fills
in default values.

There is a custom extension to the schema to report deprecated options.

The config dependencies are implemented as a separate pass. While it's
technically possible to express the dependencies in the schema itself,
the error messages are not very helpful and it makes the schema much
harder to read.

Phases no longer define `config_options`. New options should be added to
the schema. Since the default values are populated automatically during
validation, there is no need to duplicate them into the code.

The `pungi-config-validate` script is updated to use the schema and
report errors even for deeply nested fields.

The dependencies are updated: pungi now depends on `python-jsonschema`
(which is already available in Fedora).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-01 10:56:15 +02:00
Lubomír Sedlář
463088d580 Refactor failables
This is a breaking change as big part of current failable_deliverables
options will be ignored.

There is no change for buildinstall and creatiso phase.

Failability for artifacts in other phases is now configured per
artifact. It already works correctly for ostree and ostree_installer
phases (even per-arch). For OSBS phase there is currently only a binary
switch as it does not handle multiple arches yet. When it gains that
support, the option should contain list of non-blocking architectures.

For live images, live media and image build phases each config block can
configure list of failable arches. If the list is not empty, it can
fail. Once we have a way to fail only some arches, the config will not
need to change.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-07-27 13:06:01 +02:00
Lubomír Sedlář
084994538b Add common global settings for images
The live_images, live_media and image_build phases have same options
that need to be specified on each image. This leads to a lot of
duplication in the config file. This patch adds global settings and
phase-level settings that allow to significantly reduce duplication.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-18 18:11:29 +02:00
Lubomír Sedlář
18b6020ac5 Improve logging of failable deliverables
The pungi.global.log should show subvariant for each failed deliverable
(if available). When the compose finishes, there is a new log file in
logs/global/deliverables.json containing details about all deliverables
as triples of (variant, arch, subvariant).

 * `required` lists all deliverables that can not fail
 * `attempted` lists all failable deliverables that were started
 * `failed` is a subset of `attempted` and only contains deliverables
   that failed

If the compose fails, the lists may be incomplete.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-13 13:45:30 +02:00
Lubomír Sedlář
98f7ef739e Refactor checking for failable deliverables
This uses a new common function that will ensure consistent logging
without duplicating the functionality all over the place.

A couple tests are updated to verify that correct data is logged.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-24 10:12:20 +01:00
Lubomír Sedlář
652987f2cc Reuse helper in all tests
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-23 12:00:06 +01:00
Dennis Gilmore
6230b0ff3e Merge #232 Improve logging by adding subvariants 2016-03-22 14:15:26 +00:00
Lubomír Sedlář
ecbf08c6f8 Add compose type to release for images
Move getting of the precomputed value to a single place in Compose
class. The value now has format `date[.type_suffix].respin`.

Resolves: rhbz#1319924
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-22 08:57:40 +01:00
Lubomír Sedlář
f1974d3f03 [live-media] Use subvariants in logging
Print the subvariant together with variant UID and arches. This should
make the global log easier to understand.

The output of koji command should now be split into multiple files in
logs/{arch}/livemedia-{variant}-{subvariant}.{arch}.log instead of
having them all in one file.

The tests were updated to use more sensible value for the subvariant
field.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-17 09:04:50 +01:00
Lubomír Sedlář
c1b8fbe4f9 Update tests to match the subvariant
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-10 16:59:34 -06:00
Lubomír Sedlář
7e2cd28fb8 [live-media] Support release set to None globally
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-09 13:08:22 +01:00
Lubomír Sedlář
86bb816417 [live-media] Add global settings
These can be overriden for a particular image, but in general case they
can simplify the config.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-04 08:15:25 +01:00
Lubomír Sedlář
595845e104 [live-media] Rename test case
Originally it was copied from image build phase, and was not renamed.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-04 08:15:25 +01:00
Lubomír Sedlář
e9292fc942 [tests] Dummy compose is no longer private
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-22 19:11:04 +01:00
Lubomír Sedlář
90954ddf0e [tests] Use real paths module in testing
This avoids problems of mock paths having different API that the real
ones.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-22 19:11:04 +01:00
Lubomír Sedlář
8661d294ab [tests] Move dummy testing compose into separate module
This greatly reduces duplication, as a single copy is now shared among
all test cases.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-22 19:10:12 +01:00
Lubomír Sedlář
0fb8049fdd Update tests
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-19 08:51:03 +01:00
Lubomír Sedlář
37ac0d7bfe [live-media] Update tests to use $basearch
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-18 16:17:30 +01:00
Lubomír Sedlář
8ef2e7e308 [live-media] Properly calculate iso dir
The method expects to be called with a arch, not just variant.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 13:58:42 +01:00
Lubomír Sedlář
ac29bf05ee [live-media] Koji option is ksfile, not kickstart
In config, it is still called kickstart to be consistent with how
image-build calls it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 10:40:04 +01:00
Lubomír Sedlář
6ec151cde2 [live-media] Use install tree from another variant
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 10:24:12 +01:00
Lubomír Sedlář
0c4bd42f80 [live-media] Put images into iso dir
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 10:11:38 +01:00
Lubomír Sedlář
91faa0fafe [image-build] Koji expects arches as a comma separated string
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 07:50:02 +01:00
Lubomír Sedlář
b35264cfba [live-media] Version is required argument
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-10 18:29:29 +01:00
Lubomír Sedlář
439622d576 [live-media] Add live media phase
This phase builds live media in Koji using the Live Media Creator. It
runs in parallel with current live images, create ISO and image build
phases.

The documentation is updated to explain how to configure this.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-01 13:28:49 +01:00