Commit Graph

56 Commits

Author SHA1 Message Date
Ozan Unsal
ce45fdc39a
Fix unittest errors
Signed-off-by: Ozan Unsal <ounsal@redhat.com>

(cherry picked from commit aa0aae3d3e (centos_master))
2023-11-10 16:57:51 +02:00
Lubomír Sedlář
41048f60b7
Update to work with latest mock
The `called_once` attribute now raises an exception. Switch to
`assert_called_once` method. Also replace `assertTrue(x.called)` with
`x.assert_called()`.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 51628a974d)
2023-11-10 16:50:34 +02:00
soksanichenko
b61e59d676 - Use unittest.mock instead external mock 2022-11-11 15:32:00 +02:00
Haibo Lin
e42e65783d image_build: Allow reusing old image_build results
JIRA: RHELCMP-5970
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-10-20 14:07:02 +08:00
Haibo Lin
c0193c9fca Fix flake8 complaints - E501
E501 line too long (92 > 88 characters)
E501 line too long (103 > 88 characters)
...

JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 14:36:46 +08:00
Haibo Lin
3eddcfccd8 Fix flake8 complaints - F401
F401 'dnf' imported but unused
F401 'imp' imported but unused
F401 'os' imported but unused
F401 'subprocess' imported but unused
F401 'sys' imported but unused
F401 'yum' imported but unused

JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 11:48:31 +08:00
Haibo Lin
38142d30ba Format tests with black
JIRA: COMPOSE-4086
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-05 17:29:15 +08:00
Haibo Lin
0ed70fc8b6 tests: drop sys.path modification
It's unnecessary to add the path manually.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-12-10 10:53:38 +08:00
Lubomír Sedlář
2f54745715 Remove usage of unittest2 on Python 3
It was needed to provide assertItemsEqual method. Starting with Python
3.2, there's assertCountEqual that does the same thing. Six provides a
helper that will dispatch to the existing method. With this change,
unittest 2 is only needed on Python 2.6 to backport the method.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 15:25:18 +02:00
Lubomír Sedlář
43fb06e00f Remove shebangs from test files
They should be executed with a runner like nose, not directly.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 15:24:17 +02:00
Lubomír Sedlář
3efcfede6a image-build: Support repo/install_tree as path
If a repo or install tree is specified as an absolute path on the local
filesystem, we should either translate it using the configured mappings,
or if no mapping matches, it should be return unchanged. A variant name
can not start with a slash, so attempting that translation does not make
much sense.

JIRA: COMPOSE-3290
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-25 08:18:47 +01:00
Lubomír Sedlář
ca7d6256e5 Move resolving git reference to config validation
Instead of multiple places handling the same thing duplicating the
logic, it's better to do it once upfront. This allows easy caching of
the results.

Additional advantage of this approach is that the config dump will
include resolved URLs. The original reference will still be available in
the copy of the original config.

JIRA: COMPOSE-3065
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-21 13:25:35 +01:00
Lubomír Sedlář
e3de4dcccf Report failed failable deliverables as errors
It does not abort the compose, but it's still an error. It should be
marked as such in the log. This will allow easier searching for the
failures.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-17 09:31:50 +02:00
Lubomír Sedlář
cdfa3cb45f image-build: Drop suffixes from configuration
The file extension in configuration is only used to tell Pungi which
files from the task results should be downloaded. The user has to get it
right or the phase will fail. Each format has a single valid suffix.

Pungi should not require users to specify the suffix, since it can just
as well just know the right value.

The old configuration will continue working, only the extension will be
ignored.

Fixes: https://pagure.io/pungi/issue/753
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-11-01 12:47:36 +01:00
Qixiang Wan
b81e94e808 image-build: add arch name(s) in image config file name
Pungi write image config file with name of <format>-<name>.cfg, if there
are two or more image configs present for different arches under the same
variant and with same format & name, the config file can be overwritten,
and result in invalid image conf file.

Example:

image_build = {
    '^Server$': [
        {
            'image-build': {
                'format': [('qcow2', 'qcow2'),],
                'name': 'fedora-guest-image',
                'target': 'guest-fedora-26-image',
                'version': '26',
                'ksurl': "git://git.example.com/ks.git?fedora#HEAD",
                'kickstart': "fedora-26-kvm.ks",
                'ksversion': 'f26',
                'distro': 'fedora-26',
                'disk-size': '10',
                'arches': ['x86_64'],
                'repo': ["http://example.com/linux/fedora/26/Everything/x86_64/os", ]
            }
        },
        {
           'image-build': {
                'format': [('qcow2', 'qcow2'),],
                'name': 'fedora-guest-image',
                'target': 'guest-fedora-26-image',
                'version': '26',
                'ksurl': "git://git.example.com/ks.git?fedora#HEAD",
                'kickstart': "fedora-26-kvm.ks",
                'ksversion': 'f26',
                'distro': 'fedora-26',
                'disk-size': '10',
                'arches': ['ppc64le'],
            }
        },
    ],
}

In this case, config file "qcow2_guest-fedora-26-image.cfg" will be
created for both x86_64 and ppc64le under the same variant dir, and
there is a high chance it will be over-written while Pungi creating the
koji task. We can add arch name(s) in config filename to avoid that.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-09-03 23:51:08 +08:00
Lubomír Sedlář
f6d07c1651 Be explicit about generating release for images
The config now uses similar logic what previous commit did for OSTree.
Also we should report error when an unknown generator is used.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-13 10:17:59 +02:00
Lubomír Sedlář
8418b68fb0 image-build: Pass arches around as a list
Instead of joining the arches as a comma separated string and splitting
it again later. Ultimately we do need the original format to pass to
koji wrapper, but we can produce that value later.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-24 08:50:21 +01:00
Lubomír Sedlář
6c708549c8 image-build: Expand arches for can_fail
We need to work with a list of strings, not a comma-delimited single
string.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-24 08:50:21 +01:00
Lubomír Sedlář
356b78d440 image-build: Allow failure only on some arches
This uses the --can-fail option in koji. Failing an optional image will
not abort whole task. If the whole task fails (or there is a problem on
the compose side), we abort unless all arches are optional.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-14 10:14:30 +01:00
Lubomír Sedlář
721932a573 image-build: Use install_tree from parent for nested variants
Buildinstall only runs for top-level variants. Addons, optionals and
integrated layered products must reuse install tree from their parent,
because otherwise there would be no boot.iso.

Fixes: #472
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-14 10:04:01 +01:00
Lubomír Sedlář
c338219ef0 config: Report unknown options as warnings
Given the way config files can include other files, it is entirely
possible to use the same config file for both Pungi and Distill. Pungi
will however complain about unknown options.

This patch reverts part of c38bb480 and moves deprecation handling back
into schema validation. The validation method then returns a list of
errors and a list of warnings.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-13 11:58:08 +01:00
Lubomír Sedlář
2bd18b1b03 image-build: Reduce duplication
All tests should print complete diffs on failure, so there is no need to
define this in each test separately.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-28 09:28:46 +01:00
Qixiang Wan
80fa723b1d Include phase name in log for some phases
Phases createiso, liveimages, image_build, ostree_installer and osbs are
done in parallel, logs from these phases are mixed and and it's not
obvious which log message belongs to which phase. This change adds phase
name in log message for these phases.

The new mixin 'PhaseLoggerMixin' is added to extend a Pungi phase with a
logging logger which copy handlers from compose's logger but with
formatter changed.

Fixes: #58
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-11-23 16:47:22 +08:00
Lubomír Sedlář
13871b64fb compose: Drop unused argument
The `get_variants()` method had a `recursive` argument with default
value of `False. However, this argument had no effect and the method
always returned all variants recursively.

We can just drop the argument. All callers are updated to not supply the
argument. Should any need for getting the top-level variants only arise,
they can be accessed as the `variants` attribute directly on the Compose
object.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 13:39:01 +01:00
Lubomír Sedlář
bd00920c62 compose: Search all nested variants
The code to search for install tree and repo for image-build and
live-media was only looking at top-level variants. Addons, optional or
integrated layered products could not have been found. This would lead
to error messages such as "There is no variant Server-optional to get
repo from when building live image for Client" even though the variant
exists.

Various tests are updated to exercise this edge case.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 13:18:34 +01:00
Patrick Uiterwijk
59dbd9d35f Remove executable permissions on test scripts
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2016-10-25 10:42:16 +00:00
Dennis Gilmore
6aeab9ee9d Merge #381 Automatically generate missing image version 2016-09-06 15:19:55 +00:00
Lubomír Sedlář
223a015898 Automatically generate missing image version
If the configuration does not specify version for images or live media,
Pungi will create a default value based on `release_version`. If label
is used for the compose, the milestone from it will be appended to the
version (unless it's RC).

This change is backwards compatible: nothing changes when version is set
in configuration. If the version was missing before, building the
artifacts would fail. With this patch, default values will be supplied.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-01 15:13:04 +02:00
Lubomír Sedlář
f9a6c8418f Add JSON Schema for configuration
The schema is written in Python to reduce duplication. When
configuration is loaded, the validation checks if it's correct and fills
in default values.

There is a custom extension to the schema to report deprecated options.

The config dependencies are implemented as a separate pass. While it's
technically possible to express the dependencies in the schema itself,
the error messages are not very helpful and it makes the schema much
harder to read.

Phases no longer define `config_options`. New options should be added to
the schema. Since the default values are populated automatically during
validation, there is no need to duplicate them into the code.

The `pungi-config-validate` script is updated to use the schema and
report errors even for deeply nested fields.

The dependencies are updated: pungi now depends on `python-jsonschema`
(which is already available in Fedora).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-01 10:56:15 +02:00
Lubomír Sedlář
463088d580 Refactor failables
This is a breaking change as big part of current failable_deliverables
options will be ignored.

There is no change for buildinstall and creatiso phase.

Failability for artifacts in other phases is now configured per
artifact. It already works correctly for ostree and ostree_installer
phases (even per-arch). For OSBS phase there is currently only a binary
switch as it does not handle multiple arches yet. When it gains that
support, the option should contain list of non-blocking architectures.

For live images, live media and image build phases each config block can
configure list of failable arches. If the list is not empty, it can
fail. Once we have a way to fail only some arches, the config will not
need to change.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-07-27 13:06:01 +02:00
Lubomír Sedlář
d5512fd6ff Use unittest2 if available
The module backports features to Python 2.6 and 2.7. If it is available,
the tests will use it. If it is not available, it will fall back to
regular unittest. On Python 2.7, the tests pass anyway. On Python 2.6,
there are failures with Python 2.6.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-25 13:41:49 +02:00
Lubomír Sedlář
b634a18a7f Stop using str.format
On Python 2.6, it requires the format placeholder to have explicit index
of argument, so using % formatting is easier.

There are a couple places where the method is still used because the
same argument is used twice.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-25 13:41:49 +02:00
Lubomír Sedlář
bb4afea4f1 [image-build] Allow using external install trees
This is needed for the twoweek Fedora atomic release to point to regular
nightly composes.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-23 18:30:14 +02:00
Lubomír Sedlář
084994538b Add common global settings for images
The live_images, live_media and image_build phases have same options
that need to be specified on each image. This leads to a lot of
duplication in the config file. This patch adds global settings and
phase-level settings that allow to significantly reduce duplication.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-18 18:11:29 +02:00
Lubomír Sedlář
18b6020ac5 Improve logging of failable deliverables
The pungi.global.log should show subvariant for each failed deliverable
(if available). When the compose finishes, there is a new log file in
logs/global/deliverables.json containing details about all deliverables
as triples of (variant, arch, subvariant).

 * `required` lists all deliverables that can not fail
 * `attempted` lists all failable deliverables that were started
 * `failed` is a subset of `attempted` and only contains deliverables
   that failed

If the compose fails, the lists may be incomplete.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-13 13:45:30 +02:00
Lubomír Sedlář
98f7ef739e Refactor checking for failable deliverables
This uses a new common function that will ensure consistent logging
without duplicating the functionality all over the place.

A couple tests are updated to verify that correct data is logged.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-24 10:12:20 +01:00
Lubomír Sedlář
652987f2cc Reuse helper in all tests
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-23 12:00:06 +01:00
Dennis Gilmore
6230b0ff3e Merge #232 Improve logging by adding subvariants 2016-03-22 14:15:26 +00:00
Lubomír Sedlář
ecbf08c6f8 Add compose type to release for images
Move getting of the precomputed value to a single place in Compose
class. The value now has format `date[.type_suffix].respin`.

Resolves: rhbz#1319924
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-22 08:57:40 +01:00
Lubomír Sedlář
ea7f98ef4e [image-build] Use subvariants in logging output
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-17 10:19:28 +01:00
Lubomír Sedlář
4a4ef23e3c [image-build] Fix resolving git urls
After config was modified to work with sections, the resolving broke.
This patch fixes it and adds a test to catch this problem in the future.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-29 08:23:26 +01:00
Lubomír Sedlář
e9292fc942 [tests] Dummy compose is no longer private
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-22 19:11:04 +01:00
Lubomír Sedlář
90954ddf0e [tests] Use real paths module in testing
This avoids problems of mock paths having different API that the real
ones.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-22 19:11:04 +01:00
Lubomír Sedlář
8661d294ab [tests] Move dummy testing compose into separate module
This greatly reduces duplication, as a single copy is now shared among
all test cases.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-22 19:10:12 +01:00
Lubomír Sedlář
0fb8049fdd Update tests
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-19 08:51:03 +01:00
Lubomír Sedlář
e9a42ad54f [image-build] Fix tests
This patch updates tests so that they pass, and fixes a couple places in
image build that did not work correctly with the nested configuration.
The documentation is reformatted so that it horizontal scrolling is less
likely.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 13:22:26 +01:00
Lubomír Sedlář
3baa386ed4 [image-build] Fix failable tests
A missing input value was causing tests to not check the expected
condition (even though they were still passing).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-08 10:54:24 +01:00
Dennis Gilmore
2b897ec6ea Merge #135 Add live media support 2016-02-02 10:15:56 +00:00
Lubomír Sedlář
ae30c07553 [koji-wrapper] Use more descriptive method names
The methods mentioning image build are generic and can work for other
task types.

get_image_build_paths -> get_image_paths
run_create_image_cmd -> run_blocking_cmd

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-29 09:02:02 +01:00
Lubomír Sedlář
a698c7cc0a [image-build] Allow running image build scratch tasks
Issue #130

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-26 12:47:59 +01:00