almalinux/pungi
14
1
Fork 0
Code Issues Pull Requests Releases Activity
e7a58ccd07
pungi/tests/test_imagebuildphase.py

997 lines
40 KiB
Python
Raw Normal View History

[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
# -*- coding: utf-8 -*-
import mock
Remove usage of unittest2 on Python 3 It was needed to provide assertItemsEqual method. Starting with Python 3.2, there's assertCountEqual that does the same thing. Six provides a helper that will dispatch to the existing method. With this change, unittest 2 is only needed on Python 2.6 to backport the method. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 12:45:03 +00:00
import six
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
import os
import sys
from pungi.phases.image_build import ImageBuildPhase, CreateImageBuildThread
Reuse helper in all tests Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-22 11:18:31 +00:00
from tests.helpers import DummyCompose, PungiTestCase, boom
[image-build] Optionally do not break whole compose The configuration can now specify image-build as a deliverable that can fail but not abort the whole compose. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 16:14:51 +00:00
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
[tests] Use real paths module in testing This avoids problems of mock paths having different API that the real ones. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 14:15:36 +00:00
class TestImageBuildPhase(PungiTestCase):
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
image-build: Reduce duplication All tests should print complete diffs on failure, so there is no need to define this in each test separately. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-28 08:28:46 +00:00
def setUp(self):
super(TestImageBuildPhase, self).setUp()
self.maxDiff = None
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
@mock.patch('pungi.phases.image_build.ThreadPool')
def test_image_build(self, ThreadPool):
[tests] Dummy compose is no longer private Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-22 14:58:34 +00:00
compose = DummyCompose(self.topdir, {
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
'image_build': {
'^Client|Server$': [
{
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'image-build': {
'format': [('docker', 'tar.xz')],
'name': 'Fedora-Docker-Base',
'target': 'f24',
'version': 'Rawhide',
'ksurl': 'git://git.fedorahosted.org/git/spin-kickstarts.git',
'kickstart': "fedora-docker-base.ks",
'distro': 'Fedora-20',
Add JSON Schema for configuration The schema is written in Python to reduce duplication. When configuration is loaded, the validation checks if it's correct and fills in default values. There is a custom extension to the schema to report deprecated options. The config dependencies are implemented as a separate pass. While it's technically possible to express the dependencies in the schema itself, the error messages are not very helpful and it makes the schema much harder to read. Phases no longer define `config_options`. New options should be added to the schema. Since the default values are populated automatically during validation, there is no need to duplicate them into the code. The `pungi-config-validate` script is updated to use the schema and report errors even for deeply nested fields. The dependencies are updated: pungi now depends on `python-jsonschema` (which is already available in Fedora). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-22 14:08:25 +00:00
'disk_size': 3,
'failable': ['x86_64'],
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
}
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
}
]
},
'koji_profile': 'koji',
})
config: Report unknown options as warnings Given the way config files can include other files, it is entirely possible to use the same config file for both Pungi and Distill. Pungi will however complain about unknown options. This patch reverts part of c38bb480 and moves deprecation handling back into schema validation. The validation method then returns a list of errors and a list of warnings. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-07 14:57:35 +00:00
self.assertValidConfig(compose.conf)
Add JSON Schema for configuration The schema is written in Python to reduce duplication. When configuration is loaded, the validation checks if it's correct and fills in default values. There is a custom extension to the schema to report deprecated options. The config dependencies are implemented as a separate pass. While it's technically possible to express the dependencies in the schema itself, the error messages are not very helpful and it makes the schema much harder to read. Phases no longer define `config_options`. New options should be added to the schema. Since the default values are populated automatically during validation, there is no need to duplicate them into the code. The `pungi-config-validate` script is updated to use the schema and report errors even for deeply nested fields. The dependencies are updated: pungi now depends on `python-jsonschema` (which is already available in Fedora). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-22 14:08:25 +00:00
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
phase = ImageBuildPhase(compose)
phase.run()
# assert at least one thread was started
self.assertTrue(phase.pool.add.called)
client_args = {
"image_conf": {
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'image-build': {
[tests] Use real paths module in testing This avoids problems of mock paths having different API that the real ones. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 14:15:36 +00:00
'install_tree': self.topdir + '/compose/Client/$arch/os',
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'kickstart': 'fedora-docker-base.ks',
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'format': ['docker'],
[tests] Use real paths module in testing This avoids problems of mock paths having different API that the real ones. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 14:15:36 +00:00
'repo': self.topdir + '/compose/Client/$arch/os',
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'variant': compose.variants['Client'],
'target': 'f24',
'disk_size': 3,
'name': 'Fedora-Docker-Base',
image-build: Pass arches around as a list Instead of joining the arches as a comma separated string and splitting it again later. Ultimately we do need the original format to pass to koji wrapper, but we can produce that value later. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-24 07:36:33 +00:00
'arches': ['amd64'],
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'version': 'Rawhide',
'ksurl': 'git://git.fedorahosted.org/git/spin-kickstarts.git',
'distro': 'Fedora-20',
image-build: Pass arches around as a list Instead of joining the arches as a comma separated string and splitting it again later. Ultimately we do need the original format to pass to koji wrapper, but we can produce that value later. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-24 07:36:33 +00:00
'can_fail': ['x86_64'],
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
}
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
},
image-build: add arch name(s) in image config file name Pungi write image config file with name of <format>-<name>.cfg, if there are two or more image configs present for different arches under the same variant and with same format & name, the config file can be overwritten, and result in invalid image conf file. Example: image_build = { '^Server$': [ { 'image-build': { 'format': [('qcow2', 'qcow2'),], 'name': 'fedora-guest-image', 'target': 'guest-fedora-26-image', 'version': '26', 'ksurl': "git://git.example.com/ks.git?fedora#HEAD", 'kickstart': "fedora-26-kvm.ks", 'ksversion': 'f26', 'distro': 'fedora-26', 'disk-size': '10', 'arches': ['x86_64'], 'repo': ["http://example.com/linux/fedora/26/Everything/x86_64/os", ] } }, { 'image-build': { 'format': [('qcow2', 'qcow2'),], 'name': 'fedora-guest-image', 'target': 'guest-fedora-26-image', 'version': '26', 'ksurl': "git://git.example.com/ks.git?fedora#HEAD", 'kickstart': "fedora-26-kvm.ks", 'ksversion': 'f26', 'distro': 'fedora-26', 'disk-size': '10', 'arches': ['ppc64le'], } }, ], } In this case, config file "qcow2_guest-fedora-26-image.cfg" will be created for both x86_64 and ppc64le under the same variant dir, and there is a high chance it will be over-written while Pungi creating the koji task. We can add arch name(s) in config filename to avoid that. Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-08-31 08:10:52 +00:00
"conf_file": self.topdir + '/work/image-build/Client/docker_Fedora-Docker-Base_amd64.cfg',
[tests] Use real paths module in testing This avoids problems of mock paths having different API that the real ones. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 14:15:36 +00:00
"image_dir": self.topdir + '/compose/Client/%(arch)s/images',
"relative_image_dir": 'Client/%(arch)s/images',
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
"link_type": 'hardlink-or-copy',
[image-build] Allow running image build scratch tasks Issue #130 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-26 11:46:50 +00:00
"scratch": False,
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
}
server_args = {
"image_conf": {
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'image-build': {
[tests] Use real paths module in testing This avoids problems of mock paths having different API that the real ones. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 14:15:36 +00:00
'install_tree': self.topdir + '/compose/Server/$arch/os',
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'kickstart': 'fedora-docker-base.ks',
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'format': ['docker'],
[tests] Use real paths module in testing This avoids problems of mock paths having different API that the real ones. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 14:15:36 +00:00
'repo': self.topdir + '/compose/Server/$arch/os',
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'variant': compose.variants['Server'],
'target': 'f24',
'disk_size': 3,
'name': 'Fedora-Docker-Base',
image-build: Pass arches around as a list Instead of joining the arches as a comma separated string and splitting it again later. Ultimately we do need the original format to pass to koji wrapper, but we can produce that value later. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-24 07:36:33 +00:00
'arches': ['amd64', 'x86_64'],
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'version': 'Rawhide',
'ksurl': 'git://git.fedorahosted.org/git/spin-kickstarts.git',
'distro': 'Fedora-20',
image-build: Pass arches around as a list Instead of joining the arches as a comma separated string and splitting it again later. Ultimately we do need the original format to pass to koji wrapper, but we can produce that value later. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-24 07:36:33 +00:00
'can_fail': ['x86_64'],
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
}
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
},
image-build: add arch name(s) in image config file name Pungi write image config file with name of <format>-<name>.cfg, if there are two or more image configs present for different arches under the same variant and with same format & name, the config file can be overwritten, and result in invalid image conf file. Example: image_build = { '^Server$': [ { 'image-build': { 'format': [('qcow2', 'qcow2'),], 'name': 'fedora-guest-image', 'target': 'guest-fedora-26-image', 'version': '26', 'ksurl': "git://git.example.com/ks.git?fedora#HEAD", 'kickstart': "fedora-26-kvm.ks", 'ksversion': 'f26', 'distro': 'fedora-26', 'disk-size': '10', 'arches': ['x86_64'], 'repo': ["http://example.com/linux/fedora/26/Everything/x86_64/os", ] } }, { 'image-build': { 'format': [('qcow2', 'qcow2'),], 'name': 'fedora-guest-image', 'target': 'guest-fedora-26-image', 'version': '26', 'ksurl': "git://git.example.com/ks.git?fedora#HEAD", 'kickstart': "fedora-26-kvm.ks", 'ksversion': 'f26', 'distro': 'fedora-26', 'disk-size': '10', 'arches': ['ppc64le'], } }, ], } In this case, config file "qcow2_guest-fedora-26-image.cfg" will be created for both x86_64 and ppc64le under the same variant dir, and there is a high chance it will be over-written while Pungi creating the koji task. We can add arch name(s) in config filename to avoid that. Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-08-31 08:10:52 +00:00
"conf_file": self.topdir + '/work/image-build/Server/docker_Fedora-Docker-Base_amd64-x86_64.cfg',
[tests] Use real paths module in testing This avoids problems of mock paths having different API that the real ones. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 14:15:36 +00:00
"image_dir": self.topdir + '/compose/Server/%(arch)s/images',
"relative_image_dir": 'Server/%(arch)s/images',
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
"link_type": 'hardlink-or-copy',
[image-build] Allow running image build scratch tasks Issue #130 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-26 11:46:50 +00:00
"scratch": False,
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
}
Remove usage of unittest2 on Python 3 It was needed to provide assertItemsEqual method. Starting with Python 3.2, there's assertCountEqual that does the same thing. Six provides a helper that will dispatch to the existing method. With this change, unittest 2 is only needed on Python 2.6 to backport the method. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 12:45:03 +00:00
six.assertCountEqual(
self,
phase.pool.queue_put.mock_calls,
[mock.call((compose, client_args)), mock.call((compose, server_args))],
)
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
Add common global settings for images The live_images, live_media and image_build phases have same options that need to be specified on each image. This leads to a lot of duplication in the config file. This patch adds global settings and phase-level settings that allow to significantly reduce duplication. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-14 12:23:42 +00:00
@mock.patch('pungi.phases.image_build.ThreadPool')
def test_image_build_phase_global_options(self, ThreadPool):
compose = DummyCompose(self.topdir, {
'image_build_ksurl': 'git://git.fedorahosted.org/git/spin-kickstarts.git',
Be explicit about generating release for images The config now uses similar logic what previous commit did for OSTree. Also we should report error when an unknown generator is used. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-07 13:33:43 +00:00
'image_build_release': '!RELEASE_FROM_LABEL_DATE_TYPE_RESPIN',
Add common global settings for images The live_images, live_media and image_build phases have same options that need to be specified on each image. This leads to a lot of duplication in the config file. This patch adds global settings and phase-level settings that allow to significantly reduce duplication. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-14 12:23:42 +00:00
'image_build_target': 'f24',
'image_build_version': 'Rawhide',
'image_build': {
'^Server$': [
{
'image-build': {
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'format': ['docker'],
Add common global settings for images The live_images, live_media and image_build phases have same options that need to be specified on each image. This leads to a lot of duplication in the config file. This patch adds global settings and phase-level settings that allow to significantly reduce duplication. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-14 12:23:42 +00:00
'name': 'Fedora-Docker-Base',
'kickstart': "fedora-docker-base.ks",
'distro': 'Fedora-20',
'disk_size': 3
}
}
]
},
'koji_profile': 'koji',
})
config: Report unknown options as warnings Given the way config files can include other files, it is entirely possible to use the same config file for both Pungi and Distill. Pungi will however complain about unknown options. This patch reverts part of c38bb480 and moves deprecation handling back into schema validation. The validation method then returns a list of errors and a list of warnings. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-07 14:57:35 +00:00
self.assertValidConfig(compose.conf)
Add JSON Schema for configuration The schema is written in Python to reduce duplication. When configuration is loaded, the validation checks if it's correct and fills in default values. There is a custom extension to the schema to report deprecated options. The config dependencies are implemented as a separate pass. While it's technically possible to express the dependencies in the schema itself, the error messages are not very helpful and it makes the schema much harder to read. Phases no longer define `config_options`. New options should be added to the schema. Since the default values are populated automatically during validation, there is no need to duplicate them into the code. The `pungi-config-validate` script is updated to use the schema and report errors even for deeply nested fields. The dependencies are updated: pungi now depends on `python-jsonschema` (which is already available in Fedora). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-22 14:08:25 +00:00
Add common global settings for images The live_images, live_media and image_build phases have same options that need to be specified on each image. This leads to a lot of duplication in the config file. This patch adds global settings and phase-level settings that allow to significantly reduce duplication. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-14 12:23:42 +00:00
phase = ImageBuildPhase(compose)
phase.run()
# assert at least one thread was started
self.assertTrue(phase.pool.add.called)
server_args = {
"image_conf": {
'image-build': {
'install_tree': self.topdir + '/compose/Server/$arch/os',
'kickstart': 'fedora-docker-base.ks',
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'format': ['docker'],
Add common global settings for images The live_images, live_media and image_build phases have same options that need to be specified on each image. This leads to a lot of duplication in the config file. This patch adds global settings and phase-level settings that allow to significantly reduce duplication. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-14 12:23:42 +00:00
'repo': self.topdir + '/compose/Server/$arch/os',
'variant': compose.variants['Server'],
'target': 'f24',
'disk_size': 3,
'name': 'Fedora-Docker-Base',
image-build: Pass arches around as a list Instead of joining the arches as a comma separated string and splitting it again later. Ultimately we do need the original format to pass to koji wrapper, but we can produce that value later. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-24 07:36:33 +00:00
'arches': ['amd64', 'x86_64'],
Add common global settings for images The live_images, live_media and image_build phases have same options that need to be specified on each image. This leads to a lot of duplication in the config file. This patch adds global settings and phase-level settings that allow to significantly reduce duplication. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-14 12:23:42 +00:00
'version': 'Rawhide',
'ksurl': 'git://git.fedorahosted.org/git/spin-kickstarts.git',
'distro': 'Fedora-20',
'release': '20151203.t.0',
}
},
image-build: add arch name(s) in image config file name Pungi write image config file with name of <format>-<name>.cfg, if there are two or more image configs present for different arches under the same variant and with same format & name, the config file can be overwritten, and result in invalid image conf file. Example: image_build = { '^Server$': [ { 'image-build': { 'format': [('qcow2', 'qcow2'),], 'name': 'fedora-guest-image', 'target': 'guest-fedora-26-image', 'version': '26', 'ksurl': "git://git.example.com/ks.git?fedora#HEAD", 'kickstart': "fedora-26-kvm.ks", 'ksversion': 'f26', 'distro': 'fedora-26', 'disk-size': '10', 'arches': ['x86_64'], 'repo': ["http://example.com/linux/fedora/26/Everything/x86_64/os", ] } }, { 'image-build': { 'format': [('qcow2', 'qcow2'),], 'name': 'fedora-guest-image', 'target': 'guest-fedora-26-image', 'version': '26', 'ksurl': "git://git.example.com/ks.git?fedora#HEAD", 'kickstart': "fedora-26-kvm.ks", 'ksversion': 'f26', 'distro': 'fedora-26', 'disk-size': '10', 'arches': ['ppc64le'], } }, ], } In this case, config file "qcow2_guest-fedora-26-image.cfg" will be created for both x86_64 and ppc64le under the same variant dir, and there is a high chance it will be over-written while Pungi creating the koji task. We can add arch name(s) in config filename to avoid that. Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-08-31 08:10:52 +00:00
"conf_file": self.topdir + '/work/image-build/Server/docker_Fedora-Docker-Base_amd64-x86_64.cfg',
Add common global settings for images The live_images, live_media and image_build phases have same options that need to be specified on each image. This leads to a lot of duplication in the config file. This patch adds global settings and phase-level settings that allow to significantly reduce duplication. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-14 12:23:42 +00:00
"image_dir": self.topdir + '/compose/Server/%(arch)s/images',
"relative_image_dir": 'Server/%(arch)s/images',
"link_type": 'hardlink-or-copy',
"scratch": False,
Automatically generate missing image version If the configuration does not specify version for images or live media, Pungi will create a default value based on `release_version`. If label is used for the compose, the milestone from it will be appended to the version (unless it's RC). This change is backwards compatible: nothing changes when version is set in configuration. If the version was missing before, building the artifacts would fail. With this patch, default values will be supplied. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-30 07:51:36 +00:00
}
Remove usage of unittest2 on Python 3 It was needed to provide assertItemsEqual method. Starting with Python 3.2, there's assertCountEqual that does the same thing. Six provides a helper that will dispatch to the existing method. With this change, unittest 2 is only needed on Python 2.6 to backport the method. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 12:45:03 +00:00
self.assertEqual(
phase.pool.queue_put.mock_calls, [mock.call((compose, server_args))]
)
Automatically generate missing image version If the configuration does not specify version for images or live media, Pungi will create a default value based on `release_version`. If label is used for the compose, the milestone from it will be appended to the version (unless it's RC). This change is backwards compatible: nothing changes when version is set in configuration. If the version was missing before, building the artifacts would fail. With this patch, default values will be supplied. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-30 07:51:36 +00:00
@mock.patch('pungi.phases.image_build.ThreadPool')
def test_image_build_phase_missing_version(self, ThreadPool):
compose = DummyCompose(self.topdir, {
'image_build_ksurl': 'git://git.fedorahosted.org/git/spin-kickstarts.git',
Be explicit about generating release for images The config now uses similar logic what previous commit did for OSTree. Also we should report error when an unknown generator is used. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-07 13:33:43 +00:00
'image_build_release': '!RELEASE_FROM_LABEL_DATE_TYPE_RESPIN',
Automatically generate missing image version If the configuration does not specify version for images or live media, Pungi will create a default value based on `release_version`. If label is used for the compose, the milestone from it will be appended to the version (unless it's RC). This change is backwards compatible: nothing changes when version is set in configuration. If the version was missing before, building the artifacts would fail. With this patch, default values will be supplied. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-30 07:51:36 +00:00
'image_build_target': 'f24',
'image_build': {
'^Server$': [
{
'image-build': {
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'format': 'docker',
Automatically generate missing image version If the configuration does not specify version for images or live media, Pungi will create a default value based on `release_version`. If label is used for the compose, the milestone from it will be appended to the version (unless it's RC). This change is backwards compatible: nothing changes when version is set in configuration. If the version was missing before, building the artifacts would fail. With this patch, default values will be supplied. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-30 07:51:36 +00:00
'name': 'Fedora-Docker-Base',
'kickstart': "fedora-docker-base.ks",
'distro': 'Fedora-20',
'disk_size': 3
}
}
]
},
'koji_profile': 'koji',
})
phase = ImageBuildPhase(compose)
phase.run()
# assert at least one thread was started
self.assertTrue(phase.pool.add.called)
server_args = {
"image_conf": {
'image-build': {
'install_tree': self.topdir + '/compose/Server/$arch/os',
'kickstart': 'fedora-docker-base.ks',
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'format': ['docker'],
Automatically generate missing image version If the configuration does not specify version for images or live media, Pungi will create a default value based on `release_version`. If label is used for the compose, the milestone from it will be appended to the version (unless it's RC). This change is backwards compatible: nothing changes when version is set in configuration. If the version was missing before, building the artifacts would fail. With this patch, default values will be supplied. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-30 07:51:36 +00:00
'repo': self.topdir + '/compose/Server/$arch/os',
'variant': compose.variants['Server'],
'target': 'f24',
'disk_size': 3,
'name': 'Fedora-Docker-Base',
image-build: Pass arches around as a list Instead of joining the arches as a comma separated string and splitting it again later. Ultimately we do need the original format to pass to koji wrapper, but we can produce that value later. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-24 07:36:33 +00:00
'arches': ['amd64', 'x86_64'],
Automatically generate missing image version If the configuration does not specify version for images or live media, Pungi will create a default value based on `release_version`. If label is used for the compose, the milestone from it will be appended to the version (unless it's RC). This change is backwards compatible: nothing changes when version is set in configuration. If the version was missing before, building the artifacts would fail. With this patch, default values will be supplied. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-30 07:51:36 +00:00
'version': '25',
'ksurl': 'git://git.fedorahosted.org/git/spin-kickstarts.git',
'distro': 'Fedora-20',
Add common global settings for images The live_images, live_media and image_build phases have same options that need to be specified on each image. This leads to a lot of duplication in the config file. This patch adds global settings and phase-level settings that allow to significantly reduce duplication. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-14 12:23:42 +00:00
'release': '20151203.t.0',
}
},
image-build: add arch name(s) in image config file name Pungi write image config file with name of <format>-<name>.cfg, if there are two or more image configs present for different arches under the same variant and with same format & name, the config file can be overwritten, and result in invalid image conf file. Example: image_build = { '^Server$': [ { 'image-build': { 'format': [('qcow2', 'qcow2'),], 'name': 'fedora-guest-image', 'target': 'guest-fedora-26-image', 'version': '26', 'ksurl': "git://git.example.com/ks.git?fedora#HEAD", 'kickstart': "fedora-26-kvm.ks", 'ksversion': 'f26', 'distro': 'fedora-26', 'disk-size': '10', 'arches': ['x86_64'], 'repo': ["http://example.com/linux/fedora/26/Everything/x86_64/os", ] } }, { 'image-build': { 'format': [('qcow2', 'qcow2'),], 'name': 'fedora-guest-image', 'target': 'guest-fedora-26-image', 'version': '26', 'ksurl': "git://git.example.com/ks.git?fedora#HEAD", 'kickstart': "fedora-26-kvm.ks", 'ksversion': 'f26', 'distro': 'fedora-26', 'disk-size': '10', 'arches': ['ppc64le'], } }, ], } In this case, config file "qcow2_guest-fedora-26-image.cfg" will be created for both x86_64 and ppc64le under the same variant dir, and there is a high chance it will be over-written while Pungi creating the koji task. We can add arch name(s) in config filename to avoid that. Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-08-31 08:10:52 +00:00
"conf_file": self.topdir + '/work/image-build/Server/docker_Fedora-Docker-Base_amd64-x86_64.cfg',
Add common global settings for images The live_images, live_media and image_build phases have same options that need to be specified on each image. This leads to a lot of duplication in the config file. This patch adds global settings and phase-level settings that allow to significantly reduce duplication. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-14 12:23:42 +00:00
"image_dir": self.topdir + '/compose/Server/%(arch)s/images',
"relative_image_dir": 'Server/%(arch)s/images',
"link_type": 'hardlink-or-copy',
"scratch": False,
}
Remove usage of unittest2 on Python 3 It was needed to provide assertItemsEqual method. Starting with Python 3.2, there's assertCountEqual that does the same thing. Six provides a helper that will dispatch to the existing method. With this change, unittest 2 is only needed on Python 2.6 to backport the method. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 12:45:03 +00:00
self.assertEqual(
phase.pool.queue_put.mock_calls, [mock.call((compose, server_args))]
)
Add common global settings for images The live_images, live_media and image_build phases have same options that need to be specified on each image. This leads to a lot of duplication in the config file. This patch adds global settings and phase-level settings that allow to significantly reduce duplication. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-14 12:23:42 +00:00
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
@mock.patch('pungi.phases.image_build.ThreadPool')
def test_image_build_filter_all_variants(self, ThreadPool):
[tests] Dummy compose is no longer private Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-22 14:58:34 +00:00
compose = DummyCompose(self.topdir, {
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
'image_build': {
'^Client|Server$': [
{
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'image-build': {
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'format': ['docker'],
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'name': 'Fedora-Docker-Base',
'target': 'f24',
'version': 'Rawhide',
'ksurl': 'git://git.fedorahosted.org/git/spin-kickstarts.git',
'kickstart': "fedora-docker-base.ks",
'distro': 'Fedora-20',
'disk_size': 3,
'arches': ['non-existing'],
}
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
}
]
},
'koji_profile': 'koji',
})
config: Report unknown options as warnings Given the way config files can include other files, it is entirely possible to use the same config file for both Pungi and Distill. Pungi will however complain about unknown options. This patch reverts part of c38bb480 and moves deprecation handling back into schema validation. The validation method then returns a list of errors and a list of warnings. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-07 14:57:35 +00:00
self.assertValidConfig(compose.conf)
Add JSON Schema for configuration The schema is written in Python to reduce duplication. When configuration is loaded, the validation checks if it's correct and fills in default values. There is a custom extension to the schema to report deprecated options. The config dependencies are implemented as a separate pass. While it's technically possible to express the dependencies in the schema itself, the error messages are not very helpful and it makes the schema much harder to read. Phases no longer define `config_options`. New options should be added to the schema. Since the default values are populated automatically during validation, there is no need to duplicate them into the code. The `pungi-config-validate` script is updated to use the schema and report errors even for deeply nested fields. The dependencies are updated: pungi now depends on `python-jsonschema` (which is already available in Fedora). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-22 14:08:25 +00:00
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
phase = ImageBuildPhase(compose)
phase.run()
# assert at least one thread was started
self.assertFalse(phase.pool.add.called)
self.assertFalse(phase.pool.queue_put.called)
[image-build] Take install tree from another variant Add a configuration option to use install tree from another variant. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-08 12:37:09 +00:00
@mock.patch('pungi.phases.image_build.ThreadPool')
def test_image_build_set_install_tree(self, ThreadPool):
[tests] Dummy compose is no longer private Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-22 14:58:34 +00:00
compose = DummyCompose(self.topdir, {
[image-build] Take install tree from another variant Add a configuration option to use install tree from another variant. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-08 12:37:09 +00:00
'image_build': {
'^Server$': [
{
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'image-build': {
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'format': ['docker'],
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'name': 'Fedora-Docker-Base',
'target': 'f24',
'version': 'Rawhide',
'ksurl': 'git://git.fedorahosted.org/git/spin-kickstarts.git',
'kickstart': "fedora-docker-base.ks",
'distro': 'Fedora-20',
'disk_size': 3,
'arches': ['x86_64'],
compose: Search all nested variants The code to search for install tree and repo for image-build and live-media was only looking at top-level variants. Addons, optional or integrated layered products could not have been found. This would lead to error messages such as "There is no variant Server-optional to get repo from when building live image for Client" even though the variant exists. Various tests are updated to exercise this edge case. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 09:20:55 +00:00
'install_tree_from': 'Server-optional',
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
}
[image-build] Take install tree from another variant Add a configuration option to use install tree from another variant. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-08 12:37:09 +00:00
}
]
},
'koji_profile': 'koji',
})
compose: Drop unused argument The `get_variants()` method had a `recursive` argument with default value of `False. However, this argument had no effect and the method always returned all variants recursively. We can just drop the argument. All callers are updated to not supply the argument. Should any need for getting the top-level variants only arise, they can be accessed as the `variants` attribute directly on the Compose object. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 12:39:01 +00:00
compose.setup_optional()
[image-build] Take install tree from another variant Add a configuration option to use install tree from another variant. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-08 12:37:09 +00:00
config: Report unknown options as warnings Given the way config files can include other files, it is entirely possible to use the same config file for both Pungi and Distill. Pungi will however complain about unknown options. This patch reverts part of c38bb480 and moves deprecation handling back into schema validation. The validation method then returns a list of errors and a list of warnings. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-07 14:57:35 +00:00
self.assertValidConfig(compose.conf)
Add JSON Schema for configuration The schema is written in Python to reduce duplication. When configuration is loaded, the validation checks if it's correct and fills in default values. There is a custom extension to the schema to report deprecated options. The config dependencies are implemented as a separate pass. While it's technically possible to express the dependencies in the schema itself, the error messages are not very helpful and it makes the schema much harder to read. Phases no longer define `config_options`. New options should be added to the schema. Since the default values are populated automatically during validation, there is no need to duplicate them into the code. The `pungi-config-validate` script is updated to use the schema and report errors even for deeply nested fields. The dependencies are updated: pungi now depends on `python-jsonschema` (which is already available in Fedora). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-22 14:08:25 +00:00
[image-build] Take install tree from another variant Add a configuration option to use install tree from another variant. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-08 12:37:09 +00:00
phase = ImageBuildPhase(compose)
phase.run()
# assert at least one thread was started
self.assertTrue(phase.pool.add.called)
self.assertTrue(phase.pool.queue_put.called_once)
args, kwargs = phase.pool.queue_put.call_args
self.assertEqual(args[0][0], compose)
self.assertDictEqual(args[0][1], {
"image_conf": {
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'image-build': {
compose: Search all nested variants The code to search for install tree and repo for image-build and live-media was only looking at top-level variants. Addons, optional or integrated layered products could not have been found. This would lead to error messages such as "There is no variant Server-optional to get repo from when building live image for Client" even though the variant exists. Various tests are updated to exercise this edge case. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 09:20:55 +00:00
'install_tree': self.topdir + '/compose/Server-optional/$arch/os',
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'kickstart': 'fedora-docker-base.ks',
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'format': ['docker'],
[tests] Use real paths module in testing This avoids problems of mock paths having different API that the real ones. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 14:15:36 +00:00
'repo': self.topdir + '/compose/Server/$arch/os',
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'variant': compose.variants['Server'],
'target': 'f24',
'disk_size': 3,
'name': 'Fedora-Docker-Base',
image-build: Pass arches around as a list Instead of joining the arches as a comma separated string and splitting it again later. Ultimately we do need the original format to pass to koji wrapper, but we can produce that value later. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-24 07:36:33 +00:00
'arches': ['x86_64'],
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'version': 'Rawhide',
'ksurl': 'git://git.fedorahosted.org/git/spin-kickstarts.git',
'distro': 'Fedora-20',
}
[image-build] Take install tree from another variant Add a configuration option to use install tree from another variant. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-08 12:37:09 +00:00
},
image-build: add arch name(s) in image config file name Pungi write image config file with name of <format>-<name>.cfg, if there are two or more image configs present for different arches under the same variant and with same format & name, the config file can be overwritten, and result in invalid image conf file. Example: image_build = { '^Server$': [ { 'image-build': { 'format': [('qcow2', 'qcow2'),], 'name': 'fedora-guest-image', 'target': 'guest-fedora-26-image', 'version': '26', 'ksurl': "git://git.example.com/ks.git?fedora#HEAD", 'kickstart': "fedora-26-kvm.ks", 'ksversion': 'f26', 'distro': 'fedora-26', 'disk-size': '10', 'arches': ['x86_64'], 'repo': ["http://example.com/linux/fedora/26/Everything/x86_64/os", ] } }, { 'image-build': { 'format': [('qcow2', 'qcow2'),], 'name': 'fedora-guest-image', 'target': 'guest-fedora-26-image', 'version': '26', 'ksurl': "git://git.example.com/ks.git?fedora#HEAD", 'kickstart': "fedora-26-kvm.ks", 'ksversion': 'f26', 'distro': 'fedora-26', 'disk-size': '10', 'arches': ['ppc64le'], } }, ], } In this case, config file "qcow2_guest-fedora-26-image.cfg" will be created for both x86_64 and ppc64le under the same variant dir, and there is a high chance it will be over-written while Pungi creating the koji task. We can add arch name(s) in config filename to avoid that. Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-08-31 08:10:52 +00:00
"conf_file": self.topdir + '/work/image-build/Server/docker_Fedora-Docker-Base_x86_64.cfg',
[tests] Use real paths module in testing This avoids problems of mock paths having different API that the real ones. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 14:15:36 +00:00
"image_dir": self.topdir + '/compose/Server/%(arch)s/images',
"relative_image_dir": 'Server/%(arch)s/images',
[image-build] Use repo from another variant The config can now additionally specify other variants whose repos will be passed on to koji. The previous way of specifying extra repos is still available, as is the automatic adding of repo for current variant. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 08:03:04 +00:00
"link_type": 'hardlink-or-copy',
[image-build] Allow running image build scratch tasks Issue #130 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-26 11:46:50 +00:00
"scratch": False,
[image-build] Use repo from another variant The config can now additionally specify other variants whose repos will be passed on to koji. The previous way of specifying extra repos is still available, as is the automatic adding of repo for current variant. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 08:03:04 +00:00
})
image-build: Support repo/install_tree as path If a repo or install tree is specified as an absolute path on the local filesystem, we should either translate it using the configured mappings, or if no mapping matches, it should be return unchanged. A variant name can not start with a slash, so attempting that translation does not make much sense. JIRA: COMPOSE-3290 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-21 09:00:03 +00:00
@mock.patch("pungi.phases.image_build.ThreadPool")
def test_image_build_set_install_tree_from_path(self, ThreadPool):
compose = DummyCompose(self.topdir, {
"image_build": {
"^Server$": [
{
"image-build": {
"format": ["docker"],
"name": "Fedora-Docker-Base",
"target": "f24",
"version": "Rawhide",
"ksurl": "git://git.fedorahosted.org/git/spin-kickstarts.git",
"kickstart": "fedora-docker-base.ks",
"distro": "Fedora-20",
"disk_size": 3,
"arches": ["x86_64"],
"install_tree_from": "/my/tree",
}
}
]
},
"koji_profile": "koji",
"translate_paths": [("/my", "http://example.com")],
})
self.assertValidConfig(compose.conf)
phase = ImageBuildPhase(compose)
phase.run()
# assert at least one thread was started
self.assertTrue(phase.pool.add.called)
self.assertTrue(phase.pool.queue_put.called_once)
args, kwargs = phase.pool.queue_put.call_args
self.assertEqual(args[0][0], compose)
self.assertDictEqual(args[0][1], {
"image_conf": {
"image-build": {
"install_tree": "http://example.com/tree",
"kickstart": "fedora-docker-base.ks",
"format": ["docker"],
"repo": self.topdir + "/compose/Server/$arch/os",
"variant": compose.variants["Server"],
"target": "f24",
"disk_size": 3,
"name": "Fedora-Docker-Base",
"arches": ["x86_64"],
"version": "Rawhide",
"ksurl": "git://git.fedorahosted.org/git/spin-kickstarts.git",
"distro": "Fedora-20",
}
},
"conf_file": self.topdir + "/work/image-build/Server/docker_Fedora-Docker-Base_x86_64.cfg",
"image_dir": self.topdir + "/compose/Server/%(arch)s/images",
"relative_image_dir": "Server/%(arch)s/images",
"link_type": "hardlink-or-copy",
"scratch": False,
})
[image-build] Use repo from another variant The config can now additionally specify other variants whose repos will be passed on to koji. The previous way of specifying extra repos is still available, as is the automatic adding of repo for current variant. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 08:03:04 +00:00
@mock.patch('pungi.phases.image_build.ThreadPool')
def test_image_build_set_extra_repos(self, ThreadPool):
[tests] Dummy compose is no longer private Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-22 14:58:34 +00:00
compose = DummyCompose(self.topdir, {
[image-build] Use repo from another variant The config can now additionally specify other variants whose repos will be passed on to koji. The previous way of specifying extra repos is still available, as is the automatic adding of repo for current variant. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 08:03:04 +00:00
'image_build': {
'^Server$': [
{
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'image-build': {
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'format': ['docker'],
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'name': 'Fedora-Docker-Base',
'target': 'f24',
'version': 'Rawhide',
'ksurl': 'git://git.fedorahosted.org/git/spin-kickstarts.git',
'kickstart': "fedora-docker-base.ks",
'distro': 'Fedora-20',
'disk_size': 3,
'arches': ['x86_64'],
compose: Search all nested variants The code to search for install tree and repo for image-build and live-media was only looking at top-level variants. Addons, optional or integrated layered products could not have been found. This would lead to error messages such as "There is no variant Server-optional to get repo from when building live image for Client" even though the variant exists. Various tests are updated to exercise this edge case. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 09:20:55 +00:00
'repo_from': ['Everything', 'Server-optional'],
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
}
[image-build] Use repo from another variant The config can now additionally specify other variants whose repos will be passed on to koji. The previous way of specifying extra repos is still available, as is the automatic adding of repo for current variant. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 08:03:04 +00:00
}
]
},
'koji_profile': 'koji',
})
compose: Drop unused argument The `get_variants()` method had a `recursive` argument with default value of `False. However, this argument had no effect and the method always returned all variants recursively. We can just drop the argument. All callers are updated to not supply the argument. Should any need for getting the top-level variants only arise, they can be accessed as the `variants` attribute directly on the Compose object. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 12:39:01 +00:00
compose.setup_optional()
[image-build] Use repo from another variant The config can now additionally specify other variants whose repos will be passed on to koji. The previous way of specifying extra repos is still available, as is the automatic adding of repo for current variant. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 08:03:04 +00:00
config: Report unknown options as warnings Given the way config files can include other files, it is entirely possible to use the same config file for both Pungi and Distill. Pungi will however complain about unknown options. This patch reverts part of c38bb480 and moves deprecation handling back into schema validation. The validation method then returns a list of errors and a list of warnings. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-07 14:57:35 +00:00
self.assertValidConfig(compose.conf)
Add JSON Schema for configuration The schema is written in Python to reduce duplication. When configuration is loaded, the validation checks if it's correct and fills in default values. There is a custom extension to the schema to report deprecated options. The config dependencies are implemented as a separate pass. While it's technically possible to express the dependencies in the schema itself, the error messages are not very helpful and it makes the schema much harder to read. Phases no longer define `config_options`. New options should be added to the schema. Since the default values are populated automatically during validation, there is no need to duplicate them into the code. The `pungi-config-validate` script is updated to use the schema and report errors even for deeply nested fields. The dependencies are updated: pungi now depends on `python-jsonschema` (which is already available in Fedora). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-22 14:08:25 +00:00
[image-build] Use repo from another variant The config can now additionally specify other variants whose repos will be passed on to koji. The previous way of specifying extra repos is still available, as is the automatic adding of repo for current variant. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 08:03:04 +00:00
phase = ImageBuildPhase(compose)
phase.run()
# assert at least one thread was started
self.assertTrue(phase.pool.add.called)
self.assertTrue(phase.pool.queue_put.called_once)
args, kwargs = phase.pool.queue_put.call_args
self.assertEqual(args[0][0], compose)
self.assertDictEqual(args[0][1], {
"image_conf": {
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'image-build': {
[tests] Use real paths module in testing This avoids problems of mock paths having different API that the real ones. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 14:15:36 +00:00
'install_tree': self.topdir + '/compose/Server/$arch/os',
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'kickstart': 'fedora-docker-base.ks',
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'format': ['docker'],
[tests] Use real paths module in testing This avoids problems of mock paths having different API that the real ones. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 14:15:36 +00:00
'repo': ','.join([self.topdir + '/compose/Everything/$arch/os',
compose: Search all nested variants The code to search for install tree and repo for image-build and live-media was only looking at top-level variants. Addons, optional or integrated layered products could not have been found. This would lead to error messages such as "There is no variant Server-optional to get repo from when building live image for Client" even though the variant exists. Various tests are updated to exercise this edge case. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 09:20:55 +00:00
self.topdir + '/compose/Server-optional/$arch/os',
[tests] Use real paths module in testing This avoids problems of mock paths having different API that the real ones. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 14:15:36 +00:00
self.topdir + '/compose/Server/$arch/os']),
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'variant': compose.variants['Server'],
'target': 'f24',
'disk_size': 3,
'name': 'Fedora-Docker-Base',
image-build: Pass arches around as a list Instead of joining the arches as a comma separated string and splitting it again later. Ultimately we do need the original format to pass to koji wrapper, but we can produce that value later. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-24 07:36:33 +00:00
'arches': ['x86_64'],
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'version': 'Rawhide',
'ksurl': 'git://git.fedorahosted.org/git/spin-kickstarts.git',
'distro': 'Fedora-20',
}
[image-build] Use repo from another variant The config can now additionally specify other variants whose repos will be passed on to koji. The previous way of specifying extra repos is still available, as is the automatic adding of repo for current variant. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 08:03:04 +00:00
},
image-build: add arch name(s) in image config file name Pungi write image config file with name of <format>-<name>.cfg, if there are two or more image configs present for different arches under the same variant and with same format & name, the config file can be overwritten, and result in invalid image conf file. Example: image_build = { '^Server$': [ { 'image-build': { 'format': [('qcow2', 'qcow2'),], 'name': 'fedora-guest-image', 'target': 'guest-fedora-26-image', 'version': '26', 'ksurl': "git://git.example.com/ks.git?fedora#HEAD", 'kickstart': "fedora-26-kvm.ks", 'ksversion': 'f26', 'distro': 'fedora-26', 'disk-size': '10', 'arches': ['x86_64'], 'repo': ["http://example.com/linux/fedora/26/Everything/x86_64/os", ] } }, { 'image-build': { 'format': [('qcow2', 'qcow2'),], 'name': 'fedora-guest-image', 'target': 'guest-fedora-26-image', 'version': '26', 'ksurl': "git://git.example.com/ks.git?fedora#HEAD", 'kickstart': "fedora-26-kvm.ks", 'ksversion': 'f26', 'distro': 'fedora-26', 'disk-size': '10', 'arches': ['ppc64le'], } }, ], } In this case, config file "qcow2_guest-fedora-26-image.cfg" will be created for both x86_64 and ppc64le under the same variant dir, and there is a high chance it will be over-written while Pungi creating the koji task. We can add arch name(s) in config filename to avoid that. Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-08-31 08:10:52 +00:00
"conf_file": self.topdir + '/work/image-build/Server/docker_Fedora-Docker-Base_x86_64.cfg',
[tests] Use real paths module in testing This avoids problems of mock paths having different API that the real ones. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 14:15:36 +00:00
"image_dir": self.topdir + '/compose/Server/%(arch)s/images',
"relative_image_dir": 'Server/%(arch)s/images',
[image-build] Take install tree from another variant Add a configuration option to use install tree from another variant. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-08 12:37:09 +00:00
"link_type": 'hardlink-or-copy',
[image-build] Allow running image build scratch tasks Issue #130 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-26 11:46:50 +00:00
"scratch": False,
[image-build] Take install tree from another variant Add a configuration option to use install tree from another variant. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-08 12:37:09 +00:00
})
[image-build] Allow dynamic release for images When the release is explicitly set to None, generate a value from date and respin. The documentation is updated to explain how it works. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-21 13:30:44 +00:00
@mock.patch('pungi.phases.image_build.ThreadPool')
[image-build] Allow using external install trees This is needed for the twoweek Fedora atomic release to point to regular nightly composes. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-23 13:23:12 +00:00
def test_image_build_set_external_install_tree(self, ThreadPool):
compose = DummyCompose(self.topdir, {
'image_build': {
'^Server$': [
{
'image-build': {
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'format': ['docker'],
[image-build] Allow using external install trees This is needed for the twoweek Fedora atomic release to point to regular nightly composes. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-23 13:23:12 +00:00
'name': 'Fedora-Docker-Base',
'target': 'f24',
'version': 'Rawhide',
'ksurl': 'git://git.fedorahosted.org/git/spin-kickstarts.git',
'kickstart': "fedora-docker-base.ks",
'distro': 'Fedora-20',
'disk_size': 3,
'arches': ['x86_64'],
'install_tree_from': 'http://example.com/install-tree/',
}
}
]
},
'koji_profile': 'koji',
})
config: Report unknown options as warnings Given the way config files can include other files, it is entirely possible to use the same config file for both Pungi and Distill. Pungi will however complain about unknown options. This patch reverts part of c38bb480 and moves deprecation handling back into schema validation. The validation method then returns a list of errors and a list of warnings. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-07 14:57:35 +00:00
self.assertValidConfig(compose.conf)
Add JSON Schema for configuration The schema is written in Python to reduce duplication. When configuration is loaded, the validation checks if it's correct and fills in default values. There is a custom extension to the schema to report deprecated options. The config dependencies are implemented as a separate pass. While it's technically possible to express the dependencies in the schema itself, the error messages are not very helpful and it makes the schema much harder to read. Phases no longer define `config_options`. New options should be added to the schema. Since the default values are populated automatically during validation, there is no need to duplicate them into the code. The `pungi-config-validate` script is updated to use the schema and report errors even for deeply nested fields. The dependencies are updated: pungi now depends on `python-jsonschema` (which is already available in Fedora). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-22 14:08:25 +00:00
[image-build] Allow using external install trees This is needed for the twoweek Fedora atomic release to point to regular nightly composes. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-23 13:23:12 +00:00
phase = ImageBuildPhase(compose)
phase.run()
# assert at least one thread was started
self.assertTrue(phase.pool.add.called)
self.assertTrue(phase.pool.queue_put.called_once)
args, kwargs = phase.pool.queue_put.call_args
self.assertEqual(args[0][0], compose)
self.assertDictEqual(args[0][1], {
"image_conf": {
'image-build': {
'install_tree': 'http://example.com/install-tree/',
'kickstart': 'fedora-docker-base.ks',
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'format': ['docker'],
[image-build] Allow using external install trees This is needed for the twoweek Fedora atomic release to point to regular nightly composes. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-23 13:23:12 +00:00
'repo': ','.join([self.topdir + '/compose/Server/$arch/os']),
'variant': compose.variants['Server'],
'target': 'f24',
'disk_size': 3,
'name': 'Fedora-Docker-Base',
image-build: Pass arches around as a list Instead of joining the arches as a comma separated string and splitting it again later. Ultimately we do need the original format to pass to koji wrapper, but we can produce that value later. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-24 07:36:33 +00:00
'arches': ['x86_64'],
[image-build] Allow using external install trees This is needed for the twoweek Fedora atomic release to point to regular nightly composes. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-23 13:23:12 +00:00
'version': 'Rawhide',
'ksurl': 'git://git.fedorahosted.org/git/spin-kickstarts.git',
'distro': 'Fedora-20',
}
},
image-build: add arch name(s) in image config file name Pungi write image config file with name of <format>-<name>.cfg, if there are two or more image configs present for different arches under the same variant and with same format & name, the config file can be overwritten, and result in invalid image conf file. Example: image_build = { '^Server$': [ { 'image-build': { 'format': [('qcow2', 'qcow2'),], 'name': 'fedora-guest-image', 'target': 'guest-fedora-26-image', 'version': '26', 'ksurl': "git://git.example.com/ks.git?fedora#HEAD", 'kickstart': "fedora-26-kvm.ks", 'ksversion': 'f26', 'distro': 'fedora-26', 'disk-size': '10', 'arches': ['x86_64'], 'repo': ["http://example.com/linux/fedora/26/Everything/x86_64/os", ] } }, { 'image-build': { 'format': [('qcow2', 'qcow2'),], 'name': 'fedora-guest-image', 'target': 'guest-fedora-26-image', 'version': '26', 'ksurl': "git://git.example.com/ks.git?fedora#HEAD", 'kickstart': "fedora-26-kvm.ks", 'ksversion': 'f26', 'distro': 'fedora-26', 'disk-size': '10', 'arches': ['ppc64le'], } }, ], } In this case, config file "qcow2_guest-fedora-26-image.cfg" will be created for both x86_64 and ppc64le under the same variant dir, and there is a high chance it will be over-written while Pungi creating the koji task. We can add arch name(s) in config filename to avoid that. Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-08-31 08:10:52 +00:00
"conf_file": self.topdir + '/work/image-build/Server/docker_Fedora-Docker-Base_x86_64.cfg',
[image-build] Allow using external install trees This is needed for the twoweek Fedora atomic release to point to regular nightly composes. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-23 13:23:12 +00:00
"image_dir": self.topdir + '/compose/Server/%(arch)s/images',
"relative_image_dir": 'Server/%(arch)s/images',
"link_type": 'hardlink-or-copy',
"scratch": False,
})
@mock.patch('pungi.phases.image_build.ThreadPool')
[image-build] Allow dynamic release for images When the release is explicitly set to None, generate a value from date and respin. The documentation is updated to explain how it works. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-21 13:30:44 +00:00
def test_image_build_create_release(self, ThreadPool):
[tests] Dummy compose is no longer private Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-22 14:58:34 +00:00
compose = DummyCompose(self.topdir, {
[image-build] Allow dynamic release for images When the release is explicitly set to None, generate a value from date and respin. The documentation is updated to explain how it works. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-21 13:30:44 +00:00
'image_build': {
'^Server$': [
{
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'image-build': {
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'format': ['docker'],
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'name': 'Fedora-Docker-Base',
'target': 'f24',
'version': 'Rawhide',
'ksurl': 'git://git.fedorahosted.org/git/spin-kickstarts.git',
'kickstart': "fedora-docker-base.ks",
'distro': 'Fedora-20',
'disk_size': 3,
'arches': ['x86_64'],
'release': None,
}
[image-build] Allow dynamic release for images When the release is explicitly set to None, generate a value from date and respin. The documentation is updated to explain how it works. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-21 13:30:44 +00:00
}
]
},
'koji_profile': 'koji',
})
config: Report unknown options as warnings Given the way config files can include other files, it is entirely possible to use the same config file for both Pungi and Distill. Pungi will however complain about unknown options. This patch reverts part of c38bb480 and moves deprecation handling back into schema validation. The validation method then returns a list of errors and a list of warnings. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-07 14:57:35 +00:00
self.assertValidConfig(compose.conf)
Add JSON Schema for configuration The schema is written in Python to reduce duplication. When configuration is loaded, the validation checks if it's correct and fills in default values. There is a custom extension to the schema to report deprecated options. The config dependencies are implemented as a separate pass. While it's technically possible to express the dependencies in the schema itself, the error messages are not very helpful and it makes the schema much harder to read. Phases no longer define `config_options`. New options should be added to the schema. Since the default values are populated automatically during validation, there is no need to duplicate them into the code. The `pungi-config-validate` script is updated to use the schema and report errors even for deeply nested fields. The dependencies are updated: pungi now depends on `python-jsonschema` (which is already available in Fedora). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-22 14:08:25 +00:00
[image-build] Allow dynamic release for images When the release is explicitly set to None, generate a value from date and respin. The documentation is updated to explain how it works. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-21 13:30:44 +00:00
phase = ImageBuildPhase(compose)
phase.run()
# assert at least one thread was started
self.assertTrue(phase.pool.add.called)
self.assertTrue(phase.pool.queue_put.called_once)
args, kwargs = phase.pool.queue_put.call_args
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
self.assertEqual(args[0][1].get('image_conf', {}).get('image-build', {}).get('release'),
Add compose type to release for images Move getting of the precomputed value to a single place in Compose class. The value now has format `date[.type_suffix].respin`. Resolves: rhbz#1319924 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-22 07:56:51 +00:00
'20151203.t.0')
[image-build] Allow dynamic release for images When the release is explicitly set to None, generate a value from date and respin. The documentation is updated to explain how it works. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-21 13:30:44 +00:00
Be explicit about generating release for images The config now uses similar logic what previous commit did for OSTree. Also we should report error when an unknown generator is used. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-07 13:33:43 +00:00
@mock.patch('pungi.phases.image_build.ThreadPool')
def test_image_build_create_release_with_explicit_config(self, ThreadPool):
compose = DummyCompose(self.topdir, {
'image_build': {
'^Server$': [
{
'image-build': {
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'format': ['docker'],
Be explicit about generating release for images The config now uses similar logic what previous commit did for OSTree. Also we should report error when an unknown generator is used. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-07 13:33:43 +00:00
'name': 'Fedora-Docker-Base',
'target': 'f24',
'version': 'Rawhide',
'ksurl': 'git://git.fedorahosted.org/git/spin-kickstarts.git',
'kickstart': "fedora-docker-base.ks",
'distro': 'Fedora-20',
'disk_size': 3,
'arches': ['x86_64'],
'release': '!RELEASE_FROM_LABEL_DATE_TYPE_RESPIN',
}
}
]
},
'koji_profile': 'koji',
})
self.assertValidConfig(compose.conf)
phase = ImageBuildPhase(compose)
phase.run()
# assert at least one thread was started
self.assertTrue(phase.pool.add.called)
self.assertTrue(phase.pool.queue_put.called_once)
args, kwargs = phase.pool.queue_put.call_args
self.assertEqual(args[0][1].get('image_conf', {}).get('image-build', {}).get('release'),
'20151203.t.0')
[image-build] Allow running image build scratch tasks Issue #130 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-26 11:46:50 +00:00
@mock.patch('pungi.phases.image_build.ThreadPool')
def test_image_build_scratch_build(self, ThreadPool):
[tests] Dummy compose is no longer private Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-22 14:58:34 +00:00
compose = DummyCompose(self.topdir, {
[image-build] Allow running image build scratch tasks Issue #130 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-26 11:46:50 +00:00
'image_build': {
'^Server$': [
{
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'image-build': {
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'format': ['docker'],
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'name': 'Fedora-Docker-Base',
'target': 'f24',
'version': 'Rawhide',
'ksurl': 'git://git.fedorahosted.org/git/spin-kickstarts.git',
'kickstart': "fedora-docker-base.ks",
'distro': 'Fedora-20',
'disk_size': 3,
'arches': ['x86_64'],
'scratch': True,
}
[image-build] Allow running image build scratch tasks Issue #130 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-26 11:46:50 +00:00
}
]
},
'koji_profile': 'koji',
})
config: Report unknown options as warnings Given the way config files can include other files, it is entirely possible to use the same config file for both Pungi and Distill. Pungi will however complain about unknown options. This patch reverts part of c38bb480 and moves deprecation handling back into schema validation. The validation method then returns a list of errors and a list of warnings. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-07 14:57:35 +00:00
self.assertValidConfig(compose.conf)
Add JSON Schema for configuration The schema is written in Python to reduce duplication. When configuration is loaded, the validation checks if it's correct and fills in default values. There is a custom extension to the schema to report deprecated options. The config dependencies are implemented as a separate pass. While it's technically possible to express the dependencies in the schema itself, the error messages are not very helpful and it makes the schema much harder to read. Phases no longer define `config_options`. New options should be added to the schema. Since the default values are populated automatically during validation, there is no need to duplicate them into the code. The `pungi-config-validate` script is updated to use the schema and report errors even for deeply nested fields. The dependencies are updated: pungi now depends on `python-jsonschema` (which is already available in Fedora). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-22 14:08:25 +00:00
[image-build] Allow running image build scratch tasks Issue #130 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-26 11:46:50 +00:00
phase = ImageBuildPhase(compose)
phase.run()
# assert at least one thread was started
self.assertTrue(phase.pool.add.called)
self.assertTrue(phase.pool.queue_put.called_once)
args, kwargs = phase.pool.queue_put.call_args
self.assertTrue(args[0][1].get('scratch'))
image-build: Use install_tree from parent for nested variants Buildinstall only runs for top-level variants. Addons, optionals and integrated layered products must reuse install tree from their parent, because otherwise there would be no boot.iso. Fixes: #472 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-24 13:53:30 +00:00
@mock.patch('pungi.phases.image_build.ThreadPool')
def test_image_build_optional(self, ThreadPool):
compose = DummyCompose(self.topdir, {
'image_build': {
'^Server-optional$': [
{
'image-build': {
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'format': ['docker'],
image-build: Use install_tree from parent for nested variants Buildinstall only runs for top-level variants. Addons, optionals and integrated layered products must reuse install tree from their parent, because otherwise there would be no boot.iso. Fixes: #472 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-24 13:53:30 +00:00
'name': 'Fedora-Docker-Base',
'target': 'f24',
'version': 'Rawhide',
'ksurl': 'git://git.fedorahosted.org/git/spin-kickstarts.git',
'kickstart': "fedora-docker-base.ks",
'distro': 'Fedora-20',
'disk_size': 3,
'failable': ['x86_64'],
}
}
]
},
'koji_profile': 'koji',
})
compose.setup_optional()
self.assertValidConfig(compose.conf)
phase = ImageBuildPhase(compose)
phase.run()
# assert at least one thread was started
self.assertTrue(phase.pool.add.called)
server_args = {
"image_conf": {
'image-build': {
'install_tree': self.topdir + '/compose/Server/$arch/os',
'kickstart': 'fedora-docker-base.ks',
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'format': ['docker'],
image-build: Use install_tree from parent for nested variants Buildinstall only runs for top-level variants. Addons, optionals and integrated layered products must reuse install tree from their parent, because otherwise there would be no boot.iso. Fixes: #472 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-24 13:53:30 +00:00
'repo': self.topdir + '/compose/Server-optional/$arch/os',
'variant': compose.all_variants['Server-optional'],
'target': 'f24',
'disk_size': 3,
'name': 'Fedora-Docker-Base',
image-build: Pass arches around as a list Instead of joining the arches as a comma separated string and splitting it again later. Ultimately we do need the original format to pass to koji wrapper, but we can produce that value later. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-24 07:36:33 +00:00
'arches': ['x86_64'],
image-build: Use install_tree from parent for nested variants Buildinstall only runs for top-level variants. Addons, optionals and integrated layered products must reuse install tree from their parent, because otherwise there would be no boot.iso. Fixes: #472 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-24 13:53:30 +00:00
'version': 'Rawhide',
'ksurl': 'git://git.fedorahosted.org/git/spin-kickstarts.git',
'distro': 'Fedora-20',
image-build: Pass arches around as a list Instead of joining the arches as a comma separated string and splitting it again later. Ultimately we do need the original format to pass to koji wrapper, but we can produce that value later. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-24 07:36:33 +00:00
'can_fail': ['x86_64'],
image-build: Use install_tree from parent for nested variants Buildinstall only runs for top-level variants. Addons, optionals and integrated layered products must reuse install tree from their parent, because otherwise there would be no boot.iso. Fixes: #472 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-24 13:53:30 +00:00
}
},
image-build: add arch name(s) in image config file name Pungi write image config file with name of <format>-<name>.cfg, if there are two or more image configs present for different arches under the same variant and with same format & name, the config file can be overwritten, and result in invalid image conf file. Example: image_build = { '^Server$': [ { 'image-build': { 'format': [('qcow2', 'qcow2'),], 'name': 'fedora-guest-image', 'target': 'guest-fedora-26-image', 'version': '26', 'ksurl': "git://git.example.com/ks.git?fedora#HEAD", 'kickstart': "fedora-26-kvm.ks", 'ksversion': 'f26', 'distro': 'fedora-26', 'disk-size': '10', 'arches': ['x86_64'], 'repo': ["http://example.com/linux/fedora/26/Everything/x86_64/os", ] } }, { 'image-build': { 'format': [('qcow2', 'qcow2'),], 'name': 'fedora-guest-image', 'target': 'guest-fedora-26-image', 'version': '26', 'ksurl': "git://git.example.com/ks.git?fedora#HEAD", 'kickstart': "fedora-26-kvm.ks", 'ksversion': 'f26', 'distro': 'fedora-26', 'disk-size': '10', 'arches': ['ppc64le'], } }, ], } In this case, config file "qcow2_guest-fedora-26-image.cfg" will be created for both x86_64 and ppc64le under the same variant dir, and there is a high chance it will be over-written while Pungi creating the koji task. We can add arch name(s) in config filename to avoid that. Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-08-31 08:10:52 +00:00
"conf_file": self.topdir + '/work/image-build/Server-optional/docker_Fedora-Docker-Base_x86_64.cfg',
image-build: Use install_tree from parent for nested variants Buildinstall only runs for top-level variants. Addons, optionals and integrated layered products must reuse install tree from their parent, because otherwise there would be no boot.iso. Fixes: #472 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-24 13:53:30 +00:00
"image_dir": self.topdir + '/compose/Server-optional/%(arch)s/images',
"relative_image_dir": 'Server-optional/%(arch)s/images',
"link_type": 'hardlink-or-copy',
"scratch": False,
}
Remove usage of unittest2 on Python 3 It was needed to provide assertItemsEqual method. Starting with Python 3.2, there's assertCountEqual that does the same thing. Six provides a helper that will dispatch to the existing method. With this change, unittest 2 is only needed on Python 2.6 to backport the method. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 12:45:03 +00:00
self.assertEqual(
phase.pool.queue_put.mock_calls, [mock.call((compose, server_args))]
)
image-build: Use install_tree from parent for nested variants Buildinstall only runs for top-level variants. Addons, optionals and integrated layered products must reuse install tree from their parent, because otherwise there would be no boot.iso. Fixes: #472 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-24 13:53:30 +00:00
image-build: Expand arches for can_fail We need to work with a list of strings, not a comma-delimited single string. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-23 15:52:57 +00:00
@mock.patch('pungi.phases.image_build.ThreadPool')
def test_failable_star(self, ThreadPool):
compose = DummyCompose(self.topdir, {
'image_build': {
'^Server$': [
{
'image-build': {
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'format': ['docker'],
image-build: Expand arches for can_fail We need to work with a list of strings, not a comma-delimited single string. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-23 15:52:57 +00:00
'name': 'Fedora-Docker-Base',
'target': 'f24',
'version': 'Rawhide',
'ksurl': 'git://git.fedorahosted.org/git/spin-kickstarts.git',
'kickstart': "fedora-docker-base.ks",
'distro': 'Fedora-20',
'disk_size': 3,
'failable': ['*'],
}
}
]
},
'koji_profile': 'koji',
})
compose.setup_optional()
self.assertValidConfig(compose.conf)
phase = ImageBuildPhase(compose)
phase.run()
# assert at least one thread was started
self.assertTrue(phase.pool.add.called)
server_args = {
"image_conf": {
'image-build': {
'install_tree': self.topdir + '/compose/Server/$arch/os',
'kickstart': 'fedora-docker-base.ks',
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'format': ['docker'],
image-build: Expand arches for can_fail We need to work with a list of strings, not a comma-delimited single string. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-23 15:52:57 +00:00
'repo': self.topdir + '/compose/Server/$arch/os',
'variant': compose.all_variants['Server'],
'target': 'f24',
'disk_size': 3,
'name': 'Fedora-Docker-Base',
image-build: Pass arches around as a list Instead of joining the arches as a comma separated string and splitting it again later. Ultimately we do need the original format to pass to koji wrapper, but we can produce that value later. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-24 07:36:33 +00:00
'arches': ['amd64', 'x86_64'],
image-build: Expand arches for can_fail We need to work with a list of strings, not a comma-delimited single string. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-23 15:52:57 +00:00
'version': 'Rawhide',
'ksurl': 'git://git.fedorahosted.org/git/spin-kickstarts.git',
'distro': 'Fedora-20',
image-build: Pass arches around as a list Instead of joining the arches as a comma separated string and splitting it again later. Ultimately we do need the original format to pass to koji wrapper, but we can produce that value later. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-24 07:36:33 +00:00
'can_fail': ['amd64', 'x86_64'],
image-build: Expand arches for can_fail We need to work with a list of strings, not a comma-delimited single string. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-23 15:52:57 +00:00
}
},
image-build: add arch name(s) in image config file name Pungi write image config file with name of <format>-<name>.cfg, if there are two or more image configs present for different arches under the same variant and with same format & name, the config file can be overwritten, and result in invalid image conf file. Example: image_build = { '^Server$': [ { 'image-build': { 'format': [('qcow2', 'qcow2'),], 'name': 'fedora-guest-image', 'target': 'guest-fedora-26-image', 'version': '26', 'ksurl': "git://git.example.com/ks.git?fedora#HEAD", 'kickstart': "fedora-26-kvm.ks", 'ksversion': 'f26', 'distro': 'fedora-26', 'disk-size': '10', 'arches': ['x86_64'], 'repo': ["http://example.com/linux/fedora/26/Everything/x86_64/os", ] } }, { 'image-build': { 'format': [('qcow2', 'qcow2'),], 'name': 'fedora-guest-image', 'target': 'guest-fedora-26-image', 'version': '26', 'ksurl': "git://git.example.com/ks.git?fedora#HEAD", 'kickstart': "fedora-26-kvm.ks", 'ksversion': 'f26', 'distro': 'fedora-26', 'disk-size': '10', 'arches': ['ppc64le'], } }, ], } In this case, config file "qcow2_guest-fedora-26-image.cfg" will be created for both x86_64 and ppc64le under the same variant dir, and there is a high chance it will be over-written while Pungi creating the koji task. We can add arch name(s) in config filename to avoid that. Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-08-31 08:10:52 +00:00
"conf_file": self.topdir + '/work/image-build/Server/docker_Fedora-Docker-Base_amd64-x86_64.cfg',
image-build: Expand arches for can_fail We need to work with a list of strings, not a comma-delimited single string. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-23 15:52:57 +00:00
"image_dir": self.topdir + '/compose/Server/%(arch)s/images',
"relative_image_dir": 'Server/%(arch)s/images',
"link_type": 'hardlink-or-copy',
"scratch": False,
}
Remove usage of unittest2 on Python 3 It was needed to provide assertItemsEqual method. Starting with Python 3.2, there's assertCountEqual that does the same thing. Six provides a helper that will dispatch to the existing method. With this change, unittest 2 is only needed on Python 2.6 to backport the method. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 12:45:03 +00:00
self.assertEqual(
phase.pool.queue_put.mock_calls, [mock.call((compose, server_args))]
)
image-build: Expand arches for can_fail We need to work with a list of strings, not a comma-delimited single string. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-23 15:52:57 +00:00
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
[tests] Use real paths module in testing This avoids problems of mock paths having different API that the real ones. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 14:15:36 +00:00
class TestCreateImageBuildThread(PungiTestCase):
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
[tests] Use real paths module in testing This avoids problems of mock paths having different API that the real ones. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 14:15:36 +00:00
@mock.patch('pungi.phases.image_build.get_mtime')
@mock.patch('pungi.phases.image_build.get_file_size')
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
@mock.patch('pungi.phases.image_build.KojiWrapper')
@mock.patch('pungi.phases.image_build.Linker')
[tests] Use real paths module in testing This avoids problems of mock paths having different API that the real ones. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 14:15:36 +00:00
def test_process(self, Linker, KojiWrapper, get_file_size, get_mtime):
[tests] Dummy compose is no longer private Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-22 14:58:34 +00:00
compose = DummyCompose(self.topdir, {
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
'koji_profile': 'koji'
})
pool = mock.Mock()
cmd = {
"image_conf": {
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'image-build': {
'install_tree': '/ostree/$arch/Client',
'kickstart': 'fedora-docker-base.ks',
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'format': ['docker', 'qcow2'],
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'repo': '/ostree/$arch/Client',
'variant': compose.variants['Client'],
'target': 'f24',
'disk_size': 3,
'name': 'Fedora-Docker-Base',
image-build: Pass arches around as a list Instead of joining the arches as a comma separated string and splitting it again later. Ultimately we do need the original format to pass to koji wrapper, but we can produce that value later. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-24 07:36:33 +00:00
'arches': ['amd64', 'x86_64'],
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'version': 'Rawhide',
'ksurl': 'git://git.fedorahosted.org/git/spin-kickstarts.git',
'distro': 'Fedora-20',
[image-build] Use subvariants in logging output Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-17 08:21:35 +00:00
'subvariant': 'KDE',
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
}
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
},
"conf_file": 'amd64,x86_64-Client-Fedora-Docker-Base-docker',
[tests] Use real paths module in testing This avoids problems of mock paths having different API that the real ones. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 14:15:36 +00:00
"image_dir": self.topdir + '/compose/Client/%(arch)s/images',
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
"relative_image_dir": 'image_dir/Client/%(arch)s',
"link_type": 'hardlink-or-copy',
[image-build] Allow running image build scratch tasks Issue #130 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-26 11:46:50 +00:00
"scratch": False,
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
}
koji_wrapper = KojiWrapper.return_value
[koji-wrapper] Use more descriptive method names The methods mentioning image build are generic and can work for other task types. get_image_build_paths -> get_image_paths run_create_image_cmd -> run_blocking_cmd Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-28 13:58:24 +00:00
koji_wrapper.run_blocking_cmd.return_value = {
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
"retcode": 0,
"output": None,
"task_id": 1234,
}
[koji-wrapper] Use more descriptive method names The methods mentioning image build are generic and can work for other task types. get_image_build_paths -> get_image_paths run_create_image_cmd -> run_blocking_cmd Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-28 13:58:24 +00:00
koji_wrapper.get_image_paths.return_value = {
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
'amd64': [
'/koji/task/1235/tdl-amd64.xml',
'/koji/task/1235/Fedora-Docker-Base-20160103.amd64.qcow2',
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'/koji/task/1235/Fedora-Docker-Base-20160103.amd64.tar.gz'
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
],
'x86_64': [
'/koji/task/1235/tdl-x86_64.xml',
'/koji/task/1235/Fedora-Docker-Base-20160103.x86_64.qcow2',
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'/koji/task/1235/Fedora-Docker-Base-20160103.x86_64.tar.gz'
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
]
}
linker = Linker.return_value
[tests] Use real paths module in testing This avoids problems of mock paths having different API that the real ones. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 14:15:36 +00:00
get_file_size.return_value = 1024
get_mtime.return_value = 13579
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
t = CreateImageBuildThread(pool)
[tests] Use real paths module in testing This avoids problems of mock paths having different API that the real ones. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 14:15:36 +00:00
with mock.patch('time.sleep'):
t.process((compose, cmd), 1)
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
Remove usage of unittest2 on Python 3 It was needed to provide assertItemsEqual method. Starting with Python 3.2, there's assertCountEqual that does the same thing. Six provides a helper that will dispatch to the existing method. With this change, unittest 2 is only needed on Python 2.6 to backport the method. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 12:45:03 +00:00
self.assertEqual(
[image-build] Use subvariants in logging output Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-17 08:21:35 +00:00
koji_wrapper.get_image_build_cmd.call_args_list,
[mock.call(cmd['image_conf'],
conf_file_dest='amd64,x86_64-Client-Fedora-Docker-Base-docker',
scratch=False)]
)
Remove usage of unittest2 on Python 3 It was needed to provide assertItemsEqual method. Starting with Python 3.2, there's assertCountEqual that does the same thing. Six provides a helper that will dispatch to the existing method. With this change, unittest 2 is only needed on Python 2.6 to backport the method. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 12:45:03 +00:00
self.assertEqual(
[image-build] Use subvariants in logging output Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-17 08:21:35 +00:00
koji_wrapper.run_blocking_cmd.call_args_list,
[mock.call(koji_wrapper.get_image_build_cmd.return_value,
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
log_file=self.topdir + '/logs/amd64-x86_64/imagebuild-Client-KDE-docker-qcow2.amd64-x86_64.log')]
[image-build] Use subvariants in logging output Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-17 08:21:35 +00:00
)
Remove usage of unittest2 on Python 3 It was needed to provide assertItemsEqual method. Starting with Python 3.2, there's assertCountEqual that does the same thing. Six provides a helper that will dispatch to the existing method. With this change, unittest 2 is only needed on Python 2.6 to backport the method. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 12:45:03 +00:00
six.assertCountEqual(
self,
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
linker.mock_calls,
Stop using str.format On Python 2.6, it requires the format placeholder to have explicit index of argument, so using % formatting is easier. There are a couple places where the method is still used because the same argument is used twice. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-25 11:37:39 +00:00
[mock.call.link(
'/koji/task/1235/Fedora-Docker-Base-20160103.amd64.qcow2',
self.topdir + '/compose/Client/amd64/images/Fedora-Docker-Base-20160103.amd64.qcow2',
link_type='hardlink-or-copy'),
mock.call.link(
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'/koji/task/1235/Fedora-Docker-Base-20160103.amd64.tar.gz',
self.topdir + '/compose/Client/amd64/images/Fedora-Docker-Base-20160103.amd64.tar.gz',
Stop using str.format On Python 2.6, it requires the format placeholder to have explicit index of argument, so using % formatting is easier. There are a couple places where the method is still used because the same argument is used twice. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-25 11:37:39 +00:00
link_type='hardlink-or-copy'),
mock.call.link(
'/koji/task/1235/Fedora-Docker-Base-20160103.x86_64.qcow2',
self.topdir + '/compose/Client/x86_64/images/Fedora-Docker-Base-20160103.x86_64.qcow2',
link_type='hardlink-or-copy'),
mock.call.link(
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'/koji/task/1235/Fedora-Docker-Base-20160103.x86_64.tar.gz',
self.topdir + '/compose/Client/x86_64/images/Fedora-Docker-Base-20160103.x86_64.tar.gz',
Stop using str.format On Python 2.6, it requires the format placeholder to have explicit index of argument, so using % formatting is easier. There are a couple places where the method is still used because the same argument is used twice. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-25 11:37:39 +00:00
link_type='hardlink-or-copy')])
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
image_relative_paths = {
'image_dir/Client/amd64/Fedora-Docker-Base-20160103.amd64.qcow2': {
'format': 'qcow2',
'type': 'qcow2',
'arch': 'amd64',
},
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'image_dir/Client/amd64/Fedora-Docker-Base-20160103.amd64.tar.gz': {
'format': 'tar.gz',
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
'type': 'docker',
'arch': 'amd64',
},
'image_dir/Client/x86_64/Fedora-Docker-Base-20160103.x86_64.qcow2': {
'format': 'qcow2',
'type': 'qcow2',
'arch': 'x86_64',
},
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'image_dir/Client/x86_64/Fedora-Docker-Base-20160103.x86_64.tar.gz': {
'format': 'tar.gz',
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
'type': 'docker',
'arch': 'x86_64',
},
}
# Assert there are 4 images added to manifest and the arguments are sane
self.assertEqual(len(compose.im.add.call_args_list), 4)
for call in compose.im.add.call_args_list:
_, kwargs = call
image = kwargs['image']
self.assertEqual(kwargs['variant'], 'Client')
self.assertIn(kwargs['arch'], ('amd64', 'x86_64'))
self.assertEqual(kwargs['arch'], image.arch)
self.assertIn(image.path, image_relative_paths)
data = image_relative_paths.pop(image.path)
self.assertEqual(data['format'], image.format)
self.assertEqual(data['type'], image.type)
[image-build] Use subvariants in logging output Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-17 08:21:35 +00:00
self.assertEqual('KDE', image.subvariant)
[image-build] Use single koji task per variant Given a list of arches, koji can build multiple images in one go (automatically starting children tasks for each one). This causes a bunch of changes: * The configuration no longer allows changing config based on architecture, only variants are allowed. It is however possible to filter which arches are used for the building in the variant. * The configuration files for koji image-build are stored in work/image-build/$variant (not split based on arch). This patch also changes the option name that is passed to koji image-build: the repos should be specified under key `repo` (without the trailing slash). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-05 08:27:20 +00:00
[tests] Use real paths module in testing This avoids problems of mock paths having different API that the real ones. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 14:15:36 +00:00
self.assertTrue(os.path.isdir(self.topdir + '/compose/Client/amd64/images'))
self.assertTrue(os.path.isdir(self.topdir + '/compose/Client/x86_64/images'))
[image-build] Stop creating wrong arch dirs There is no such arch as $arch or %(arch)s. Pagure: #124 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-19 07:24:02 +00:00
[image-build] Optionally do not break whole compose The configuration can now specify image-build as a deliverable that can fail but not abort the whole compose. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 16:14:51 +00:00
@mock.patch('pungi.phases.image_build.KojiWrapper')
@mock.patch('pungi.phases.image_build.Linker')
def test_process_handle_fail(self, Linker, KojiWrapper):
Refactor failables This is a breaking change as big part of current failable_deliverables options will be ignored. There is no change for buildinstall and creatiso phase. Failability for artifacts in other phases is now configured per artifact. It already works correctly for ostree and ostree_installer phases (even per-arch). For OSBS phase there is currently only a binary switch as it does not handle multiple arches yet. When it gains that support, the option should contain list of non-blocking architectures. For live images, live media and image build phases each config block can configure list of failable arches. If the list is not empty, it can fail. Once we have a way to fail only some arches, the config will not need to change. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-06-24 07:44:40 +00:00
compose = DummyCompose(self.topdir, {'koji_profile': 'koji'})
[image-build] Optionally do not break whole compose The configuration can now specify image-build as a deliverable that can fail but not abort the whole compose. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 16:14:51 +00:00
pool = mock.Mock()
cmd = {
"image_conf": {
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'image-build': {
'install_tree': '/ostree/$arch/Client',
'kickstart': 'fedora-docker-base.ks',
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'format': ['docker', 'qcow2'],
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'repo': '/ostree/$arch/Client',
'variant': compose.variants['Client'],
'target': 'f24',
'disk_size': 3,
'name': 'Fedora-Docker-Base',
image-build: Pass arches around as a list Instead of joining the arches as a comma separated string and splitting it again later. Ultimately we do need the original format to pass to koji wrapper, but we can produce that value later. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-24 07:36:33 +00:00
'arches': ['amd64', 'x86_64'],
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'version': 'Rawhide',
'ksurl': 'git://git.fedorahosted.org/git/spin-kickstarts.git',
'distro': 'Fedora-20',
image-build: Pass arches around as a list Instead of joining the arches as a comma separated string and splitting it again later. Ultimately we do need the original format to pass to koji wrapper, but we can produce that value later. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-24 07:36:33 +00:00
"can_fail": ['amd64', 'x86_64'],
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
}
[image-build] Optionally do not break whole compose The configuration can now specify image-build as a deliverable that can fail but not abort the whole compose. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 16:14:51 +00:00
},
"conf_file": 'amd64,x86_64-Client-Fedora-Docker-Base-docker',
"image_dir": '/image_dir/Client/%(arch)s',
"relative_image_dir": 'image_dir/Client/%(arch)s',
"link_type": 'hardlink-or-copy',
[image-build] Fix failable tests A missing input value was causing tests to not check the expected condition (even though they were still passing). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-08 09:54:24 +00:00
'scratch': False,
[image-build] Optionally do not break whole compose The configuration can now specify image-build as a deliverable that can fail but not abort the whole compose. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 16:14:51 +00:00
}
koji_wrapper = KojiWrapper.return_value
[koji-wrapper] Use more descriptive method names The methods mentioning image build are generic and can work for other task types. get_image_build_paths -> get_image_paths run_create_image_cmd -> run_blocking_cmd Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-28 13:58:24 +00:00
koji_wrapper.run_blocking_cmd.return_value = {
[image-build] Optionally do not break whole compose The configuration can now specify image-build as a deliverable that can fail but not abort the whole compose. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 16:14:51 +00:00
"retcode": 1,
"output": None,
"task_id": 1234,
}
t = CreateImageBuildThread(pool)
Refactor checking for failable deliverables This uses a new common function that will ensure consistent logging without duplicating the functionality all over the place. A couple tests are updated to verify that correct data is logged. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-24 09:12:20 +00:00
with mock.patch('time.sleep'):
t.process((compose, cmd), 1)
Report failed failable deliverables as errors It does not abort the compose, but it's still an error. It should be marked as such in the log. This will allow easier searching for the failures. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-11 13:37:21 +00:00
pool._logger.error.assert_has_calls([
Improve logging of failable deliverables The pungi.global.log should show subvariant for each failed deliverable (if available). When the compose finishes, there is a new log file in logs/global/deliverables.json containing details about all deliverables as triples of (variant, arch, subvariant). * `required` lists all deliverables that can not fail * `attempted` lists all failable deliverables that were started * `failed` is a subset of `attempted` and only contains deliverables that failed If the compose fails, the lists may be incomplete. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-13 11:44:17 +00:00
mock.call('[FAIL] Image build (variant Client, arch *, subvariant Client) failed, but going on anyway.'),
Refactor checking for failable deliverables This uses a new common function that will ensure consistent logging without duplicating the functionality all over the place. A couple tests are updated to verify that correct data is logged. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-24 09:12:20 +00:00
mock.call('ImageBuild task failed: 1234. See %s for more details.'
% (os.path.join(self.topdir,
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'logs/amd64-x86_64/imagebuild-Client-Client-docker-qcow2.amd64-x86_64.log'))),
Refactor checking for failable deliverables This uses a new common function that will ensure consistent logging without duplicating the functionality all over the place. A couple tests are updated to verify that correct data is logged. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-24 09:12:20 +00:00
])
[image-build] Optionally do not break whole compose The configuration can now specify image-build as a deliverable that can fail but not abort the whole compose. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 16:14:51 +00:00
@mock.patch('pungi.phases.image_build.KojiWrapper')
@mock.patch('pungi.phases.image_build.Linker')
def test_process_handle_exception(self, Linker, KojiWrapper):
Refactor failables This is a breaking change as big part of current failable_deliverables options will be ignored. There is no change for buildinstall and creatiso phase. Failability for artifacts in other phases is now configured per artifact. It already works correctly for ostree and ostree_installer phases (even per-arch). For OSBS phase there is currently only a binary switch as it does not handle multiple arches yet. When it gains that support, the option should contain list of non-blocking architectures. For live images, live media and image build phases each config block can configure list of failable arches. If the list is not empty, it can fail. Once we have a way to fail only some arches, the config will not need to change. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-06-24 07:44:40 +00:00
compose = DummyCompose(self.topdir, {'koji_profile': 'koji'})
[image-build] Optionally do not break whole compose The configuration can now specify image-build as a deliverable that can fail but not abort the whole compose. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 16:14:51 +00:00
pool = mock.Mock()
cmd = {
"image_conf": {
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'image-build': {
'install_tree': '/ostree/$arch/Client',
'kickstart': 'fedora-docker-base.ks',
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'format': ['docker', 'qcow2'],
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'repo': '/ostree/$arch/Client',
'variant': compose.variants['Client'],
'target': 'f24',
'disk_size': 3,
'name': 'Fedora-Docker-Base',
image-build: Pass arches around as a list Instead of joining the arches as a comma separated string and splitting it again later. Ultimately we do need the original format to pass to koji wrapper, but we can produce that value later. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-24 07:36:33 +00:00
'arches': ['amd64', 'x86_64'],
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
'version': 'Rawhide',
'ksurl': 'git://git.fedorahosted.org/git/spin-kickstarts.git',
'distro': 'Fedora-20',
image-build: Pass arches around as a list Instead of joining the arches as a comma separated string and splitting it again later. Ultimately we do need the original format to pass to koji wrapper, but we can produce that value later. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-24 07:36:33 +00:00
'can_fail': ['amd64', 'x86_64'],
[image-build] Fix tests This patch updates tests so that they pass, and fixes a couple places in image build that did not work correctly with the nested configuration. The documentation is reformatted so that it horizontal scrolling is less likely. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:22:03 +00:00
}
[image-build] Optionally do not break whole compose The configuration can now specify image-build as a deliverable that can fail but not abort the whole compose. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 16:14:51 +00:00
},
"conf_file": 'amd64,x86_64-Client-Fedora-Docker-Base-docker',
"image_dir": '/image_dir/Client/%(arch)s',
"relative_image_dir": 'image_dir/Client/%(arch)s',
"link_type": 'hardlink-or-copy',
[image-build] Fix failable tests A missing input value was causing tests to not check the expected condition (even though they were still passing). Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-08 09:54:24 +00:00
'scratch': False,
[image-build] Optionally do not break whole compose The configuration can now specify image-build as a deliverable that can fail but not abort the whole compose. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 16:14:51 +00:00
}
koji_wrapper = KojiWrapper.return_value
[koji-wrapper] Use more descriptive method names The methods mentioning image build are generic and can work for other task types. get_image_build_paths -> get_image_paths run_create_image_cmd -> run_blocking_cmd Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-28 13:58:24 +00:00
koji_wrapper.run_blocking_cmd.side_effect = boom
[image-build] Optionally do not break whole compose The configuration can now specify image-build as a deliverable that can fail but not abort the whole compose. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 16:14:51 +00:00
t = CreateImageBuildThread(pool)
Refactor checking for failable deliverables This uses a new common function that will ensure consistent logging without duplicating the functionality all over the place. A couple tests are updated to verify that correct data is logged. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-24 09:12:20 +00:00
with mock.patch('time.sleep'):
t.process((compose, cmd), 1)
Report failed failable deliverables as errors It does not abort the compose, but it's still an error. It should be marked as such in the log. This will allow easier searching for the failures. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-11 13:37:21 +00:00
pool._logger.error.assert_has_calls([
Improve logging of failable deliverables The pungi.global.log should show subvariant for each failed deliverable (if available). When the compose finishes, there is a new log file in logs/global/deliverables.json containing details about all deliverables as triples of (variant, arch, subvariant). * `required` lists all deliverables that can not fail * `attempted` lists all failable deliverables that were started * `failed` is a subset of `attempted` and only contains deliverables that failed If the compose fails, the lists may be incomplete. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-13 11:44:17 +00:00
mock.call('[FAIL] Image build (variant Client, arch *, subvariant Client) failed, but going on anyway.'),
Refactor checking for failable deliverables This uses a new common function that will ensure consistent logging without duplicating the functionality all over the place. A couple tests are updated to verify that correct data is logged. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-24 09:12:20 +00:00
mock.call('BOOM'),
])
[image-build] Optionally do not break whole compose The configuration can now specify image-build as a deliverable that can fail but not abort the whole compose. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 16:14:51 +00:00
image-build: Allow failure only on some arches This uses the --can-fail option in koji. Failing an optional image will not abort whole task. If the whole task fails (or there is a problem on the compose side), we abort unless all arches are optional. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-28 14:28:00 +00:00
@mock.patch('pungi.phases.image_build.KojiWrapper')
@mock.patch('pungi.phases.image_build.Linker')
def test_process_handle_fail_only_one_optional(self, Linker, KojiWrapper):
compose = DummyCompose(self.topdir, {'koji_profile': 'koji'})
pool = mock.Mock()
cmd = {
"image_conf": {
'image-build': {
'install_tree': '/ostree/$arch/Client',
'kickstart': 'fedora-docker-base.ks',
image-build: Drop suffixes from configuration The file extension in configuration is only used to tell Pungi which files from the task results should be downloaded. The user has to get it right or the phase will fail. Each format has a single valid suffix. Pungi should not require users to specify the suffix, since it can just as well just know the right value. The old configuration will continue working, only the extension will be ignored. Fixes: https://pagure.io/pungi/issue/753 Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 12:23:49 +00:00
'format': ['docker', 'qcow2'],
image-build: Allow failure only on some arches This uses the --can-fail option in koji. Failing an optional image will not abort whole task. If the whole task fails (or there is a problem on the compose side), we abort unless all arches are optional. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-28 14:28:00 +00:00
'repo': '/ostree/$arch/Client',
'variant': compose.variants['Client'],
'target': 'f24',
'disk_size': 3,
'name': 'Fedora-Docker-Base',
image-build: Pass arches around as a list Instead of joining the arches as a comma separated string and splitting it again later. Ultimately we do need the original format to pass to koji wrapper, but we can produce that value later. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-24 07:36:33 +00:00
'arches': ['amd64', 'x86_64'],
image-build: Allow failure only on some arches This uses the --can-fail option in koji. Failing an optional image will not abort whole task. If the whole task fails (or there is a problem on the compose side), we abort unless all arches are optional. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-28 14:28:00 +00:00
'version': 'Rawhide',
'ksurl': 'git://git.fedorahosted.org/git/spin-kickstarts.git',
'distro': 'Fedora-20',
image-build: Pass arches around as a list Instead of joining the arches as a comma separated string and splitting it again later. Ultimately we do need the original format to pass to koji wrapper, but we can produce that value later. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-24 07:36:33 +00:00
'can_fail': ['amd64'],
image-build: Allow failure only on some arches This uses the --can-fail option in koji. Failing an optional image will not abort whole task. If the whole task fails (or there is a problem on the compose side), we abort unless all arches are optional. Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-28 14:28:00 +00:00
}
},
"conf_file": 'amd64,x86_64-Client-Fedora-Docker-Base-docker',
"image_dir": '/image_dir/Client/%(arch)s',
"relative_image_dir": 'image_dir/Client/%(arch)s',
"link_type": 'hardlink-or-copy',
'scratch': False,
}
koji_wrapper = KojiWrapper.return_value
koji_wrapper.run_blocking_cmd.return_value = {
"retcode": 1,
"output": None,
"task_id": 1234,
}
t = CreateImageBuildThread(pool)
with self.assertRaises(RuntimeError):
with mock.patch('time.sleep'):
t.process((compose, cmd), 1)
Reference in New Issue Copy Permalink