Compare commits

...

9 Commits

Author SHA1 Message Date
Ondrej Nosek
6e0a9385f2 4.3.7 release
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2022-12-09 13:50:53 +01:00
Lubomír Sedlář
8be0d84f8a
osbuild: test passing of rich repos from configuration
Test that "rich" repositories defined as dicts in the configuration
stay as dicts in the arguments passed to the osbuild phase.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2022-11-28 14:47:11 +01:00
Tomáš Hozza
8f0906be53
osbuild: support specifying package_sets for repos
The `koji-osbuild` plugin supports additional formats for the `repo`
property since v4 [1]. Specifically, a repo can be specified as a
dictionary with `baseurl` key and `package_sets` list containing
specific package set names, that the repository should be used for.

Extend the configuration schema to reflect the plugin change.
Extend the documentation to cover the new repository format.
Extend an existing unit test to specify additional repository using the
added format.

[1] https://github.com/osbuild/koji-osbuild/pull/82

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2022-11-28 14:47:11 +01:00
Tomáš Hozza
e3072c3d5f
osbuild: don't use util.get_repo_urls()
Don't use `util.get_repo_urls()` to resolve provided repositories, but
implement osbuild-specific variant of the function named
`_get_repo_urls(). The reason is that the function from `utils`
transforms repositories defined as dicts to strings, which is
undesired for osbuild. The requirement for osbuild is to preserve the
dict as is, just to resolve the string in `baseurl` to the actual
repository URL.

Add a unit test covering the newly added function. It is inspired by a
similar test from `test_util.py`.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2022-11-28 14:47:11 +01:00
Tomáš Hozza
ef6d40dce4
osbuild: update schema and config documentation
The `koji-osbuild` Hub schema has been relaxed a bit in the latest
release (v11). Adjust the schema in Pungi to reflect changes in
`koji-osbuild`.

For more information on the changes in `koji-osbuild`, see:
https://github.com/osbuild/koji-osbuild/pull/108

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2022-11-28 14:17:42 +01:00
Lubomír Sedlář
df6664098d Speed up tests by 30 seconds
The retry test for CTS doesn't actually need to wait. Let's mock the
sleep function.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-11-23 11:48:12 +01:00
Lubomír Sedlář
147df93f75 Stop sending compose paths to CTS
The tracking service will reject it as it's not an HTTP URL. Let's not
even try.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-11-23 11:48:12 +01:00
Lubomír Sedlář
dd8c1002d4 Report errors from CTS
If the service returns a status code indicating a user error, report
that and do not retry.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-11-23 11:48:12 +01:00
Lubomír Sedlář
12e3a46390 createiso: Create Joliet tree with xorriso
This structure is important for isoinfo -J, which is in turn called by
virt-install.

This can be tested by using a bootable ISO by modifying it with a dummy
additional file and preserving boot records:

    $ xorriso -indev netinst.iso -outdev test.iso -boot_image any replay -map setup.py setup.py -end
    ...
    $ isoinfo -J -i test.iso
    isoinfo: Unable to find Joliet SVD
    $ rm test.iso
    $ xorriso -indev netinst.iso -outdev test.iso -joliet on -boot_image any replay -map setup.py setup.py -end
    ...
    $ isoinfo -J -i test.iso
    $

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=2144105
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-11-22 12:58:46 +01:00
10 changed files with 239 additions and 13 deletions

View File

@ -53,7 +53,7 @@ copyright = u'2016, Red Hat, Inc.'
# The short X.Y version. # The short X.Y version.
version = '4.3' version = '4.3'
# The full version, including alpha/beta/rc tags. # The full version, including alpha/beta/rc tags.
release = '4.3.6' release = '4.3.7'
# The language for content autogenerated by Sphinx. Refer to documentation # The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages. # for a list of supported languages.

View File

@ -1607,8 +1607,23 @@ OSBuild Composer for building images
* ``release`` -- release part of the final NVR. If neither this option nor * ``release`` -- release part of the final NVR. If neither this option nor
the global ``osbuild_release`` is set, Koji will automatically generate a the global ``osbuild_release`` is set, Koji will automatically generate a
value. value.
* ``repo`` -- a list of repository URLs from which to consume packages for * ``repo`` -- a list of repositories from which to consume packages for
building the image. By default only the variant repository is used. building the image. By default only the variant repository is used.
The list items may use one of the following formats:
* String with just the repository URL.
* Dictionary with the following keys:
* ``baseurl`` -- URL of the repository.
* ``package_sets`` -- a list of package set names to use for this
repository. Package sets are an internal concept of Image Builder
and are used in image definitions. If specified, the repository is
used by Image Builder only for the pipeline with the same name.
For example, specifying the ``build`` package set name will make
the repository to be used only for the build environment in which
the image will be built. (optional)
* ``arches`` -- list of architectures for which to build the image. By * ``arches`` -- list of architectures for which to build the image. By
default, the variant arches are used. This option can only restrict it, default, the variant arches are used. This option can only restrict it,
not add a new one. not add a new one.
@ -1641,13 +1656,13 @@ OSBuild Composer for building images
* ``tenant_id`` -- Azure tenant ID to upload the image to * ``tenant_id`` -- Azure tenant ID to upload the image to
* ``subscription_id`` -- Azure subscription ID to upload the image to * ``subscription_id`` -- Azure subscription ID to upload the image to
* ``resource_group`` -- Azure resource group to upload the image to * ``resource_group`` -- Azure resource group to upload the image to
* ``location`` -- Azure location to upload the image to * ``location`` -- Azure location of the resource group (optional)
* ``image_name`` -- Image name of the uploaded Azure image (optional) * ``image_name`` -- Image name of the uploaded Azure image (optional)
* **GCP upload options** -- upload to Google Cloud Platform. * **GCP upload options** -- upload to Google Cloud Platform.
* ``region`` -- GCP region to upload the image to * ``region`` -- GCP region to upload the image to
* ``bucket`` -- GCP bucket to upload the image to * ``bucket`` -- GCP bucket to upload the image to (optional)
* ``share_with_accounts`` -- list of GCP accounts to share the image * ``share_with_accounts`` -- list of GCP accounts to share the image
with with
* ``image_name`` -- Image name of the uploaded GCP image (optional) * ``image_name`` -- Image name of the uploaded GCP image (optional)

View File

@ -1,5 +1,5 @@
Name: pungi Name: pungi
Version: 4.3.6 Version: 4.3.7
Release: 1%{?dist} Release: 1%{?dist}
Summary: Distribution compose tool Summary: Distribution compose tool
@ -111,6 +111,23 @@ pytest
cd tests && ./test_compose.sh cd tests && ./test_compose.sh
%changelog %changelog
* Fri Dec 09 2022 Ondřej Nosek <onosek@redhat.com>
- osbuild: test passing of rich repos from configuration (lsedlar)
- osbuild: support specifying `package_sets` for repos (thozza)
- osbuild: don't use `util.get_repo_urls()` (thozza)
- osbuild: update schema and config documentation (thozza)
- Speed up tests by 30 seconds (lsedlar)
- Stop sending compose paths to CTS (lsedlar)
- Report errors from CTS (lsedlar)
- createiso: Create Joliet tree with xorriso (lsedlar)
- init: Filter comps for modular variants with tags (lsedlar)
- Retry failed cts requests (hlin)
- Ignore existing kerberos ticket for CTS auth (lsedlar)
- osbuild: support specifying upload_options (thozza)
- osbuild: accept only a single image type in the configuration (thozza)
- Add Jenkinsfile for CI (hlin)
- profiler: Flush stdout before printing (lsedlar)
* Fri Aug 26 2022 Lubomír Sedlář <lsedlar@redhat.com> - 4.3.6-1 * Fri Aug 26 2022 Lubomír Sedlář <lsedlar@redhat.com> - 4.3.6-1
- pkgset: Report better error when module is missing an arch (lsedlar) - pkgset: Report better error when module is missing an arch (lsedlar)
- osbuild: add support for building ostree artifacts (ondrej) - osbuild: add support for building ostree artifacts (ondrej)

View File

@ -1188,14 +1188,36 @@ def make_schema():
}, },
"arches": {"$ref": "#/definitions/list_of_strings"}, "arches": {"$ref": "#/definitions/list_of_strings"},
"release": {"type": "string"}, "release": {"type": "string"},
"repo": {"$ref": "#/definitions/list_of_strings"}, "repo": {
"type": "array",
"items": {
"oneOf": [
{
"type": "object",
"additionalProperties": False,
"required": ["baseurl"],
"properties": {
"baseurl": {"type": "string"},
"package_sets": {
"type": "array",
"items": {"type": "string"},
},
},
},
{"type": "string"},
]
},
},
"failable": {"$ref": "#/definitions/list_of_strings"}, "failable": {"$ref": "#/definitions/list_of_strings"},
"subvariant": {"type": "string"}, "subvariant": {"type": "string"},
"ostree_url": {"type": "string"}, "ostree_url": {"type": "string"},
"ostree_ref": {"type": "string"}, "ostree_ref": {"type": "string"},
"ostree_parent": {"type": "string"}, "ostree_parent": {"type": "string"},
"upload_options": { "upload_options": {
"oneOf": [ # this should be really 'oneOf', but the minimal
# required properties in AWSEC2 and GCP options
# overlap.
"anyOf": [
# AWSEC2UploadOptions # AWSEC2UploadOptions
{ {
"type": "object", "type": "object",
@ -1234,7 +1256,6 @@ def make_schema():
"tenant_id", "tenant_id",
"subscription_id", "subscription_id",
"resource_group", "resource_group",
"location",
], ],
"properties": { "properties": {
"tenant_id": {"type": "string"}, "tenant_id": {"type": "string"},
@ -1250,7 +1271,7 @@ def make_schema():
{ {
"type": "object", "type": "object",
"additionalProperties": False, "additionalProperties": False,
"required": ["region", "bucket"], "required": ["region"],
"properties": { "properties": {
"region": {"type": "string"}, "region": {"type": "string"},
"bucket": {"type": "string"}, "bucket": {"type": "string"},

View File

@ -61,6 +61,12 @@ except ImportError:
def retry_request(method, url, data=None, auth=None): def retry_request(method, url, data=None, auth=None):
request_method = getattr(requests, method) request_method = getattr(requests, method)
rv = request_method(url, json=data, auth=auth) rv = request_method(url, json=data, auth=auth)
if rv.status_code >= 400 and rv.status_code < 500:
try:
error = rv.json()["message"]
except ValueError:
error = rv.text
raise RuntimeError("CTS responded with %d: %s" % (rv.status_code, error))
rv.raise_for_status() rv.raise_for_status()
return rv return rv
@ -168,6 +174,9 @@ def update_compose_url(compose_id, compose_dir, conf):
url = os.path.join(cts_url, "api/1/composes", compose_id) url = os.path.join(cts_url, "api/1/composes", compose_id)
tp = conf.get("translate_paths", None) tp = conf.get("translate_paths", None)
compose_url = translate_path_raw(tp, compose_dir) compose_url = translate_path_raw(tp, compose_dir)
if compose_url == compose_dir:
# We do not have a URL, do not attempt the update.
return
data = { data = {
"action": "set_url", "action": "set_url",
"compose_url": compose_url, "compose_url": compose_url,

View File

@ -125,6 +125,8 @@ def write_xorriso_commands(opts):
emit(f, "-outdev %s" % os.path.join(opts.output_dir, opts.iso_name)) emit(f, "-outdev %s" % os.path.join(opts.output_dir, opts.iso_name))
emit(f, "-boot_image any replay") emit(f, "-boot_image any replay")
emit(f, "-volid %s" % opts.volid) emit(f, "-volid %s" % opts.volid)
# isoinfo -J uses the Joliet tree, and it's used by virt-install
emit(f, "-joliet on")
with open(opts.graft_points) as gp: with open(opts.graft_points) as gp:
for line in gp: for line in gp:

View File

@ -27,6 +27,35 @@ class OSBuildPhase(
arches = set(image_conf["arches"]) & arches arches = set(image_conf["arches"]) & arches
return sorted(arches) return sorted(arches)
@staticmethod
def _get_repo_urls(compose, repos, arch="$basearch"):
"""
Get list of repos with resolved repo URLs. Preserve repos defined
as dicts.
"""
resolved_repos = []
for repo in repos:
if isinstance(repo, dict):
try:
url = repo["baseurl"]
except KeyError:
raise RuntimeError(
"`baseurl` is required in repo dict %s" % str(repo)
)
url = util.get_repo_url(compose, url, arch=arch)
if url is None:
raise RuntimeError("Failed to resolve repo URL for %s" % str(repo))
repo["baseurl"] = url
resolved_repos.append(repo)
else:
repo = util.get_repo_url(compose, repo, arch=arch)
if repo is None:
raise RuntimeError("Failed to resolve repo URL for %s" % repo)
resolved_repos.append(repo)
return resolved_repos
def _get_repo(self, image_conf, variant): def _get_repo(self, image_conf, variant):
""" """
Get a list of repos. First included are those explicitly listed in Get a list of repos. First included are those explicitly listed in
@ -38,7 +67,7 @@ class OSBuildPhase(
if not variant.is_empty and variant.uid not in repos: if not variant.is_empty and variant.uid not in repos:
repos.append(variant.uid) repos.append(variant.uid)
return util.get_repo_urls(self.compose, repos, arch="$arch") return OSBuildPhase._get_repo_urls(self.compose, repos, arch="$arch")
def run(self): def run(self):
for variant in self.compose.get_variants(): for variant in self.compose.get_variants():

View File

@ -25,7 +25,7 @@ packages = sorted(packages)
setup( setup(
name="pungi", name="pungi",
version="4.3.6", version="4.3.7",
description="Distribution compose tool", description="Distribution compose tool",
url="https://pagure.io/pungi", url="https://pagure.io/pungi",
author="Dennis Gilmore", author="Dennis Gilmore",

View File

@ -628,6 +628,7 @@ class ComposeTestCase(unittest.TestCase):
ci_copy = dict(self.ci_json) ci_copy = dict(self.ci_json)
ci_copy["header"]["version"] = "1.2" ci_copy["header"]["version"] = "1.2"
mocked_response = mock.MagicMock() mocked_response = mock.MagicMock()
mocked_response.status_code = 200
mocked_response.text = json.dumps(self.ci_json) mocked_response.text = json.dumps(self.ci_json)
mocked_requests.post.return_value = mocked_response mocked_requests.post.return_value = mocked_response
@ -811,6 +812,7 @@ class TracebackTest(unittest.TestCase):
class RetryRequestTest(unittest.TestCase): class RetryRequestTest(unittest.TestCase):
@mock.patch("time.sleep", new=lambda x: x)
@mock.patch("pungi.compose.requests") @mock.patch("pungi.compose.requests")
def test_retry_timeout(self, mocked_requests): def test_retry_timeout(self, mocked_requests):
mocked_requests.post.side_effect = [ mocked_requests.post.side_effect = [
@ -827,3 +829,17 @@ class RetryRequestTest(unittest.TestCase):
], ],
) )
self.assertEqual(rv.status_code, 200) self.assertEqual(rv.status_code, 200)
@mock.patch("pungi.compose.requests")
def test_no_retry_on_client_error(self, mocked_requests):
mocked_requests.post.side_effect = [
mock.Mock(status_code=400, json=lambda: {"message": "You made a mistake"}),
]
url = "http://locahost/api/1/composes/"
with self.assertRaises(RuntimeError):
retry_request("post", url)
self.assertEqual(
mocked_requests.mock_calls,
[mock.call.post(url, json=None, auth=None)],
)

View File

@ -3,14 +3,76 @@
import mock import mock
import os import os
import shutil
import tempfile
import unittest
import koji as orig_koji import koji as orig_koji
from tests import helpers from tests import helpers
from pungi import compose
from pungi.phases import osbuild from pungi.phases import osbuild
from pungi.checks import validate from pungi.checks import validate
class OSBuildPhaseHelperFuncsTest(unittest.TestCase):
@mock.patch("pungi.compose.ComposeInfo")
def setUp(self, ci):
self.tmp_dir = tempfile.mkdtemp()
conf = {"translate_paths": [(self.tmp_dir, "http://example.com")]}
ci.return_value.compose.respin = 0
ci.return_value.compose.id = "RHEL-8.0-20180101.n.0"
ci.return_value.compose.date = "20160101"
ci.return_value.compose.type = "nightly"
ci.return_value.compose.type_suffix = ".n"
ci.return_value.compose.label = "RC-1.0"
ci.return_value.compose.label_major_version = "1"
compose_dir = os.path.join(self.tmp_dir, ci.return_value.compose.id)
self.compose = compose.Compose(conf, compose_dir)
server_variant = mock.Mock(uid="Server", type="variant")
client_variant = mock.Mock(uid="Client", type="variant")
self.compose.all_variants = {
"Server": server_variant,
"Client": client_variant,
}
def tearDown(self):
shutil.rmtree(self.tmp_dir)
def test__get_repo_urls(self):
repos = [
"http://example.com/repo",
"Server",
{
"baseurl": "Client",
"package_sets": ["build"],
},
{
"baseurl": "ftp://example.com/linux/repo",
"package_sets": ["build"],
},
]
expect = [
"http://example.com/repo",
"http://example.com/RHEL-8.0-20180101.n.0/compose/Server/$basearch/os",
{
"baseurl": "http://example.com/RHEL-8.0-20180101.n.0/compose/Client/"
+ "$basearch/os",
"package_sets": ["build"],
},
{
"baseurl": "ftp://example.com/linux/repo",
"package_sets": ["build"],
},
]
self.assertEqual(
osbuild.OSBuildPhase._get_repo_urls(self.compose, repos), expect
)
class OSBuildPhaseTest(helpers.PungiTestCase): class OSBuildPhaseTest(helpers.PungiTestCase):
@mock.patch("pungi.phases.osbuild.ThreadPool") @mock.patch("pungi.phases.osbuild.ThreadPool")
def test_run(self, ThreadPool): def test_run(self, ThreadPool):
@ -124,6 +186,49 @@ class OSBuildPhaseTest(helpers.PungiTestCase):
) )
self.assertNotEqual(validate(compose.conf), ([], [])) self.assertNotEqual(validate(compose.conf), ([], []))
@mock.patch("pungi.phases.osbuild.ThreadPool")
def test_rich_repos(self, ThreadPool):
repo = {"baseurl": "http://example.com/repo", "package_sets": ["build"]}
cfg = {
"name": "test-image",
"distro": "rhel-8",
"version": "1",
"target": "image-target",
"arches": ["x86_64"],
"image_types": ["qcow2"],
"repo": [repo],
}
compose = helpers.DummyCompose(
self.topdir, {"osbuild": {"^Everything$": [cfg]}}
)
self.assertValidConfig(compose.conf)
pool = ThreadPool.return_value
phase = osbuild.OSBuildPhase(compose)
phase.run()
self.assertEqual(len(pool.add.call_args_list), 1)
self.assertEqual(
pool.queue_put.call_args_list,
[
mock.call(
(
compose,
compose.variants["Everything"],
cfg,
["x86_64"],
"1",
None,
"image-target",
[repo, self.topdir + "/compose/Everything/$arch/os"],
[],
),
),
],
)
class RunOSBuildThreadTest(helpers.PungiTestCase): class RunOSBuildThreadTest(helpers.PungiTestCase):
def setUp(self): def setUp(self):
@ -189,7 +294,13 @@ class RunOSBuildThreadTest(helpers.PungiTestCase):
"1", # version "1", # version
"15", # release "15", # release
"image-target", "image-target",
[self.topdir + "/compose/Everything/$arch/os"], [
self.topdir + "/compose/Everything/$arch/os",
{
"baseurl": self.topdir + "/compose/Everything/$arch/os",
"package_sets": ["build"],
},
],
["x86_64"], ["x86_64"],
), ),
1, 1,
@ -211,7 +322,13 @@ class RunOSBuildThreadTest(helpers.PungiTestCase):
["aarch64", "x86_64"], ["aarch64", "x86_64"],
opts={ opts={
"release": "15", "release": "15",
"repo": [self.topdir + "/compose/Everything/$arch/os"], "repo": [
self.topdir + "/compose/Everything/$arch/os",
{
"baseurl": self.topdir + "/compose/Everything/$arch/os",
"package_sets": ["build"],
},
],
}, },
), ),
mock.call.save_task_id(1234), mock.call.save_task_id(1234),