almalinux/pungi
14
1
Fork 0
Code Issues Pull Requests Releases Activity

osbs: Rework configuration for image pushes

Browse Source 
Embedding the registry configuration into OSBS config itself is
simple, but makes it impossible to reuse the same configuration for
multiple different composes.

A nice example is a nightly pushing images to a testing registry, and
production compose building the same images but pushing to staging
location. The original design requires duplication of all the
configuration just because registries are different.

With this option, the push information is stored in a separate option as
a mapping from NVR patterns to arbitrary data. The patterns are used to
match finished builds to registry.

The old configuration is marked as deprecated in code and will
eventually be removed. The deprecation handling in config validation
does not allow emitting warnings for nested values.

JIRA: COMPOSE-3394
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This commit is contained in:
Lubomír Sedlář 2019-04-05 10:23:37 +02:00
parent 959d6979d4
commit d521711957
4 changed files with 67 additions and 11 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

19
doc/configuration.rst
View File

@ -1533,14 +1533,6 @@ they are not scratch builds).
option will most likely change to list architectures that are allowed
to fail.
It is possible to configure extra information about where to push the image
(unless it is a scratch build). Pungi will take any value in ``registry``
key in the configuration and collect them across all built images. The data
will be saved into ``logs/global/osbs-registries.json`` as a mapping from
Koji NVR to the registry data. The same data is also sent to the message
bus on ``osbs-request-push`` topic once the compose finishes successfully.
Handling the message and performing the actual push is outside of scope for
Pungi.
The configuration will pass other attributes directly to the Koji task.
This includes ``name``, ``version``, ``scratch`` and ``priority``.
@ -1551,6 +1543,17 @@ they are not scratch builds).
variant uid, Pungi will create the .repo file for that variant. ``gpgkey``
can be specified to enable gpgcheck in repo files for variants.
**osbs_registries**
(*dict*) -- It is possible to configure extra information about where to
push the image (unless it is a scratch build). For each finished build,
Pungi will try to match NVR against a key in this mapping (using shell-style
globbing) and take the corresponding value and collect them across all built
images. The data will be saved into ``logs/global/osbs-registries.json`` as
a mapping from Koji NVR to the registry data. The same data is also sent to
the message bus on ``osbs-request-push`` topic once the compose finishes
successfully. Handling the message and performing the actual push is outside
of scope for Pungi.
Example config
--------------

8
pungi/checks.py
View File

@ -1225,6 +1225,14 @@ def make_schema():
},
"additionalProperties": False,
},
"osbs_registries": {
"type": "object",
"patternProperties": {
# There are no restrictions on what the value can be.
".+": {}
},
"additionalProperties": False,
},
"extra_files": _variant_arch_mapping({
"type": "array",

18
pungi/phases/osbs.py
View File

@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
import fnmatch
import json
import os
from kobo.threads import ThreadPool, WorkerThread
@ -58,6 +59,16 @@ class OSBSPhase(PhaseLoggerMixin, ConfigGuardedPhase):
)
def get_registry(compose, nvr, fallback=None):
"""Get a configured registry for the image from config matching given NVR.
If not present, return fallback value.
"""
for pattern, registry in compose.conf.get("osbs_registries", {}).items():
if fnmatch.fnmatch(nvr, pattern):
return registry
return fallback
class OSBSThread(WorkerThread):
def process(self, item, num):
compose, variant, config = item
@ -79,6 +90,7 @@ class OSBSThread(WorkerThread):
gpgkey = config.pop('gpgkey', None)
repos = [self._get_repo(compose, v, gpgkey=gpgkey)
for v in [variant.uid] + shortcuts.force_list(config.pop('repo', []))]
# Deprecated in 4.1.36
registry = config.pop("registry", None)
config['yum_repourls'] = repos
@ -98,8 +110,10 @@ class OSBSThread(WorkerThread):
scratch = config.get('scratch', False)
nvr = self._add_metadata(variant, task_id, compose, scratch)
if nvr and registry:
self.pool.registries[nvr] = registry
if nvr:
registry = get_registry(compose, nvr, registry)
if registry:
self.pool.registries[nvr] = registry
self.pool.log_info('[DONE ] %s' % msg)

33
tests/test_osbs_phase.py
View File

@ -360,7 +360,7 @@ class OSBSThreadTest(helpers.PungiTestCase):
self.assertIn("baseurl=http://example.com/repo\n", f)
@mock.patch("pungi.phases.osbs.kojiwrapper.KojiWrapper")
def test_run_with_registry(self, KojiWrapper):
def test_run_with_deprecated_registry(self, KojiWrapper):
cfg = {
"url": "git://example.com/repo?#BEEFCAFE",
"target": "f24-docker-candidate",
@ -390,6 +390,37 @@ class OSBSThreadTest(helpers.PungiTestCase):
self._assertRepoFile(["Server", "Everything"])
self.assertEqual(self.t.pool.registries, {"my-name-1.0-1": {"foo": "bar"}})
@mock.patch("pungi.phases.osbs.kojiwrapper.KojiWrapper")
def test_run_with_registry(self, KojiWrapper):
cfg = {
"url": "git://example.com/repo?#BEEFCAFE",
"target": "f24-docker-candidate",
"git_branch": "f24-docker",
"name": "my-name",
"version": "1.0",
"repo": ["Everything", "http://pkgs.example.com/my.repo"],
}
self.compose.conf["osbs_registries"] = {"my-name-1.0-*": [{"foo": "bar"}]}
self._setupMock(KojiWrapper)
self._assertConfigCorrect(cfg)
self.t.process((self.compose, self.compose.variants["Server"], cfg), 1)
options = {
"name": "my-name",
"version": "1.0",
"git_branch": "f24-docker",
"yum_repourls": [
"http://root/work/global/tmp-Server/compose-rpms-Server-1.repo",
"http://root/work/global/tmp-Everything/compose-rpms-Everything-1.repo",
"http://pkgs.example.com/my.repo",
]
}
self._assertCorrectCalls(options)
self._assertCorrectMetadata()
self._assertRepoFile(["Server", "Everything"])
self.assertEqual(self.t.pool.registries, {"my-name-1.0-1": [{"foo": "bar"}]})
@mock.patch('pungi.phases.osbs.kojiwrapper.KojiWrapper')
def test_run_with_extra_repos_in_list(self, KojiWrapper):
cfg = {