almalinux/pungi
14
1
Fork 0
Code Issues Pull Requests Releases Activity

Backport more patches

Browse Source 
* Allow empty modular variants
* Add option to disable multilib
This commit is contained in:
Lubomír Sedlář 2018-03-14 15:01:30 +01:00
parent a45af95feb
commit 55cc5c7668
12 changed files with 272 additions and 19 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
0001-Support-multiple-sources-in-one-variant.patch
View File

@ -1,7 +1,7 @@
From 659eb0215a7a53628533c195cb6dc6e461d8be27 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Lubom=C3=ADr=20Sedl=C3=A1=C5=99?= <lsedlar@redhat.com>
Date: Tue, 7 Nov 2017 14:16:37 +0100
Subject: [PATCH 1/8] Support multiple sources in one variant
Subject: [PATCH 01/11] Support multiple sources in one variant
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

2
0002-Remove-comps-groups-from-purely-modular-variants.patch
View File

@ -1,7 +1,7 @@
From de5dcc7e9ebb3c5b7201b404f302b9ac7dcab722 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Lubom=C3=ADr=20Sedl=C3=A1=C5=99?= <lsedlar@redhat.com>
Date: Fri, 2 Mar 2018 08:33:37 +0100
Subject: [PATCH 2/8] Remove comps groups from purely modular variants
Subject: [PATCH 02/11] Remove comps groups from purely modular variants
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

2
0003-pkgset-Correctly-detect-single-tag-for-variant.patch
View File

@ -1,7 +1,7 @@
From adcb2e23312914535dd71b15d4705c8101055836 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Lubom=C3=ADr=20Sedl=C3=A1=C5=99?= <lsedlar@redhat.com>
Date: Tue, 6 Mar 2018 08:47:17 +0100
Subject: [PATCH 3/8] pkgset: Correctly detect single tag for variant
Subject: [PATCH 03/11] pkgset: Correctly detect single tag for variant
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

2
0004-image-build-Accept-tar.xz-extension-for-docker-image.patch
View File

@ -1,7 +1,7 @@
From fde41452c0bb030eb3467a87eaf25d7f789cba52 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Lubom=C3=ADr=20Sedl=C3=A1=C5=99?= <lsedlar@redhat.com>
Date: Thu, 8 Mar 2018 09:07:48 +0100
Subject: [PATCH 4/8] image-build: Accept tar.xz extension for docker images
Subject: [PATCH 04/11] image-build: Accept tar.xz extension for docker images
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

2
0005-Write-package-whitelist-for-each-variant.patch
View File

@ -1,7 +1,7 @@
From 63329d48c3bc1c72a7bacd654a3ce6e93f6041e7 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Lubom=C3=ADr=20Sedl=C3=A1=C5=99?= <lsedlar@redhat.com>
Date: Wed, 7 Mar 2018 12:35:33 +0100
Subject: [PATCH 5/8] Write package whitelist for each variant
Subject: [PATCH 05/11] Write package whitelist for each variant
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

2
0006-gather-Honor-package-whitelist.patch
View File

@ -1,7 +1,7 @@
From 1bfea4523b803917e37f81f83519721848012674 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Lubom=C3=ADr=20Sedl=C3=A1=C5=99?= <lsedlar@redhat.com>
Date: Wed, 7 Mar 2018 13:42:09 +0100
Subject: [PATCH 6/8] gather: Honor package whitelist
Subject: [PATCH 06/11] gather: Honor package whitelist
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

2
0007-pkgset-Remove-check-for-unique-name.patch
View File

@ -1,7 +1,7 @@
From 74b0d14095733c66c54d47edaac69ef056f55332 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Lubom=C3=ADr=20Sedl=C3=A1=C5=99?= <lsedlar@redhat.com>
Date: Wed, 7 Mar 2018 13:58:53 +0100
Subject: [PATCH 7/8] pkgset: Remove check for unique name
Subject: [PATCH 07/11] pkgset: Remove check for unique name
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

2
0008-pkgset-Merge-initial-package-set-without-checks.patch
View File

@ -1,7 +1,7 @@
From 40c8f95b2bba62f454c7f996409e8bf1d775eee4 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Lubom=C3=ADr=20Sedl=C3=A1=C5=99?= <lsedlar@redhat.com>
Date: Wed, 7 Mar 2018 13:59:12 +0100
Subject: [PATCH 8/8] pkgset: Merge initial package set without checks
Subject: [PATCH 08/11] pkgset: Merge initial package set without checks
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

18
0001-gather-Fix-package-set-whitelist.patch → 0009-gather-Fix-package-set-whitelist.patch
View File

@ -1,7 +1,7 @@
From 56e00505e0f757f2376f6127808f472b3fb67900 Mon Sep 17 00:00:00 2001
From 2dca0b0e3106f3c7252a5d0ad56f3943940e3fd0 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Lubom=C3=ADr=20Sedl=C3=A1=C5=99?= <lsedlar@redhat.com>
Date: Fri, 9 Mar 2018 13:45:03 +0100
Subject: [PATCH] gather: Fix package set whitelist
Subject: [PATCH 09/11] gather: Fix package set whitelist
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
@ -19,7 +19,7 @@ Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
4 files changed, 23 insertions(+), 26 deletions(-)
diff --git a/pungi/gather_dnf.py b/pungi/gather_dnf.py
index f4dda5b3..2c2d7691 100644
index 1023c57d..a0ea6fd1 100644
--- a/pungi/gather_dnf.py
+++ b/pungi/gather_dnf.py
@@ -19,7 +19,7 @@ from itertools import count
@ -49,10 +49,10 @@ index f4dda5b3..2c2d7691 100644
for queue in all_queues:
setattr(self, queue, getattr(self, queue).filter(pkg=to_keep).latest().apply())
diff --git a/pungi/phases/gather/methods/method_deps.py b/pungi/phases/gather/methods/method_deps.py
index 1f38f9a1..f6a765a9 100644
index d38343f3..8c29cfde 100644
--- a/pungi/phases/gather/methods/method_deps.py
+++ b/pungi/phases/gather/methods/method_deps.py
@@ -23,7 +23,7 @@ from kobo.rpmlib import parse_nvra
@@ -22,7 +22,7 @@ from kobo.pkgset import SimpleRpmWrapper, RpmWrapper
from pungi.util import rmtree, get_arch_variant_data
from pungi.wrappers.pungi import PungiWrapper
@ -61,7 +61,7 @@ index 1f38f9a1..f6a765a9 100644
import pungi.phases.gather
import pungi.phases.gather.method
@@ -109,8 +109,11 @@ def write_pungi_config(compose, arch, variant, packages, groups, filter_packages
@@ -90,8 +90,11 @@ def write_pungi_config(compose, arch, variant, packages, groups, filter_packages
package_whitelist = set()
if variant.pkgset:
@ -76,10 +76,10 @@ index 1f38f9a1..f6a765a9 100644
pungi_wrapper.write_kickstart(
ks_path=pungi_cfg, repos=repos, groups=groups, packages=packages_str,
diff --git a/tests/test_gather.py b/tests/test_gather.py
index 81df6324..1daf3c60 100644
index e73ae9c3..fb59dc17 100644
--- a/tests/test_gather.py
+++ b/tests/test_gather.py
@@ -1853,18 +1853,14 @@ class DNFDepsolvingTestCase(DepsolvingBase, unittest.TestCase):
@@ -1798,18 +1798,14 @@ class DNFDepsolvingTestCase(DepsolvingBase, unittest.TestCase):
"dummy-bash",
]
package_whitelist = [
@ -106,7 +106,7 @@ index 81df6324..1daf3c60 100644
]
pkg_map = self.go(packages, None, greedy="none", package_whitelist=package_whitelist)
@@ -1900,8 +1896,7 @@ class DNFDepsolvingTestCase(DepsolvingBase, unittest.TestCase):
@@ -1845,8 +1841,7 @@ class DNFDepsolvingTestCase(DepsolvingBase, unittest.TestCase):
def test_package_whitelist(self):
packages = ['*']
whitelist = [

123
0010-buildinstall-Add-option-to-disable-it.patch Normal file
View File

@ -0,0 +1,123 @@
From 1e6a0c19e78be15e2398bd1a462b5c8b41168155 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Lubom=C3=ADr=20Sedl=C3=A1=C5=99?= <lsedlar@redhat.com>
Date: Tue, 13 Mar 2018 15:54:52 +0100
Subject: [PATCH 10/11] buildinstall: Add option to disable it
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Fixes: https://pagure.io/pungi/issue/854
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
---
doc/configuration.rst | 12 ++++++++++++
pungi/checks.py | 1 +
pungi/phases/buildinstall.py | 8 ++++++++
tests/test_buildinstall.py | 27 +++++++++++++++++++++++++++
4 files changed, 48 insertions(+)
diff --git a/doc/configuration.rst b/doc/configuration.rst
index dbe5ba08..e5f1e9e0 100644
--- a/doc/configuration.rst
+++ b/doc/configuration.rst
@@ -504,6 +504,11 @@ Options
task using HTTP and set the output directory for this task to
``buildinstall_topdir``. Once the runroot task finishes, Pungi will copy
the results of runroot tasks to the compose working directory.
+**buildinstall_skip**
+ (*list*) -- mapping that defines which variants and arches to skip during
+ buildinstall; format: ``[(variant_uid_regex, {arch|*: True})]``. This is
+ only supported for lorax.
+
Example
-------
@@ -525,6 +530,13 @@ Example
})
]
+ # Don't run buildinstall phase for Modular variant
+ buildinstall_skip = [
+ ('^Modular', {
+ '*': True
+ })
+ ]
+
.. note::
diff --git a/pungi/checks.py b/pungi/checks.py
index 9b317ab6..2ff4f697 100644
--- a/pungi/checks.py
+++ b/pungi/checks.py
@@ -727,6 +727,7 @@ def make_schema():
"buildinstall_topdir": {"type": "string"},
"buildinstall_kickstart": {"$ref": "#/definitions/str_or_scm_dict"},
"buildinstall_use_guestmount": {"type": "boolean", "default": True},
+ "buildinstall_skip": _variant_arch_mapping({"type": "boolean"}),
"global_ksurl": {"type": "string"},
"global_version": {"type": "string"},
diff --git a/pungi/phases/buildinstall.py b/pungi/phases/buildinstall.py
index 69813c2a..3ec523bf 100644
--- a/pungi/phases/buildinstall.py
+++ b/pungi/phases/buildinstall.py
@@ -134,10 +134,18 @@ class BuildinstallPhase(PhaseBase):
repo_baseurl = translate_path(self.compose, repo_baseurl)
if self.buildinstall_method == "lorax":
+
buildarch = get_valid_arches(arch)[0]
for variant in self.compose.get_variants(arch=arch, types=['variant']):
if variant.is_empty:
continue
+
+ skip = get_arch_variant_data(self.compose.conf, "buildinstall_skip", arch, variant)
+ if skip == [True]:
+ self.compose.log_info(
+ 'Skipping buildinstall for %s.%s due to config option' % (variant, arch))
+ continue
+
volid = get_volid(self.compose, arch, variant=variant, disc_type=disc_type)
commands.append(
(variant,
diff --git a/tests/test_buildinstall.py b/tests/test_buildinstall.py
index 335b0d2d..aa8f527b 100644
--- a/tests/test_buildinstall.py
+++ b/tests/test_buildinstall.py
@@ -43,6 +43,33 @@ class TestBuildinstallPhase(PungiTestCase):
self.assertTrue(phase.skip())
+ @mock.patch('pungi.phases.buildinstall.ThreadPool')
+ @mock.patch('pungi.phases.buildinstall.LoraxWrapper')
+ @mock.patch('pungi.phases.buildinstall.get_volid')
+ def test_skip_option(self, get_volid, loraxCls, poolCls):
+ compose = BuildInstallCompose(self.topdir, {
+ 'bootable': True,
+ 'buildinstall_method': 'lorax',
+ 'buildinstall_skip': [
+ ('^Server$', {
+ 'amd64': True
+ }),
+ ('^Client$', {
+ '*': True,
+ }),
+ ]
+ })
+
+ get_volid.return_value = 'vol_id'
+ loraxCls.return_value.get_lorax_cmd.return_value = ['lorax', '...']
+
+ phase = BuildinstallPhase(compose)
+
+ phase.run()
+
+ pool = poolCls.return_value
+ self.assertEqual(1, len(pool.queue_put.mock_calls))
+
def test_does_not_skip_on_bootable(self):
compose = BuildInstallCompose(self.topdir, {'bootable': True})
compose.just_phases = None
--
2.13.6

124
0011-pkgset-Allow-empty-list-of-modules.patch Normal file
View File

@ -0,0 +1,124 @@
From 4481d1145e1ea4b0bc3ac3696491a01e633f6397 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Lubom=C3=ADr=20Sedl=C3=A1=C5=99?= <lsedlar@redhat.com>
Date: Wed, 14 Mar 2018 08:17:50 +0100
Subject: [PATCH 11/11] pkgset: Allow empty list of modules
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
This should indicate that it's a modular variant, but there is no
modular content yet. We don't want to treat that as Everything.
The end result will be an empty repository.
Fixes: https://pagure.io/pungi/issue/871
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
---
pungi/phases/gather/sources/source_comps.py | 2 +-
pungi/phases/gather/sources/source_module.py | 2 +-
pungi/phases/pkgset/sources/source_koji.py | 2 +-
pungi/wrappers/variants.py | 8 ++++++--
share/variants.dtd | 2 +-
tests/helpers.py | 1 +
6 files changed, 11 insertions(+), 6 deletions(-)
diff --git a/pungi/phases/gather/sources/source_comps.py b/pungi/phases/gather/sources/source_comps.py
index 447e0a47..0b4a87d5 100644
--- a/pungi/phases/gather/sources/source_comps.py
+++ b/pungi/phases/gather/sources/source_comps.py
@@ -39,7 +39,7 @@ class GatherSourceComps(pungi.phases.gather.source.GatherSourceBase):
comps = CompsWrapper(self.compose.paths.work.comps(arch=arch))
- is_modular = variant and not variant.groups and variant.modules
+ is_modular = variant and not variant.groups and variant.modules is not None
if variant is not None and (variant.groups or variant.type != 'variant' or is_modular):
# Get packages for a particular variant. We want to skip the
# filtering if the variant is top-level and has no groups (to use
diff --git a/pungi/phases/gather/sources/source_module.py b/pungi/phases/gather/sources/source_module.py
index 57740fca..ba6db442 100644
--- a/pungi/phases/gather/sources/source_module.py
+++ b/pungi/phases/gather/sources/source_module.py
@@ -41,7 +41,7 @@ class GatherSourceModule(pungi.phases.gather.source.GatherSourceBase):
compatible_arches = pungi.arch.get_compatible_arches(arch, multilib=True)
- if variant is not None and variant.modules:
+ if variant is not None and variant.modules is not None:
variant.arch_mmds.setdefault(arch, {})
# Contains per-module RPMs added to variant.
diff --git a/pungi/phases/pkgset/sources/source_koji.py b/pungi/phases/pkgset/sources/source_koji.py
index 2ce14be6..4d97b020 100644
--- a/pungi/phases/pkgset/sources/source_koji.py
+++ b/pungi/phases/pkgset/sources/source_koji.py
@@ -260,7 +260,7 @@ def populate_global_pkgset(compose, koji_wrapper, path_prefix, event_id):
if pdc_modules:
with open(pdc_module_file, 'w') as f:
json.dump(pdc_modules, f)
- if not variant_tags[variant]:
+ if not variant_tags[variant] and variant.modules is None:
variant_tags[variant].extend(force_list(compose.conf["pkgset_koji_tag"]))
# Add global tag(s) if supplied.
diff --git a/pungi/wrappers/variants.py b/pungi/wrappers/variants.py
index b32b1f59..6bd00800 100755
--- a/pungi/wrappers/variants.py
+++ b/pungi/wrappers/variants.py
@@ -71,7 +71,7 @@ class VariantsXmlParser(object):
"type": str(variant_node.attrib["type"]),
"arches": [str(i) for i in variant_node.xpath("arches/arch/text()")],
"groups": [],
- "modules": [],
+ "modules": None,
"environments": [],
"buildinstallpackages": [],
"is_empty": bool(variant_node.attrib.get("is_empty", False)),
@@ -110,6 +110,7 @@ class VariantsXmlParser(object):
"glob": self._is_true(module_node.attrib.get("glob", "false"))
}
+ variant_dict["modules"] = variant_dict["modules"] or []
variant_dict["modules"].append(module)
for environments_node in variant_node.xpath("environments"):
@@ -283,7 +284,10 @@ class Variant(object):
return result
def get_modules(self, arch=None, types=None, recursive=False):
- """Return list of groups, default types is ["self"]"""
+ """Return list of modules, default types is ["self"]"""
+
+ if self.modules is None:
+ return []
types = types or ["self"]
result = copy.deepcopy(self.modules)
diff --git a/share/variants.dtd b/share/variants.dtd
index 197e4c4b..f8e4a5f3 100644
--- a/share/variants.dtd
+++ b/share/variants.dtd
@@ -27,7 +27,7 @@
uservisible (true|false) #IMPLIED
>
-<!ELEMENT modules (module)+>
+<!ELEMENT modules (module)*>
<!ELEMENT module (#PCDATA)>
<!ATTLIST module
diff --git a/tests/helpers.py b/tests/helpers.py
index b82de42f..4ed0ae3f 100644
--- a/tests/helpers.py
+++ b/tests/helpers.py
@@ -39,6 +39,7 @@ class MockVariant(mock.Mock):
self.arch_mmds = {}
self.variants = {}
self.pkgset = mock.Mock(rpms_by_arch={})
+ self.modules = None
def __str__(self):
return self.uid
--
2.13.6

10
pungi.spec
View File

@ -1,6 +1,6 @@
Name: pungi
Version: 4.1.22
Release: 8%{?dist}
Release: 9%{?dist}
Summary: Distribution compose tool
Group: Development/Tools
@ -15,7 +15,9 @@ Patch4: 0005-Write-package-whitelist-for-each-variant.patch
Patch5: 0006-gather-Honor-package-whitelist.patch
Patch6: 0007-pkgset-Remove-check-for-unique-name.patch
Patch7: 0008-pkgset-Merge-initial-package-set-without-checks.patch
Patch8: 0001-gather-Fix-package-set-whitelist.patch
Patch8: 0009-gather-Fix-package-set-whitelist.patch
Patch9: 0010-buildinstall-Add-option-to-disable-it.patch
Patch10: 0011-pkgset-Allow-empty-list-of-modules.patch
BuildRequires: python3-nose
BuildRequires: python3-mock
BuildRequires: python2-devel
@ -176,6 +178,10 @@ rm -rf %{buildroot}%{python2_sitelib}/%{name}_utils
%{_bindir}/%{name}-wait-for-signed-ostree-handler
%changelog
* Wed Mar 14 2018 Lubomír Sedlář <lsedlar@redhat.com> - 4.1.22-9
- Allow empty modular variants
- Add option to disable multilib
* Fri Mar 09 2018 Lubomír Sedlář <lsedlar@redhat.com> - 4.1.22-8
- Fix package set whitelist