Commit Graph

81 Commits

Author SHA1 Message Date
Lubomír Sedlář
fc86e03e44
pkgset: ignore events for modular content tags
Generally we want all packages to come from particular event.

There are two exceptions: packages configured via `pkgset_koji_builds`
are pulled in by exact NVR and skip event; and modules in
`pkgset_koji_modules` are pulled in by NSVC and also ignore events.

However, the modular content tag did honor event, and could lead to a
crashed compose if the content tag did not exist at the configured
event.

This patch is a slightly too big hammer. It ignores events for all
modules, not just ones configured by explicit NSVC. It's not a huge deal
as the content tags are created before the corresponding module build is
created, and once all rpm builds are tagged into the content tag, MBS
will never change it again.

JIRA: RHELCMP-12765
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit b32c8f3e5e)
2024-08-30 13:30:38 +03:00
Lubomír Sedlář
548441644b
pkgset: Ignore duplicated module builds
If the module tag contains the same module build multiple times (because
it's in multiple tags in the inheritance), Pungi will not process that
correctly and try to include the same NSVC in the compose multiple
times. That leads to a crash.

This patch adds another step to the inheritance filter to ensure the
result contains each module only once.

JIRA: RHELCMP-12768
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 935da7c246)
2024-08-30 13:30:36 +03:00
Timothée Ravier
aba5a7a093
Fix minor Ruff/flake8 warnings
```
pungi/checks.py:575:17: F601 [*] Dictionary key literal `"type"` repeated
pungi/phases/pkgset/pkgsets.py:617:12: E721 Do not compare types, use `isinstance()`
tests/test_pkgset_source_koji.py:241:16: E721 Do not compare types, use `isinstance()`
tests/test_pkgset_source_koji.py:244:16: E721 Do not compare types, use `isinstance()`
tests/test_pkgset_source_koji.py:370:16: E721 Do not compare types, use `isinstance()`
tests/test_pkgset_source_koji.py:374:20: E721 Do not compare types, use `isinstance()`
```

Signed-off-by: Timothée Ravier <tim@siosm.fr>
(cherry picked from commit 2534ddee99)
2024-08-30 13:30:32 +03:00
ae527a2e01
- The unittests are fixed 2023-11-10 18:08:03 +02:00
Ozan Unsal
ce45fdc39a
Fix unittest errors
Signed-off-by: Ozan Unsal <ounsal@redhat.com>

(cherry picked from commit aa0aae3d3e (centos_master))
2023-11-10 16:57:51 +02:00
soksanichenko
b14e85324c - Fix unittests 2022-11-08 14:57:52 +02:00
soksanichenko
5a19ad2258 - Fix unittests 2022-11-08 12:47:14 +02:00
soksanichenko
ee9c9a74e6 - Fix unittests 2022-11-07 23:55:26 +02:00
soksanichenko
323d31df2b Merge branch 'master' into a8_updated
# Conflicts:
#	pungi.spec
#	pungi/wrappers/kojiwrapper.py
#	setup.py
#	tests/test_extra_isos_phase.py
#	tests/test_pkgset_pkgsets.py
2022-11-07 23:38:38 +02:00
soksanichenko
ff946d3f7b - Unittests are fixed 2022-11-07 20:15:37 +02:00
soksanichenko
ede91bcd03 - Right name of the class in constructor 2022-11-07 20:03:59 +02:00
soksanichenko
b49ffee06d - Mock of Koji is moved to the separate modules, classes
- Unittests for mock of Koji are moved to the separate
2022-11-07 19:24:39 +02:00
Filip Valder
a7c111643d Supersede ModuleStream loading with ModuleIndex
- Use ModuleIndex's update_from_file/update_from_string instead of ModuleStream's
read_file/read_string which is deprecated.
- Extend tests to work with real module streams instead of mocks.

Signed-off-by: Filip Valder <fvalder@redhat.com>
2021-08-12 16:32:38 +02:00
stepan_oksanichenko
cc4d99441c LNX-108: Add multiarch support to pungi
@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: Ibfd540454941922d790ae4e56cc0992c0c85635d
2021-05-24 18:07:11 +03:00
stepan_oksanichenko
305103a38e LNX-286: Prepare pungi configuration and setup Jenkins job for AlmaLinux 8.4 beta
- Some modules can be absent in koji env but be present in variants.xml.
  And Pungi will fail in this case. So we must filter out those modules
  from expected modules list by list from pungi build config

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: I22c15c42868412e34fd554030130bd7c3e25b8ef
2021-04-23 13:03:05 +03:00
Haibo Lin
035b37c566 Cancel koji tasks when pungi terminated
JIRA: RHELCMP-4148
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-03-23 14:47:48 +08:00
oshyshatskyi
903db91c0f LNX-102: Patch pungi tool to use local koji mock
Instead of koji.mbox use local koji-like wrapper.

@BS-LINKED-5ff8b8cb6f44affc6c5e9a7a
@BS-TARGET-CL8

Change-Id: I82a2bc8bc71ae06240656898f3df71bb28bcb9e9
2021-01-25 16:17:33 +02:00
Haibo Lin
1e1c8533ac tests: Fix tests for old version of git
GitSCMTestCaseReal failed with error 'Unknown option: -C' when building
rpm for el7.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-08-18 12:38:20 +08:00
Haibo Lin
f7167fa3b6 Allow including scratch module builds
JIRA: RHELCMP-439
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-07-17 09:08:36 +08:00
Haibo Lin
3543f8fb3e pkgset: Reuse pkgset repos
JIRA: COMPOSE-4158
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-03-13 17:29:53 +08:00
Haibo Lin
3eddcfccd8 Fix flake8 complaints - F401
F401 'dnf' imported but unused
F401 'imp' imported but unused
F401 'os' imported but unused
F401 'subprocess' imported but unused
F401 'sys' imported but unused
F401 'yum' imported but unused

JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 11:48:31 +08:00
Haibo Lin
38142d30ba Format tests with black
JIRA: COMPOSE-4086
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-05 17:29:15 +08:00
Haibo Lin
52f82ccc6e pkgset: Ignore deleted module builds
JIRA: COMPOSE-4058
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-14 16:23:55 +08:00
Haibo Lin
0ed70fc8b6 tests: drop sys.path modification
It's unnecessary to add the path manually.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-12-10 10:53:38 +08:00
Lubomír Sedlář
2f54745715 Remove usage of unittest2 on Python 3
It was needed to provide assertItemsEqual method. Starting with Python
3.2, there's assertCountEqual that does the same thing. Six provides a
helper that will dispatch to the existing method. With this change,
unittest 2 is only needed on Python 2.6 to backport the method.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 15:25:18 +02:00
Lubomír Sedlář
8ab7d9f7ba Move import of modulemd to a separate module
This should make it possible to only import the library only when it's
really needed.

DNF does not work with libmodulemd v2. If we import libmodulemd2 and
then dnf, the program will just hang forever. We only need DNF in
pungi-gather, where libmodulemd is not needed, and also where we do need
libmodulemd, we don't have DNF.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:40:31 +02:00
Lubomír Sedlář
21d45eb243 pkgset: Allow filtering modules from Koji tags
Add a configuration option to enable skipping some modules found in the
configured tag.

Fixes: https://pagure.io/pungi/issue/1260
JIRA: COMPOSE-3794
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-24 10:13:00 +02:00
Lubomír Sedlář
054b91c798 Fix tests
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
a446593698 pkgset: Update Koji source to create multiple pkgsets
With this patch, there should be a separate package set for each tag
that is consumed.

Generally each module will create a separate package set, with the
exception of -devel modules that will be in the same set as their
non-devel version.

Variants no longer need to keep their own package set objects. Instead
they now include a set of package set names that should be used for the
variant. This can replace the whitelist mechanism of deps gather method.

JIRA: COMPOSE-3620
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
eed2aa2753 pkgset: Add object representing a package set on disk
Once a package set repo is written to disk, let's use this object that
connects the repository path with the mapping of packages.

This change also makes it explicit where the dependency on package set
repos are. In the original codebase, any part of code could generate a
path to the repo, even if that repo has not yet been written.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
90393c4b49 Update code to pass around a list of pkgsets
This opens up a path to having multiple package sets in the compose. The
pkgset phase now creates a list of them (although at this time there is
always a single item in that list).

Any consumer of the package sets objects is updated to handle a list.
Generally this means an extra loop.

JIRA: COMPOSE-3620
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
e674c2f574 pkgset: Create repos in a single place
Both pkgset sources use the same logic to create per-arch repos. There
is no reason to have that code in both places.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-08-06 09:24:28 +02:00
Lubomír Sedlář
8994aa5d88 pkgset: Added modules to variant in correct format
The data parsed from variants.xml uses a different format that what we
added in `_add_module_to_variant`. This leads to crashes later.

JIRA: COMPOSE-3746
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-08-05 12:06:26 +02:00
Lubomír Sedlář
6efaae19fd Remove debug mode
This was already discouraged to not be used, and is a bad idea in
current setup anyway. Removing this can simplify the code.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-24 13:36:23 +02:00
Lubomír Sedlář
61e3cb0ef1 Port to libmodulemd v2
Fixes: https://pagure.io/pungi/issue/1225
JIRA: COMPOSE-3662
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-15 15:27:05 +02:00
Lubomír Sedlář
da78b99fc0 pkgset: Load modulemd only when needed
We can avoid parsing source modulemd information since we can get the
same information from the Koji build info.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-11 14:25:42 +02:00
Lubomír Sedlář
04baa2a4db Stop storing modulemd without arch
Historically each variant had a list of modules. This is no longer
needed and can be dropped. We can also stop logging the modulemd since
we know it was retrieved from Koji and not modified locally.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-11 14:25:42 +02:00
Lubomír Sedlář
82580ed5b3 pkgset: Create arch repos in parallel
This patch reuses the existing createrepo_num_threads options to limit
maximum number of parallel createrepo processes.

Fixes: https://pagure.io/pungi/issue/955
JIRA: COMPOSE-2575
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-19 15:42:46 +02:00
Lubomír Sedlář
2c3e6a5a74 pkgset: Use highest pickle protocol
Higher protocols should be more efficient in terms of performance and
storage size. Since we don't really care about interoperability with
different python version, we can safely go to the highest version.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-18 09:33:11 +02:00
Lubomír Sedlář
e68e17d8fe pkgset: Check for unused module patterns across all tags
Since there can be multiple tags, the check must be done once for all of
them at the same time. Otherwise any module found only in some and not
all tags would raise this error.

The code builds a set of all existing patterns and then removes items
from it. If there is something left once all tags are processed, it
means such patterns were not matched by anything.

JIRA: COMPOSE-3609
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-13 10:10:30 +02:00
Lubomír Sedlář
4e5f74b78d pkgset: Set correct nsvc for devel modules
We need to update the source modulemd to match what we want. For most
modules the name will be correct already, but in case of devel modules
the original name will still be present there.

Fixes: COMPOSE-3596
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-31 16:02:19 +02:00
Lubomír Sedlář
d0e8472ab5 Remove invalid escape sequences
This emits a warning on Python 3.8. Let's just fix it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-15 15:47:44 +02:00
Lubomír Sedlář
a9b9ec97fb pkgset: Fix whitelist for modules
The prefix checking only works if there are no streams that would share
prefixes. Let's instead check the value as a whole. There is extra
complexity from the fact that version and context may not be specified.

The stream as specified in input is processed to replace dashes (`-`)
with underscores (`_`) to match how the builds are imported into Koji.

JIRA: COMPOSE-3547
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-06 09:48:44 +02:00
Lubomír Sedlář
f858cea466 pkgset: Do not overwrite version in module
This value is important to get correct path to metadata.

JIRA: COMPOSE-3541
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-30 12:22:13 +02:00
Lubomír Sedlář
d7ef86293e tests: Skip tests if libmodulemd is not available
These tests need the library, and will crash without it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-25 10:41:09 +01:00
Lubomír Sedlář
66a127c313 pkgset: Refactor hiding unused modular packages
A module build can create packages that are tagged in the content tag,
but should not be included in the module. Originally Pungi didn't know
what exactly the module contains and so it needed to apply filters to
exclude stuff that was definitely out.

With getting the final MMD from Koji, we can actually make this a bit
more strict by only keeping packages that we know we need.

When processing each content tag, we can put into package set only
packages that are included in some module using that tag. This should
work with -devel modules as well. Both the regular and -devel modules
will contribute to the set and thus all packages will go to the package
set.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-25 08:48:16 +01:00
Lubomír Sedlář
9229699078 pkgset: Load final modulemd files from Koji
Instead of loading the "source" modulemd, always get the final file for
each architecture from Koji.

Logging the downloaded files locally is no longer necessary, when
debugging a problem we can find the files in the respective builds in
Koji.

Tests are updated to work with new code, and an obsolete test is
removed.

JIRA: COMPOSE-3147
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-25 08:48:16 +01:00
Lubomír Sedlář
039b8d44b3 pkgset: Stop loading list of module RPMs
The list is not needed for anything anymore, and it only takes time and
space in memory.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-14 09:08:44 -04:00
Jan Kaluza
31bafa29c5 Allow setting wildcard as a module name in variants to include all the modules.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2019-02-07 13:59:57 +01:00
Jan Kaluza
5fc0f915c6 Get the NSVC from Koji module CG build metadata
Stream can contain dash sign and when MBS imports such NSVC to Koji, the
dash is replaced with underscore. The current Pungi code does not
respect that and tries to use the stream from Koji directly, which
results in wrong stream being using in some Pungi internal data.

In this PR, the NSVC is taken from module metadata section of CG Koji
build, which contains real stream including the dashes.

Merges: https://pagure.io/pungi/pull-request/1072
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-10-23 15:26:37 +02:00