Commit Graph

2868 Commits

Author SHA1 Message Date
Lubomír Sedlář
ff526d1dd2 Regenerate test fixture repo
Make dummy-bash -> dummy-glibc dependency require archful. This avoids
potential race condition where order of dependency processing can result in
different packages being pulled in. The tests where this could happen are
updated.

Make dummy-glibc-debuginfo depend on dummy-glibc-debuginfo-common.

The filenames for the repo no longer include hash, and sqlite databases are not
generated.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:46:21 +02:00
Lubomír Sedlář
8ab7d9f7ba Move import of modulemd to a separate module
This should make it possible to only import the library only when it's
really needed.

DNF does not work with libmodulemd v2. If we import libmodulemd2 and
then dnf, the program will just hang forever. We only need DNF in
pungi-gather, where libmodulemd is not needed, and also where we do need
libmodulemd, we don't have DNF.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:40:31 +02:00
Lubomír Sedlář
f822ee324a ostree: Run commands in universal_newlines mode
This will mean the output is returned as unicode (decoded as UTF-8).
Thus Kobo will not have to do any decoding. This should work around
possible errors with breaking multibyte unicode character sequences into
different chunks.

Relates: https://pagure.io/releng/failed-composes/issue/237
Relates: https://github.com/release-engineering/kobo/issues/119
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 09:30:53 +02:00
Lubomír Sedlář
21d45eb243 pkgset: Allow filtering modules from Koji tags
Add a configuration option to enable skipping some modules found in the
configured tag.

Fixes: https://pagure.io/pungi/issue/1260
JIRA: COMPOSE-3794
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-24 10:13:00 +02:00
Lubomír Sedlář
44e551317a pkgset: Initialize path_prefix to empty value
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-23 09:40:36 +02:00
Lubomír Sedlář
2b112d53f7 Allow loading overrides for module defaults
This patch adds a new config option. This is expected to be a name of
subdirectory in the repo with module defaults. If supplied, overrides
from that location are loaded every time defaults are loaded.

This raises the minimal required version of libmodulemd to 2.8.0

JIRA: COMPOSE-3828
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-23 09:23:39 +02:00
Haibo Lin
260df24859 Move pkgset skipping logs to separate file
There will be a new log file logs/global/excluding-arch.global.log

Fixes: https://pagure.io/pungi/issue/1251
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-09-20 16:02:41 +08:00
Lubomír Sedlář
150f5f0cb6 Fix crash when pkgset phase is skipped
JIRA: COMPOSE-3832
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-18 15:47:16 +02:00
Lubomír Sedlář
3e263f33f2 Resolve symlinks to images
Koji started to create relative symlinks to images created in various
tasks. We need to check for that and potentially link the actual file.

JIRA: COMPOSE-3822
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-11 15:36:28 +02:00
Lubomír Sedlář
e814e1445a 4.1.39 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-09 09:30:45 +02:00
Lubomír Sedlář
2d99edc8d8 Fix getting platforms from lookaside
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-06 11:14:16 +02:00
Lubomír Sedlář
dfd4ff6016 extra-files: Simplify iterating over variants
Finding variants with particular architecture is not simple. Swapping
the two loops makes it faster, and additionally it fixes to correctly
work with sources.

JIRA: COMPOSE-3787
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-04 15:45:15 +02:00
Lubomír Sedlář
b8e41d9b1b createiso: Make media.repo sticky
This means the file will be included on all ISOs when splitting.

JIRA: COMPOSE-3787
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-04 15:45:15 +02:00
Lubomír Sedlář
7aa65f00c2 hybrid: Re-add getting platform from lookaside repos
This is needed when a compose does not include any modules itself, but
is using a lookaside with modules.

JIRA: COMPOSE-3720
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:40:46 +02:00
Lubomír Sedlář
1423105802 pkgset: Remove fast_merge method
This is not called anywhere anymore. Instead of merging these package
sets, they exist as separate entities.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
465ecf229c Add test for materialized pkgset
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
63a8b7b6c9 Remove package whitelist
This was a workaround to make some packages from the global repo
invisible for depsolving. This is now handled by packages being in
different repos. We can select which repos are enabled at which point.
This achieves the same result, but much faster.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
cec3efed51 paths: Remove arch_repo path
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
a426a83ed6 gather: Remove use of arch_repo
The repo was used to speed up creating lookaside repo from a variant.
This uses a similar approach as createrepo phase: selecting the last
available package set and using that data.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
2d0ffb56ca ostree-installer: Remove usage of arch_repo
Pass all pkgset repos as input to the task.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
26ddd46acb ostree: Remove arch_repo path usage
Pass all pkgset repos as input to the task.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
ab9122be2a buildinstall: Remove arch_repo usage
Simply use all existing package set repos as input for the runroot
task. The command line gets a bit long, but the actual behaviour should
remain the same.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
3824eab15b createrepo: Remove arch_repo usage
There is no longer a single repo with all packages. This means that the
metadata has to be loaded from another location.

When taking packages from Koji, we can assume that the non-modular
package tag will be processed last. The repo for this tag will be used.
This has better chance of being useful than using a random module.

For repo sources, there is only one package set anyway, so this change
makes no difference.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
054b91c798 Fix tests
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
11779f6644 pkgset: Update processing file cache
The cache is now always saved when the repo is created on disk. The
loading procedure in Koji source is updated to look for cache for
correct tag.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
fe2df01e8b Remove pickling from source repos
The pickled pkgset was only ever used for debug mode. We don't use that
anymore. Let's stop writing files that are not used anywhere.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
6fa478e688 gather: Port hybrid method
Update the code to consume multiple source repos.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
a446593698 pkgset: Update Koji source to create multiple pkgsets
With this patch, there should be a separate package set for each tag
that is consumed.

Generally each module will create a separate package set, with the
exception of -devel modules that will be in the same set as their
non-devel version.

Variants no longer need to keep their own package set objects. Instead
they now include a set of package set names that should be used for the
variant. This can replace the whitelist mechanism of deps gather method.

JIRA: COMPOSE-3620
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
eed2aa2753 pkgset: Add object representing a package set on disk
Once a package set repo is written to disk, let's use this object that
connects the repository path with the mapping of packages.

This change also makes it explicit where the dependency on package set
repos are. In the original codebase, any part of code could generate a
path to the repo, even if that repo has not yet been written.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
3f665937b2 pkgset: Add name to package set
This name will serve as an identifier for the group of packages.

For Koji package sets, it should the name of the tag from which the
packages come. For package sets based on repos a dummy constant name is
used.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
be61da0192 paths: Add function to generate path to pkgset repo
This will be used to hold a repository for different package sets.

JIRA: COMPOSE-3620
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
90393c4b49 Update code to pass around a list of pkgsets
This opens up a path to having multiple package sets in the compose. The
pkgset phase now creates a list of them (although at this time there is
always a single item in that list).

Any consumer of the package sets objects is updated to handle a list.
Generally this means an extra loop.

JIRA: COMPOSE-3620
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Pat Riehecky
a99bf8c828 Store the parsed variants for possible later re-use
Merges: https://pagure.io/pungi/pull-request/1257
Signed-off-by: Pat Riehecky <riehecky@fnal.gov>
2019-09-02 13:44:24 +02:00
Pat Riehecky
b420986aa4 compose: additional logging on variants being processed
Merges: https://pagure.io/pungi/pull-request/1254
Signed-off-by: Pat Riehecky <riehecky@fnal.gov>
2019-09-02 13:41:22 +02:00
Haibo Lin
53a93f016a Refactor code for removing repoclosure cache
For 'yum' backend, only cache dirs following repoclosure-$COMPOSE_ID-$variant.$arch
name convention are created, e.g. repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64

But for 'dnf' backend, the dir name looks like
repoclosure-$COMPOSE_ID-$variant.$arch-$suffix and there are other files
created, e.g.
    repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64-df9fe164317e314e
    repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64-filenames.solvx
    repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64.solv

JIRA: COMPOSE-2565
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-08-23 11:00:17 +08:00
Haibo Lin
82349626c6 Delete cache dirs even though repoclosure command failed
JIRA: COMPOSE-2565
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-08-22 08:37:53 +00:00
Haibo Lin
29224b02ff Delete lookaside cache dir
If --lookaside option passed to repoclosure command, extra cache dir
will be created and it should be deleted too.

JIRA: COMPOSE-2565
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-08-22 14:24:40 +08:00
Haibo Lin
048698d885 Delete repoclosure cache
JIRA: COMPOSE-2565
Merges: https://pagure.io/pungi/pull-request/1253
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-08-15 08:42:16 +02:00
Lubomír Sedlář
eeaee1c20f Add function to get all arches in a compose
The same logic is used in two places. This deserves a common function.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-08-06 13:30:08 +02:00
Lubomír Sedlář
51d638d5db Remove not-used hack to workaround required option
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-08-06 09:25:29 +02:00
Lubomír Sedlář
e674c2f574 pkgset: Create repos in a single place
Both pkgset sources use the same logic to create per-arch repos. There
is no reason to have that code in both places.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-08-06 09:24:28 +02:00
Lubomír Sedlář
8994aa5d88 pkgset: Added modules to variant in correct format
The data parsed from variants.xml uses a different format that what we
added in `_add_module_to_variant`. This leads to crashes later.

JIRA: COMPOSE-3746
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-08-05 12:06:26 +02:00
Haibo Lin
c00162413c Add compose_id to repoclosure/lookaside directory names
JIRA: COMPOSE-3616
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-08-05 15:57:27 +08:00
Lubomír Sedlář
e9a363bfde Fix shadowed variable
This could result in incorrect log messages being printed.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-24 13:45:27 +02:00
Lubomír Sedlář
27d015543f Remove now unused msg variables
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-24 13:45:00 +02:00
Lubomír Sedlář
6efaae19fd Remove debug mode
This was already discouraged to not be used, and is a bad idea in
current setup anyway. Removing this can simplify the code.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-24 13:36:23 +02:00
Lubomír Sedlář
0891bfbe59 Remove dead code
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-24 12:37:53 +02:00
Lubomír Sedlář
b514e20833 pkgset: Check for empty module index
Instead of testing its return value. Future version of libmodulemd will
raise an exception instead of returning empty data.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-24 07:49:12 +02:00
Lubomír Sedlář
c6d6367932 gather: Mark repos as containing module hotfixes
This should prevent DNF from doing any unwanted magic with the contents.
Ideally we want it to completely ignore the modular metadata in there,
but there seems to be no way to do that.

This should at least prevent it from hiding non-modular packages that
are masked by some default stream.

Fixes: https://pagure.io/pungi/issue/1241
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-22 12:35:36 +02:00
Haibo Lin
3811c0a176 config: Deprecate bootable option
JIRA: COMPOSE-2635
Fixes: https://pagure.io/pungi/issue/976
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-07-22 16:51:38 +08:00