Commit Graph

150 Commits

Author SHA1 Message Date
Haibo Lin 41a629969c Format code base with black
https://black.readthedocs.io/en/stable/

JIRA: COMPOSE-4086
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-05 17:35:47 +08:00
Haibo Lin 52f82ccc6e pkgset: Ignore deleted module builds
JIRA: COMPOSE-4058
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-14 16:23:55 +08:00
Haibo Lin 01ab1d2e24 Hide latest koji package via pkgset_koji_builds option
JIRA: COMPOSE-4010
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-09 07:33:46 +00:00
Lubomír Sedlář 02ace28fe4 pkgset: Hardlink downloaded packages
When pungi-gather (or old pungi) download the packages from repos into
work/$arch/, they are linked to work/global/. This was using link_type
configuration option.

However if that is set to symlink, we get a relative symlink in
work/global/ which is later copied under the compose/ directory. Since
it's a relative symlink, it gets broken by this.

The fix is to hardlink the downloaded packages instead in the first
step. Since both the source and destination are in work/ directory, we
know it's the same volume and hardlinks should work. There is a fallback
to copy just to be sure, but it shouldn't ever be used.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-06 10:49:25 +01:00
Lubomír Sedlář d34b0d7900 pkgset: Fix running without any koji tags
All places in the code assume the option to not be required except for
this one line.

An obsolete comment is removed as well.

Relates: https://pagure.io/releng/failed-composes/issue/477
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-11 09:30:37 +01:00
Lubomír Sedlář 6771a21916 pkgset: Only reuse valid old repo
Instead of just checking that the repo directory exists, make sure the
repodata subdirectory is in there. If it's missing, then createrepo_c
has nothing to use anyway, and it may help avoid issues.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-22 15:32:13 +02:00
Lubomír Sedlář 8ab7d9f7ba Move import of modulemd to a separate module
This should make it possible to only import the library only when it's
really needed.

DNF does not work with libmodulemd v2. If we import libmodulemd2 and
then dnf, the program will just hang forever. We only need DNF in
pungi-gather, where libmodulemd is not needed, and also where we do need
libmodulemd, we don't have DNF.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:40:31 +02:00
Lubomír Sedlář 21d45eb243 pkgset: Allow filtering modules from Koji tags
Add a configuration option to enable skipping some modules found in the
configured tag.

Fixes: https://pagure.io/pungi/issue/1260
JIRA: COMPOSE-3794
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-24 10:13:00 +02:00
Lubomír Sedlář 44e551317a pkgset: Initialize path_prefix to empty value
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-23 09:40:36 +02:00
Lubomír Sedlář 2b112d53f7 Allow loading overrides for module defaults
This patch adds a new config option. This is expected to be a name of
subdirectory in the repo with module defaults. If supplied, overrides
from that location are loaded every time defaults are loaded.

This raises the minimal required version of libmodulemd to 2.8.0

JIRA: COMPOSE-3828
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-23 09:23:39 +02:00
Lubomír Sedlář 150f5f0cb6 Fix crash when pkgset phase is skipped
JIRA: COMPOSE-3832
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-18 15:47:16 +02:00
Lubomír Sedlář 1423105802 pkgset: Remove fast_merge method
This is not called anywhere anymore. Instead of merging these package
sets, they exist as separate entities.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář 465ecf229c Add test for materialized pkgset
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář 054b91c798 Fix tests
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář 11779f6644 pkgset: Update processing file cache
The cache is now always saved when the repo is created on disk. The
loading procedure in Koji source is updated to look for cache for
correct tag.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář fe2df01e8b Remove pickling from source repos
The pickled pkgset was only ever used for debug mode. We don't use that
anymore. Let's stop writing files that are not used anywhere.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář a446593698 pkgset: Update Koji source to create multiple pkgsets
With this patch, there should be a separate package set for each tag
that is consumed.

Generally each module will create a separate package set, with the
exception of -devel modules that will be in the same set as their
non-devel version.

Variants no longer need to keep their own package set objects. Instead
they now include a set of package set names that should be used for the
variant. This can replace the whitelist mechanism of deps gather method.

JIRA: COMPOSE-3620
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář eed2aa2753 pkgset: Add object representing a package set on disk
Once a package set repo is written to disk, let's use this object that
connects the repository path with the mapping of packages.

This change also makes it explicit where the dependency on package set
repos are. In the original codebase, any part of code could generate a
path to the repo, even if that repo has not yet been written.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář 3f665937b2 pkgset: Add name to package set
This name will serve as an identifier for the group of packages.

For Koji package sets, it should the name of the tag from which the
packages come. For package sets based on repos a dummy constant name is
used.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář 90393c4b49 Update code to pass around a list of pkgsets
This opens up a path to having multiple package sets in the compose. The
pkgset phase now creates a list of them (although at this time there is
always a single item in that list).

Any consumer of the package sets objects is updated to handle a list.
Generally this means an extra loop.

JIRA: COMPOSE-3620
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář eeaee1c20f Add function to get all arches in a compose
The same logic is used in two places. This deserves a common function.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-08-06 13:30:08 +02:00
Lubomír Sedlář 51d638d5db Remove not-used hack to workaround required option
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-08-06 09:25:29 +02:00
Lubomír Sedlář e674c2f574 pkgset: Create repos in a single place
Both pkgset sources use the same logic to create per-arch repos. There
is no reason to have that code in both places.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-08-06 09:24:28 +02:00
Lubomír Sedlář 8994aa5d88 pkgset: Added modules to variant in correct format
The data parsed from variants.xml uses a different format that what we
added in `_add_module_to_variant`. This leads to crashes later.

JIRA: COMPOSE-3746
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-08-05 12:06:26 +02:00
Lubomír Sedlář 27d015543f Remove now unused msg variables
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-24 13:45:00 +02:00
Lubomír Sedlář 6efaae19fd Remove debug mode
This was already discouraged to not be used, and is a bad idea in
current setup anyway. Removing this can simplify the code.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-24 13:36:23 +02:00
Lubomír Sedlář 0891bfbe59 Remove dead code
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-24 12:37:53 +02:00
Haibo Lin 3811c0a176 config: Deprecate bootable option
JIRA: COMPOSE-2635
Fixes: https://pagure.io/pungi/issue/976
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-07-22 16:51:38 +08:00
Lubomír Sedlář 61e3cb0ef1 Port to libmodulemd v2
Fixes: https://pagure.io/pungi/issue/1225
JIRA: COMPOSE-3662
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-15 15:27:05 +02:00
Lubomír Sedlář da78b99fc0 pkgset: Load modulemd only when needed
We can avoid parsing source modulemd information since we can get the
same information from the Koji build info.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-11 14:25:42 +02:00
Lubomír Sedlář 04baa2a4db Stop storing modulemd without arch
Historically each variant had a list of modules. This is no longer
needed and can be dropped. We can also stop logging the modulemd since
we know it was retrieved from Koji and not modified locally.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-11 14:25:42 +02:00
Lubomír Sedlář a31fe998d5 Remove unused import
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-11 14:25:42 +02:00
Lubomír Sedlář 49d0ab797c pkgset: Include module metadata in the repos
If the package set repo contains any modular package, the module
metadata is added there as well.

This is needed to accomodate change in DNF that refuses to work with
repo with modular packages if the metadata is not there. This DNF change
can cause issues in buildinstall phase.

Related: https://bugzilla.redhat.com/show_bug.cgi?id=1623128

The hybrid solver is modified to not create a separate repo with the
module metadata anymore, since it will be available in the repo with
packages. This also allows us to drop code to look into lookaside repos.

We still need to iterate over local modules in order to find out what
platform should be used.

JIRA: COMPOSE-3621
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-01 09:09:17 +02:00
Lubomír Sedlář 82580ed5b3 pkgset: Create arch repos in parallel
This patch reuses the existing createrepo_num_threads options to limit
maximum number of parallel createrepo processes.

Fixes: https://pagure.io/pungi/issue/955
JIRA: COMPOSE-2575
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-19 15:42:46 +02:00
Lubomír Sedlář 2c3e6a5a74 pkgset: Use highest pickle protocol
Higher protocols should be more efficient in terms of performance and
storage size. Since we don't really care about interoperability with
different python version, we can safely go to the highest version.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-18 09:33:11 +02:00
Lubomír Sedlář e68e17d8fe pkgset: Check for unused module patterns across all tags
Since there can be multiple tags, the check must be done once for all of
them at the same time. Otherwise any module found only in some and not
all tags would raise this error.

The code builds a set of all existing patterns and then removes items
from it. If there is something left once all tags are processed, it
means such patterns were not matched by anything.

JIRA: COMPOSE-3609
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-13 10:10:30 +02:00
Lubomír Sedlář 2a9490526a pkgset: Make serialization more resilient
If the package set is loaded from a pickle, it will not contain this
key, and thus it can not be again written down into a new pickle. While
this is not a common use case, it's still better not to crash in that
case.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-11 08:45:16 +02:00
Lubomír Sedlář 4e5f74b78d pkgset: Set correct nsvc for devel modules
We need to update the source modulemd to match what we want. For most
modules the name will be correct already, but in case of devel modules
the original name will still be present there.

Fixes: COMPOSE-3596
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-31 16:02:19 +02:00
Lubomír Sedlář 33471c38bb pkgset: Ignore modules without metadata in Koji
This is fairly similar to a package only being built for particular
arches.

Fixes: https://pagure.io/pungi/issue/1198
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-23 13:51:15 +02:00
Lubomír Sedlář d0e8472ab5 Remove invalid escape sequences
This emits a warning on Python 3.8. Let's just fix it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-15 15:47:44 +02:00
Lubomír Sedlář a9b9ec97fb pkgset: Fix whitelist for modules
The prefix checking only works if there are no streams that would share
prefixes. Let's instead check the value as a whole. There is extra
complexity from the fact that version and context may not be specified.

The stream as specified in input is processed to replace dashes (`-`)
with underscores (`_`) to match how the builds are imported into Koji.

JIRA: COMPOSE-3547
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-06 09:48:44 +02:00
Lubomír Sedlář 2ae742af04 pkgset: Fix filtering excluded modular packages
For modular tags we only include packages that are included in the
module. Originally the filter was taking package names from
rpms.artifacts section of MMD.

This however does not work correctly, as there can easily be module
which lists foo.src but does not want to include foo.x86_64 or other
arches.

This patch fixes this particular problem by included arch in the set of
packages to be kept.

JIRA: COMPOSE-3543
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-02 13:22:13 +02:00
Lubomír Sedlář f858cea466 pkgset: Do not overwrite version in module
This value is important to get correct path to metadata.

JIRA: COMPOSE-3541
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-30 12:22:13 +02:00
Lubomír Sedlář b73d2d7f11 pkgset: Treat modular version as number for sorting
The scheme for generating versions has changed multiple times. MBS is
careful to only modify them so that they always compare correctly. This
only works though if the versions are treated as numbers.

This should be safe in that non-numbers should never be encountered as
module version. Libmodulemd internally stores the version as int (or
some version of int).

JIRA: COMPOSE-3540
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-26 14:51:24 +02:00
Lubomír Sedlář 1f95c33e2a gather: Use wildcard for repo selection
If package source is set to repos, honor wildcard and apply that block
every time.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-01 09:49:53 +02:00
Lubomír Sedlář 66a127c313 pkgset: Refactor hiding unused modular packages
A module build can create packages that are tagged in the content tag,
but should not be included in the module. Originally Pungi didn't know
what exactly the module contains and so it needed to apply filters to
exclude stuff that was definitely out.

With getting the final MMD from Koji, we can actually make this a bit
more strict by only keeping packages that we know we need.

When processing each content tag, we can put into package set only
packages that are included in some module using that tag. This should
work with -devel modules as well. Both the regular and -devel modules
will contribute to the set and thus all packages will go to the package
set.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-25 08:48:16 +01:00
Lubomír Sedlář 9f9b784e64 Remove configuration for devel modules
They should be loaded in the same way as regular modules straight from
Koji.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-25 08:48:16 +01:00
Lubomír Sedlář 9229699078 pkgset: Load final modulemd files from Koji
Instead of loading the "source" modulemd, always get the final file for
each architecture from Koji.

Logging the downloaded files locally is no longer necessary, when
debugging a problem we can find the files in the respective builds in
Koji.

Tests are updated to work with new code, and an obsolete test is
removed.

JIRA: COMPOSE-3147
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-25 08:48:16 +01:00
Lubomír Sedlář 039b8d44b3 pkgset: Stop loading list of module RPMs
The list is not needed for anything anymore, and it only takes time and
space in memory.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-14 09:08:44 -04:00
Lubomír Sedlář 0fc797a06f pkgset: Only load cache once
The file in old compose does not change, there is no need to load it
again for each tag.

One consequence of this change is that the same cache will be used by
all package sets. They add stuff to the cache as they use it. However
that should not be a problem for the same reason it's okay to use the
cache in the first place. If two packages have the same path, they are
actually the same file and it's okay to reuse the data.

JIRA: COMPOSE-3374
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-12 08:57:35 -04:00