Commit Graph

2533 Commits

Author SHA1 Message Date
Haibo Lin
3901c227f0 Fix crash on unsigned packages
JIRA: COMPOSE-3869
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-10-21 17:02:59 +08:00
Juliana Rodrigueiro
f8dcda9dcb doc: Make it clear that the field 'repo' has to be empty for 'file' backend
Leaving the field 'repo' out for the 'file' backend makes the compose fail.

    [ERROR   ] Compose run failed: 'repo'

Signed-off-by: Juliana Rodrigueiro <juliana.rodrigueiro@intra2net.com>
2019-10-18 10:30:25 +02:00
Haibo Lin
2ff1f2fac3 gather: Make depsolving parallel
JIRA: COMPOSE-3827
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-10-18 14:20:32 +08:00
Lubomír Sedlář
014560f8bd doc: Add full examples of compose configuration
JIRA: COMPOSE-3797
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-14 11:04:39 +02:00
Owen W. Taylor
72bf795bd4 ostree-install: allow configuring additional depenencies for runroot
A lorax template used for the ostree-installer might need an additional
package dependency (e.g., flatpak to embed a flatpak repository) - add
a config key 'extra_runroot_pkgs' to the ostree installer configuration
to allow supplementing the set of packages installed into the runroot.

Signed-off-by: Owen W. Taylor <otaylor@fishsoup.net>
2019-10-10 15:13:38 -04:00
Lubomír Sedlář
c346492df4 buildinstall: Allow customizing dracut arguments
JIRA: COMPOSE-3853
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-09 15:42:10 +02:00
Lubomír Sedlář
908a6a759d Use custom assertion for checking content of files
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 15:25:44 +02:00
Lubomír Sedlář
562b770b8d Drop custom ANY object
There is already one such thing in mock library.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 15:25:18 +02:00
Lubomír Sedlář
2f54745715 Remove usage of unittest2 on Python 3
It was needed to provide assertItemsEqual method. Starting with Python
3.2, there's assertCountEqual that does the same thing. Six provides a
helper that will dispatch to the existing method. With this change,
unittest 2 is only needed on Python 2.6 to backport the method.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 15:25:18 +02:00
Lubomír Sedlář
43fb06e00f Remove shebangs from test files
They should be executed with a runner like nose, not directly.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 15:24:17 +02:00
Lubomír Sedlář
26962d94ca gather: Resolve dependencies of debug packages
Until now, the behaviour was that all debuginfo from a build would be
included if at least one package with the same arch was included.

This resulted in many debuginfo packages being included even though
their corresponding package was not present.

With this patch, we will always pull in debugsource, but foo-debuginfo
will only be included if foo is included for the same arch. As a
consequence, it is necessary to resolve dependencies of debuginfo
packages. There are cases where foo-debuginfo needs foo-debuginfo-common
for example.

This change means that DNF and YUM backends are no longer identical in
the output. The tests where this is demonstrated are duplicated and
their results are modified.

JIRA: COMPOSE-3823
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:46:21 +02:00
Lubomír Sedlář
4413b0df24 gather: Fix bare except block
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:46:21 +02:00
Lubomír Sedlář
89fcb79aca gather: Fix formatting
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:46:21 +02:00
Lubomír Sedlář
428f8297e1 gather: Simplify adding packages to result
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:46:21 +02:00
Lubomír Sedlář
c7f3f38822 gather: Remove unused import
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:46:21 +02:00
Lubomír Sedlář
ff526d1dd2 Regenerate test fixture repo
Make dummy-bash -> dummy-glibc dependency require archful. This avoids
potential race condition where order of dependency processing can result in
different packages being pulled in. The tests where this could happen are
updated.

Make dummy-glibc-debuginfo depend on dummy-glibc-debuginfo-common.

The filenames for the repo no longer include hash, and sqlite databases are not
generated.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:46:21 +02:00
Lubomír Sedlář
8ab7d9f7ba Move import of modulemd to a separate module
This should make it possible to only import the library only when it's
really needed.

DNF does not work with libmodulemd v2. If we import libmodulemd2 and
then dnf, the program will just hang forever. We only need DNF in
pungi-gather, where libmodulemd is not needed, and also where we do need
libmodulemd, we don't have DNF.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:40:31 +02:00
Lubomír Sedlář
f822ee324a ostree: Run commands in universal_newlines mode
This will mean the output is returned as unicode (decoded as UTF-8).
Thus Kobo will not have to do any decoding. This should work around
possible errors with breaking multibyte unicode character sequences into
different chunks.

Relates: https://pagure.io/releng/failed-composes/issue/237
Relates: https://github.com/release-engineering/kobo/issues/119
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 09:30:53 +02:00
Lubomír Sedlář
21d45eb243 pkgset: Allow filtering modules from Koji tags
Add a configuration option to enable skipping some modules found in the
configured tag.

Fixes: https://pagure.io/pungi/issue/1260
JIRA: COMPOSE-3794
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-24 10:13:00 +02:00
Lubomír Sedlář
44e551317a pkgset: Initialize path_prefix to empty value
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-23 09:40:36 +02:00
Lubomír Sedlář
2b112d53f7 Allow loading overrides for module defaults
This patch adds a new config option. This is expected to be a name of
subdirectory in the repo with module defaults. If supplied, overrides
from that location are loaded every time defaults are loaded.

This raises the minimal required version of libmodulemd to 2.8.0

JIRA: COMPOSE-3828
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-23 09:23:39 +02:00
Haibo Lin
260df24859 Move pkgset skipping logs to separate file
There will be a new log file logs/global/excluding-arch.global.log

Fixes: https://pagure.io/pungi/issue/1251
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-09-20 16:02:41 +08:00
Lubomír Sedlář
150f5f0cb6 Fix crash when pkgset phase is skipped
JIRA: COMPOSE-3832
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-18 15:47:16 +02:00
Lubomír Sedlář
3e263f33f2 Resolve symlinks to images
Koji started to create relative symlinks to images created in various
tasks. We need to check for that and potentially link the actual file.

JIRA: COMPOSE-3822
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-11 15:36:28 +02:00
Lubomír Sedlář
e814e1445a 4.1.39 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-09 09:30:45 +02:00
Lubomír Sedlář
2d99edc8d8 Fix getting platforms from lookaside
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-06 11:14:16 +02:00
Lubomír Sedlář
dfd4ff6016 extra-files: Simplify iterating over variants
Finding variants with particular architecture is not simple. Swapping
the two loops makes it faster, and additionally it fixes to correctly
work with sources.

JIRA: COMPOSE-3787
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-04 15:45:15 +02:00
Lubomír Sedlář
b8e41d9b1b createiso: Make media.repo sticky
This means the file will be included on all ISOs when splitting.

JIRA: COMPOSE-3787
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-04 15:45:15 +02:00
Lubomír Sedlář
7aa65f00c2 hybrid: Re-add getting platform from lookaside repos
This is needed when a compose does not include any modules itself, but
is using a lookaside with modules.

JIRA: COMPOSE-3720
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:40:46 +02:00
Lubomír Sedlář
1423105802 pkgset: Remove fast_merge method
This is not called anywhere anymore. Instead of merging these package
sets, they exist as separate entities.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
465ecf229c Add test for materialized pkgset
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
63a8b7b6c9 Remove package whitelist
This was a workaround to make some packages from the global repo
invisible for depsolving. This is now handled by packages being in
different repos. We can select which repos are enabled at which point.
This achieves the same result, but much faster.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
cec3efed51 paths: Remove arch_repo path
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
a426a83ed6 gather: Remove use of arch_repo
The repo was used to speed up creating lookaside repo from a variant.
This uses a similar approach as createrepo phase: selecting the last
available package set and using that data.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
2d0ffb56ca ostree-installer: Remove usage of arch_repo
Pass all pkgset repos as input to the task.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
26ddd46acb ostree: Remove arch_repo path usage
Pass all pkgset repos as input to the task.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
ab9122be2a buildinstall: Remove arch_repo usage
Simply use all existing package set repos as input for the runroot
task. The command line gets a bit long, but the actual behaviour should
remain the same.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
3824eab15b createrepo: Remove arch_repo usage
There is no longer a single repo with all packages. This means that the
metadata has to be loaded from another location.

When taking packages from Koji, we can assume that the non-modular
package tag will be processed last. The repo for this tag will be used.
This has better chance of being useful than using a random module.

For repo sources, there is only one package set anyway, so this change
makes no difference.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
054b91c798 Fix tests
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
11779f6644 pkgset: Update processing file cache
The cache is now always saved when the repo is created on disk. The
loading procedure in Koji source is updated to look for cache for
correct tag.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
fe2df01e8b Remove pickling from source repos
The pickled pkgset was only ever used for debug mode. We don't use that
anymore. Let's stop writing files that are not used anywhere.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
6fa478e688 gather: Port hybrid method
Update the code to consume multiple source repos.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
a446593698 pkgset: Update Koji source to create multiple pkgsets
With this patch, there should be a separate package set for each tag
that is consumed.

Generally each module will create a separate package set, with the
exception of -devel modules that will be in the same set as their
non-devel version.

Variants no longer need to keep their own package set objects. Instead
they now include a set of package set names that should be used for the
variant. This can replace the whitelist mechanism of deps gather method.

JIRA: COMPOSE-3620
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
eed2aa2753 pkgset: Add object representing a package set on disk
Once a package set repo is written to disk, let's use this object that
connects the repository path with the mapping of packages.

This change also makes it explicit where the dependency on package set
repos are. In the original codebase, any part of code could generate a
path to the repo, even if that repo has not yet been written.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
3f665937b2 pkgset: Add name to package set
This name will serve as an identifier for the group of packages.

For Koji package sets, it should the name of the tag from which the
packages come. For package sets based on repos a dummy constant name is
used.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
be61da0192 paths: Add function to generate path to pkgset repo
This will be used to hold a repository for different package sets.

JIRA: COMPOSE-3620
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
90393c4b49 Update code to pass around a list of pkgsets
This opens up a path to having multiple package sets in the compose. The
pkgset phase now creates a list of them (although at this time there is
always a single item in that list).

Any consumer of the package sets objects is updated to handle a list.
Generally this means an extra loop.

JIRA: COMPOSE-3620
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Pat Riehecky
a99bf8c828 Store the parsed variants for possible later re-use
Merges: https://pagure.io/pungi/pull-request/1257
Signed-off-by: Pat Riehecky <riehecky@fnal.gov>
2019-09-02 13:44:24 +02:00
Pat Riehecky
b420986aa4 compose: additional logging on variants being processed
Merges: https://pagure.io/pungi/pull-request/1254
Signed-off-by: Pat Riehecky <riehecky@fnal.gov>
2019-09-02 13:41:22 +02:00
Haibo Lin
53a93f016a Refactor code for removing repoclosure cache
For 'yum' backend, only cache dirs following repoclosure-$COMPOSE_ID-$variant.$arch
name convention are created, e.g. repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64

But for 'dnf' backend, the dir name looks like
repoclosure-$COMPOSE_ID-$variant.$arch-$suffix and there are other files
created, e.g.
    repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64-df9fe164317e314e
    repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64-filenames.solvx
    repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64.solv

JIRA: COMPOSE-2565
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-08-23 11:00:17 +08:00