Commit Graph

750 Commits

Author SHA1 Message Date
Haibo Lin
3f111b559f Check dependency of --just-phase
JIRA: COMPOSE-4020
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-06 08:52:47 +00:00
Haibo Lin
794d151bef Remove productimg phase
JIRA: COMPOSE-4004
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-03 11:42:38 +00:00
Jan Kaluza
817acdbbac Allow setting runroot_method per Pungi phase.
The `runroot_method` now accepts `dict` value with phase name as a key
and runroot method as a value. For backward compatibility, the `str`
value is still supported.

The new `global_runroot_method` option has been added which defines
the runroot method in case it is not set in `dict` in the `runroot_method`.

This commit allows running `createiso` phase locally while keeping the other
phases in Koji.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-01-02 14:10:20 +00:00
Haibo Lin
b415e31f9d ostree: Fix arg passed to scm.get_dir_from_scm
A compose object should be passed instead of logger.

JIRA: COMPOSE-4050
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-02 13:13:14 +00:00
Haibo Lin
12828849d6 gather: Improve logging for gathering
Adding arch and variant to log message to make it clearer.

JIRA: COMPOSE-4009
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-12-16 08:56:22 +00:00
Haibo Lin
0ed70fc8b6 tests: drop sys.path modification
It's unnecessary to add the path manually.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-12-10 10:53:38 +08:00
Haibo Lin
3cf16eb42d cleanup: refactor scripts via entry points
Fixes: https://pagure.io/pungi/issue/1045
JIRA: COMPOSE-2946
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-12-10 10:53:31 +08:00
Lubomír Sedlář
aefe9b186d repoclosure: Parse all fus logs
Originally the list of solvables for fus was growing with each iteration
and nothing was ever removed. That later changed so that fus iterations
are only done on newly added stuff. It's great for performance, but
means that the last log is not a superset of all others.

To get all dependency problems we need to look into all log files, not
just the last one.

JIRA: COMPOSE-3964
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-12-02 11:08:55 +01:00
Lubomír Sedlář
51b1144b70 runroot: Log different commands to different files
JIRA: COMPOSE-3980
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-28 15:12:05 +01:00
Haibo Lin
6afbe6d20a buildinstall: Log message for boot config change
JIRA: COMPOSE-3945
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-21 12:16:05 +08:00
Haibo Lin
0c040e0a69 livemedia: Remove title option
Fixes: https://pagure.io/pungi/issue/1293
JIRA: COMPOSE-3947
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-20 15:33:28 +08:00
Haibo Lin
242100eb72 Make sure repoclosure cache cleaned when running as root
When running repoclosure as root user, it will use other dir instead of
the one returned by getCacheDir().

For yum, with --tempcache option could let the cache dir returned by
getCacheDir() always be used.

For dnf, there's no such an option and we have to handle it specially.

JIRA: COMPOSE-3922
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-20 14:13:19 +08:00
Haibo Lin
e0dd20dffe Prefix repoclosure cache dir with compose id
JIRA: COMPOSE-3922
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-20 13:51:13 +08:00
Lubomír Sedlář
c87d299a20 hybrid: Download remote files when getting platform
When probing lookasides for platform definition, we need to make sure it
works for repos specified as HTTP urls. Createrepo doesn't seem to
automatically download the repodata, so we have to help it.

JIRA: COMPOSE-3958
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-19 08:51:18 +01:00
Lubomír Sedlář
4473b05f10 gather: Use fresh cache for each sequence of fus runs
Each depsolved tree will be using its own cache for fus. This should
still allow for faster loading of metadata after first iteration, but
should prevent errors from using cached files meant for another variant
or architecture. The cache is deleted after the last iteration.

JIRA: COMPOSE-3959
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-18 14:39:47 +01:00
Haibo Lin
722411dfcd tests: Add test for compose logger setup
PR#1267 brings in some uncovered code which introduced
an issue fixed by PR#1292.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-12 18:03:13 +08:00
Haibo Lin
2cbd75803a gather: Do not write metadata if gather phase skipped
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-07 16:19:19 +08:00
Lubomír Sedlář
cba3f1c88f Write global metadata about extra files
This is basically collecting all individual extra_files.json and putting
their content into a single location in
compose/metadata/extra_files.json. The file format is part of productmd
1.23.

JIRA: COMPOSE-3831
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-05 09:11:01 +01:00
Lubomír Sedlář
20c3614fb3 scm: Add backend for downloading archives from Koji
Tests and documentation included.

JIRA: COMPOSE-3816
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-05 08:51:26 +01:00
Lubomír Sedlář
39e8f6f710 Pass compose to SCM wrappers
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-05 08:51:26 +01:00
Haibo Lin
b05295c202 Remove misleading warning about size of the ISO
JIRA: COMPOSE-3885
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-04 14:22:59 +08:00
Haibo Lin
114df77c6b Retry watching koji tasks on server outage
Fixes: https://pagure.io/pungi/issue/1285
JIRA: COMPOSE-3896
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-01 15:57:48 +08:00
Haibo Lin
3901c227f0 Fix crash on unsigned packages
JIRA: COMPOSE-3869
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-10-21 17:02:59 +08:00
Owen W. Taylor
72bf795bd4 ostree-install: allow configuring additional depenencies for runroot
A lorax template used for the ostree-installer might need an additional
package dependency (e.g., flatpak to embed a flatpak repository) - add
a config key 'extra_runroot_pkgs' to the ostree installer configuration
to allow supplementing the set of packages installed into the runroot.

Signed-off-by: Owen W. Taylor <otaylor@fishsoup.net>
2019-10-10 15:13:38 -04:00
Lubomír Sedlář
c346492df4 buildinstall: Allow customizing dracut arguments
JIRA: COMPOSE-3853
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-09 15:42:10 +02:00
Lubomír Sedlář
908a6a759d Use custom assertion for checking content of files
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 15:25:44 +02:00
Lubomír Sedlář
562b770b8d Drop custom ANY object
There is already one such thing in mock library.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 15:25:18 +02:00
Lubomír Sedlář
2f54745715 Remove usage of unittest2 on Python 3
It was needed to provide assertItemsEqual method. Starting with Python
3.2, there's assertCountEqual that does the same thing. Six provides a
helper that will dispatch to the existing method. With this change,
unittest 2 is only needed on Python 2.6 to backport the method.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 15:25:18 +02:00
Lubomír Sedlář
43fb06e00f Remove shebangs from test files
They should be executed with a runner like nose, not directly.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 15:24:17 +02:00
Lubomír Sedlář
26962d94ca gather: Resolve dependencies of debug packages
Until now, the behaviour was that all debuginfo from a build would be
included if at least one package with the same arch was included.

This resulted in many debuginfo packages being included even though
their corresponding package was not present.

With this patch, we will always pull in debugsource, but foo-debuginfo
will only be included if foo is included for the same arch. As a
consequence, it is necessary to resolve dependencies of debuginfo
packages. There are cases where foo-debuginfo needs foo-debuginfo-common
for example.

This change means that DNF and YUM backends are no longer identical in
the output. The tests where this is demonstrated are duplicated and
their results are modified.

JIRA: COMPOSE-3823
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:46:21 +02:00
Lubomír Sedlář
ff526d1dd2 Regenerate test fixture repo
Make dummy-bash -> dummy-glibc dependency require archful. This avoids
potential race condition where order of dependency processing can result in
different packages being pulled in. The tests where this could happen are
updated.

Make dummy-glibc-debuginfo depend on dummy-glibc-debuginfo-common.

The filenames for the repo no longer include hash, and sqlite databases are not
generated.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:46:21 +02:00
Lubomír Sedlář
8ab7d9f7ba Move import of modulemd to a separate module
This should make it possible to only import the library only when it's
really needed.

DNF does not work with libmodulemd v2. If we import libmodulemd2 and
then dnf, the program will just hang forever. We only need DNF in
pungi-gather, where libmodulemd is not needed, and also where we do need
libmodulemd, we don't have DNF.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:40:31 +02:00
Lubomír Sedlář
f822ee324a ostree: Run commands in universal_newlines mode
This will mean the output is returned as unicode (decoded as UTF-8).
Thus Kobo will not have to do any decoding. This should work around
possible errors with breaking multibyte unicode character sequences into
different chunks.

Relates: https://pagure.io/releng/failed-composes/issue/237
Relates: https://github.com/release-engineering/kobo/issues/119
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 09:30:53 +02:00
Lubomír Sedlář
21d45eb243 pkgset: Allow filtering modules from Koji tags
Add a configuration option to enable skipping some modules found in the
configured tag.

Fixes: https://pagure.io/pungi/issue/1260
JIRA: COMPOSE-3794
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-24 10:13:00 +02:00
Lubomír Sedlář
2b112d53f7 Allow loading overrides for module defaults
This patch adds a new config option. This is expected to be a name of
subdirectory in the repo with module defaults. If supplied, overrides
from that location are loaded every time defaults are loaded.

This raises the minimal required version of libmodulemd to 2.8.0

JIRA: COMPOSE-3828
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-23 09:23:39 +02:00
Haibo Lin
260df24859 Move pkgset skipping logs to separate file
There will be a new log file logs/global/excluding-arch.global.log

Fixes: https://pagure.io/pungi/issue/1251
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-09-20 16:02:41 +08:00
Lubomír Sedlář
465ecf229c Add test for materialized pkgset
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
63a8b7b6c9 Remove package whitelist
This was a workaround to make some packages from the global repo
invisible for depsolving. This is now handled by packages being in
different repos. We can select which repos are enabled at which point.
This achieves the same result, but much faster.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
a426a83ed6 gather: Remove use of arch_repo
The repo was used to speed up creating lookaside repo from a variant.
This uses a similar approach as createrepo phase: selecting the last
available package set and using that data.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
2d0ffb56ca ostree-installer: Remove usage of arch_repo
Pass all pkgset repos as input to the task.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
26ddd46acb ostree: Remove arch_repo path usage
Pass all pkgset repos as input to the task.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
ab9122be2a buildinstall: Remove arch_repo usage
Simply use all existing package set repos as input for the runroot
task. The command line gets a bit long, but the actual behaviour should
remain the same.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
3824eab15b createrepo: Remove arch_repo usage
There is no longer a single repo with all packages. This means that the
metadata has to be loaded from another location.

When taking packages from Koji, we can assume that the non-modular
package tag will be processed last. The repo for this tag will be used.
This has better chance of being useful than using a random module.

For repo sources, there is only one package set anyway, so this change
makes no difference.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
054b91c798 Fix tests
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
a446593698 pkgset: Update Koji source to create multiple pkgsets
With this patch, there should be a separate package set for each tag
that is consumed.

Generally each module will create a separate package set, with the
exception of -devel modules that will be in the same set as their
non-devel version.

Variants no longer need to keep their own package set objects. Instead
they now include a set of package set names that should be used for the
variant. This can replace the whitelist mechanism of deps gather method.

JIRA: COMPOSE-3620
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
eed2aa2753 pkgset: Add object representing a package set on disk
Once a package set repo is written to disk, let's use this object that
connects the repository path with the mapping of packages.

This change also makes it explicit where the dependency on package set
repos are. In the original codebase, any part of code could generate a
path to the repo, even if that repo has not yet been written.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
3f665937b2 pkgset: Add name to package set
This name will serve as an identifier for the group of packages.

For Koji package sets, it should the name of the tag from which the
packages come. For package sets based on repos a dummy constant name is
used.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
90393c4b49 Update code to pass around a list of pkgsets
This opens up a path to having multiple package sets in the compose. The
pkgset phase now creates a list of them (although at this time there is
always a single item in that list).

Any consumer of the package sets objects is updated to handle a list.
Generally this means an extra loop.

JIRA: COMPOSE-3620
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Haibo Lin
53a93f016a Refactor code for removing repoclosure cache
For 'yum' backend, only cache dirs following repoclosure-$COMPOSE_ID-$variant.$arch
name convention are created, e.g. repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64

But for 'dnf' backend, the dir name looks like
repoclosure-$COMPOSE_ID-$variant.$arch-$suffix and there are other files
created, e.g.
    repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64-df9fe164317e314e
    repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64-filenames.solvx
    repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64.solv

JIRA: COMPOSE-2565
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-08-23 11:00:17 +08:00
Haibo Lin
048698d885 Delete repoclosure cache
JIRA: COMPOSE-2565
Merges: https://pagure.io/pungi/pull-request/1253
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-08-15 08:42:16 +02:00