Commit Graph

758 Commits

Author SHA1 Message Date
Lubomír Sedlář
46ea0743a9 iso: Clean up cache for guestmount
When partial cleanup messes up the guestfs cache, the call to guestmount
will fail. To fix that, let's check if there is a problem first and
clean up everything if needed.

Relates: https://bugzilla.redhat.com/show_bug.cgi?id=1771976
JIRA: COMPOSE-3932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-30 10:35:42 +01:00
Lubomír Sedlář
4c6396f491 Remove deprecated warn() call
Instead warning() should be called.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-29 15:34:00 +01:00
Jan Kaluza
3cde5c3a87 Add support for new Pungi Buildinstall Koji plugin.
We would like to start generating the buildinstall phase using the safer
Koji Pungi Buildinstall plugin and stop the direct use of Runroot plugin.

The plugin so far exists only as PR for Koji:
https://pagure.io/koji/pull-request/1939

This commit adds support for this plugin when `lorax_use_koji_plugin`
is set to `True`.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-01-23 16:09:15 +01:00
Haibo Lin
6eb6511aa6 Make --task-id mandatory in get_runroot_cmd
JIRA: COMPOSE-4027
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-20 12:56:01 +08:00
Haibo Lin
bce57c2e66 Wait on runroot tasks with wait-task command
JIRA: COMPOSE-4027
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-19 17:56:31 +08:00
Haibo Lin
52f82ccc6e pkgset: Ignore deleted module builds
JIRA: COMPOSE-4058
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-14 16:23:55 +08:00
Lubomír Sedlář
e7a58ccd07 Add tests for new exclude options
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-13 10:54:11 +01:00
Jan Kaluza
fcf1442f71 Support generating ISOs when using link_type="symlink".
When `link_type = "symlink"` is used, the packages are in fact symlinks
to /mnt/koji. When graft points file is generated, the paths in this graft
points file point to symlinks and therefore symlinks are copied into the
generated ISO file instead of real files.

In this commit, the code to generate the graft points file is changed
so it resolves the symlink to real file stored on /mnt/koji. To make
this code safer, it does such resolving only in case the symlink points
outside of `compose.paths.compose.topdir()`. Therefore you can still
generate ISO file with symlink pointing to file stored within the ISO
file itself, although this is not done currently afaik.

The main reason for this is to be able to generate ISO files even
without hardlinks (which would need read-write access on /mnt/koji)
and without copying all the packages from /mnt/koji to local storage.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-01-06 12:30:13 +00:00
Haibo Lin
3f111b559f Check dependency of --just-phase
JIRA: COMPOSE-4020
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-06 08:52:47 +00:00
Haibo Lin
794d151bef Remove productimg phase
JIRA: COMPOSE-4004
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-03 11:42:38 +00:00
Jan Kaluza
817acdbbac Allow setting runroot_method per Pungi phase.
The `runroot_method` now accepts `dict` value with phase name as a key
and runroot method as a value. For backward compatibility, the `str`
value is still supported.

The new `global_runroot_method` option has been added which defines
the runroot method in case it is not set in `dict` in the `runroot_method`.

This commit allows running `createiso` phase locally while keeping the other
phases in Koji.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-01-02 14:10:20 +00:00
Haibo Lin
b415e31f9d ostree: Fix arg passed to scm.get_dir_from_scm
A compose object should be passed instead of logger.

JIRA: COMPOSE-4050
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-02 13:13:14 +00:00
Haibo Lin
12828849d6 gather: Improve logging for gathering
Adding arch and variant to log message to make it clearer.

JIRA: COMPOSE-4009
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-12-16 08:56:22 +00:00
Haibo Lin
0ed70fc8b6 tests: drop sys.path modification
It's unnecessary to add the path manually.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-12-10 10:53:38 +08:00
Haibo Lin
3cf16eb42d cleanup: refactor scripts via entry points
Fixes: https://pagure.io/pungi/issue/1045
JIRA: COMPOSE-2946
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-12-10 10:53:31 +08:00
Lubomír Sedlář
aefe9b186d repoclosure: Parse all fus logs
Originally the list of solvables for fus was growing with each iteration
and nothing was ever removed. That later changed so that fus iterations
are only done on newly added stuff. It's great for performance, but
means that the last log is not a superset of all others.

To get all dependency problems we need to look into all log files, not
just the last one.

JIRA: COMPOSE-3964
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-12-02 11:08:55 +01:00
Lubomír Sedlář
51b1144b70 runroot: Log different commands to different files
JIRA: COMPOSE-3980
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-28 15:12:05 +01:00
Haibo Lin
6afbe6d20a buildinstall: Log message for boot config change
JIRA: COMPOSE-3945
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-21 12:16:05 +08:00
Haibo Lin
0c040e0a69 livemedia: Remove title option
Fixes: https://pagure.io/pungi/issue/1293
JIRA: COMPOSE-3947
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-20 15:33:28 +08:00
Haibo Lin
242100eb72 Make sure repoclosure cache cleaned when running as root
When running repoclosure as root user, it will use other dir instead of
the one returned by getCacheDir().

For yum, with --tempcache option could let the cache dir returned by
getCacheDir() always be used.

For dnf, there's no such an option and we have to handle it specially.

JIRA: COMPOSE-3922
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-20 14:13:19 +08:00
Haibo Lin
e0dd20dffe Prefix repoclosure cache dir with compose id
JIRA: COMPOSE-3922
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-20 13:51:13 +08:00
Lubomír Sedlář
c87d299a20 hybrid: Download remote files when getting platform
When probing lookasides for platform definition, we need to make sure it
works for repos specified as HTTP urls. Createrepo doesn't seem to
automatically download the repodata, so we have to help it.

JIRA: COMPOSE-3958
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-19 08:51:18 +01:00
Lubomír Sedlář
4473b05f10 gather: Use fresh cache for each sequence of fus runs
Each depsolved tree will be using its own cache for fus. This should
still allow for faster loading of metadata after first iteration, but
should prevent errors from using cached files meant for another variant
or architecture. The cache is deleted after the last iteration.

JIRA: COMPOSE-3959
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-18 14:39:47 +01:00
Haibo Lin
722411dfcd tests: Add test for compose logger setup
PR#1267 brings in some uncovered code which introduced
an issue fixed by PR#1292.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-12 18:03:13 +08:00
Haibo Lin
2cbd75803a gather: Do not write metadata if gather phase skipped
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-07 16:19:19 +08:00
Lubomír Sedlář
cba3f1c88f Write global metadata about extra files
This is basically collecting all individual extra_files.json and putting
their content into a single location in
compose/metadata/extra_files.json. The file format is part of productmd
1.23.

JIRA: COMPOSE-3831
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-05 09:11:01 +01:00
Lubomír Sedlář
20c3614fb3 scm: Add backend for downloading archives from Koji
Tests and documentation included.

JIRA: COMPOSE-3816
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-05 08:51:26 +01:00
Lubomír Sedlář
39e8f6f710 Pass compose to SCM wrappers
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-05 08:51:26 +01:00
Haibo Lin
b05295c202 Remove misleading warning about size of the ISO
JIRA: COMPOSE-3885
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-04 14:22:59 +08:00
Haibo Lin
114df77c6b Retry watching koji tasks on server outage
Fixes: https://pagure.io/pungi/issue/1285
JIRA: COMPOSE-3896
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-01 15:57:48 +08:00
Haibo Lin
3901c227f0 Fix crash on unsigned packages
JIRA: COMPOSE-3869
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-10-21 17:02:59 +08:00
Owen W. Taylor
72bf795bd4 ostree-install: allow configuring additional depenencies for runroot
A lorax template used for the ostree-installer might need an additional
package dependency (e.g., flatpak to embed a flatpak repository) - add
a config key 'extra_runroot_pkgs' to the ostree installer configuration
to allow supplementing the set of packages installed into the runroot.

Signed-off-by: Owen W. Taylor <otaylor@fishsoup.net>
2019-10-10 15:13:38 -04:00
Lubomír Sedlář
c346492df4 buildinstall: Allow customizing dracut arguments
JIRA: COMPOSE-3853
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-09 15:42:10 +02:00
Lubomír Sedlář
908a6a759d Use custom assertion for checking content of files
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 15:25:44 +02:00
Lubomír Sedlář
562b770b8d Drop custom ANY object
There is already one such thing in mock library.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 15:25:18 +02:00
Lubomír Sedlář
2f54745715 Remove usage of unittest2 on Python 3
It was needed to provide assertItemsEqual method. Starting with Python
3.2, there's assertCountEqual that does the same thing. Six provides a
helper that will dispatch to the existing method. With this change,
unittest 2 is only needed on Python 2.6 to backport the method.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 15:25:18 +02:00
Lubomír Sedlář
43fb06e00f Remove shebangs from test files
They should be executed with a runner like nose, not directly.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 15:24:17 +02:00
Lubomír Sedlář
26962d94ca gather: Resolve dependencies of debug packages
Until now, the behaviour was that all debuginfo from a build would be
included if at least one package with the same arch was included.

This resulted in many debuginfo packages being included even though
their corresponding package was not present.

With this patch, we will always pull in debugsource, but foo-debuginfo
will only be included if foo is included for the same arch. As a
consequence, it is necessary to resolve dependencies of debuginfo
packages. There are cases where foo-debuginfo needs foo-debuginfo-common
for example.

This change means that DNF and YUM backends are no longer identical in
the output. The tests where this is demonstrated are duplicated and
their results are modified.

JIRA: COMPOSE-3823
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:46:21 +02:00
Lubomír Sedlář
ff526d1dd2 Regenerate test fixture repo
Make dummy-bash -> dummy-glibc dependency require archful. This avoids
potential race condition where order of dependency processing can result in
different packages being pulled in. The tests where this could happen are
updated.

Make dummy-glibc-debuginfo depend on dummy-glibc-debuginfo-common.

The filenames for the repo no longer include hash, and sqlite databases are not
generated.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:46:21 +02:00
Lubomír Sedlář
8ab7d9f7ba Move import of modulemd to a separate module
This should make it possible to only import the library only when it's
really needed.

DNF does not work with libmodulemd v2. If we import libmodulemd2 and
then dnf, the program will just hang forever. We only need DNF in
pungi-gather, where libmodulemd is not needed, and also where we do need
libmodulemd, we don't have DNF.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:40:31 +02:00
Lubomír Sedlář
f822ee324a ostree: Run commands in universal_newlines mode
This will mean the output is returned as unicode (decoded as UTF-8).
Thus Kobo will not have to do any decoding. This should work around
possible errors with breaking multibyte unicode character sequences into
different chunks.

Relates: https://pagure.io/releng/failed-composes/issue/237
Relates: https://github.com/release-engineering/kobo/issues/119
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 09:30:53 +02:00
Lubomír Sedlář
21d45eb243 pkgset: Allow filtering modules from Koji tags
Add a configuration option to enable skipping some modules found in the
configured tag.

Fixes: https://pagure.io/pungi/issue/1260
JIRA: COMPOSE-3794
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-24 10:13:00 +02:00
Lubomír Sedlář
2b112d53f7 Allow loading overrides for module defaults
This patch adds a new config option. This is expected to be a name of
subdirectory in the repo with module defaults. If supplied, overrides
from that location are loaded every time defaults are loaded.

This raises the minimal required version of libmodulemd to 2.8.0

JIRA: COMPOSE-3828
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-23 09:23:39 +02:00
Haibo Lin
260df24859 Move pkgset skipping logs to separate file
There will be a new log file logs/global/excluding-arch.global.log

Fixes: https://pagure.io/pungi/issue/1251
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-09-20 16:02:41 +08:00
Lubomír Sedlář
465ecf229c Add test for materialized pkgset
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
63a8b7b6c9 Remove package whitelist
This was a workaround to make some packages from the global repo
invisible for depsolving. This is now handled by packages being in
different repos. We can select which repos are enabled at which point.
This achieves the same result, but much faster.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
a426a83ed6 gather: Remove use of arch_repo
The repo was used to speed up creating lookaside repo from a variant.
This uses a similar approach as createrepo phase: selecting the last
available package set and using that data.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
2d0ffb56ca ostree-installer: Remove usage of arch_repo
Pass all pkgset repos as input to the task.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
26ddd46acb ostree: Remove arch_repo path usage
Pass all pkgset repos as input to the task.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
ab9122be2a buildinstall: Remove arch_repo usage
Simply use all existing package set repos as input for the runroot
task. The command line gets a bit long, but the actual behaviour should
remain the same.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00