Commit Graph

2529 Commits

Author SHA1 Message Date
Haibo Lin
01ab1d2e24 Hide latest koji package via pkgset_koji_builds option
JIRA: COMPOSE-4010
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-09 07:33:46 +00:00
Lubomír Sedlář
aa6a213c8d Add a new option for customizing version in .treeinfo
There was an attempt to achieve this by customizing --version and
--release arguments for Lorax, but Pungi does not take the [general] and
[release] sections from its .treeinfo. Instead it was always using
release version.

The value from this new option will be put into .treeinfo and used for
--version and --release arguments in Lorax (unless explicitly defined in
lorax_options to another value).

JIRA: COMPOSE-4029
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-07 10:55:48 +01:00
Lubomír Sedlář
2c2462970d Kickstart on RHEL 7 does not have metalink attr
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-07 10:30:16 +01:00
Jan Kaluza
fcf1442f71 Support generating ISOs when using link_type="symlink".
When `link_type = "symlink"` is used, the packages are in fact symlinks
to /mnt/koji. When graft points file is generated, the paths in this graft
points file point to symlinks and therefore symlinks are copied into the
generated ISO file instead of real files.

In this commit, the code to generate the graft points file is changed
so it resolves the symlink to real file stored on /mnt/koji. To make
this code safer, it does such resolving only in case the symlink points
outside of `compose.paths.compose.topdir()`. Therefore you can still
generate ISO file with symlink pointing to file stored within the ISO
file itself, although this is not done currently afaik.

The main reason for this is to be able to generate ISO files even
without hardlinks (which would need read-write access on /mnt/koji)
and without copying all the packages from /mnt/koji to local storage.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-01-06 12:30:13 +00:00
Lubomír Sedlář
02ace28fe4 pkgset: Hardlink downloaded packages
When pungi-gather (or old pungi) download the packages from repos into
work/$arch/, they are linked to work/global/. This was using link_type
configuration option.

However if that is set to symlink, we get a relative symlink in
work/global/ which is later copied under the compose/ directory. Since
it's a relative symlink, it gets broken by this.

The fix is to hardlink the downloaded packages instead in the first
step. Since both the source and destination are in work/ directory, we
know it's the same volume and hardlinks should work. There is a fallback
to copy just to be sure, but it shouldn't ever be used.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-06 10:49:25 +01:00
Haibo Lin
3f111b559f Check dependency of --just-phase
JIRA: COMPOSE-4020
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-06 08:52:47 +00:00
Jan Kaluza
3750d6795f Update runroot_method documentation.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-01-06 07:47:38 +01:00
Haibo Lin
794d151bef Remove productimg phase
JIRA: COMPOSE-4004
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-03 11:42:38 +00:00
Lubomír Sedlář
38f6162b46 Fix unified isos with missing images.json
The metadata file can be missing if the compose contains no images. We
need to handle that by creating a new empty file to add unified images
to.

JIRA: COMPOSE-4048
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-03 09:20:16 +01:00
Jan Kaluza
817acdbbac Allow setting runroot_method per Pungi phase.
The `runroot_method` now accepts `dict` value with phase name as a key
and runroot method as a value. For backward compatibility, the `str`
value is still supported.

The new `global_runroot_method` option has been added which defines
the runroot method in case it is not set in `dict` in the `runroot_method`.

This commit allows running `createiso` phase locally while keeping the other
phases in Koji.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-01-02 14:10:20 +00:00
Haibo Lin
b415e31f9d ostree: Fix arg passed to scm.get_dir_from_scm
A compose object should be passed instead of logger.

JIRA: COMPOSE-4050
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-02 13:13:14 +00:00
Lubomír Sedlář
b8bb4f7daa Improve detection of lookaside packages
It is possible we only ever see a package from non-lookaside repo, but
it actually is in lookaside. This patch should check for that before
adding the package to result set.

JIRA: RCM-71946
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-02 13:50:29 +01:00
Lubomír Sedlář
38c24b3038 gather: Deduplicate packages before printing
The final set of packages can contain multiple copies of the same
package if it's present in multiple repositories. Since they have the
same NVRA and were built in Koji, we know they are identical and should
merge their flags.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-02 13:36:42 +01:00
Haibo Lin
b043ac66dc Do not mention pdc_client when missing modulemd
JIRA: COMPOSE-4007
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-12-18 13:50:49 +08:00
Haibo Lin
12828849d6 gather: Improve logging for gathering
Adding arch and variant to log message to make it clearer.

JIRA: COMPOSE-4009
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-12-16 08:56:22 +00:00
Frédéric Pierret (fepitre)
9b101d554f
gather: handle metalink in kickstart repos
Signed-off-by: Frédéric Pierret (fepitre) <frederic.pierret@qubes-os.org>
2019-12-15 21:05:10 +01:00
Haibo Lin
0ed70fc8b6 tests: drop sys.path modification
It's unnecessary to add the path manually.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-12-10 10:53:38 +08:00
Haibo Lin
3cf16eb42d cleanup: refactor scripts via entry points
Fixes: https://pagure.io/pungi/issue/1045
JIRA: COMPOSE-2946
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-12-10 10:53:31 +08:00
Lubomír Sedlář
ce16e55ebd Avoid crash if a module is not available on all arches
Theoretically it is possible, and this is the bare minimum of changes
needed to survive such situation. There may be other pitfalls.

Fixes: https://pagure.io/pungi/issue/1309
JIRA: COMPOSE-4016
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-12-06 14:22:41 +01:00
Lubomír Sedlář
0f53506765 Add version constraint to productmd dependency
This should be propagated to RPM level with F30+.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-12-06 09:49:01 +01:00
Frédéric Pierret (fepitre)
fa8b2094da Allow specifying temp dir in pungi-gather
That fix invalid cross-device link when hardlinking when /tmp is a
separate filesystem

Signed-off-by: Frédéric Pierret (fepitre) <frederic.pierret@qubes-os.org>
2019-12-06 08:06:49 +01:00
Lubomír Sedlář
ea2cd448a0 4.1.41 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-12-02 14:51:35 +01:00
Lubomír Sedlář
aefe9b186d repoclosure: Parse all fus logs
Originally the list of solvables for fus was growing with each iteration
and nothing was ever removed. That later changed so that fus iterations
are only done on newly added stuff. It's great for performance, but
means that the last log is not a superset of all others.

To get all dependency problems we need to look into all log files, not
just the last one.

JIRA: COMPOSE-3964
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-12-02 11:08:55 +01:00
Lubomír Sedlář
51b1144b70 runroot: Log different commands to different files
JIRA: COMPOSE-3980
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-28 15:12:05 +01:00
Lubomír Sedlář
7f35ac622a gather: Collect and re-raise errors from gather method
When there is an exception in gathering (such as after seeing unsigned
packages in deps method), the exception was lost and the compose
continued to run until it tried to access the result and crashed on
KeyError.

Relates: https://pagure.io/releng/failed-composes/issue/587
JIRA: COMPOSE-3986
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-25 14:28:34 +01:00
Haibo Lin
6afbe6d20a buildinstall: Log message for boot config change
JIRA: COMPOSE-3945
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-21 12:16:05 +08:00
Haibo Lin
0c040e0a69 livemedia: Remove title option
Fixes: https://pagure.io/pungi/issue/1293
JIRA: COMPOSE-3947
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-20 15:33:28 +08:00
Haibo Lin
242100eb72 Make sure repoclosure cache cleaned when running as root
When running repoclosure as root user, it will use other dir instead of
the one returned by getCacheDir().

For yum, with --tempcache option could let the cache dir returned by
getCacheDir() always be used.

For dnf, there's no such an option and we have to handle it specially.

JIRA: COMPOSE-3922
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-20 14:13:19 +08:00
Haibo Lin
90187298f2 Prefix lookaside cache dir with compose id
Then it could be cleaned up leater.

JIRA: COMPOSE-3922
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-20 13:51:13 +08:00
Haibo Lin
e0dd20dffe Prefix repoclosure cache dir with compose id
JIRA: COMPOSE-3922
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-20 13:51:13 +08:00
Lubomír Sedlář
c87d299a20 hybrid: Download remote files when getting platform
When probing lookasides for platform definition, we need to make sure it
works for repos specified as HTTP urls. Createrepo doesn't seem to
automatically download the repodata, so we have to help it.

JIRA: COMPOSE-3958
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-19 08:51:18 +01:00
Lubomír Sedlář
4473b05f10 gather: Use fresh cache for each sequence of fus runs
Each depsolved tree will be using its own cache for fus. This should
still allow for faster loading of metadata after first iteration, but
should prevent errors from using cached files meant for another variant
or architecture. The cache is deleted after the last iteration.

JIRA: COMPOSE-3959
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-18 14:39:47 +01:00
Haibo Lin
722411dfcd tests: Add test for compose logger setup
PR#1267 brings in some uncovered code which introduced
an issue fixed by PR#1292.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-12 18:03:13 +08:00
Lubomír Sedlář
d34b0d7900 pkgset: Fix running without any koji tags
All places in the code assume the option to not be required except for
this one line.

An obsolete comment is removed as well.

Relates: https://pagure.io/releng/failed-composes/issue/477
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-11 09:30:37 +01:00
Lubomír Sedlář
254d0cebff Get message from LogRecord with a method
Relates: https://pagure.io/releng/failed-composes/issue/469
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-11 08:46:01 +01:00
Haibo Lin
6be2cf5118 Do not write images.json if all related phases skipped
JIRA: COMPOSE-3870
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-11 14:45:01 +08:00
Haibo Lin
2cbd75803a gather: Do not write metadata if gather phase skipped
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-07 16:19:19 +08:00
Lubomír Sedlář
cba3f1c88f Write global metadata about extra files
This is basically collecting all individual extra_files.json and putting
their content into a single location in
compose/metadata/extra_files.json. The file format is part of productmd
1.23.

JIRA: COMPOSE-3831
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-05 09:11:01 +01:00
Lubomír Sedlář
20c3614fb3 scm: Add backend for downloading archives from Koji
Tests and documentation included.

JIRA: COMPOSE-3816
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-05 08:51:26 +01:00
Lubomír Sedlář
39e8f6f710 Pass compose to SCM wrappers
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-05 08:51:26 +01:00
Haibo Lin
b05295c202 Remove misleading warning about size of the ISO
JIRA: COMPOSE-3885
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-04 14:22:59 +08:00
Haibo Lin
114df77c6b Retry watching koji tasks on server outage
Fixes: https://pagure.io/pungi/issue/1285
JIRA: COMPOSE-3896
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-01 15:57:48 +08:00
Haibo Lin
5751de1096 config-dump: Allow overwritting config via -e option
The -e option was designed for defining missing config item,
but users always try to use it to overwrite existing config.

Now it works as expected.

JIRA: COMPOSE-3718
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-10-24 18:29:47 +08:00
Lubomír Sedlář
8829fc32ab 4.1.40 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-23 09:02:41 +02:00
Lubomír Sedlář
6771a21916 pkgset: Only reuse valid old repo
Instead of just checking that the repo directory exists, make sure the
repodata subdirectory is in there. If it's missing, then createrepo_c
has nothing to use anyway, and it may help avoid issues.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-22 15:32:13 +02:00
Haibo Lin
4f96164ec7 Clean up skipping phases
It makes no sense to only skip phases needed by other phase.

Fixes: https://pagure.io/pungi/issue/1274
JIRA: COMPOSE-3871
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-10-22 16:36:01 +08:00
Haibo Lin
3901c227f0 Fix crash on unsigned packages
JIRA: COMPOSE-3869
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-10-21 17:02:59 +08:00
Juliana Rodrigueiro
f8dcda9dcb doc: Make it clear that the field 'repo' has to be empty for 'file' backend
Leaving the field 'repo' out for the 'file' backend makes the compose fail.

    [ERROR   ] Compose run failed: 'repo'

Signed-off-by: Juliana Rodrigueiro <juliana.rodrigueiro@intra2net.com>
2019-10-18 10:30:25 +02:00
Haibo Lin
2ff1f2fac3 gather: Make depsolving parallel
JIRA: COMPOSE-3827
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-10-18 14:20:32 +08:00
Lubomír Sedlář
014560f8bd doc: Add full examples of compose configuration
JIRA: COMPOSE-3797
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-14 11:04:39 +02:00