When `link_type = "symlink"` is used, the packages are in fact symlinks
to /mnt/koji. When graft points file is generated, the paths in this graft
points file point to symlinks and therefore symlinks are copied into the
generated ISO file instead of real files.
In this commit, the code to generate the graft points file is changed
so it resolves the symlink to real file stored on /mnt/koji. To make
this code safer, it does such resolving only in case the symlink points
outside of `compose.paths.compose.topdir()`. Therefore you can still
generate ISO file with symlink pointing to file stored within the ISO
file itself, although this is not done currently afaik.
The main reason for this is to be able to generate ISO files even
without hardlinks (which would need read-write access on /mnt/koji)
and without copying all the packages from /mnt/koji to local storage.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
When pungi-gather (or old pungi) download the packages from repos into
work/$arch/, they are linked to work/global/. This was using link_type
configuration option.
However if that is set to symlink, we get a relative symlink in
work/global/ which is later copied under the compose/ directory. Since
it's a relative symlink, it gets broken by this.
The fix is to hardlink the downloaded packages instead in the first
step. Since both the source and destination are in work/ directory, we
know it's the same volume and hardlinks should work. There is a fallback
to copy just to be sure, but it shouldn't ever be used.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The `runroot_method` now accepts `dict` value with phase name as a key
and runroot method as a value. For backward compatibility, the `str`
value is still supported.
The new `global_runroot_method` option has been added which defines
the runroot method in case it is not set in `dict` in the `runroot_method`.
This commit allows running `createiso` phase locally while keeping the other
phases in Koji.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
It is possible we only ever see a package from non-lookaside repo, but
it actually is in lookaside. This patch should check for that before
adding the package to result set.
JIRA: RCM-71946
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The final set of packages can contain multiple copies of the same
package if it's present in multiple repositories. Since they have the
same NVRA and were built in Koji, we know they are identical and should
merge their flags.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Theoretically it is possible, and this is the bare minimum of changes
needed to survive such situation. There may be other pitfalls.
Fixes: https://pagure.io/pungi/issue/1309
JIRA: COMPOSE-4016
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Originally the list of solvables for fus was growing with each iteration
and nothing was ever removed. That later changed so that fus iterations
are only done on newly added stuff. It's great for performance, but
means that the last log is not a superset of all others.
To get all dependency problems we need to look into all log files, not
just the last one.
JIRA: COMPOSE-3964
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When there is an exception in gathering (such as after seeing unsigned
packages in deps method), the exception was lost and the compose
continued to run until it tried to access the result and crashed on
KeyError.
Relates: https://pagure.io/releng/failed-composes/issue/587
JIRA: COMPOSE-3986
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When running repoclosure as root user, it will use other dir instead of
the one returned by getCacheDir().
For yum, with --tempcache option could let the cache dir returned by
getCacheDir() always be used.
For dnf, there's no such an option and we have to handle it specially.
JIRA: COMPOSE-3922
Signed-off-by: Haibo Lin <hlin@redhat.com>
When probing lookasides for platform definition, we need to make sure it
works for repos specified as HTTP urls. Createrepo doesn't seem to
automatically download the repodata, so we have to help it.
JIRA: COMPOSE-3958
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Each depsolved tree will be using its own cache for fus. This should
still allow for faster loading of metadata after first iteration, but
should prevent errors from using cached files meant for another variant
or architecture. The cache is deleted after the last iteration.
JIRA: COMPOSE-3959
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
All places in the code assume the option to not be required except for
this one line.
An obsolete comment is removed as well.
Relates: https://pagure.io/releng/failed-composes/issue/477
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is basically collecting all individual extra_files.json and putting
their content into a single location in
compose/metadata/extra_files.json. The file format is part of productmd
1.23.
JIRA: COMPOSE-3831
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of just checking that the repo directory exists, make sure the
repodata subdirectory is in there. If it's missing, then createrepo_c
has nothing to use anyway, and it may help avoid issues.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It makes no sense to only skip phases needed by other phase.
Fixes: https://pagure.io/pungi/issue/1274
JIRA: COMPOSE-3871
Signed-off-by: Haibo Lin <hlin@redhat.com>
A lorax template used for the ostree-installer might need an additional
package dependency (e.g., flatpak to embed a flatpak repository) - add
a config key 'extra_runroot_pkgs' to the ostree installer configuration
to allow supplementing the set of packages installed into the runroot.
Signed-off-by: Owen W. Taylor <otaylor@fishsoup.net>
Until now, the behaviour was that all debuginfo from a build would be
included if at least one package with the same arch was included.
This resulted in many debuginfo packages being included even though
their corresponding package was not present.
With this patch, we will always pull in debugsource, but foo-debuginfo
will only be included if foo is included for the same arch. As a
consequence, it is necessary to resolve dependencies of debuginfo
packages. There are cases where foo-debuginfo needs foo-debuginfo-common
for example.
This change means that DNF and YUM backends are no longer identical in
the output. The tests where this is demonstrated are duplicated and
their results are modified.
JIRA: COMPOSE-3823
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should make it possible to only import the library only when it's
really needed.
DNF does not work with libmodulemd v2. If we import libmodulemd2 and
then dnf, the program will just hang forever. We only need DNF in
pungi-gather, where libmodulemd is not needed, and also where we do need
libmodulemd, we don't have DNF.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Add a configuration option to enable skipping some modules found in the
configured tag.
Fixes: https://pagure.io/pungi/issue/1260
JIRA: COMPOSE-3794
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch adds a new config option. This is expected to be a name of
subdirectory in the repo with module defaults. If supplied, overrides
from that location are loaded every time defaults are loaded.
This raises the minimal required version of libmodulemd to 2.8.0
JIRA: COMPOSE-3828
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There will be a new log file logs/global/excluding-arch.global.log
Fixes: https://pagure.io/pungi/issue/1251
Signed-off-by: Haibo Lin <hlin@redhat.com>
Koji started to create relative symlinks to images created in various
tasks. We need to check for that and potentially link the actual file.
JIRA: COMPOSE-3822
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Jan Kaluza
Lubomír Sedlář
Haibo Lin
Frédéric Pierret (fepitre)
Owen W. Taylor