Rather than trying to use local access when it's accessible, let user
make the decision:
* if koji_cache is configured use it and download stuff
* if not, fall back to local access
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 0d3cd150bd)
With this patch, Pungi can be configured with a local directory to be
used as a cache for RPMs, and it will download packages from Koji over
HTTP instead of reading them from filesystem directly.
The files from the cache can then be hardlink as usual.
There is locking in place to avoid different composes running at the
same time to step on each other.
This is now supported for RPMs only, be it real builds or scratch
builds.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 631bb01d8f)
The exceptions from libmodulemd are not particularly helpful as they do
not contain information about what file caused it.
modulemd-yaml-error-quark: Failed to open file: Permission denied (0)
This patch should add the path to the problematic file into the message.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 14e025a5a1)
This is a rather fringe use case. If the configuration contains
pkgset_koji_builds or pkgset_koji_scratch_tasks but no pkgset_koji_tag,
the compose will be empty.
The expectation though is that the packages should be pulled.
The extra RPMs are added to all non-modular tags because they are
supposed to mask builds from the same packages (e.g. user may want to
explicitly pull in older version than tagged).
This patch adds support for composes containing only explicitly listed
builds by creating a dummy package set that is not actually using any
tag.
JIRA: RHELCMP-11385
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit f960b4d155)
Pungi expects each module to be built for all architectures by default.
Unless the module is filtered out, missing metadata for a particular
arch would cause it to crash with a incomprehensible error message. This
should make it a little better.
Relates: https://pagure.io/releng/failed-composes/issue/3889
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The pickle files are used for reusing results from old compose and the
failure should not block the compose process.
JIRA: RHELCMP-9494
Signed-off-by: Haibo Lin <hlin@redhat.com>
- Use ModuleIndex's update_from_file/update_from_string instead of ModuleStream's
read_file/read_string which is deprecated.
- Extend tests to work with real module streams instead of mocks.
Signed-off-by: Filip Valder <fvalder@redhat.com>
- Some modules can be absent in koji env but be present in variants.xml.
And Pungi will fail in this case. So we must filter out those modules
from expected modules list by list from pungi build config
@BS-NOBUILD
@BS-TARGET-CL8
Change-Id: I22c15c42868412e34fd554030130bd7c3e25b8ef
When a module is skipped from the compose, we should not add it to a
mapping of module tags. If it's there, we then spend time building a
repo for the module, and it get's passed to buildinstall, despite the
packages not being supposed to be included in the compose.
If the packages are not included in any variant, they shouldn't be
available to buildinstall either.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If packages are appearing quickly in Koji, and signing them is triggered
by automation, there may be a delay between the package being signed and
compose running. In such case it may be preferable to wait for the
signed copy rather than fail the compose.
JIRA: RHELCMP-3932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When adding extra modules via option *pkgset_koji_module_builds*, all
other versions of the same stream potentially available in a Brew tag
should be skipped.
JIRA: RHELCMP-3689
Signed-off-by: Haibo Lin <hlin@redhat.com>
Instead of koji.mbox use local koji-like wrapper.
@BS-LINKED-5ff8b8cb6f44affc6c5e9a7a
@BS-TARGET-CL8
Change-Id: I82a2bc8bc71ae06240656898f3df71bb28bcb9e9
When modules are used, there are lot of small package sets. These
package sets have usually less than 500 packages. The createrepo
part of `MaterializedPackageSet.create` executed for such small
set of packages takes around 1 second. Most of this time
the createrepo_c runs in single thread. It does the initialization,
it writes the XML files, ...
The parts of createrepo which can be run in parallel and therefore
would use all the CPUs are quite small for very small package sets.
This commit therefore executes multiple threads with
`MaterializedPackageSet.create` for these very small package sets.
This saves around 40 seconds from pkgset phase for RHEL compose.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
The current code calls `find_old_compose` followed by multiple `os.path.*`
calls to find out if particular file exists in the old compose. This
duplicates code a lot and makes it harder to read.
In this commit, new `Compose.old_compose_path` is introduced and
used instead of direct calls of `find_old_compose`.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
When pungi-gather (or old pungi) download the packages from repos into
work/$arch/, they are linked to work/global/. This was using link_type
configuration option.
However if that is set to symlink, we get a relative symlink in
work/global/ which is later copied under the compose/ directory. Since
it's a relative symlink, it gets broken by this.
The fix is to hardlink the downloaded packages instead in the first
step. Since both the source and destination are in work/ directory, we
know it's the same volume and hardlinks should work. There is a fallback
to copy just to be sure, but it shouldn't ever be used.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
All places in the code assume the option to not be required except for
this one line.
An obsolete comment is removed as well.
Relates: https://pagure.io/releng/failed-composes/issue/477
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should make it possible to only import the library only when it's
really needed.
DNF does not work with libmodulemd v2. If we import libmodulemd2 and
then dnf, the program will just hang forever. We only need DNF in
pungi-gather, where libmodulemd is not needed, and also where we do need
libmodulemd, we don't have DNF.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Add a configuration option to enable skipping some modules found in the
configured tag.
Fixes: https://pagure.io/pungi/issue/1260
JIRA: COMPOSE-3794
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The cache is now always saved when the repo is created on disk. The
loading procedure in Koji source is updated to look for cache for
correct tag.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The pickled pkgset was only ever used for debug mode. We don't use that
anymore. Let's stop writing files that are not used anywhere.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
With this patch, there should be a separate package set for each tag
that is consumed.
Generally each module will create a separate package set, with the
exception of -devel modules that will be in the same set as their
non-devel version.
Variants no longer need to keep their own package set objects. Instead
they now include a set of package set names that should be used for the
variant. This can replace the whitelist mechanism of deps gather method.
JIRA: COMPOSE-3620
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Once a package set repo is written to disk, let's use this object that
connects the repository path with the mapping of packages.
This change also makes it explicit where the dependency on package set
repos are. In the original codebase, any part of code could generate a
path to the repo, even if that repo has not yet been written.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This name will serve as an identifier for the group of packages.
For Koji package sets, it should the name of the tag from which the
packages come. For package sets based on repos a dummy constant name is
used.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Lubomír Sedlář
soksanichenko
Haibo Lin
Filip Valder
oshyshatskyi
Jan Kaluza