If the input packages contain e.g. `ansible`, and there's
`ansible-runner-service` in the package set, we don't want to pull it
in.
JIRA: RHELCMP-446
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
- Get also requires/provides of RPMs in package set.
- Store the results of gather phase as pickle file.
- Reuse old gather phase results in case Pungi configuration
did not change, the "names" of RPMs in global package set
did not change and their requires/provides did not change.
- Add `gather_allow_reuse` option to enable this feature.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
Add gather_allow_reuse, add more tests and better handling of gather_lookaside_repos.
When probing lookasides for platform definition, we need to make sure it
works for repos specified as HTTP urls. Createrepo doesn't seem to
automatically download the repodata, so we have to help it.
JIRA: COMPOSE-3958
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Each depsolved tree will be using its own cache for fus. This should
still allow for faster loading of metadata after first iteration, but
should prevent errors from using cached files meant for another variant
or architecture. The cache is deleted after the last iteration.
JIRA: COMPOSE-3959
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should make it possible to only import the library only when it's
really needed.
DNF does not work with libmodulemd v2. If we import libmodulemd2 and
then dnf, the program will just hang forever. We only need DNF in
pungi-gather, where libmodulemd is not needed, and also where we do need
libmodulemd, we don't have DNF.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is needed when a compose does not include any modules itself, but
is using a lookaside with modules.
JIRA: COMPOSE-3720
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This was a workaround to make some packages from the global repo
invisible for depsolving. This is now handled by packages being in
different repos. We can select which repos are enabled at which point.
This achieves the same result, but much faster.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
With this patch, there should be a separate package set for each tag
that is consumed.
Generally each module will create a separate package set, with the
exception of -devel modules that will be in the same set as their
non-devel version.
Variants no longer need to keep their own package set objects. Instead
they now include a set of package set names that should be used for the
variant. This can replace the whitelist mechanism of deps gather method.
JIRA: COMPOSE-3620
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Once a package set repo is written to disk, let's use this object that
connects the repository path with the mapping of packages.
This change also makes it explicit where the dependency on package set
repos are. In the original codebase, any part of code could generate a
path to the repo, even if that repo has not yet been written.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This opens up a path to having multiple package sets in the compose. The
pkgset phase now creates a list of them (although at this time there is
always a single item in that list).
Any consumer of the package sets objects is updated to handle a list.
Generally this means an extra loop.
JIRA: COMPOSE-3620
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This was already discouraged to not be used, and is a bad idea in
current setup anyway. Removing this can simplify the code.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
On Python 3 it is not possible to sort str and None or RpmWrapper.
First convert everything to strings and then sort it. The sorting is
really to simplify diffing the files, so exact order does not have to be
preserved.
Fixes: https://pagure.io/pungi/issue/1227
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the package set repo contains any modular package, the module
metadata is added there as well.
This is needed to accomodate change in DNF that refuses to work with
repo with modular packages if the metadata is not there. This DNF change
can cause issues in buildinstall phase.
Related: https://bugzilla.redhat.com/show_bug.cgi?id=1623128
The hybrid solver is modified to not create a separate repo with the
module metadata anymore, since it will be available in the repo with
packages. This also allows us to drop code to look into lookaside repos.
We still need to iterate over local modules in order to find out what
platform should be used.
JIRA: COMPOSE-3621
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Behavior before this change: debuginfo was added based on source RPMs.
Pungi would get all debuginfo for all architectures that include at
least one binary package. This had a consequence that if foo-x.x86_64
and foo-x.i686 from the same built were included, foo-y-debuginfo.i686
would be included despite foo-y only being present for x86_64.
The patch changes this to work on binary package level: for each
included package `x`, check if there is `x-debuginfo` or
`x-debugsource`, and include them. Packages added in this way have to go
through one iteration of the solver, to account for cases such as
`x-libs-debuginfo` depending on `x-debuginfo` (with only `x` starting
this chain).
To make it slightly faster, the invocation of fus is changed to only
process newly added package in each iteration. It should have no effect
on the results, and subsequent iterations are much faster if they don't
process the same stuff again and again.
JIRA: COMPOSE-3247
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
For binary packages the filters are handled at the depsolver level.
However sources and debuginfo is added later in the process, so the
filters have to be explicitly applied.
JIRA: COMPOSE-3114
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the hybrid solver is used in a situation where there are modules in
lookaside repo, but not in the compose itself, it will fail to detect
any platform. Since we are already opening the module repodata, we can
retrieve platforms from all modules in there as well.
If there are conflicts (e.g. multiple modules depending on different
platforms), an error will be reported.
JIRA: COMPOSE-3277
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is really an oversight. The list of packages from prepopulate list
should be treated the same as if the packagees were coming from comps or
additional packages.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The initial implementation is very coarse. It enables it for all
variants and all modules. That is not always wanted.
With this patch, the config file has to explicitly list the devel
modules for each variant that should have it. The variant must be
configured also to include the non-devel module (but the module may be
in lookaside so it won't be included).
We now include module metadata in the internal lookaside repo, so that
this whole thing works if one variant is built on top of another.
JIRA: COMPOSE-3034
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the configuration lists `*` in `additional_packages`, it has a
special meaning. If it's passed to fus directly, it will use it to match
all modules and RPMs that are not masked by a package available in some
default stream. Neither is good. We don't want it to match modules, and
we want even the masked packages.
The fix is to expand the wildcard to a list of NVRs and give that to
fus. It should include the package even if it is masked.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a multilib package fails to be added (broken dependencies, or it
doesn't exist at all), we want to track it and eventually stop the
iterations.
Merges: https://pagure.io/pungi/pull-request/1071
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If there are two variants with different module sets, the missing
modulemd from variant B causeing depsolving in variant A to consider the
packages as non-modular. That is wrong.
Related: https://bugzilla.redhat.com/show_bug.cgi?id=1640125
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of having one giant command line, write the long list into a
file and pass that to the solver.
The items on the input list are sorted for easy processing.
JIRA: COMPOSE-3012
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We can't drop the whole package set once the first solver iteration
finishes, because it runs for each arch separately and we need the data
for each of them. We can however delete the arch specific portion.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If we want to include the -devel modules, the original modules should be
present for the solver. However if we put them in both local and
lookaside repos, fus will get confused. Let's not include modules that
are in lookaside in the module repo created for the solving job.
Even if the modular packages are present in the local repo, they are
identical to the ones in lookaside, and so they should not make it into
the result anyway.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It's possible we ask to include module X, but it's in lookaside and as
such it should not be in the output. Therefore we need to remove it from
the variant.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
A repo does not necessarily have the location_base attribute if the
packages are next to the repodata. This can easily happen for lookaside
repos.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There's an environment variable that can suppress printing of PID for
every debug message. It provides us with no real information and only
makes the log larger.
JIRA: COMPOSE-2973
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
