When probing lookasides for platform definition, we need to make sure it
works for repos specified as HTTP urls. Createrepo doesn't seem to
automatically download the repodata, so we have to help it.
JIRA: COMPOSE-3958
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Each depsolved tree will be using its own cache for fus. This should
still allow for faster loading of metadata after first iteration, but
should prevent errors from using cached files meant for another variant
or architecture. The cache is deleted after the last iteration.
JIRA: COMPOSE-3959
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should make it possible to only import the library only when it's
really needed.
DNF does not work with libmodulemd v2. If we import libmodulemd2 and
then dnf, the program will just hang forever. We only need DNF in
pungi-gather, where libmodulemd is not needed, and also where we do need
libmodulemd, we don't have DNF.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is needed when a compose does not include any modules itself, but
is using a lookaside with modules.
JIRA: COMPOSE-3720
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This opens up a path to having multiple package sets in the compose. The
pkgset phase now creates a list of them (although at this time there is
always a single item in that list).
Any consumer of the package sets objects is updated to handle a list.
Generally this means an extra loop.
JIRA: COMPOSE-3620
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the package set repo contains any modular package, the module
metadata is added there as well.
This is needed to accomodate change in DNF that refuses to work with
repo with modular packages if the metadata is not there. This DNF change
can cause issues in buildinstall phase.
Related: https://bugzilla.redhat.com/show_bug.cgi?id=1623128
The hybrid solver is modified to not create a separate repo with the
module metadata anymore, since it will be available in the repo with
packages. This also allows us to drop code to look into lookaside repos.
We still need to iterate over local modules in order to find out what
platform should be used.
JIRA: COMPOSE-3621
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Behavior before this change: debuginfo was added based on source RPMs.
Pungi would get all debuginfo for all architectures that include at
least one binary package. This had a consequence that if foo-x.x86_64
and foo-x.i686 from the same built were included, foo-y-debuginfo.i686
would be included despite foo-y only being present for x86_64.
The patch changes this to work on binary package level: for each
included package `x`, check if there is `x-debuginfo` or
`x-debugsource`, and include them. Packages added in this way have to go
through one iteration of the solver, to account for cases such as
`x-libs-debuginfo` depending on `x-debuginfo` (with only `x` starting
this chain).
To make it slightly faster, the invocation of fus is changed to only
process newly added package in each iteration. It should have no effect
on the results, and subsequent iterations are much faster if they don't
process the same stuff again and again.
JIRA: COMPOSE-3247
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
For binary packages the filters are handled at the depsolver level.
However sources and debuginfo is added later in the process, so the
filters have to be explicitly applied.
JIRA: COMPOSE-3114
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the hybrid solver is used in a situation where there are modules in
lookaside repo, but not in the compose itself, it will fail to detect
any platform. Since we are already opening the module repodata, we can
retrieve platforms from all modules in there as well.
If there are conflicts (e.g. multiple modules depending on different
platforms), an error will be reported.
JIRA: COMPOSE-3277
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is really an oversight. The list of packages from prepopulate list
should be treated the same as if the packagees were coming from comps or
additional packages.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The initial implementation is very coarse. It enables it for all
variants and all modules. That is not always wanted.
With this patch, the config file has to explicitly list the devel
modules for each variant that should have it. The variant must be
configured also to include the non-devel module (but the module may be
in lookaside so it won't be included).
We now include module metadata in the internal lookaside repo, so that
this whole thing works if one variant is built on top of another.
JIRA: COMPOSE-3034
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the configuration lists `*` in `additional_packages`, it has a
special meaning. If it's passed to fus directly, it will use it to match
all modules and RPMs that are not masked by a package available in some
default stream. Neither is good. We don't want it to match modules, and
we want even the masked packages.
The fix is to expand the wildcard to a list of NVRs and give that to
fus. It should include the package even if it is masked.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a multilib package fails to be added (broken dependencies, or it
doesn't exist at all), we want to track it and eventually stop the
iterations.
Merges: https://pagure.io/pungi/pull-request/1071
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If there are two variants with different module sets, the missing
modulemd from variant B causeing depsolving in variant A to consider the
packages as non-modular. That is wrong.
Related: https://bugzilla.redhat.com/show_bug.cgi?id=1640125
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of having one giant command line, write the long list into a
file and pass that to the solver.
The items on the input list are sorted for easy processing.
JIRA: COMPOSE-3012
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If we want to include the -devel modules, the original modules should be
present for the solver. However if we put them in both local and
lookaside repos, fus will get confused. Let's not include modules that
are in lookaside in the module repo created for the solving job.
Even if the modular packages are present in the local repo, they are
identical to the ones in lookaside, and so they should not make it into
the result anyway.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It's possible we ask to include module X, but it's in lookaside and as
such it should not be in the output. Therefore we need to remove it from
the variant.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
A repo does not necessarily have the location_base attribute if the
packages are next to the repodata. This can easily happen for lookaside
repos.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There's an environment variable that can suppress printing of PID for
every debug message. It provides us with no real information and only
makes the log larger.
JIRA: COMPOSE-2973
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The multilib library expects us to ask about the candidate package for
another arch (should I include this as multilib?) and not the current
arch (should I include other arches for this package?).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Fus returns also RPMs in modules, but until latest version it only
worked if the package was in the same repo as the metadata. This changed
in latest version and now Pungi does not need to expand the list
anymore.
JIRA: COMPOSE-2779
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Comps file specifies a pattern for some packages. If that package is
installed, all packages matching the pattern are added as well. This can
be added to fus as another pass.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch adds a new gather method called `hybrid`, which uses a `fus`
binary, which must exist somewhere on the `$PATH`. It will call it
multiple times to add multilib packages.
The solver can handle packages, modules and comps groups as input.
However comps groups are never passed in. Pungi will expand it to a list
of packages to avoid issues with comps handling in fus. It ignores
optional packages, and if the group mentions a package that does not
exist, nothing else from the group is included.
Multilib is also handled outside of fus. Pungi will run it, parse the
packages from output, determines multilib packages and adds them as
input. Then it runs the solver again. This is done until nothing new is
added. Usually two passes should be enough.
Source packages and debuginfo are added as a final step. All debuginfo
packages from any included source are added. If the source or debuginfo
package is included in any lookaside repo, it will be skipped.
The tool expects to get a platform stream that should be provided for
modules to depend on. Pungi looks into the modules and gets the platform
from there. If there are more requests, an error is raised.
There is some missing functionality and options that are ignored.
Particularly these are:
* gather_fulltree
* gather_selfhosting
* greedy_method
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Haibo Lin
Lubomír Sedlář