Commit Graph

126 Commits

Author SHA1 Message Date
Lubomír Sedlář
b80efbfd97 hybrid: Refactor handling debuginfo packages
Behavior before this change: debuginfo was added based on source RPMs.
Pungi would get all debuginfo for all architectures that include at
least one binary package. This had a consequence that if foo-x.x86_64
and foo-x.i686 from the same built were included, foo-y-debuginfo.i686
would be included despite foo-y only being present for x86_64.

The patch changes this to work on binary package level: for each
included package `x`, check if there is `x-debuginfo` or
`x-debugsource`, and include them. Packages added in this way have to go
through one iteration of the solver, to account for cases such as
`x-libs-debuginfo` depending on `x-debuginfo` (with only `x` starting
this chain).

To make it slightly faster, the invocation of fus is changed to only
process newly added package in each iteration. It should have no effect
on the results, and subsequent iterations are much faster if they don't
process the same stuff again and again.

JIRA: COMPOSE-3247
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-18 13:56:48 +01:00
Lubomír Sedlář
4431ea8a0c hybrid: Fix opening gzipped files on Python 2.6
GzipFile works in `with` statement starting with 2.7

JIRA: RCM-51070
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-06 08:04:19 +01:00
Lubomír Sedlář
23bf01bb45 hybrid: Apply filters to debuginfo and source packages
For binary packages the filters are handled at the depsolver level.
However sources and debuginfo is added later in the process, so the
filters have to be explicitly applied.

JIRA: COMPOSE-3114
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-12 12:20:46 +01:00
Lubomír Sedlář
e573246a2a hybrid: Get platform from lookaside repos
If the hybrid solver is used in a situation where there are modules in
lookaside repo, but not in the compose itself, it will fail to detect
any platform. Since we are already opening the module repodata, we can
retrieve platforms from all modules in there as well.

If there are conflicts (e.g. multiple modules depending on different
platforms), an error will be reported.

JIRA: COMPOSE-3277
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-08 14:41:56 +01:00
Jan Kaluza
c3aa297d8c Return RPMs added to -devel module in GatherSourceModule.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2019-02-08 14:24:16 +01:00
Lubomír Sedlář
940a581bd9 gather: Link files in order for dependant variants
If there are variants that depend on another, they should be processed
in order to make sure packages from the base variant are linked first.
That way the srpm cache is populated and any package in layered variant
but with source in base will have access to correct epoch information.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-04 08:24:22 +01:00
Lubomír Sedlář
f59034b22d hybrid: Add packages from prepopulate to input
This is really an oversight. The list of packages from prepopulate list
should be treated the same as if the packagees were coming from comps or
additional packages.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-01-29 16:16:44 +01:00
Lubomír Sedlář
7c7f997d74 gather: Expand wildcards in package names for nodeps
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-08 08:44:44 +01:00
Lubomír Sedlář
a73099d446 Support more specific config for devel modules
The initial implementation is very coarse. It enables it for all
variants and all modules. That is not always wanted.

With this patch, the config file has to explicitly list the devel
modules for each variant that should have it. The variant must be
configured also to include the non-devel module (but the module may be
in lookaside so it won't be included).

We now include module metadata in the internal lookaside repo, so that
this whole thing works if one variant is built on top of another.

JIRA: COMPOSE-3034
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-30 15:58:35 +01:00
Lubomír Sedlář
04715f4906 hybrid: Remove dead code
We don't explicitly add modular packages anymore, fus is supposed to
handle that.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-26 14:56:53 +02:00
Lubomír Sedlář
e10b893adc gather: Expand wildcards in Pungi
When the configuration lists `*` in `additional_packages`, it has a
special meaning. If it's passed to fus directly, it will use it to match
all modules and RPMs that are not masked by a package available in some
default stream. Neither is good. We don't want it to match modules, and
we want even the masked packages.

The fix is to expand the wildcard to a list of NVRs and give that to
fus. It should include the package even if it is masked.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-26 13:16:12 +02:00
Lubomír Sedlář
fe39056431 repoclosure: Extract logs from hybrid solver
There is no repoclosure that correctly understands modules. The best
thing we can offer is the errors reported by the depsolver.

JIRA: COMPOSE-2321
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-26 08:42:21 +02:00
Lubomír Sedlář
12f949fe84 gather: Track multilib that doesn't exist
If a multilib package fails to be added (broken dependencies, or it
doesn't exist at all), we want to track it and eventually stop the
iterations.

Merges: https://pagure.io/pungi/pull-request/1071
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-24 08:13:56 +02:00
Lubomír Sedlář
10fa53a6ac gather: Prepare module metadata before starting depsolving
Without this we don't have artifacts from all modules when the first
modular variant is solved. That makes some modular packages appear as
bare.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-19 10:28:17 +02:00
Lubomír Sedlář
0c3e02eeb0 hybrid: Include modulemd from all variants in temporary repo
If there are two variants with different module sets, the missing
modulemd from variant B causeing depsolving in variant A to consider the
packages as non-modular. That is wrong.

Related: https://bugzilla.redhat.com/show_bug.cgi?id=1640125
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-19 10:28:17 +02:00
Lubomír Sedlář
736772f954 hybrid: Don't add debuginfo as langpacks
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-17 10:14:04 +02:00
Lubomír Sedlář
9afb8e6801 fus: Write solvables to file
Instead of having one giant command line, write the long list into a
file and pass that to the solver.

The items on the input list are sorted for easy processing.

JIRA: COMPOSE-3012
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-17 10:14:04 +02:00
Lubomír Sedlář
4f712259d7 hybrid: Honor filter_packages
JIRA: COMPOSE-3011
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-17 10:14:04 +02:00
Lubomír Sedlář
9d9f0add83 Drop the memory saving code
It actually still does not work if there is inheritance between
variants.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-16 15:35:47 +02:00
Lubomír Sedlář
844f2005a0 Save memory less agressively
We can't drop the whole package set once the first solver iteration
finishes, because it runs for each arch separately and we need the data
for each of them. We can however delete the arch specific portion.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-15 15:34:35 +02:00
Lubomír Sedlář
ce9ac35640 hybrid: Only include modules that are not in lookaside
If we want to include the -devel modules, the original modules should be
present for the solver. However if we put them in both local and
lookaside repos, fus will get confused. Let's not include modules that
are in lookaside in the module repo created for the solving job.

Even if the modular packages are present in the local repo, they are
identical to the ones in lookaside, and so they should not make it into
the result anyway.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-08 14:41:04 +02:00
Lubomír Sedlář
814103d87f Try to be more conservative about memory usage
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-08 14:41:04 +02:00
Lubomír Sedlář
3fea217b9c hybrid: Remove modules not listed by fus
It's possible we ask to include module X, but it's in lookaside and as
such it should not be in the output. Therefore we need to remove it from
the variant.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-08 14:41:04 +02:00
Lubomír Sedlář
1a161982c0 gather: Make devel modules configurable
By default nothing should change. This patch adds a new config option
that enables all this new craziness:

 * it turns of applying module filters at pkgset level
 * it creates new modules and adds them to the compose

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-08 14:41:04 +02:00
Lubomír Sedlář
06b32b5d80 pkgset: Stop prefilling RPM artifacts
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-08 14:41:04 +02:00
Lubomír Sedlář
08d65bdde6 gather: Create devel module for each normal module
The module has same S:V:C, but the name is suffixed with `-devel`. The
module should contain all packages from the module koji tag that were
not included in the actual module.

The devel module has the same dependencies as the regular module, but
also additionally depends on the original module. The API and profiles
are cleared in the new module.

In the metadata it shows the same koji tag.

The test if package goes to the module is refactored to a function to
make work with the negated case a bit easier.

There may be unneeded multilib packages in the -devel module, because
there might be buildtime dependencies between things that we don't see.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>

Clear API and profiles

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-08 14:40:13 +02:00
Lubomír Sedlář
604ec40c8e gather: Work with repos without location_base
A repo does not necessarily have the location_base attribute if the
packages are next to the repodata. This can easily happen for lookaside
repos.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-08 14:36:43 +02:00
Lubomír Sedlář
1aff7fc3ac linker: Simplify creating pool
The pool is created in multiple places, and the process is always to
create an instance and add workers to it. Let's abstract the loop in a
method.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-03 10:27:56 +02:00
Lubomír Sedlář
e2e8df3f09 gather: Hide pid of fus process
There's an environment variable that can suppress printing of PID for
every debug message. It provides us with no real information and only
makes the log larger.

JIRA: COMPOSE-2973
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-03 10:23:39 +02:00
Lubomír Sedlář
be81aeaa7d gather: Filter arches similarly to pkgset
There should not be multilib arches when checking if noarch package is
compatible with current arch. Otherwise we might be excluding a package
from x86_64 just because it does not work on i686.

JIRA: COMPOSE-2885
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-14 13:27:39 +02:00
Lubomír Sedlář
5f6ee61c70 gather: Fix multilib query for hybrid solver
The multilib library expects us to ask about the candidate package for
another arch (should I include this as multilib?) and not the current
arch (should I include other arches for this package?).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-05 16:04:40 +02:00
Lubomír Sedlář
a53dc6f1bb gather: Expand multilib lists for hybrid method
If there are globs in the blacklist or whitelist, we need to expand it
to full package name.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-05 15:08:31 +02:00
Lubomír Sedlář
1350684c31 Index arch modulemd by full NSVC
There can be multiple modules with the same name and stream. They should
all have the same version, but will have different contexts. Fus takes
only N:S as input, but should pull in all matching modules. We just need
to give it correct data in the repo.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-05 13:10:53 +02:00
Lubomír Sedlář
92968fe52d gather: Keep original rpms.json in debug mode
If we're running in debug mode and the file is already present, it
should not be modified. This means that in order to rerun the actual
gather phase the file needs to be manually deleted first. However the
much more common use is to skip gather phase (because only images should
be re-run). In that case the manifest will be preserved correctly.

JIRA: COMPOSE-2756
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-29 15:52:45 +02:00
Lubomír Sedlář
03ee632cc8 gather: Honor module whitelist
If the modulemd contains a whitelist of packages (under buildopts), it
means the packages are possibly renamed, and we need to check that list
instead of components.rpms.

Multilib does not really work now, anything with non-native arch is
skipped.

JIRA: RCM-38019
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-27 09:08:16 +02:00
Lubomír Sedlář
b12deab153 gather: Get modular packages from fus
Fus returns also RPMs in modules, but until latest version it only
worked if the package was in the same repo as the metadata. This changed
in latest version and now Pungi does not need to expand the list
anymore.

JIRA: COMPOSE-2779
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-03 08:00:37 +02:00
Lubomír Sedlář
4d4c6555e2 gather: Detect hybrid variant with additional packages
The check if a variant is hybrid (or modular only) currently only looks
at comps groups. However it's possible there will be no comps groups,
but packages will be listed explicitly in config as additional_packages.

Relates: RCM-37979
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-26 17:11:13 -04:00
Lubomír Sedlář
b772d4a773 gather: Add langpacks in hybrid solver
Comps file specifies a pattern for some packages. If that package is
installed, all packages matching the pattern are added as well. This can
be added to fus as another pass.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-23 09:08:33 -04:00
Lubomír Sedlář
9ca454007a gather: Early exit for non-comps sources
When getting list of initial packages, only run the source and do
nothing else. Additional package, system-release etc. will be added only
to comps.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-19 09:11:35 +02:00
Lubomír Sedlář
ba0193ca28 gather: Add a hybrid depsolver backend
This patch adds a new gather method called `hybrid`, which uses a `fus`
binary, which must exist somewhere on the `$PATH`. It will call it
multiple times to add multilib packages.

The solver can handle packages, modules and comps groups as input.
However comps groups are never passed in. Pungi will expand it to a list
of packages to avoid issues with comps handling in fus. It ignores
optional packages, and if the group mentions a package that does not
exist, nothing else from the group is included.

Multilib is also handled outside of fus. Pungi will run it, parse the
packages from output, determines multilib packages and adds them as
input. Then it runs the solver again. This is done until nothing new is
added. Usually two passes should be enough.

Source packages and debuginfo are added as a final step. All debuginfo
packages from any included source are added. If the source or debuginfo
package is included in any lookaside repo, it will be skipped.

The tool expects to get a platform stream that should be provided for
modules to depend on. Pungi looks into the modules and gets the platform
from there. If there are more requests, an error is raised.

There is some missing functionality and options that are ignored.
Particularly these are:

 * gather_fulltree
 * gather_selfhosting
 * greedy_method

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-16 10:21:41 +02:00
Stephen Gallagher
ec96757707 Copy modules instead of reparsing them
The Modulemd.copy() method has been available since libmodulemd 1.1
and is much faster than dumping to a string and parsing it again.

Signed-off-by: Stephen Gallagher <sgallagh@redhat.com>
Merges: https://pagure.io/pungi/pull-request/965
2018-05-31 09:04:28 +02:00
Lubomír Sedlář
4544b454f8 Stop using .message attribute on exceptions
It does not exist on Python 3. Converting the exception to string works
identically.

The validate methods on many phases are simplified by not calling the
parent (which does not do anything).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-15 14:17:46 +02:00
Lubomír Sedlář
eaf58f7d40 gather: Use another variant as lookaside
Create a temporary repository and add it as another lookaside in the
compose.

JIRA: COMPOSE-2426
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-23 10:51:56 +02:00
Lubomír Sedlář
58afece033 Only use comps repo if we really have comps
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-17 15:32:02 +02:00
Ondrej Nosek
cb3d36be5d Variant as a lookaside - configuration
Relates: COMPOSE-2425

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-04-13 14:14:43 +02:00
Lubomír Sedlář
15ccd309fa Remove comps from arch repo
The comps could potentially be different in different variants, so
instead we can create the comps repo for every variant separately and
use two repos instead of one (packages in one repository, comps in
another one).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-12 13:52:36 +02:00
Lubomír Sedlář
8f1beeb54b init: Always filter comps file
Even for Everything we want to filter the comps file to make sure we
remove the stuff that is not compatible with current arch. All groups
are still preserved in that case.

This allows us to do the filtering once in init phase than just use the
prepared file in comps source.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-10 08:21:56 +02:00
Lubomír Sedlář
aab3b04b08 gather: Use comps for given variant
The file for variant is filtered to only contain relevant parts.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-10 08:21:56 +02:00
Lubomír Sedlář
c705488505 Add all packages to whitelist for hybrid variant
If a variant contains both modules and comps groups, we need to include
builds from the compose tag in the package whitelist. However only
packages that are not already provided by any module should be added.

JIRA: COMPOSE-2435
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-09 13:10:36 +02:00
Ondrej Nosek
7798174b30 Tests fail if unittest2 library is missing
Merges: https://pagure.io/pungi/pull-request/894
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-04-09 07:47:17 +02:00