Commit Graph

80 Commits

Author SHA1 Message Date
Romain Forlot
da791ed15c Fix can't link XDEV using repos as pkgset_sources
Trying to compose from external classic repositories return an error trying the hardling from
a yum cache directory located in /tmp to the target directory in another filesystem.
This commit fixes this using the 'link' method form linker module which handle the link_type
configuration parameter instead of the hardcoded method 'hardlink'.

Change-Id: Ib79cfbd72f9def6462fddb2ae368730c55f257cd
Signed-off-by: Romain Forlot <romain.forlot@iot.bzh>
2021-04-22 14:03:12 +02:00
Lubomír Sedlář
59e2aa9607 Fix flake8 issues
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-05-26 11:49:50 +02:00
Haibo Lin
ff269a8675 gather: Make additional_packages accept debuginfo packages
Now debuginfo packages defined in additional_packages option
could be handled correctly.

JIRA: COMPOSE-4085
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-12 16:38:08 +08:00
Haibo Lin
c0193c9fca Fix flake8 complaints - E501
E501 line too long (92 > 88 characters)
E501 line too long (103 > 88 characters)
...

JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 14:36:46 +08:00
Haibo Lin
41a629969c Format code base with black
https://black.readthedocs.io/en/stable/

JIRA: COMPOSE-4086
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-05 17:35:47 +08:00
Lubomír Sedlář
4c6396f491 Remove deprecated warn() call
Instead warning() should be called.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-29 15:34:00 +01:00
Frédéric Pierret (fepitre)
c624aab945
pungi-gather: add options for excluding debug and source packages
Signed-off-by: Frédéric Pierret (fepitre) <frederic.pierret@qubes-os.org>
2020-01-11 22:13:51 +01:00
Lubomír Sedlář
b8bb4f7daa Improve detection of lookaside packages
It is possible we only ever see a package from non-lookaside repo, but
it actually is in lookaside. This patch should check for that before
adding the package to result set.

JIRA: RCM-71946
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-02 13:50:29 +01:00
Lubomír Sedlář
26962d94ca gather: Resolve dependencies of debug packages
Until now, the behaviour was that all debuginfo from a build would be
included if at least one package with the same arch was included.

This resulted in many debuginfo packages being included even though
their corresponding package was not present.

With this patch, we will always pull in debugsource, but foo-debuginfo
will only be included if foo is included for the same arch. As a
consequence, it is necessary to resolve dependencies of debuginfo
packages. There are cases where foo-debuginfo needs foo-debuginfo-common
for example.

This change means that DNF and YUM backends are no longer identical in
the output. The tests where this is demonstrated are duplicated and
their results are modified.

JIRA: COMPOSE-3823
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:46:21 +02:00
Lubomír Sedlář
4413b0df24 gather: Fix bare except block
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:46:21 +02:00
Lubomír Sedlář
89fcb79aca gather: Fix formatting
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:46:21 +02:00
Lubomír Sedlář
428f8297e1 gather: Simplify adding packages to result
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:46:21 +02:00
Lubomír Sedlář
c7f3f38822 gather: Remove unused import
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:46:21 +02:00
Lubomír Sedlář
63a8b7b6c9 Remove package whitelist
This was a workaround to make some packages from the global repo
invisible for depsolving. This is now handled by packages being in
different repos. We can select which repos are enabled at which point.
This achieves the same result, but much faster.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
7d099012aa Fix tests for DNF 3
The configuration for modules has changed.

Lookaside handling has changed, and there are now test failures. This
is a not a bug in DNF, so we need to fix it on our side.

Relates: https://bugzilla.redhat.com/show_bug.cgi?id=1603123
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-19 14:22:10 +02:00
Lubomír Sedlář
c07a4d64a1 gather: Stop pulling debuginfo and source for lookaside packages
If a package is in lookaside, we don't really need a debuginfo nor
source for it. Definitely we should not pull them from the package set
even if there is some suitable package. Therefore we should include them
in the output but marked as lookaside.

Fixes: https://pagure.io/pungi/issue/915
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-19 08:26:42 +02:00
Lubomír Sedlář
9f583eeb6d gather: Reduce logs from DNF gathering
The log file used to contain a dump of the configuration, which could
potentially be very large. This patch removes data that is directly
visible in the input kickstart file. Instead it just displays the number
of items.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-12 07:49:44 +02:00
Lubomír Sedlář
56e00505e0 gather: Fix package set whitelist
We need to include all relevant arches, not just the base one (including
noarch and src). However the list can be shortened by only listing
NEVRs, because that should be unique.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-09 15:53:31 +01:00
Lubomír Sedlář
a03a46a078 gather: Honor package whitelist
Basically everything not on the list is excluded. This has to be applied
before we filter only the latest versions (otherwise we could lose
packages that are on the whitelist).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-08 14:29:23 +01:00
Lubomír Sedlář
3201648c37 gather: Allow filtering debuginfo packages
This already works on YUM backend, and this patch makes it work for DNF
as well.

Both native and multilib debuginfo and debugsource packages will be
excluded. This matches the yum behavior.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-02-28 10:07:42 +01:00
Lubomír Sedlář
0074fe3f2c gather: Set lookaside flag focorrectly
For sources and debuginfo package the flag should be set based on the
repo we pull that particular RPM from, not for the corresponding binary
package.

There could be a case where they come from different repos, so this
would cause problems.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-01-25 15:54:00 +01:00
Lubomír Sedlář
e8fa2a13b1 gather: Try getting srpm from the same repo as rpm
When the same package is contained in the main repo for depsolving as
well as in a lookaside repo, there was a chance that we would pick a
package from main repo, but corresponding source from lookaside.
This would cause a crash when trying to link the packages into the
compose.

A solution is to try to get source rpm from the same repo as the binary
package. Only when it's not available there we get some other one.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-01-25 13:52:55 +01:00
Lubomír Sedlář
ed22e07ef9 Port to Python 3
This should make all tests pass on both Python 2 and Python 3.

Unittest2 is required on Py 2.6 and Py 3.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-24 15:15:31 +02:00
Lubomír Sedlář
d34c0a2777 gather(dnf): Remove dead code
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-09-12 08:38:08 +02:00
Lubomír Sedlář
239e6b4301 gather(dnf): Don't exclude packages from lookaside
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-09-12 08:38:08 +02:00
Lubomír Sedlář
816ab917da gather-dnf: Warn about unresolvable dependencies
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-09-12 08:38:08 +02:00
Lubomír Sedlář
c3b49f7ffb pkgset: Download packages with dnf
When using repos as gather_source, we should use DNF backend even for
constructing initial package set and to download the packages from
source repos. Without this the repos source would not be usable on
Python 3.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-16 14:32:26 +02:00
Lubomír Sedlář
a8cd78faf9 gather: Add fulltree-exclude flag to DNF backend
This is needed for correct trimming of addons and optional. A package
pulled in as a dependency but that matches something on fulltree exclude
list should have this flag. It will then be moved from addon to base
variant and therefore excluded from optional.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-16 13:06:55 +02:00
Till Maas
b26547ae77 Use only one list of patterns/rules for debug packages
Signed-off-by: Till Maas <opensource@till.name>
2017-07-26 23:18:29 +02:00
Till Maas
938531e2b2 Do not match "*-debugsource-*" as debuginfo package
These kind of packages are not expected to exist in the wild.

Signed-off-by: Till Maas <opensource@till.name>
2017-07-26 23:18:24 +02:00
Lubomír Sedlář
d92390b80b Add support for debugsource packages
These packages should behave like regular debuginfo packages (at least
for now).

Fixes: https://pagure.io/pungi/issue/684
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-26 10:34:18 +02:00
Lubomír Sedlář
3130d837c0 gather: Don't resolve dependencies in lookaside
When looking at a package in a lookaside repo, it does not make much
sense to process its dependencies. We should just assume that the
lookaside can satisfy them.

In the worst case, this could result in packages being pulled into the
compose just so that they could satisfy a dep of something in lookaside.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-05-29 12:55:38 +02:00
Lubomír Sedlář
dc410b58b5 Remove shebang from non-executable files
This fixes an rpmlint error on built package.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-05-10 09:07:16 +02:00
Lubomír Sedlář
29c339d659 gather-dnf: Log exact Requires pulling a package in
This is useful for debugging.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-26 10:04:26 +02:00
Lubomír Sedlář
fcd4c231c4 Stop importing hawkey directly
It's API is considered deprecated.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:50 +01:00
Lubomír Sedlář
27b24a489f Handle noarch debuginfo packages
Pull debuginfo packages for noarch packages as well, but only when the
debuginfo is noarch as well.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:50 +01:00
Lubomír Sedlář
d23a2f4548 Fix multilib exclude pattern to not match noarch
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:50 +01:00
Lubomír Sedlář
5533bf7ca3 Simplify add_binary_package_deps
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:50 +01:00
Lubomír Sedlář
c32bdce46d Work around a bug in DNF
https://bugzilla.redhat.com/show_bug.cgi?id=1418298

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:50 +01:00
Lubomír Sedlář
73b8928013 Expand multilib blacklist and whitelist
This way we pay once for the expansion, and lookups are then done in
constant time instead of iterating over the list over and over again.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:50 +01:00
Lubomír Sedlář
5c1d04eb00 Add matching multilib method to log
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:50 +01:00
Lubomír Sedlář
08fbdec494 Add detailed reason why package is added
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář
919e4d748e Remove extra section for adding greedy=build packages
It does not change any result for tests, and makes it work a lot faster.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář
fe026bb588 Handle Requires(pre|post)
These requires are accessible from a separate attribute, but we want to
handle them the same ways as regular Requires.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář
2f40a45708 Honor multilib_blacklist for initial packages matching a glob
We want to exclude all multilib packages matching the blacklist (that
are not noarch) from the package set.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář
d4f8e32a80 Fix handling globs in input packages
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář
a9415eb0e5 Simplify add_multilib_package method
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář
cd80d6ef5a Stop caching provides of added packages
This only works in non-greedy mode. When greedy, the same provides can
be linked to multiple packages that should be pulled in.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář
7343daba24 Clean up logging excluded packages
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář
ec67eac1cc Use enum for flags
This adds an extra dependency on python-enum34.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00