Commit Graph

2132 Commits

Author SHA1 Message Date
Lubomír Sedlář
c07a4d64a1 gather: Stop pulling debuginfo and source for lookaside packages
If a package is in lookaside, we don't really need a debuginfo nor
source for it. Definitely we should not pull them from the package set
even if there is some suitable package. Therefore we should include them
in the output but marked as lookaside.

Fixes: https://pagure.io/pungi/issue/915
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-19 08:26:42 +02:00
Lubomír Sedlář
58afece033 Only use comps repo if we really have comps
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-17 15:32:02 +02:00
Lubomír Sedlář
267ff86f04 pkgset: Use modules PDC API
Instead of the deprecated and confusing unreleasedvariants.

JIRA: COMPOSE-2363
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-17 12:16:18 +02:00
Patrick Uiterwijk
527394707d Access ci_base date via compose
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2018-04-15 02:16:23 +02:00
Lubomír Sedlář
a217eea24e Allow filtering comps for different variants
We already filter for arches, so this just extends the filter to work
with variants as well.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-13 14:56:06 +02:00
Lubomír Sedlář
d7021c5688 comps: Make filtering by attribute more generic
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-13 14:56:06 +02:00
Lubomír Sedlář
44c1e2dc6f pkgset: Dump downloaded modulemd to logs
This will allow easier inspection.

JIRA: COMPOSE-2316
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-13 14:44:51 +02:00
Lubomír Sedlář
4596020ecd Fix PEP8 warning about if not x in y
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-13 14:44:51 +02:00
Ondrej Nosek
cb3d36be5d Variant as a lookaside - configuration
Relates: COMPOSE-2425

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-04-13 14:14:43 +02:00
Lubomír Sedlář
15ccd309fa Remove comps from arch repo
The comps could potentially be different in different variants, so
instead we can create the comps repo for every variant separately and
use two repos instead of one (packages in one repository, comps in
another one).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-12 13:52:36 +02:00
Lubomír Sedlář
9daaf1e038 init: Stop creating module defaults dir twice
JIRA: COMPOSE-2447
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-12 13:38:36 +02:00
Lubomír Sedlář
9f583eeb6d gather: Reduce logs from DNF gathering
The log file used to contain a dump of the configuration, which could
potentially be very large. This patch removes data that is directly
visible in the input kickstart file. Instead it just displays the number
of items.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-12 07:49:44 +02:00
Lubomír Sedlář
d1d074ce28 Clone module defaults into work/ directory
We can not rely on config_dir being writable, and should not modify
anything in there anyway.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-11 16:29:36 +02:00
Petr Šabata
8b24c7cbf3 Update the configuration JSON schema for module_defaults_dir
Signed-off-by: Petr Šabata <contyk@redhat.com>
2018-04-11 16:29:18 +02:00
Petr Šabata
8238011bdd Update configuration docs with module_defaults_dir
Signed-off-by: Petr Šabata <contyk@redhat.com>
2018-04-11 16:29:18 +02:00
Petr Šabata
b9523ff5b0 Handle relative paths in module_defaults_dir
This now handles strings as well as sct_dicts with type=file that
include relative paths.

Signed-off-by: Petr Šabata <contyk@redhat.com>
2018-04-11 16:29:18 +02:00
Petr Šabata
0e7f770fb7 Include module defaults in the repodata
If the compose configuration includes the module_defaults_dir (an
scm_dict), clone the directory, read the module defaults contained
therein and include relevant defaults in the combined modulemd file.

Only defaults for modules present in the variant are included.

This requires libmodulemd 1.2.0+.

Merges: https://pagure.io/pungi/pull-request/891
Signed-off-by: Petr Šabata <contyk@redhat.com>
2018-04-11 16:29:17 +02:00
Lubomír Sedlář
506ac99f62 Add *.in fixtures to tarball
The recent comps related commit added a fixture file with new extension
which is not getting included in the source tarball and therefore causes
tests to fail.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-10 08:33:41 +02:00
Lubomír Sedlář
8f1beeb54b init: Always filter comps file
Even for Everything we want to filter the comps file to make sure we
remove the stuff that is not compatible with current arch. All groups
are still preserved in that case.

This allows us to do the filtering once in init phase than just use the
prepared file in comps source.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-10 08:21:56 +02:00
Lubomír Sedlář
6daee968ae docs: Describe comps processing
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-10 08:21:56 +02:00
Lubomír Sedlář
aab3b04b08 gather: Use comps for given variant
The file for variant is filtered to only contain relevant parts.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-10 08:21:56 +02:00
Lubomír Sedlář
064f922117 docs: Fix typo
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-10 08:21:56 +02:00
Lubomír Sedlář
c705488505 Add all packages to whitelist for hybrid variant
If a variant contains both modules and comps groups, we need to include
builds from the compose tag in the package whitelist. However only
packages that are not already provided by any module should be added.

JIRA: COMPOSE-2435
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-09 13:10:36 +02:00
Lubomír Sedlář
de231064b7 comps: Add tests for CompsFilter
All use cases that are actually used by pungi-koji are tested. There is
missing coverage for

 * keeping only items with matching arch
 * not reindenting the file

These aren't currently used and should be removed in the future, but
there may be other tools depending on the comps_filter executable.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-09 07:48:28 +02:00
Lubomír Sedlář
7ea4c33d87 comps: Move filtering into wrapper module
The code should not live directly in the executable, that makes it very
hard to test.

Other than the move there is no functional change.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-09 07:48:28 +02:00
Ondrej Nosek
7798174b30 Tests fail if unittest2 library is missing
Merges: https://pagure.io/pungi/pull-request/894
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-04-09 07:47:17 +02:00
Robert Marshall
6454f30714 Add unittest2 and rpmdevools to contributing doc
- Add rpmdevtools and python-unittest2 to the list of packages
  required to set up a testing environment

Signed-off-by: Robert Marshall <rmarshall@redhat.com>
2018-04-06 19:27:22 -04:00
Lubomír Sedlář
f38770c67d pkgset: Construct UID for PDC modules
We can't rely on the UID to be correctly joined with colons. There may
be historical data that still uses dashes.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-06 10:47:22 +02:00
Lubomír Sedlář
b3a3575ecf gather: Simplify creating temporary directory
There is no need to create the directory in work/, as it will get
deleted immediately. Let's move it to /tmp and use the context manager
to clean it up.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-05 15:21:39 +02:00
Lubomír Sedlář
d9e2101b08 buildinstall: Add extra repos
A new configuration option is added that allows users to point lorax to
extra repositories. This can be handy if some tools to create the
bootable image are not part of the product itself.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
JIRA: COMPOSE-2253
2018-04-05 07:56:20 +02:00
Lubomír Sedlář
1436ea2b03 tests: Use dummy modulesdir for DNF
Otherwise it tries to ensure it exists, and since the default is
/etc/dnf/modules.d, it's causing problems if the directory does not
exist and user does not have permissions to create it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-04 12:23:43 +02:00
Ondrej Nosek
1f0739831c Update tests for Python 2.6
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-03-29 16:34:52 +02:00
Ondrej Nosek
f3b5a66614 4.1.23 release
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-03-29 14:11:58 +02:00
Ondrej Nosek
9b46377fb6 Update documentation section 'contributing'
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-03-29 11:23:57 +02:00
Ondrej Nosek
e3aa2f769b Write module metadata
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-03-29 09:47:55 +02:00
Jan Kaluza
c6d507582a Support multilib in GatherSourceModule
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-03-28 13:38:46 +02:00
Lubomír Sedlář
071792bdd0 ostree: Always substitute basearch
When ref is not modified via pungi config, we read it from the treefile
and substitute in basearch.

When pungi is configured to replace it, it modifies the treefile and
then used the value from config to avoid parsing the file. This however
did not substitute the basearch value.

We can simply use one code path for getting the value. This will work
for both cases.

Fixes: https://pagure.io/pungi/issue/866
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-21 15:37:15 +01:00
Patrick Uiterwijk
f814651d91 If sigkeys is specified, require at least one
Merges: https://pagure.io/pungi/pull-request/880
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2018-03-21 15:34:58 +01:00
Jan Kaluza
9be2d6a920 Allow setting <kojitag/> in <modules/> in variants.xml to get the modules from this Koji tag.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-03-21 14:33:45 +01:00
Jan Kaluza
1574f306c7 Move Modulemd import to pungi/__init__.py to remove duplicated code.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-03-20 12:59:17 +01:00
Jan Kaluza
3f71cdd384 Use Modulemd.Module for 'variant.arch_mmds' instead of yaml dump
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
Merges: https://pagure.io/pungi/pull-request/872
2018-03-19 14:47:03 +01:00
Lubomír Sedlář
6bff4bd10e Fix modular content in non-modular variant
When allowing empty list of modules, the check for variant tags got
broken, causing Everything to no longer have an associated list of
allowed packages.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Relates: https://pagure.io/pungi/issue/862
2018-03-16 14:47:10 +01:00
Jan Kaluza
fedce5dff1 Remove the filtered RPMs from module metadata even in case all RPMs are filtered out.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-03-15 11:13:39 +01:00
Lubomír Sedlář
5c902592ae pkgset: Allow empty list of modules
This should indicate that it's a modular variant, but there is no
modular content yet. We don't want to treat that as Everything.

The end result will be an empty repository.

Fixes: https://pagure.io/pungi/issue/871
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-14 14:49:09 +01:00
Lubomír Sedlář
95bb147015 buildinstall: Add option to disable it
Fixes: https://pagure.io/pungi/issue/854
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-13 15:54:52 +01:00
Jan Kaluza
340ae4d286 Use libmodulemd instead of modulemd Python module
Merges: https://pagure.io/pungi/pull-request/851
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-03-12 13:07:33 +01:00
Lubomír Sedlář
56e00505e0 gather: Fix package set whitelist
We need to include all relevant arches, not just the base one (including
noarch and src). However the list can be shortened by only listing
NEVRs, because that should be unique.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-09 15:53:31 +01:00
Lubomír Sedlář
c83316da31 pkgset: Merge initial package set without checks
For the first pass we don't need to filter out exclusive architectures,
and we don't need to exclude source packages without any binary
packages. We just want to merge the two package sets as fast as
possible.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-08 14:29:23 +01:00
Lubomír Sedlář
b393a4246b pkgset: Remove check for unique name
We now have a way to select even older version of package (since the
newer one can be left out of the whitelist), so we can include multiple
versions of the same package into global package set.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-08 14:29:23 +01:00
Lubomír Sedlář
a03a46a078 gather: Honor package whitelist
Basically everything not on the list is excluded. This has to be applied
before we filter only the latest versions (otherwise we could lose
packages that are on the whitelist).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-08 14:29:23 +01:00