Commit Graph

522 Commits

Author SHA1 Message Date
Lubomír Sedlář
eaf58f7d40 gather: Use another variant as lookaside
Create a temporary repository and add it as another lookaside in the
compose.

JIRA: COMPOSE-2426
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-23 10:51:56 +02:00
Lubomír Sedlář
ea0964eeef buildinstall: Use metadata if skipped
If the phase is skipped, it could mean that we are doing a debug run and
we don't want to mess up the .treefile by missing arch specific images.
The other alternative is that the phase was really skipped, in which
case there will be no files generated and we already handle that fine.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-23 10:42:33 +02:00
Jan Kaluza
9915c7d644 Allow reusing pkgset FileCache from old composes.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-04-23 09:57:03 +02:00
Lubomír Sedlář
58afece033 Only use comps repo if we really have comps
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-17 15:32:02 +02:00
Lubomír Sedlář
267ff86f04 pkgset: Use modules PDC API
Instead of the deprecated and confusing unreleasedvariants.

JIRA: COMPOSE-2363
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-17 12:16:18 +02:00
Patrick Uiterwijk
527394707d Access ci_base date via compose
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2018-04-15 02:16:23 +02:00
Lubomír Sedlář
a217eea24e Allow filtering comps for different variants
We already filter for arches, so this just extends the filter to work
with variants as well.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-13 14:56:06 +02:00
Ondrej Nosek
cb3d36be5d Variant as a lookaside - configuration
Relates: COMPOSE-2425

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-04-13 14:14:43 +02:00
Lubomír Sedlář
15ccd309fa Remove comps from arch repo
The comps could potentially be different in different variants, so
instead we can create the comps repo for every variant separately and
use two repos instead of one (packages in one repository, comps in
another one).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-12 13:52:36 +02:00
Petr Šabata
0e7f770fb7 Include module defaults in the repodata
If the compose configuration includes the module_defaults_dir (an
scm_dict), clone the directory, read the module defaults contained
therein and include relevant defaults in the combined modulemd file.

Only defaults for modules present in the variant are included.

This requires libmodulemd 1.2.0+.

Merges: https://pagure.io/pungi/pull-request/891
Signed-off-by: Petr Šabata <contyk@redhat.com>
2018-04-11 16:29:17 +02:00
Lubomír Sedlář
8f1beeb54b init: Always filter comps file
Even for Everything we want to filter the comps file to make sure we
remove the stuff that is not compatible with current arch. All groups
are still preserved in that case.

This allows us to do the filtering once in init phase than just use the
prepared file in comps source.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-10 08:21:56 +02:00
Lubomír Sedlář
aab3b04b08 gather: Use comps for given variant
The file for variant is filtered to only contain relevant parts.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-10 08:21:56 +02:00
Lubomír Sedlář
de231064b7 comps: Add tests for CompsFilter
All use cases that are actually used by pungi-koji are tested. There is
missing coverage for

 * keeping only items with matching arch
 * not reindenting the file

These aren't currently used and should be removed in the future, but
there may be other tools depending on the comps_filter executable.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-09 07:48:28 +02:00
Ondrej Nosek
7798174b30 Tests fail if unittest2 library is missing
Merges: https://pagure.io/pungi/pull-request/894
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-04-09 07:47:17 +02:00
Lubomír Sedlář
f38770c67d pkgset: Construct UID for PDC modules
We can't rely on the UID to be correctly joined with colons. There may
be historical data that still uses dashes.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-06 10:47:22 +02:00
Lubomír Sedlář
d9e2101b08 buildinstall: Add extra repos
A new configuration option is added that allows users to point lorax to
extra repositories. This can be handy if some tools to create the
bootable image are not part of the product itself.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
JIRA: COMPOSE-2253
2018-04-05 07:56:20 +02:00
Lubomír Sedlář
1436ea2b03 tests: Use dummy modulesdir for DNF
Otherwise it tries to ensure it exists, and since the default is
/etc/dnf/modules.d, it's causing problems if the directory does not
exist and user does not have permissions to create it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-04 12:23:43 +02:00
Ondrej Nosek
1f0739831c Update tests for Python 2.6
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-03-29 16:34:52 +02:00
Ondrej Nosek
e3aa2f769b Write module metadata
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-03-29 09:47:55 +02:00
Jan Kaluza
c6d507582a Support multilib in GatherSourceModule
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-03-28 13:38:46 +02:00
Patrick Uiterwijk
f814651d91 If sigkeys is specified, require at least one
Merges: https://pagure.io/pungi/pull-request/880
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2018-03-21 15:34:58 +01:00
Jan Kaluza
9be2d6a920 Allow setting <kojitag/> in <modules/> in variants.xml to get the modules from this Koji tag.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-03-21 14:33:45 +01:00
Jan Kaluza
1574f306c7 Move Modulemd import to pungi/__init__.py to remove duplicated code.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-03-20 12:59:17 +01:00
Jan Kaluza
3f71cdd384 Use Modulemd.Module for 'variant.arch_mmds' instead of yaml dump
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
Merges: https://pagure.io/pungi/pull-request/872
2018-03-19 14:47:03 +01:00
Jan Kaluza
fedce5dff1 Remove the filtered RPMs from module metadata even in case all RPMs are filtered out.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-03-15 11:13:39 +01:00
Lubomír Sedlář
5c902592ae pkgset: Allow empty list of modules
This should indicate that it's a modular variant, but there is no
modular content yet. We don't want to treat that as Everything.

The end result will be an empty repository.

Fixes: https://pagure.io/pungi/issue/871
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-14 14:49:09 +01:00
Lubomír Sedlář
95bb147015 buildinstall: Add option to disable it
Fixes: https://pagure.io/pungi/issue/854
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-13 15:54:52 +01:00
Jan Kaluza
340ae4d286 Use libmodulemd instead of modulemd Python module
Merges: https://pagure.io/pungi/pull-request/851
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-03-12 13:07:33 +01:00
Lubomír Sedlář
56e00505e0 gather: Fix package set whitelist
We need to include all relevant arches, not just the base one (including
noarch and src). However the list can be shortened by only listing
NEVRs, because that should be unique.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-09 15:53:31 +01:00
Lubomír Sedlář
c83316da31 pkgset: Merge initial package set without checks
For the first pass we don't need to filter out exclusive architectures,
and we don't need to exclude source packages without any binary
packages. We just want to merge the two package sets as fast as
possible.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-08 14:29:23 +01:00
Lubomír Sedlář
a03a46a078 gather: Honor package whitelist
Basically everything not on the list is excluded. This has to be applied
before we filter only the latest versions (otherwise we could lose
packages that are on the whitelist).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-08 14:29:23 +01:00
Lubomír Sedlář
41d0139b39 Write package whitelist for each variant
If we have a package set for the variant (which happens if there are
modules), include a list of all NEVRAs in the pungi kickstart.

This can be used to make sure only packages from correct tag get into
the compose. If two packages with same name but different version get
into the compose, this can help get even older version into a particular
variant.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-08 14:29:23 +01:00
Lubomír Sedlář
bd852f4059 Remove comps groups from purely modular variants
The comps source should not return all groups when there are only
modules defined. This fixes part of the problem: non-modular packages
will not go in by default.

The second part is the comps file in the created repository. It will be
filtered to not contain any groups (because packages from there will not
be in the repo).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-02 08:46:23 +01:00
Lubomír Sedlář
3201648c37 gather: Allow filtering debuginfo packages
This already works on YUM backend, and this patch makes it work for DNF
as well.

Both native and multilib debuginfo and debugsource packages will be
excluded. This matches the yum behavior.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-02-28 10:07:42 +01:00
Ondrej Nosek
660c04368b Move ostree phase and pipelines for running phases
Signed-off-by: Ondrej Nosek <onosek@redhat.com>

Related: https://pagure.io/pungi/issue/778
2018-02-26 13:43:44 +01:00
Ondrej Nosek
5c081cb545 Other repo for OstreeInstaller
OstreeInstaller phase will be moved to a different timeslot
and therefore needs different repo not to depend on Gather
phase which runs at the same time.

Related: https://pagure.io/pungi/issue/778

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-02-26 13:10:36 +01:00
Adam Williamson
a1d559fb93 Correct fix for volume ID substition sorting by length
The previous attempt - caed78e - is not really correct. It sorts
the dict item tuples according to the alphabetical sort order of
the first item of each tuple (reversed). This will always work
when both substitutions *start* with the same characters, as in
the case of two strings that start with the same characters but
have a different length, the shorter one sorts alphabetically
first, and we reverse that. But it is not safe if the shorter
substitution doesn't start with the same characters, as in the
case I put in the tests: we should sort 'zzzaaaaaazzz' before
'aaaaaa' (and hence apply the 'zzzaaaaaazzz' substitution to a
volume ID that contains that string and not the 'aaaaaa' one),
but the previous commit did not.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-02-23 14:48:04 -08:00
Ondrej Nosek
caed78e11a Ordering processing for volume ID substitutions
Related: https://pagure.io/pungi/issue/840

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-02-22 16:46:58 +01:00
Jan Kaluza
ef058d1f9b Fail early if input packages are unsigned
Use 'get_packages_to_gather' to fail early if these packages are not
signed with right key. This prevents us from having to wait for the
repo to be created and depsolving to finish. Unsigned dependencies will
still be reported later than previously.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-02-13 12:35:55 +01:00
Jan Kaluza
c75f4a1e96 Allow composing from tag with unsigned packages
There can be packages in the tag that will not end up in the compose.
Instead of failing immediately with error, this patch delays the check
until after depsolving finishes and only checks packages that will
really be included.

This is not an issue for nodeps compose, as that already pulls in only
packages that will be composed and nothing else.

Merges: https://pagure.io/pungi/pull-request/843
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-02-13 12:35:17 +01:00
Ondrej Nosek
c7cc200246 Ostree can use pkgset repos
Related: https://pagure.io/pungi/issue/778

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-02-08 15:23:15 +01:00
Lubomír Sedlář
364d7f5229 Support multiple sources in one variant
With this patch the gather_source option is no longer used. Instead, all
sources are always used. If they return at least some input packages,
then a configured method is used and the returned lists of packages from
all sources are merged.

The method used for gathering can be configured for each variant and
gather source separately.

Additional packages are only added to the comps source.

Each gathering step is logged separately. All the logs are preserved for
later inspection.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-02-01 14:31:09 +01:00
Ondrej Nosek
f301158974 ostree/utils: Drop timestamps from generated repo names - tests
Issue: https://pagure.io/pungi/issue/811

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-01-17 16:48:22 +01:00
Lubomír Sedlář
03293c725b config: Make pkgset_koji_tag optional
There are valid use cases for not specifying this option: specifically a
modular compose will get the tags to use from modules listed in the
variants file.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-01-12 15:50:21 +01:00
Ondrej Nosek
51cd359057 ostree: Add force_new_commit option - test added
Relates: https://pagure.io/pungi/issue/811

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-01-11 15:08:07 +01:00
Ondrej Nosek
d4d264eb33 Improve logging for unsigned packages
Relates: https://pagure.io/pungi/issue/820

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-01-03 11:08:00 +01:00
Ondrej Nosek
2152e7ea26 Fall back to mount if guestmount is not available
Relates: https://pagure.io/pungi/issue/803

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-01-02 17:13:47 +01:00
Ondrej Nosek
116e7ca3bd El-Torito boot information on s390x
Relates: https://github.com/rhinstaller/lorax/pull/236

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2017-12-14 10:08:30 +01:00
Ondrej Nosek
59c162d46f Remove strace from buildinstall runroot
The package is not really needed.

Relates: https://pagure.io/pungi/issue/799

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2017-12-11 17:36:28 +01:00
Lubomír Sedlář
e15a49defd tests: Use correct python version for config validation test
When tests are running using Py3, the script should be called with Py3
as well.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-12-06 13:26:43 +01:00