Commit Graph

639 Commits

Author SHA1 Message Date
Jan Kaluza
37c89dfde6 Add 'pkgset_koji_builds' option to include extra builds in a compose
This PR adds new pkgset_koji_builds configuration option.

This option allows setting list of extra Koji build NVRs which will be
included in a compose. This is useful in two cases:

a) It allows generating standard composes with few packages update to
certain version to test how the compose behaves when the package is
updated for real.

b) It allows generating compose consisting only from particular builds
when pkgset_koji_tag = '' or None. This is useful when one want to
regenerate the compose with packages which are not tagged in single Koji
tag. This is very useful for ODCS when reproducing old composes.

Merges: https://pagure.io/pungi/pull-request/1049
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-09-21 14:06:49 +02:00
Lubomír Sedlář
2a65b8fb7d ostree: Reduce duplication in tests
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-19 13:07:03 +02:00
Lubomír Sedlář
36c347eb79 ostree: Use --touch-if-changed
There are three different cases:

 * we expect commitid and it's there
 * we expect commitid and it's missing
 * we don't expect commitid

This patch helps differentiate between the second two. In former one we
should report an error and mark the phase as failed. The latter is
perfectly fine and no error should be reported

Fixes: https://pagure.io/pungi/issue/1046
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-19 13:07:03 +02:00
Lubomír Sedlář
a53dc6f1bb gather: Expand multilib lists for hybrid method
If there are globs in the blacklist or whitelist, we need to expand it
to full package name.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-05 15:08:31 +02:00
Lubomír Sedlář
1350684c31 Index arch modulemd by full NSVC
There can be multiple modules with the same name and stream. They should
all have the same version, but will have different contexts. Fus takes
only N:S as input, but should pull in all matching modules. We just need
to give it correct data in the repo.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-05 13:10:53 +02:00
Lubomír Sedlář
fbb739ef17 pkgset: Apply whitelist to modules in the tag
This patch changes the behaviour when both module tag and NSV?C? is
specified. The NSVC are used as a whitelist and only matching modules
will be included in the compose.

Additionally this patch adds filtering based on inheritance: when
finding the latest module for each N:S combination, only the top tag in
which the module is tagged is used. Even if a newer build is available
somewhere deeper in the inheritance, it's not going to be used.

Example inheritance and tagged modules

    f29-compose (foo:1:2018:cafe)
    └─ f29-candidate (foo:1:2019:cafe)

The compose will use 2018 version, because it's in the topmost tag.

JIRA: COMPOSE-2685
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-03 12:19:50 +02:00
Lubomír Sedlář
3419762830 extra_iso: Set unified flag in metadata
This will avoid conflict in productmd that two images share the same
attributes.

JIRA: COMPOSE-2908
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-31 08:46:21 +02:00
Lubomír Sedlář
828557b4d6 pkgset: Respect koji event when searching for modules
In the search result, we should ignore any module build that finished
after the event that we are working with.

Fixes: https://pagure.io/pungi/issue/999
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-30 14:58:45 +02:00
Jan Kaluza
d2f392fac8 Use dogpile.cache to cache the listTaggedRPMS calls if possible
If the same tag is queried with the same event, Pungi can cache the
response and call the API again. Particularly for small composes this
can save up significant amount of time.

Merges: https://pagure.io/pungi/pull-request/1022
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-08-30 13:21:01 +02:00
Lubomír Sedlář
92968fe52d gather: Keep original rpms.json in debug mode
If we're running in debug mode and the file is already present, it
should not be modified. This means that in order to rerun the actual
gather phase the file needs to be manually deleted first. However the
much more common use is to skip gather phase (because only images should
be re-run). In that case the manifest will be preserved correctly.

JIRA: COMPOSE-2756
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-29 15:52:45 +02:00
Lubomír Sedlář
7c5020e82d Reduce duplication in tests
Set the dummy compose into non-debug mode in a single place, and only
change it when needed.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-29 15:52:45 +02:00
Lubomír Sedlář
5346e000f0 Create non-bootable ISO for variant without buildinstall
If the configuration specifically excludes a variant from buildinstall,
but does not also disable ISO creation, we should just create a
non-bootable ISO instead of reporting a warning.

JIRA: COMPOSE-2887
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-28 08:56:03 +02:00
Lubomír Sedlář
45fa9d3273 Clean up after yum tests
The tests create empty directories in current working directory, but
they never clean them up. Instead we can switch to a temporary location
that will be cleaned up automatically.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-27 09:16:40 +02:00
Lubomír Sedlář
4b05e7b9fd extra-iso: Rename test data file
The file is not being included in the source tarball, which breaks
internal jenkins and will eventually cause the test to fail at RPM build
time.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-15 14:57:43 +02:00
Lubomír Sedlář
ac0d5e4ede createiso: Use correct python version
On some systems we need to use python3, in other places it can be
another version. Instead of guessing, let's look at shebang line in
lorax executable and use the same.

JIRA: COMPOSE-2852
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-15 08:37:12 +02:00
Lubomír Sedlář
b85cd7ff9f ostree: Update tests for working with YAML file
Merges: https://pagure.io/pungi/pull-request/1019
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-14 16:13:23 +02:00
Lubomír Sedlář
2fa1f09827 createrepo: Allow passing arbitrary arguments
This could be used to enable zchunk generation, which can require up to
4 different options. Instead of hardcoding every single one, let's just
allow more direct access to the executed command.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-08 16:15:51 +02:00
Lubomír Sedlář
b12deab153 gather: Get modular packages from fus
Fus returns also RPMs in modules, but until latest version it only
worked if the package was in the same repo as the metadata. This changed
in latest version and now Pungi does not need to expand the list
anymore.

JIRA: COMPOSE-2779
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-03 08:00:37 +02:00
Lubomír Sedlář
5926858b58 util: Remove escaping spaces from volume ID
This was not configurable for users, and the default was always used,
which meant no escaping. Might as well just remove the dead code.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-03 07:57:09 +02:00
Lubomír Sedlář
add9835b56 Allow removing non-alnum chars from volid
This mimics similar change in lorax.

JIRA: RCM-36970
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-03 07:57:09 +02:00
Lubomír Sedlář
10bdb370ba extra-isos: Include treeinfo pointing to all variants
This will be used by Anaconda to consume multiple repos for
installation.

JIRA: RCM-36970
JIRA: COMPOSE-2753
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>

Fixup
2018-08-03 07:57:09 +02:00
Lubomír Sedlář
b772d4a773 gather: Add langpacks in hybrid solver
Comps file specifies a pattern for some packages. If that package is
installed, all packages matching the pattern are added as well. This can
be added to fus as another pass.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-23 09:08:33 -04:00
Lubomír Sedlář
16ac225013 comps: Add get_langpacks function
This also simplifies the test data by removing a lot of the langpacks.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-23 09:08:33 -04:00
Lubomír Sedlář
7d099012aa Fix tests for DNF 3
The configuration for modules has changed.

Lookaside handling has changed, and there are now test failures. This
is a not a bug in DNF, so we need to fix it on our side.

Relates: https://bugzilla.redhat.com/show_bug.cgi?id=1603123
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-19 14:22:10 +02:00
Lubomír Sedlář
0b9652f2d6 tests: Use unittest2 when available
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-18 12:55:34 +02:00
Jan Kaluza
6080b45178 Fix Koji search for modules with dash in stream
MBS replaces the dash by underscore. We have to match that.

Merges: https://pagure.io/pungi/pull-request/1005
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-07-18 10:31:19 +02:00
Lubomír Sedlář
23c454ff67 buildinstall: Make output world readable
This makes it possible to run a compose as non-root user, plus removes
the need for workarounds to publish the results directly.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-17 10:12:52 +02:00
Lubomír Sedlář
e3de4dcccf Report failed failable deliverables as errors
It does not abort the compose, but it's still an error. It should be
marked as such in the log. This will allow easier searching for the
failures.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-17 09:31:50 +02:00
Lubomír Sedlář
ba0193ca28 gather: Add a hybrid depsolver backend
This patch adds a new gather method called `hybrid`, which uses a `fus`
binary, which must exist somewhere on the `$PATH`. It will call it
multiple times to add multilib packages.

The solver can handle packages, modules and comps groups as input.
However comps groups are never passed in. Pungi will expand it to a list
of packages to avoid issues with comps handling in fus. It ignores
optional packages, and if the group mentions a package that does not
exist, nothing else from the group is included.

Multilib is also handled outside of fus. Pungi will run it, parse the
packages from output, determines multilib packages and adds them as
input. Then it runs the solver again. This is done until nothing new is
added. Usually two passes should be enough.

Source packages and debuginfo are added as a final step. All debuginfo
packages from any included source are added. If the source or debuginfo
package is included in any lookaside repo, it will be skipped.

The tool expects to get a platform stream that should be provided for
modules to depend on. Pungi looks into the modules and gets the platform
from there. If there are more requests, an error is raised.

There is some missing functionality and options that are ignored.
Particularly these are:

 * gather_fulltree
 * gather_selfhosting
 * greedy_method

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-16 10:21:41 +02:00
Lubomír Sedlář
d2849d3826 osbs: Generate unique repo names
And include variant in repo file name. The whole path is unique already,
but not the filename itself.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-09 13:21:58 +02:00
Sinny Kumari
1759c1ba80 Expand version field during image_build using version_generator
We can specify !VERSION_FROM_VERSION in version field during
image_build to expand it to correct release number without any label
information.

Also implemented !RELEASE_FROM_DATE_RESPIN to provide correct
release number. This helps to keep Atomic Host media files name
produced by image_build during bodhi updates compose run
consistent with nightly run.

Fixes: https://pagure.io/pungi/issue/987
Merges: https://pagure.io/pungi/pull-request/995

Signed-off-by: Sinny Kumari <sinny@redhat.com>
2018-07-09 12:56:00 +02:00
Lubomír Sedlář
98e7106f3e pkgset: Apply module filters on pkgset level
If a module says to filter a package out, we can do it immediately when
getting the build information from Koji.

This avoids a possible problem of something pulling the module package
in as a dependency, but it should also make the package set slightly
smaller.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-25 15:15:42 +02:00
Lubomír Sedlář
470c0ab3be init: Validate whitespace in comps groups
If a package name contains leading or trailing whitespace, it will
eventually lead to issues: pungi will try to include that group, but
since it does not exist, the packages will not make it in.

The root cause is hard to find. Better report an error immediately.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-25 11:58:52 +02:00
Lubomír Sedlář
096075848b createrepo: Include empty modules
A module without any RPMs is still valid and should be included. It
should also be in the metadata.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-22 12:54:39 +02:00
Lubomír Sedlář
38f1a8509e createiso: Break hardlinks by copying files
If a file has multiple hard links, genisoimage will put the wrong number
on the ISO. This patch can work around it by copying hard-linked files
into a temporary staging directory.

JIRA: COMPOSE-2610
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-20 12:40:43 +02:00
Martin Curlej
d8c03f6239 pkgset: Query Koji instead of PDC
PDC is deprecated in upstream. The usecase for getting list of modules
by NS, NSV or NSVC can however be satisfied by querying modules imported
into Koji.

This makes it possible to deprecate PDC configuration.

Merges: https://pagure.io/pungi/pull-request/985
Signed-off-by: Martin Curlej <mcurlej@redhat.com>
2018-06-18 15:17:04 +02:00
Lubomír Sedlář
4a61de1e8a variants: Reject values with whitespace
If there is leading or trailing whitespace in a comps group name, it
will not be included in the compose and there will even be no error
message. Whitespace on module name results in a failure.

To avoid these errors, validating the variants file will now also check
that there is no whitespace in significant places, and abort the compose
if there a problem.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-12 12:48:14 +02:00
Lubomír Sedlář
eed97f357c init: Test that init phase correctly clones defaults
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-07 10:25:35 +02:00
Lubomír Sedlář
13f5018ce1 init: Add tests for cloning module defaults
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-07 10:25:35 +02:00
Lubomír Sedlář
cd493bc4c1 init: Add validation for module defaults
After cloning the repository with defaults, open each file and try to
check if there is more than one definition for the same module. It's not
a problem for the compose process, but consumers of the compose would
get confused and possibly explode. Better alert people early.

Conceptually this should be part of the test phase, but that would mean
waiting for the compose to finish before reporting the error. The
earlier the better.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-07 10:25:35 +02:00
Lubomír Sedlář
c10a4ca337 ostree-installer: Skip comps repo if there are no comps
When the compose is not using comps, we can't pass the comps to lorax,
since it doesn't exist and it would cause a crash.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-07 09:27:14 +02:00
Lubomír Sedlář
663a07068e kojiwrapper: Call chmod recursively
Related: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-04 13:11:37 +02:00
Lubomír Sedlář
71f2c61020 Add test for getting licenses from a repo
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-04 08:26:26 +02:00
Lubomír Sedlář
102fec83b3 kojiwrapper: Don't mark runroot as successful by chmod
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-31 14:50:03 +02:00
Lubomír Sedlář
cbcebe90e1 Allow extracting koji event from another compose
When multiple composes are chained, they should reuse the same event.
However it is tricky as the value would have to be passed by hand. This
patch makes it possible to read the value from another compose (the
first one in the chain).

JIRA: COMPOSE-2571
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-31 11:11:21 +02:00
Lubomír Sedlář
92b5ad2e05 kojiwrapper: Make result of runroot world readable
The commands in runroot run as root every time. If they create files
that are not readable to other users, the reset of compose could have
problems with it if it does not run as root too. Particularly updates
composes in Bodhi run under apache user.

Relates: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-29 10:31:42 +02:00
Lubomír Sedlář
e53da69db3 osbs: Add nvr to metadata
All the components are there already separately, but having the full NVR
should simplify searching the metadata with grep.

JIRA: COMPOSE-2519
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-28 13:16:01 +02:00
Patrick Uiterwijk
f1cd1ae562 Always get old compose with release type suffix
Signed-off-by: Patrick Uiterwijk <patrick@puiterwijk.org>
Merges: https://pagure.io/pungi/pull-request/792
2018-05-25 09:49:14 +02:00
Patrick Uiterwijk
288d9ecc90 Make ostree_installer check if buildinstall is skipped correctly
The _skipped attribute is only set after the buildinstall phase is started.

Fixes: #909
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2018-05-24 13:25:34 +02:00
Lubomír Sedlář
fdea7878f1 comps-wrapper: Make tests pass on EL6
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-22 12:34:39 +02:00
Ondrej Nosek
c85d80f3c2 Handling multiple modules with the same NSV - PDC
Signed-off-by: Ondrej Nosek <onosek@redhat.com>

JIRA: COMPOSE-2510
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-05-21 11:24:46 +02:00
Lubomír Sedlář
b4e746aa71 createrepo: Allow disabling SQLite database
This is an optimization for Yum. DNF does not care at all.

The behaviour is configurable, but the default depends on gather
backend, as that is what users should be using to consume the packages
from the repo.

Fixes: https://pagure.io/pungi/issue/951
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-21 08:13:29 +02:00
Lubomír Sedlář
1afb709404 init: Drop database from comps repo
It should not be needed there, since the repo is empty anyway. A test is
added for the variant specific comps repo.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-21 08:13:29 +02:00
Lubomír Sedlář
5324c6441f arch: Make i386 map to i686 instead of athlon
This is used for mapping basearch to binary arch.

One use for this is when running depsolving to tell yum/dnf what arch to
work with. With this change it will get i686 instead of athlon when
working on i386 basearch.

The other use case is finding arches compatible with given basearch. The
change will remove athlon from the list for i386.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-16 15:22:31 +02:00
Lubomír Sedlář
2862ae0b28 ostree-installer: Use Python function to copy
This should give us better error reporting. The `copy_all` function
should preserve permissions on all files.

Relates: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-16 12:35:32 +02:00
Lubomír Sedlář
b2f995d516 Add a phase for creating extra ISOs
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-16 12:20:01 +02:00
Ondrej Nosek
b2190c1c3f Validation of parameter skip_phases
It checks config file value 'skip_phases' for valid phases names.
Also checks command-line attribute 'skip-phase' of 'bin/pungi-koji'.

JIRA: COMPOSE-2493

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-05-15 13:30:00 +02:00
Lubomír Sedlář
905064775a arch: Remove mocks in tests
This was added back when rpmutils module was used. Since we migrated to
Python 3 we have included the code directly in Pungi (because it was
coming from yum which is not going to Py3 any time soon). The mocks can
be dropped.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-09 13:49:29 +02:00
Lubomír Sedlář
7c237c2c63 ostree-installer: Allow overwriting buildinstall
Relates: https://pagure.io/pungi/issue/909
Relates: https://pagure.io/pungi/issue/695
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-07 15:22:01 +02:00
Lubomír Sedlář
543154d597 ostree-installer: Work with skipped buildinstall
Fixes: https://pagure.io/pungi/issue/909
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-07 12:29:20 +02:00
Lubomír Sedlář
076be762ec createrepo: Use less verbose logs
There is really no need to write out megabytes of logs that are not
really interesting. This should also help the parallelization. With the
verbose log createrepo fills the output buffer and needs to wait for the
busy python program to read it first.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-07 08:38:23 +02:00
Lubomír Sedlář
bd9a0ceda2 pkgset: Create global repo in parallel to merging pkgsets
The work in creating the repo is done in a separate process. This can
easily use two threads.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-07 08:38:23 +02:00
Lubomír Sedlář
dc557b809a ostree-installer: Copy files without owner
The files created in koji runroot will be owned by root. If the compose
is done under different user, there could be a problem with copying the
files preserving the owner. Let's just copy them without that.

Fixes: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-04 14:18:24 +02:00
Lubomír Sedlář
99b6e44a30 createiso: Skip if buildinstall fails
If the ISO is meant to be bootable but lorax fails, there's no point in
creating the ISO as it will not behave as expected.

Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1574585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-04 08:45:45 +02:00
Lubomír Sedlář
490f079c44 Update tests for libmodulemd 1.4.0
It got a little picky about allowed inputs. We need full RPM NEVRA in
all tests.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-02 15:36:52 +02:00
Lubomír Sedlář
ed8fffb6d1 koji-wrapper: Log failed subtasks
If the parent task is successful, there can still be failed child tasks
for failable arches. We need to log those and potentially mark the
compose as incomplete.

Fixes: https://pagure.io/pungi/issue/874
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-02 08:27:44 +02:00
Lubomír Sedlář
b8555b7869 modules: Allow context in variants XML
JIRA: COMPOSE-2508
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-26 09:26:37 +02:00
Lubomír Sedlář
eaf58f7d40 gather: Use another variant as lookaside
Create a temporary repository and add it as another lookaside in the
compose.

JIRA: COMPOSE-2426
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-23 10:51:56 +02:00
Lubomír Sedlář
ea0964eeef buildinstall: Use metadata if skipped
If the phase is skipped, it could mean that we are doing a debug run and
we don't want to mess up the .treefile by missing arch specific images.
The other alternative is that the phase was really skipped, in which
case there will be no files generated and we already handle that fine.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-23 10:42:33 +02:00
Jan Kaluza
9915c7d644 Allow reusing pkgset FileCache from old composes.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-04-23 09:57:03 +02:00
Lubomír Sedlář
58afece033 Only use comps repo if we really have comps
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-17 15:32:02 +02:00
Lubomír Sedlář
267ff86f04 pkgset: Use modules PDC API
Instead of the deprecated and confusing unreleasedvariants.

JIRA: COMPOSE-2363
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-17 12:16:18 +02:00
Patrick Uiterwijk
527394707d Access ci_base date via compose
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2018-04-15 02:16:23 +02:00
Lubomír Sedlář
a217eea24e Allow filtering comps for different variants
We already filter for arches, so this just extends the filter to work
with variants as well.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-13 14:56:06 +02:00
Ondrej Nosek
cb3d36be5d Variant as a lookaside - configuration
Relates: COMPOSE-2425

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-04-13 14:14:43 +02:00
Lubomír Sedlář
15ccd309fa Remove comps from arch repo
The comps could potentially be different in different variants, so
instead we can create the comps repo for every variant separately and
use two repos instead of one (packages in one repository, comps in
another one).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-12 13:52:36 +02:00
Petr Šabata
0e7f770fb7 Include module defaults in the repodata
If the compose configuration includes the module_defaults_dir (an
scm_dict), clone the directory, read the module defaults contained
therein and include relevant defaults in the combined modulemd file.

Only defaults for modules present in the variant are included.

This requires libmodulemd 1.2.0+.

Merges: https://pagure.io/pungi/pull-request/891
Signed-off-by: Petr Šabata <contyk@redhat.com>
2018-04-11 16:29:17 +02:00
Lubomír Sedlář
8f1beeb54b init: Always filter comps file
Even for Everything we want to filter the comps file to make sure we
remove the stuff that is not compatible with current arch. All groups
are still preserved in that case.

This allows us to do the filtering once in init phase than just use the
prepared file in comps source.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-10 08:21:56 +02:00
Lubomír Sedlář
aab3b04b08 gather: Use comps for given variant
The file for variant is filtered to only contain relevant parts.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-10 08:21:56 +02:00
Lubomír Sedlář
de231064b7 comps: Add tests for CompsFilter
All use cases that are actually used by pungi-koji are tested. There is
missing coverage for

 * keeping only items with matching arch
 * not reindenting the file

These aren't currently used and should be removed in the future, but
there may be other tools depending on the comps_filter executable.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-09 07:48:28 +02:00
Ondrej Nosek
7798174b30 Tests fail if unittest2 library is missing
Merges: https://pagure.io/pungi/pull-request/894
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-04-09 07:47:17 +02:00
Lubomír Sedlář
f38770c67d pkgset: Construct UID for PDC modules
We can't rely on the UID to be correctly joined with colons. There may
be historical data that still uses dashes.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-06 10:47:22 +02:00
Lubomír Sedlář
d9e2101b08 buildinstall: Add extra repos
A new configuration option is added that allows users to point lorax to
extra repositories. This can be handy if some tools to create the
bootable image are not part of the product itself.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
JIRA: COMPOSE-2253
2018-04-05 07:56:20 +02:00
Lubomír Sedlář
1436ea2b03 tests: Use dummy modulesdir for DNF
Otherwise it tries to ensure it exists, and since the default is
/etc/dnf/modules.d, it's causing problems if the directory does not
exist and user does not have permissions to create it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-04 12:23:43 +02:00
Ondrej Nosek
1f0739831c Update tests for Python 2.6
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-03-29 16:34:52 +02:00
Ondrej Nosek
e3aa2f769b Write module metadata
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-03-29 09:47:55 +02:00
Jan Kaluza
c6d507582a Support multilib in GatherSourceModule
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-03-28 13:38:46 +02:00
Patrick Uiterwijk
f814651d91 If sigkeys is specified, require at least one
Merges: https://pagure.io/pungi/pull-request/880
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2018-03-21 15:34:58 +01:00
Jan Kaluza
9be2d6a920 Allow setting <kojitag/> in <modules/> in variants.xml to get the modules from this Koji tag.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-03-21 14:33:45 +01:00
Jan Kaluza
1574f306c7 Move Modulemd import to pungi/__init__.py to remove duplicated code.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-03-20 12:59:17 +01:00
Jan Kaluza
3f71cdd384 Use Modulemd.Module for 'variant.arch_mmds' instead of yaml dump
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
Merges: https://pagure.io/pungi/pull-request/872
2018-03-19 14:47:03 +01:00
Jan Kaluza
fedce5dff1 Remove the filtered RPMs from module metadata even in case all RPMs are filtered out.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-03-15 11:13:39 +01:00
Lubomír Sedlář
5c902592ae pkgset: Allow empty list of modules
This should indicate that it's a modular variant, but there is no
modular content yet. We don't want to treat that as Everything.

The end result will be an empty repository.

Fixes: https://pagure.io/pungi/issue/871
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-14 14:49:09 +01:00
Lubomír Sedlář
95bb147015 buildinstall: Add option to disable it
Fixes: https://pagure.io/pungi/issue/854
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-13 15:54:52 +01:00
Jan Kaluza
340ae4d286 Use libmodulemd instead of modulemd Python module
Merges: https://pagure.io/pungi/pull-request/851
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-03-12 13:07:33 +01:00
Lubomír Sedlář
56e00505e0 gather: Fix package set whitelist
We need to include all relevant arches, not just the base one (including
noarch and src). However the list can be shortened by only listing
NEVRs, because that should be unique.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-09 15:53:31 +01:00
Lubomír Sedlář
c83316da31 pkgset: Merge initial package set without checks
For the first pass we don't need to filter out exclusive architectures,
and we don't need to exclude source packages without any binary
packages. We just want to merge the two package sets as fast as
possible.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-08 14:29:23 +01:00
Lubomír Sedlář
a03a46a078 gather: Honor package whitelist
Basically everything not on the list is excluded. This has to be applied
before we filter only the latest versions (otherwise we could lose
packages that are on the whitelist).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-08 14:29:23 +01:00
Lubomír Sedlář
41d0139b39 Write package whitelist for each variant
If we have a package set for the variant (which happens if there are
modules), include a list of all NEVRAs in the pungi kickstart.

This can be used to make sure only packages from correct tag get into
the compose. If two packages with same name but different version get
into the compose, this can help get even older version into a particular
variant.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-08 14:29:23 +01:00
Lubomír Sedlář
bd852f4059 Remove comps groups from purely modular variants
The comps source should not return all groups when there are only
modules defined. This fixes part of the problem: non-modular packages
will not go in by default.

The second part is the comps file in the created repository. It will be
filtered to not contain any groups (because packages from there will not
be in the repo).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-02 08:46:23 +01:00