We need to update the source modulemd to match what we want. For most
modules the name will be correct already, but in case of devel modules
the original name will still be present there.
Fixes: COMPOSE-3596
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is fairly similar to a package only being built for particular
arches.
Fixes: https://pagure.io/pungi/issue/1198
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The prefix checking only works if there are no streams that would share
prefixes. Let's instead check the value as a whole. There is extra
complexity from the fact that version and context may not be specified.
The stream as specified in input is processed to replace dashes (`-`)
with underscores (`_`) to match how the builds are imported into Koji.
JIRA: COMPOSE-3547
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
For modular tags we only include packages that are included in the
module. Originally the filter was taking package names from
rpms.artifacts section of MMD.
This however does not work correctly, as there can easily be module
which lists foo.src but does not want to include foo.x86_64 or other
arches.
This patch fixes this particular problem by included arch in the set of
packages to be kept.
JIRA: COMPOSE-3543
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The scheme for generating versions has changed multiple times. MBS is
careful to only modify them so that they always compare correctly. This
only works though if the versions are treated as numbers.
This should be safe in that non-numbers should never be encountered as
module version. Libmodulemd internally stores the version as int (or
some version of int).
JIRA: COMPOSE-3540
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
A module build can create packages that are tagged in the content tag,
but should not be included in the module. Originally Pungi didn't know
what exactly the module contains and so it needed to apply filters to
exclude stuff that was definitely out.
With getting the final MMD from Koji, we can actually make this a bit
more strict by only keeping packages that we know we need.
When processing each content tag, we can put into package set only
packages that are included in some module using that tag. This should
work with -devel modules as well. Both the regular and -devel modules
will contribute to the set and thus all packages will go to the package
set.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of loading the "source" modulemd, always get the final file for
each architecture from Koji.
Logging the downloaded files locally is no longer necessary, when
debugging a problem we can find the files in the respective builds in
Koji.
Tests are updated to work with new code, and an obsolete test is
removed.
JIRA: COMPOSE-3147
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The file in old compose does not change, there is no need to load it
again for each tag.
One consequence of this change is that the same cache will be used by
all package sets. They add stuff to the cache as they use it. However
that should not be a problem for the same reason it's okay to use the
cache in the first place. If two packages have the same path, they are
actually the same file and it's okay to reuse the data.
JIRA: COMPOSE-3374
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Also remove the useless listTaggedRPMs call which has been used *only*
to catch the very rare error which happened from time to time when
we used PDC in Pungi few years ago. This rare error is not relevant
anymore now.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
It may make sense to break a big compose into smaller chunks that can be
done independently. This script allows describing the smaller parts,
runs them with correct dependencies and arranges the result to look like
a single big compose.
All parts use the same koji event, that is either obtained from Koji, or
from command line argument.
JIRA: COMPOSE-2654
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Stream can contain dash sign and when MBS imports such NSVC to Koji, the
dash is replaced with underscore. The current Pungi code does not
respect that and tries to use the stream from Koji directly, which
results in wrong stream being using in some Pungi internal data.
In this PR, the NSVC is taken from module metadata section of CG Koji
build, which contains real stream including the dashes.
Merges: https://pagure.io/pungi/pull-request/1072
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
By default nothing should change. This patch adds a new config option
that enables all this new craziness:
* it turns of applying module filters at pkgset level
* it creates new modules and adds them to the compose
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The pool is created in multiple places, and the process is always to
create an instance and add workers to it. Let's abstract the loop in a
method.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This PR adds new pkgset_koji_builds configuration option.
This option allows setting list of extra Koji build NVRs which will be
included in a compose. This is useful in two cases:
a) It allows generating standard composes with few packages update to
certain version to test how the compose behaves when the package is
updated for real.
b) It allows generating compose consisting only from particular builds
when pkgset_koji_tag = '' or None. This is useful when one want to
regenerate the compose with packages which are not tagged in single Koji
tag. This is very useful for ODCS when reproducing old composes.
Merges: https://pagure.io/pungi/pull-request/1049
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
This patch changes the behaviour when both module tag and NSV?C? is
specified. The NSVC are used as a whitelist and only matching modules
will be included in the compose.
Additionally this patch adds filtering based on inheritance: when
finding the latest module for each N:S combination, only the top tag in
which the module is tagged is used. Even if a newer build is available
somewhere deeper in the inheritance, it's not going to be used.
Example inheritance and tagged modules
f29-compose (foo:1:2018:cafe)
└─ f29-candidate (foo:1:2019:cafe)
The compose will use 2018 version, because it's in the topmost tag.
JIRA: COMPOSE-2685
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
In the search result, we should ignore any module build that finished
after the event that we are working with.
Fixes: https://pagure.io/pungi/issue/999
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the same tag is queried with the same event, Pungi can cache the
response and call the API again. Particularly for small composes this
can save up significant amount of time.
Merges: https://pagure.io/pungi/pull-request/1022
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
If a module says to filter a package out, we can do it immediately when
getting the build information from Koji.
This avoids a possible problem of something pulling the module package
in as a dependency, but it should also make the package set slightly
smaller.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
PDC is deprecated in upstream. The usecase for getting list of modules
by NS, NSV or NSVC can however be satisfied by querying modules imported
into Koji.
This makes it possible to deprecate PDC configuration.
Merges: https://pagure.io/pungi/pull-request/985
Signed-off-by: Martin Curlej <mcurlej@redhat.com>
The `add_noarch` option of `get_valid_arches` is broken and doesn't
really do anything (noarch is always present in the result).
This causes packages that have ExclusiveArch including noarch to
actually not be excluded. They should be.
Changing this globally could have a very big impact. Therefore we can
hide it behind a configuration option so that it's opt-in.
JIRA: COMPOSE-2457
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
With one compose combining traditional and modular content there might
be different requirements for tag inheritance. This patch adds a new
option that controls whether builds in modular tags should be inherited.
It defaults to False, which is the right option for current MBS
behavior.
JIRA: COMPOSE-2148
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The code checked with `assert` that there is only one module matching
given NSV. In actual package that would not do anything and we would
silently pick the first value.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This speeds up the compose quite a bit and there is no need for the
database anyway.
Merges: https://pagure.io/pungi/pull-request/922
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
The comps could potentially be different in different variants, so
instead we can create the comps repo for every variant separately and
use two repos instead of one (packages in one repository, comps in
another one).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We can't rely on the UID to be correctly joined with colons. There may
be historical data that still uses dashes.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should indicate that it's a modular variant, but there is no
modular content yet. We don't want to treat that as Everything.
The end result will be an empty repository.
Fixes: https://pagure.io/pungi/issue/871
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
For the first pass we don't need to filter out exclusive architectures,
and we don't need to exclude source packages without any binary
packages. We just want to merge the two package sets as fast as
possible.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
