This was a workaround to make some packages from the global repo
invisible for depsolving. This is now handled by packages being in
different repos. We can select which repos are enabled at which point.
This achieves the same result, but much faster.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
With this patch, there should be a separate package set for each tag
that is consumed.
Generally each module will create a separate package set, with the
exception of -devel modules that will be in the same set as their
non-devel version.
Variants no longer need to keep their own package set objects. Instead
they now include a set of package set names that should be used for the
variant. This can replace the whitelist mechanism of deps gather method.
JIRA: COMPOSE-3620
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Once a package set repo is written to disk, let's use this object that
connects the repository path with the mapping of packages.
This change also makes it explicit where the dependency on package set
repos are. In the original codebase, any part of code could generate a
path to the repo, even if that repo has not yet been written.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This opens up a path to having multiple package sets in the compose. The
pkgset phase now creates a list of them (although at this time there is
always a single item in that list).
Any consumer of the package sets objects is updated to handle a list.
Generally this means an extra loop.
JIRA: COMPOSE-3620
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This was already discouraged to not be used, and is a bad idea in
current setup anyway. Removing this can simplify the code.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
On Python 3 it is not possible to sort str and None or RpmWrapper.
First convert everything to strings and then sort it. The sorting is
really to simplify diffing the files, so exact order does not have to be
preserved.
Fixes: https://pagure.io/pungi/issue/1227
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We can't drop the whole package set once the first solver iteration
finishes, because it runs for each arch separately and we need the data
for each of them. We can however delete the arch specific portion.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The check if a variant is hybrid (or modular only) currently only looks
at comps groups. However it's possible there will be no comps groups,
but packages will be listed explicitly in config as additional_packages.
Relates: RCM-37979
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The comps could potentially be different in different variants, so
instead we can create the comps repo for every variant separately and
use two repos instead of one (packages in one repository, comps in
another one).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a variant contains both modules and comps groups, we need to include
builds from the compose tag in the package whitelist. However only
packages that are not already provided by any module should be added.
JIRA: COMPOSE-2435
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is no need to create the directory in work/, as it will get
deleted immediately. Let's move it to /tmp and use the context manager
to clean it up.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We need to include all relevant arches, not just the base one (including
noarch and src). However the list can be shortened by only listing
NEVRs, because that should be unique.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If we have a package set for the variant (which happens if there are
modules), include a list of all NEVRAs in the pungi kickstart.
This can be used to make sure only packages from correct tag get into
the compose. If two packages with same name but different version get
into the compose, this can help get even older version into a particular
variant.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Use 'get_packages_to_gather' to fail early if these packages are not
signed with right key. This prevents us from having to wait for the
repo to be created and depsolving to finish. Unsigned dependencies will
still be reported later than previously.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
There can be packages in the tag that will not end up in the compose.
Instead of failing immediately with error, this patch delays the check
until after depsolving finishes and only checks packages that will
really be included.
This is not an issue for nodeps compose, as that already pulls in only
packages that will be composed and nothing else.
Merges: https://pagure.io/pungi/pull-request/843
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
With this patch the gather_source option is no longer used. Instead, all
sources are always used. If they return at least some input packages,
then a configured method is used and the returned lists of packages from
all sources are merged.
The method used for gathering can be configured for each variant and
gather source separately.
Additional packages are only added to the comps source.
Each gathering step is logged separately. All the logs are preserved for
later inspection.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
`pungi-gather` (the tool that underlies both the `pkgset` and `gather`
phases) contains profiling code that will log statistics about how long
different function calls take. However, pungi-koji did not contain a
way to pass the ``--profiler`` argument to enable this.
This change adds a new configuration option ``gather_profiler`` which,
when set to true, simply passes the argument to `pungi-koji`. Hopefully
this can help shed some light on what is happening in some of our
longer-running composes.
Merges: https://pagure.io/pungi/pull-request/727
Signed-off-by: Ralph Bean <rbean@redhat.com>
Due to overwriting an existing variable the logs are getting duplicated
line about missing comps packages instead of announcement of gathering
being finished. Rename the variable to fix the problem.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When a package mentioned in comps is not available in the package set,
print a warning about this. Additionally there is a config option that
allows to turn this warning into a fatal error.
Fixes: https://pagure.io/pungi/issue/50
Fixes: https://pagure.io/pungi/issue/683
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There's no reason for reading the whole log of depsolving into memory
just to split it into lines and process one line at a time.
We can just as well read it in chunks.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Running depsolving with no requested inputs will only lead to a hard to
decipher error. We should instead explicitly tell the user that there is
a problem.
Unit tests are added to add to test this functionality.
Relates: #585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Calling it gather_backend is similar to repoclosure_backend we already
have. It's also more obvious what it does.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Add compose.mkdtemp which creates and returns a unique temporary
directory under <compose_topdir>/work/{global,<arch>}/tmp[-<variant>]/
Change tempfile.mkdtemp in code to compose.mkdtemp, so temporary
dirs are created under the compose's workdir, this makes it more
easier to maintain the temporary dirs/files especially when compose
fails with amount of temporary files left there.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
The address is no longer correct. We can just as well simply point to
the web page describing the license.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The schema is written in Python to reduce duplication. When
configuration is loaded, the validation checks if it's correct and fills
in default values.
There is a custom extension to the schema to report deprecated options.
The config dependencies are implemented as a separate pass. While it's
technically possible to express the dependencies in the schema itself,
the error messages are not very helpful and it makes the schema much
harder to read.
Phases no longer define `config_options`. New options should be added to
the schema. Since the default values are populated automatically during
validation, there is no need to duplicate them into the code.
The `pungi-config-validate` script is updated to use the schema and
report errors even for deeply nested fields.
The dependencies are updated: pungi now depends on `python-jsonschema`
(which is already available in Fedora).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is now a single option `multilib`, that maps variants and arches
to multilib methods. This replaces old `multilib_methods` option.
Multilib arches are implicitly deduced instead of using the
`multilib_arches` option.
The test compose is updated to only enable multilib on Server and its
addons.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
CHANGE: createrepo_c is config option now defaults to True.
ACTION: You can safely remove 'createrepo_c = True' from config files.
Set 'createrepo_c = False' if you need legacy createrepo.
CHANGE: createrepo_checksum config option is now mandatory.
ACTION: Add 'createrepo_checksum = "sha256"' (or "sha") to config files.
Haibo Lin
Lubomír Sedlář
Ondrej Nosek
Jan Kaluza
Ralph Bean
Daniel Mach
Qixiang Wan
Adam Miller
Dennis Gilmore