If a variant contains both modules and comps groups, we need to include
builds from the compose tag in the package whitelist. However only
packages that are not already provided by any module should be added.
JIRA: COMPOSE-2435
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is no need to create the directory in work/, as it will get
deleted immediately. Let's move it to /tmp and use the context manager
to clean it up.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We need to include all relevant arches, not just the base one (including
noarch and src). However the list can be shortened by only listing
NEVRs, because that should be unique.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If we have a package set for the variant (which happens if there are
modules), include a list of all NEVRAs in the pungi kickstart.
This can be used to make sure only packages from correct tag get into
the compose. If two packages with same name but different version get
into the compose, this can help get even older version into a particular
variant.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Use 'get_packages_to_gather' to fail early if these packages are not
signed with right key. This prevents us from having to wait for the
repo to be created and depsolving to finish. Unsigned dependencies will
still be reported later than previously.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
There can be packages in the tag that will not end up in the compose.
Instead of failing immediately with error, this patch delays the check
until after depsolving finishes and only checks packages that will
really be included.
This is not an issue for nodeps compose, as that already pulls in only
packages that will be composed and nothing else.
Merges: https://pagure.io/pungi/pull-request/843
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
With this patch the gather_source option is no longer used. Instead, all
sources are always used. If they return at least some input packages,
then a configured method is used and the returned lists of packages from
all sources are merged.
The method used for gathering can be configured for each variant and
gather source separately.
Additional packages are only added to the comps source.
Each gathering step is logged separately. All the logs are preserved for
later inspection.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
`pungi-gather` (the tool that underlies both the `pkgset` and `gather`
phases) contains profiling code that will log statistics about how long
different function calls take. However, pungi-koji did not contain a
way to pass the ``--profiler`` argument to enable this.
This change adds a new configuration option ``gather_profiler`` which,
when set to true, simply passes the argument to `pungi-koji`. Hopefully
this can help shed some light on what is happening in some of our
longer-running composes.
Merges: https://pagure.io/pungi/pull-request/727
Signed-off-by: Ralph Bean <rbean@redhat.com>
Due to overwriting an existing variable the logs are getting duplicated
line about missing comps packages instead of announcement of gathering
being finished. Rename the variable to fix the problem.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When a package mentioned in comps is not available in the package set,
print a warning about this. Additionally there is a config option that
allows to turn this warning into a fatal error.
Fixes: https://pagure.io/pungi/issue/50
Fixes: https://pagure.io/pungi/issue/683
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There's no reason for reading the whole log of depsolving into memory
just to split it into lines and process one line at a time.
We can just as well read it in chunks.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Running depsolving with no requested inputs will only lead to a hard to
decipher error. We should instead explicitly tell the user that there is
a problem.
Unit tests are added to add to test this functionality.
Relates: #585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Calling it gather_backend is similar to repoclosure_backend we already
have. It's also more obvious what it does.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Add compose.mkdtemp which creates and returns a unique temporary
directory under <compose_topdir>/work/{global,<arch>}/tmp[-<variant>]/
Change tempfile.mkdtemp in code to compose.mkdtemp, so temporary
dirs are created under the compose's workdir, this makes it more
easier to maintain the temporary dirs/files especially when compose
fails with amount of temporary files left there.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
The address is no longer correct. We can just as well simply point to
the web page describing the license.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The schema is written in Python to reduce duplication. When
configuration is loaded, the validation checks if it's correct and fills
in default values.
There is a custom extension to the schema to report deprecated options.
The config dependencies are implemented as a separate pass. While it's
technically possible to express the dependencies in the schema itself,
the error messages are not very helpful and it makes the schema much
harder to read.
Phases no longer define `config_options`. New options should be added to
the schema. Since the default values are populated automatically during
validation, there is no need to duplicate them into the code.
The `pungi-config-validate` script is updated to use the schema and
report errors even for deeply nested fields.
The dependencies are updated: pungi now depends on `python-jsonschema`
(which is already available in Fedora).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is now a single option `multilib`, that maps variants and arches
to multilib methods. This replaces old `multilib_methods` option.
Multilib arches are implicitly deduced instead of using the
`multilib_arches` option.
The test compose is updated to only enable multilib on Server and its
addons.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
CHANGE: createrepo_c is config option now defaults to True.
ACTION: You can safely remove 'createrepo_c = True' from config files.
Set 'createrepo_c = False' if you need legacy createrepo.
CHANGE: createrepo_checksum config option is now mandatory.
ACTION: Add 'createrepo_checksum = "sha256"' (or "sha") to config files.