We already filtered a list of packages compatible with the architecture.
There's no need to do that again (with bugs). Instead of using the
global package set we should just restrict the code to an arch package
set.
This should not break anything. If a package is included such that it's
in global but not arch package set, the compose would crash as linking
packages takes paths from the arch package set.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The list of RPMs in a module can contain multilib packages, but they are
never included because we only ask for compatible arches without
multilib. For x86_64 that only adds noarch, but not i686.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Without this there is almost no information on what's happening. This
patch should provide basic debugging capabilities.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should make all tests pass on both Python 2 and Python 3.
Unittest2 is required on Py 2.6 and Py 3.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the package name matches, we should take the package either if
architecture matches or the package is noarch.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
`pungi-gather` (the tool that underlies both the `pkgset` and `gather`
phases) contains profiling code that will log statistics about how long
different function calls take. However, pungi-koji did not contain a
way to pass the ``--profiler`` argument to enable this.
This change adds a new configuration option ``gather_profiler`` which,
when set to true, simply passes the argument to `pungi-koji`. Hopefully
this can help shed some light on what is happening in some of our
longer-running composes.
Merges: https://pagure.io/pungi/pull-request/727
Signed-off-by: Ralph Bean <rbean@redhat.com>
Due to overwriting an existing variable the logs are getting duplicated
line about missing comps packages instead of announcement of gathering
being finished. Rename the variable to fix the problem.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When a package mentioned in comps is not available in the package set,
print a warning about this. Additionally there is a config option that
allows to turn this warning into a fatal error.
Fixes: https://pagure.io/pungi/issue/50
Fixes: https://pagure.io/pungi/issue/683
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There's no reason for reading the whole log of depsolving into memory
just to split it into lines and process one line at a time.
We can just as well read it in chunks.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When there is a noarch subpackage, all compatible debuginfo would be
pulled in, which is not desirable.
Example: Server.x86_64 needs pkg.x86_64 and pkg-data.noarch. We only
want pkg-debuginfo.x86_64, but without this patch even
pkg-debuginfo.i686 would get in.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When there are no groups, we shouldn't try to read comps file (because
it may very well not be there).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When gather_method is set to nodeps, we should not ignore the comps
group that the method received. Instead it should find out which
packages are in those groups and take them into the compose.
In order for this to be of any reasonable use, the comps file needs to
include all dependencies for the packages.
Fixes: #653
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The get_system_release_packages function can never be called without a
variant, so it makes no sense to check for that condition.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
* add comments explaining what's going on
* break too long lines
* simplify the logic where possible
* use with statement to work with files
* remove commented out and unused code
* introduce helpers to reduce code duplication
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We never need anything more than the ID, so passing around a big
complicated object is not necessary.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Running depsolving with no requested inputs will only lead to a hard to
decipher error. We should instead explicitly tell the user that there is
a problem.
Unit tests are added to add to test this functionality.
Relates: #585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Do not create empty skeleton dirs for empty variants which we do for rpm
variants in some phases (some others already have the check):
1. createiso phase
2. extra_files phase
3. gather phase
In addtion to this, compose metadata (composeinfo.json) doesn't include
variant paths which don't exist or are just empty dirs now.
Fixes: #497
Merges: #572
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Calling it gather_backend is similar to repoclosure_backend we already
have. It's also more obvious what it does.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is a follow up of 8bc65a8be5. When
resolving packages that go into optional variant, we only want to skip
filtering groups when working on a top-level variant.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Add compose.mkdtemp which creates and returns a unique temporary
directory under <compose_topdir>/work/{global,<arch>}/tmp[-<variant>]/
Change tempfile.mkdtemp in code to compose.mkdtemp, so temporary
dirs are created under the compose's workdir, this makes it more
easier to maintain the temporary dirs/files especially when compose
fails with amount of temporary files left there.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
The `get_variants()` method had a `recursive` argument with default
value of `False. However, this argument had no effect and the method
always returned all variants recursively.
We can just drop the argument. All callers are updated to not supply the
argument. Should any need for getting the top-level variants only arise,
they can be accessed as the `variants` attribute directly on the Compose
object.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The address is no longer correct. We can just as well simply point to
the web page describing the license.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The schema is written in Python to reduce duplication. When
configuration is loaded, the validation checks if it's correct and fills
in default values.
There is a custom extension to the schema to report deprecated options.
The config dependencies are implemented as a separate pass. While it's
technically possible to express the dependencies in the schema itself,
the error messages are not very helpful and it makes the schema much
harder to read.
Phases no longer define `config_options`. New options should be added to
the schema. Since the default values are populated automatically during
validation, there is no need to duplicate them into the code.
The `pungi-config-validate` script is updated to use the schema and
report errors even for deeply nested fields.
The dependencies are updated: pungi now depends on `python-jsonschema`
(which is already available in Fedora).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When package has incompatible arch for a tree, the same error message
could be raised from two different places. This patch adds more
information to the message, so that it is easier to find the actual
problem.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Make all imports either use full package name starting with `pungi` or
use explicitly relative import. This will avoid issues when importing a
module that shadows another module on PYTHONPATH.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the `<groups></groups>` section is not specified in the variants XML
file, all groups will be used in this variant. The section must be
omitted completely, not just empty. This is (and was) correct according
to the DTD, it just lead to crash before.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Even if gather phase is skipped, it will still generate a rpms.json
metadata file. It will have no payload, but the header structure is
there.
Createrepo phase had to be updated to ignore variants that have no RPMs
listed in metadata.
There is one more fix for writing treeinfo files: if repomd.xml files
are not generated, it will no longer crash.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The variants.xml file can list a variant with is_empty="true" and no
groups. If such variant is found, not package gathering will be run for
it, and no repos will be created.
This only makes sense for a variant that will have some other
deliverables like live media or images.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is now a single option `multilib`, that maps variants and arches
to multilib methods. This replaces old `multilib_methods` option.
Multilib arches are implicitly deduced instead of using the
`multilib_arches` option.
The test compose is updated to only enable multilib on Server and its
addons.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Fedora variants needs both general fedora-release and
fedora-release-$variant packages installed. This patch disables the code
that would throw fedora-release away (assuming correct config is set).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
On case-insensitive filesystems it is not such a good idea to have
directories that only differ in case. Packages should be always split
into lowercased directories.
The test data is modified to include some packages starting with
uppercase letters. The example in code can be verified by running
`nosetests --with-doctest`.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
CHANGE: Rename product_* config options to release_* for consistency with productmd.
ACTION: Rename product_name, product_short, product_version, product_is_layered to release_* in config files.
Rename //variant/product to //variant/release in variants XML.
CHANGE: createrepo_c is config option now defaults to True.
ACTION: You can safely remove 'createrepo_c = True' from config files.
Set 'createrepo_c = False' if you need legacy createrepo.
CHANGE: createrepo_checksum config option is now mandatory.
ACTION: Add 'createrepo_checksum = "sha256"' (or "sha") to config files.
Lubomír Sedlář
Jan Kaluza
Ralph Bean
Qixiang Wan
Daniel Mach
Adam Miller
Dennis Gilmore