Modular variants can either be specified by a list of modules, or by a
list of Koji tags. In terms of comps preprocessing there should not be
any difference between the two.
Resolves: https://pagure.io/pungi/issue/1640
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
- Remove validation for modules obsoletes
We can have multiple obsoletes for one module
- Add unit tests to cover basic scenarios for
modules defaults && obsoletes
- Add additional check for invalid yaml file
in Defaults. Previously, empty list of default would
be returned when invalid yaml is present in Defaults
directory.
- Using MergeIndex for Obsoletes only (for now).
https://pagure.io/pungi/issue/1592
Signed-off-by: Marek Kulik <mkulik@redhat.com>
It was needed to provide assertItemsEqual method. Starting with Python
3.2, there's assertCountEqual that does the same thing. Six provides a
helper that will dispatch to the existing method. With this change,
unittest 2 is only needed on Python 2.6 to backport the method.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should make it possible to only import the library only when it's
really needed.
DNF does not work with libmodulemd v2. If we import libmodulemd2 and
then dnf, the program will just hang forever. We only need DNF in
pungi-gather, where libmodulemd is not needed, and also where we do need
libmodulemd, we don't have DNF.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This was already discouraged to not be used, and is a bad idea in
current setup anyway. Removing this can simplify the code.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Starting tests just to run mock functions slows the tests down for no
good reason. Let's instead mock the runner and run the dummy tasks
serially.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The environment in comps for a variant can refer to groups in parent
variant (either for addons, or because of other configuration). We
should not remove the groups in this case.
This requires changes in two places:
* teaching `comps_filter` about groups that should not be removed
* fixing writing comps so that it does not actually change the data as
well
JIRA: COMPOSE-2612
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a package name contains leading or trailing whitespace, it will
eventually lead to issues: pungi will try to include that group, but
since it does not exist, the packages will not make it in.
The root cause is hard to find. Better report an error immediately.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
After cloning the repository with defaults, open each file and try to
check if there is more than one definition for the same module. It's not
a problem for the compose process, but consumers of the compose would
get confused and possibly explode. Better alert people early.
Conceptually this should be part of the test phase, but that would mean
waiting for the compose to finish before reporting the error. The
earlier the better.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It should not be needed there, since the repo is empty anyway. A test is
added for the variant specific comps repo.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is really no need to write out megabytes of logs that are not
really interesting. This should also help the parallelization. With the
verbose log createrepo fills the output buffer and needs to wait for the
busy python program to read it first.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The comps could potentially be different in different variants, so
instead we can create the comps repo for every variant separately and
use two repos instead of one (packages in one repository, comps in
another one).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the compose configuration includes the module_defaults_dir (an
scm_dict), clone the directory, read the module defaults contained
therein and include relevant defaults in the combined modulemd file.
Only defaults for modules present in the variant are included.
This requires libmodulemd 1.2.0+.
Merges: https://pagure.io/pungi/pull-request/891
Signed-off-by: Petr Šabata <contyk@redhat.com>
Even for Everything we want to filter the comps file to make sure we
remove the stuff that is not compatible with current arch. All groups
are still preserved in that case.
This allows us to do the filtering once in init phase than just use the
prepared file in comps source.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The comps source should not return all groups when there are only
modules defined. This fixes part of the problem: non-modular packages
will not go in by default.
The second part is the comps file in the created repository. It will be
filtered to not contain any groups (because packages from there will not
be in the repo).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should make all tests pass on both Python 2 and Python 3.
Unittest2 is required on Py 2.6 and Py 3.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When variants XML lists a group that does not match any known group in
input comps, report a warning. This is not necessarily a problem in
itself, but having this information in the log can help debug problems.
Relates: #585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The optional variant can defined by just adding has_optional=True into
variant xml. In such case it has no comps groups and Pungi would copy
the original file unmodified. This leads to extra packages being pulled
into the optional variant.
In this case the correct solution is to filter the comps and remove all
groups.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The same information can be inferred from definitions in variants.xml:
if the variant has no groups defined, we include packages from all
groups. By the same logic we can also include all groups in the comps
file.
The config validation is updated to give a hint on how to remove the
option from the configuration.
Relates: #29
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The schema is written in Python to reduce duplication. When
configuration is loaded, the validation checks if it's correct and fills
in default values.
There is a custom extension to the schema to report deprecated options.
The config dependencies are implemented as a separate pass. While it's
technically possible to express the dependencies in the schema itself,
the error messages are not very helpful and it makes the schema much
harder to read.
Phases no longer define `config_options`. New options should be added to
the schema. Since the default values are populated automatically during
validation, there is no need to duplicate them into the code.
The `pungi-config-validate` script is updated to use the schema and
report errors even for deeply nested fields.
The dependencies are updated: pungi now depends on `python-jsonschema`
(which is already available in Fedora).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Check once before iterating through the variants. This greatly
simplifies the tests as each function now has one less code path.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When creating the comps repo, use a separate log file for each variant
instead of overwriting the same global file again and again.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The configuration can specify a list of variants for which the original
comps file will be copied without any modification.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
soksanichenko
Lubomír Sedlář
Marek Kulik
Filip Valder
Haibo Lin
Petr Šabata
Patrick Uiterwijk