Commit Graph

35 Commits

Author SHA1 Message Date
Haibo Lin
b217470464 Format code
Code didn't get well formatted when jenkins unusable.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-03-02 18:31:45 +08:00
Haibo Lin
c0193c9fca Fix flake8 complaints - E501
E501 line too long (92 > 88 characters)
E501 line too long (103 > 88 characters)
...

JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 14:36:46 +08:00
Haibo Lin
3eddcfccd8 Fix flake8 complaints - F401
F401 'dnf' imported but unused
F401 'imp' imported but unused
F401 'os' imported but unused
F401 'subprocess' imported but unused
F401 'sys' imported but unused
F401 'yum' imported but unused

JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 11:48:31 +08:00
Haibo Lin
38142d30ba Format tests with black
JIRA: COMPOSE-4086
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-05 17:29:15 +08:00
Haibo Lin
0ed70fc8b6 tests: drop sys.path modification
It's unnecessary to add the path manually.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-12-10 10:53:38 +08:00
Lubomír Sedlář
2f54745715 Remove usage of unittest2 on Python 3
It was needed to provide assertItemsEqual method. Starting with Python
3.2, there's assertCountEqual that does the same thing. Six provides a
helper that will dispatch to the existing method. With this change,
unittest 2 is only needed on Python 2.6 to backport the method.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 15:25:18 +02:00
Lubomír Sedlář
43fb06e00f Remove shebangs from test files
They should be executed with a runner like nose, not directly.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 15:24:17 +02:00
Haibo Lin
3811c0a176 config: Deprecate bootable option
JIRA: COMPOSE-2635
Fixes: https://pagure.io/pungi/issue/976
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-07-22 16:51:38 +08:00
Lubomír Sedlář
bf1b3e8421 Remove remaining mentions of runroot option
This also cleans up the runroot method detection code to not rely on the
now removed option.

JIRA: COMPOSE-2634
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-02 09:43:13 +02:00
Haibo Lin
c4ed2bf3b2 config: Deprecate runroot option
Fixes: https://pagure.io/pungi/issue/975
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-06-27 18:00:07 +08:00
Haibo Lin
ce066707c1 config: Deprecate release_is_layered option
Fixes: https://pagure.io/pungi/issue/977
Merges: https://pagure.io/pungi/pull-request/1204
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-06-06 13:45:49 +02:00
Lubomír Sedlář
ca7d6256e5 Move resolving git reference to config validation
Instead of multiple places handling the same thing duplicating the
logic, it's better to do it once upfront. This allows easy caching of
the results.

Additional advantage of this approach is that the config dump will
include resolved URLs. The original reference will still be available in
the copy of the original config.

JIRA: COMPOSE-3065
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-21 13:25:35 +01:00
Lubomír Sedlář
fa752eb2b5 Configure image name per variant
Up to now it was possible to change the pattern for all images, but
there are use-cases where different variants might want different names.
For example there could be one main variant that should only have
product name in the ISO filename, but addons should still be marked with
variant name.

JIRA: COMPOSE-3041
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-05 08:48:36 +01:00
Ondrej Nosek
b2190c1c3f Validation of parameter skip_phases
It checks config file value 'skip_phases' for valid phases names.
Also checks command-line attribute 'skip-phase' of 'bin/pungi-koji'.

JIRA: COMPOSE-2493

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-05-15 13:30:00 +02:00
Ondrej Nosek
cb3d36be5d Variant as a lookaside - configuration
Relates: COMPOSE-2425

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-04-13 14:14:43 +02:00
Lubomír Sedlář
364d7f5229 Support multiple sources in one variant
With this patch the gather_source option is no longer used. Instead, all
sources are always used. If they return at least some input packages,
then a configured method is used and the returned lists of packages from
all sources are merged.

The method used for gathering can be configured for each variant and
gather source separately.

Additional packages are only added to the comps source.

Each gathering step is logged separately. All the logs are preserved for
later inspection.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-02-01 14:31:09 +01:00
Lubomír Sedlář
03293c725b config: Make pkgset_koji_tag optional
There are valid use cases for not specifying this option: specifically a
modular compose will get the tags to use from modules listed in the
variants file.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-01-12 15:50:21 +01:00
Lubomír Sedlář
d5a0316877 Use dnf backend for repoclosure on PY3
When yum is not available, it makes no sense to offer it as an option.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-12-06 13:26:43 +01:00
Ondrej Nosek
91ee1fb854 Remove deprecated options
Options that are currently marked as deprecated do not have any effect
anymore (other than printing warning). We should remove them and update
the message so that we can mark options as deprecated even when they
still work.

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2017-11-16 11:12:34 +01:00
Lubomír Sedlář
daf162503c Reject yum gather backend on Python 3
It will not run, and having a nice error message is better than a
cryptic crash.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-24 15:15:31 +02:00
Lubomír Sedlář
ed22e07ef9 Port to Python 3
This should make all tests pass on both Python 2 and Python 3.

Unittest2 is required on Py 2.6 and Py 3.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-24 15:15:31 +02:00
Lubomír Sedlář
2819311d6e config: Allow comps_file for any gather_source
It only affects gathering packages when gather_source is set to comps,
but it could still make sense to include the file in the repository even
if the not used for gathering.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-18 15:24:24 +02:00
Qixiang Wan
18bd37ff2c pkgset: Allow populating packages from multiple koji tags
One of the use cases is https://pagure.io/odcs/issue/7.

Merges: #660
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-06-29 08:27:34 +02:00
Lubomír Sedlář
bab2a125d4 osbs: Make git_branch required option
In order to avoid conflicting tags, OSBS allows only one build for a
repo/branch pair at the same time. To avoid race conditions, we should
make sure we always pass in the branch. This commit makes it a required
option.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-24 10:05:25 +02:00
Lubomír Sedlář
98f40f6138 ostree: Autogenerate a version
If the value is a particular magic string, we will expand it to a proper
value.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-13 10:17:59 +02:00
Qixiang Wan
2f5d6d7dcd unify repo and repo_from options
Config option 'repo' and 'repo_from' are used in several phases, merge
them with one option 'repo'. 'append' in schema is used for appending
the values from deprecated options to 'repo', so it won't break on any
existing config files that have the old options of 'repo_from' and
'source_repo_from' (which is an alias of 'repo_from').

And 'repo' schema is updated to support repo dict as the value or an
item in the values, a repo dict is just a dict contains repo options,
'baseurl' is required in the dict, like:

{"baseurl": "http://example.com/url/to/repo"}

or:

{"baseurl": "Serer"}

currently this is used in ostree phase to support extra repo options
like:

{"baseurl": "Server", "exclude": "systemd-container"}

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-29 10:12:32 +08:00
Qixiang Wan
2aacefd9cd checks.py: show warning message for alias option
Show warning message for any alias option find in config instance.
Example warning message:

WARNING: Config option 'product_name' is deprecated and now an alias to
'release_name', please use 'release_name' instead. In:
{'release_name': 'dummy product', 'product_name': 'dummy product'}

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-29 10:12:32 +08:00
Lubomír Sedlář
e3fe67be53 repoclosure: add option to use dnf backend
This adds a new option repoclosure_backend that changes what tool is
used for repoclosure.

Checking build dependencies is currently not supported, as `dnf` does
not have the corresponding option.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-26 13:39:12 +01:00
Lubomír Sedlář
c338219ef0 config: Report unknown options as warnings
Given the way config files can include other files, it is entirely
possible to use the same config file for both Pungi and Distill. Pungi
will however complain about unknown options.

This patch reverts part of c38bb480 and moves deprecation handling back
into schema validation. The validation method then returns a list of
errors and a list of warnings.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-13 11:58:08 +01:00
Colin Walters
7bcbe30fd2 Expose lorax's --rootfs-size argument
This is going to be necessary for reworking the Atomic Host ISO,
see: https://pagure.io/fedora-lorax-templates/pull-request/6

Signed-off-by: Colin Walters <walters@verbum.org>
2016-11-17 10:17:39 -05:00
Lubomír Sedlář
c38bb4809b config: Don't abort on deprecated options
These options are in fact removed and have no effect anymore. This patch
changes the validation to print a warning that the option was removed
and what should be done instead. It no longer stops the whole compose.

The validation script still rejects configuration files with these
removed keys.

This change means we no longer check these removals with the JSON schema
(as that makes it hard to determine where exactly the problem is).

Fixes: #438
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-20 12:51:12 +02:00
Lubomír Sedlář
50e9294057 config: Validate variant regular expressions
When the mapping in configuration specifies incorrect regular expression
to match regular expressions, we should raise an error immediately and
not wait until the part of config is actually used.

This patch does not cover `live_media`, `image_build` and `osbs`
sections, as they use plain dicts and not the list of tuples format.

Fixes: #424
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-12 09:38:59 +02:00
Lubomír Sedlář
a042906717 init: Remove keep_original_comps option
The same information can be inferred from definitions in variants.xml:
if the variant has no groups defined, we include packages from all
groups. By the same logic we can also include all groups in the comps
file.

The config validation is updated to give a hint on how to remove the
option from the configuration.

Relates: #29
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-22 13:35:28 +02:00
Lubomír Sedlář
c118adc705 tests: Use unittest2 consistently
Without importing this module, the tests are going to fail on Python
2.6.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-22 10:28:01 +02:00
Lubomír Sedlář
f9a6c8418f Add JSON Schema for configuration
The schema is written in Python to reduce duplication. When
configuration is loaded, the validation checks if it's correct and fills
in default values.

There is a custom extension to the schema to report deprecated options.

The config dependencies are implemented as a separate pass. While it's
technically possible to express the dependencies in the schema itself,
the error messages are not very helpful and it makes the schema much
harder to read.

Phases no longer define `config_options`. New options should be added to
the schema. Since the default values are populated automatically during
validation, there is no need to duplicate them into the code.

The `pungi-config-validate` script is updated to use the schema and
report errors even for deeply nested fields.

The dependencies are updated: pungi now depends on `python-jsonschema`
(which is already available in Fedora).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-01 10:56:15 +02:00