Running depsolving with no requested inputs will only lead to a hard to
decipher error. We should instead explicitly tell the user that there is
a problem.
Unit tests are added to add to test this functionality.
Relates: #585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When variants XML lists a group that does not match any known group in
input comps, report a warning. This is not necessarily a problem in
itself, but having this information in the log can help debug problems.
Relates: #585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Config option 'repo' and 'repo_from' are used in several phases, merge
them with one option 'repo'. 'append' in schema is used for appending
the values from deprecated options to 'repo', so it won't break on any
existing config files that have the old options of 'repo_from' and
'source_repo_from' (which is an alias of 'repo_from').
And 'repo' schema is updated to support repo dict as the value or an
item in the values, a repo dict is just a dict contains repo options,
'baseurl' is required in the dict, like:
{"baseurl": "http://example.com/url/to/repo"}
or:
{"baseurl": "Serer"}
currently this is used in ostree phase to support extra repo options
like:
{"baseurl": "Server", "exclude": "systemd-container"}
Signed-off-by: Qixiang Wan <qwan@redhat.com>
If 'append' is defined for a property, append the values from append
options to the property. Note: The property must support to be a list
of values.
For example:
with schema:
schema = {
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Pungi Configuration",
"type": "object",
"definitions": {
"list_of_strings": {
"type": "array",
"items": {"type": "string"},
},
"strings": {
"anyOf": [
{"type": "string"},
{"$ref": "#/definitions/list_of_strings"},
]
},
},
"properties": {
"release_name": {"type": "string"},
"repo": {"$ref": "#/definitions/strings", "append": "repo_from"}
},
"additionalProperties": False,
}
and config:
repo = "http://url/to/repo"
repo_from = "Server"
config will be updated to:
repo = ["http://url/to/repo", "Server"]
It supports multiple append options too, like:
"repo": {
"$ref": "#/definitions/strings",
"append": ["repo_from", "source_repo_from"],
}
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Show warning message for any alias option find in config instance.
Example warning message:
WARNING: Config option 'product_name' is deprecated and now an alias to
'release_name', please use 'release_name' instead. In:
{'release_name': 'dummy product', 'product_name': 'dummy product'}
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Do not create empty skeleton dirs for empty variants which we do for rpm
variants in some phases (some others already have the check):
1. createiso phase
2. extra_files phase
3. gather phase
In addtion to this, compose metadata (composeinfo.json) doesn't include
variant paths which don't exist or are just empty dirs now.
Fixes: #497
Merges: #572
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Calling it gather_backend is similar to repoclosure_backend we already
have. It's also more obvious what it does.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We also rename the old multilib module used by dnf code to multilib_yum
to make it clear what is imported where.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This way we pay once for the expansion, and lookups are then done in
constant time instead of iterating over the list over and over again.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is no guarantee __del__ will ever be called, and we were leaving a
ton of stuff in /tmp. With this patch we pass the temporary directories
explictly and make sure they are deleted at the end.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
These requires are accessible from a separate attribute, but we want to
handle them the same ways as regular Requires.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We want to exclude all multilib packages matching the blacklist (that
are not noarch) from the package set.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This only works in non-greedy mode. When greedy, the same provides can
be linked to multiple packages that should be pulled in.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
* Remove dead imports
* Remove dead code
* Fix extra empty lines
* Fix formatting
* Break too long lines
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
the second most demanding filter
it is called 8297 times and gets 189s out of 15min run which is 21%
spent here
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
this is most demanding filter
it is called 10826 times and gets 210.277010s out of 15min run which is
23% spent here
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Split the function for getting implant MD5 into two functions, so that
we can have access to other information retrieved by checkisomd5
command. A test case is added for both functions.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch refactors logic for creating a temporary mount point,
mounting an image, running arbitrary code on it, unmounting the image
and removing the mount point. It immediately uses it in the buildinstall
phase.
Similar mounting is present in product_img phase as well, but due to
different usage pattern it's not changed yet.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Koji expects the value as comma separated list, otherwise it will ignore
it. This makes it possible for the image building to fail on per arch
basis.
Relates: #128
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Currently if there are multiple ostree installers for the same
architecture, the logfiles all end up in the same location and overwrite
each other.
This patch moves the logs to logs/<arch>/<variant>/ostree_installer-X/
for a unique value of X so that there can be multiple runs even for the
same tree.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When even after retries the unmounting still fails, this patch runs `ls
-lA`, `fuser -vm` and `lsof +D` on the directory to give some idea of
what's blocking it.
Relates: #559
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the image can not be unmounted because the device is busy, we should
retry. There will be increasing pauses between the attempts. At most 10
attempts will be done before giving up.
Fixes: #559
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the checkisomd5 command exits successfully but returns a wrong value,
we should catch and log that. In theory this should be impossible, but
we have seen it in production.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If task does not produce the expected number of images, it should fail
at Koji level. Also, this check does not really work when some
architectures are allowed to fail.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
live_images: additional_repos -> repo
ostree: source_repo_from -> repo_from
extra_source_repos -> repo
ostree_installer: source_repo_from -> repo_from
With the change, the phases have consolidate option names for variant
repos and external repos.
Old option names will continue to work, old names will be converted
to new names after validation automatically if new options are not
specified in config.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
When a property has 'alias' defined, and it's not present in instance,
if the alias property is present, add the property with value from alias
property before remove the alias property from instance.
Examples:
with schema:
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Pungi Configuration",
"type": "object",
"properties": {
"release_name": {"type": "string", "alias": "product_name"},
},
"required": ["release_name"],
"additionalProperties": False,
}
1. config = {"release_name": "dummy product"}:
validate pass, config not changed after validation.
2. config = {"product_name": "dummy product"}:
validate pass, config updated to the following after validation:
config: {"release_name": "dummy product"}
3. config = {"name": "dummy product"}:
validate fail, errror message is "Failed validation in : 'release_name' is a required property",
and warning message is "WARNING: Unrecognized config option: name."
4. config = {"product_name": "dummy product", "release_name": "dummy product"}
validate fail, error message is "Failed validation in : product_name is an alias of release_name, only one can be used."
Signed-off-by: Qixiang Wan <qwan@redhat.com>
The line of log printing can result in "No handlers could be found for
logger" warning in quiet mode when custom paths module is enabled,
because file handler is setup after that, since this line is only
printed to screen (when pungi-koji is not ran in quiet mode) and doesn't
contain important info, it doesn't hurt to remove it.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
This is a follow up of 8bc65a8be5. When
resolving packages that go into optional variant, we only want to skip
filtering groups when working on a top-level variant.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
For different cases where runroot is used it's now possible to set
custom weight. The usecase for this is to avoid one builder taking too
many tasks. Especially buildinstall is quite resource intensive, so one
builder taking multiple tasks at the same time leads to very slow
compose time.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The optional variant can defined by just adding has_optional=True into
variant xml. In such case it has no comps groups and Pungi would copy
the original file unmodified. This leads to extra packages being pulled
into the optional variant.
In this case the correct solution is to filter the comps and remove all
groups.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of creating and configuring the logger at module import time, we
can only get the logger if it's actually needed and configure it from
the main script.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
In multiple situations we need to create temporary files or directories
that should not be preserved after compose is finished. Let's add
context managers that ensure these get cleaned up.
This fixes tests leaving garbage around in /tmp.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
