If a phase is started successfully, it needs to be stopped as well. In
most cases when `stop` is called immediately after `start`, this is not
a problem.
Only when something else happens while a phase is runnning and this
something fails, Pungi will deadlock and never exit. This something
could be another phase or just main thread raising an exception.
Fixes: #625
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When there is a typo in the comps file, instead of crashing with a
non-descript KeyError we should raise a nice error with details about
the problem.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Occasionally we have seen the mount command fail. The default error
message says to set some environment variables and try again. We can
just always set the environment and only print the output on failure.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When looking at a package in a lookaside repo, it does not make much
sense to process its dependencies. We should just assume that the
lookaside can satisfy them.
In the worst case, this could result in packages being pulled into the
compose just so that they could satisfy a dep of something in lookaside.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If keytab is used for authentication, other commands than runroot can
possibly fail due to the credentials cache being overwritten.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The get_system_release_packages function can never be called without a
variant, so it makes no sense to check for that condition.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
* add comments explaining what's going on
* break too long lines
* simplify the logic where possible
* use with statement to work with files
* remove commented out and unused code
* introduce helpers to reduce code duplication
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the koji profile we are using is configured to use keytab, we should
run koji executable with a fresh credentials cache. Otherwise we risk a
race condition as multiple processes will trample over the same
directory in /tmp/krbcc_0.
This is currently only implemented for calling `koji runroot`. We might
need to do it for other commands as well (currently there is a sleep to
avoid the race condition for other commands).
Fixes: https://pagure.io/releng/issue/6715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
In order to avoid conflicting tags, OSBS allows only one build for a
repo/branch pair at the same time. To avoid race conditions, we should
make sure we always pass in the branch. This commit makes it a required
option.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of adding a new config option, we can just reuse the existing
`media_checksums` value. If the value is good for image checksums, it
should work for extra files as well.
Relates: #591
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the config contains a git url pointing to a non-existing branch,
pungi will fail to get commit hash from that branch and die with a
confusing error message.
Fixes: #583
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When we fail to generate a volume ID that fits in 32 characters, the
error message should include the options that were considered. It could
show that there might be a substitution that could fix the problem.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We have some hooks yield ConfigOptionWarning. When it happens within
anyOf validator, anyOf validator yield ValidationError and reports the
config as incorrect. We need to overwrite it to pass not break.
Fixes: #598
Merges: #599
Signed-off-by: Qixiang Wan <qwan@redhat.com>
The config now uses similar logic what previous commit did for OSTree.
Also we should report error when an unknown generator is used.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Running depsolving with no requested inputs will only lead to a hard to
decipher error. We should instead explicitly tell the user that there is
a problem.
Unit tests are added to add to test this functionality.
Relates: #585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When variants XML lists a group that does not match any known group in
input comps, report a warning. This is not necessarily a problem in
itself, but having this information in the log can help debug problems.
Relates: #585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Config option 'repo' and 'repo_from' are used in several phases, merge
them with one option 'repo'. 'append' in schema is used for appending
the values from deprecated options to 'repo', so it won't break on any
existing config files that have the old options of 'repo_from' and
'source_repo_from' (which is an alias of 'repo_from').
And 'repo' schema is updated to support repo dict as the value or an
item in the values, a repo dict is just a dict contains repo options,
'baseurl' is required in the dict, like:
{"baseurl": "http://example.com/url/to/repo"}
or:
{"baseurl": "Serer"}
currently this is used in ostree phase to support extra repo options
like:
{"baseurl": "Server", "exclude": "systemd-container"}
Signed-off-by: Qixiang Wan <qwan@redhat.com>
If 'append' is defined for a property, append the values from append
options to the property. Note: The property must support to be a list
of values.
For example:
with schema:
schema = {
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Pungi Configuration",
"type": "object",
"definitions": {
"list_of_strings": {
"type": "array",
"items": {"type": "string"},
},
"strings": {
"anyOf": [
{"type": "string"},
{"$ref": "#/definitions/list_of_strings"},
]
},
},
"properties": {
"release_name": {"type": "string"},
"repo": {"$ref": "#/definitions/strings", "append": "repo_from"}
},
"additionalProperties": False,
}
and config:
repo = "http://url/to/repo"
repo_from = "Server"
config will be updated to:
repo = ["http://url/to/repo", "Server"]
It supports multiple append options too, like:
"repo": {
"$ref": "#/definitions/strings",
"append": ["repo_from", "source_repo_from"],
}
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Show warning message for any alias option find in config instance.
Example warning message:
WARNING: Config option 'product_name' is deprecated and now an alias to
'release_name', please use 'release_name' instead. In:
{'release_name': 'dummy product', 'product_name': 'dummy product'}
Signed-off-by: Qixiang Wan <qwan@redhat.com>
If there are import errors for DNF, multilib or other related package,
we can assume the tests are running on EPEL. The DNF tests should be
skipped in that case.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We also rename the old multilib module used by dnf code to multilib_yum
to make it clear what is imported where.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is no guarantee __del__ will ever be called, and we were leaving a
ton of stuff in /tmp. With this patch we pass the temporary directories
explictly and make sure they are deleted at the end.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
These requires are accessible from a separate attribute, but we want to
handle them the same ways as regular Requires.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This only works in non-greedy mode. When greedy, the same provides can
be linked to multiple packages that should be pulled in.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
With this script it's possible to add additional files into an ISO file.
If the file happens to be ks.cfg, the boot configs are tweaked so that
the kickstart is actually used.
Resolves: #503
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch refactors logic for creating a temporary mount point,
mounting an image, running arbitrary code on it, unmounting the image
and removing the mount point. It immediately uses it in the buildinstall
phase.
Similar mounting is present in product_img phase as well, but due to
different usage pattern it's not changed yet.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Currently if there are multiple ostree installers for the same
architecture, the logfiles all end up in the same location and overwrite
each other.
This patch moves the logs to logs/<arch>/<variant>/ostree_installer-X/
for a unique value of X so that there can be multiple runs even for the
same tree.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When even after retries the unmounting still fails, this patch runs `ls
-lA`, `fuser -vm` and `lsof +D` on the directory to give some idea of
what's blocking it.
Relates: #559
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the image can not be unmounted because the device is busy, we should
retry. There will be increasing pauses between the attempts. At most 10
attempts will be done before giving up.
Fixes: #559
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the checkisomd5 command exits successfully but returns a wrong value,
we should catch and log that. In theory this should be impossible, but
we have seen it in production.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
live_images: additional_repos -> repo
ostree: source_repo_from -> repo_from
extra_source_repos -> repo
ostree_installer: source_repo_from -> repo_from
With the change, the phases have consolidate option names for variant
repos and external repos.
Old option names will continue to work, old names will be converted
to new names after validation automatically if new options are not
specified in config.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
When a property has 'alias' defined, and it's not present in instance,
if the alias property is present, add the property with value from alias
property before remove the alias property from instance.
Examples:
with schema:
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Pungi Configuration",
"type": "object",
"properties": {
"release_name": {"type": "string", "alias": "product_name"},
},
"required": ["release_name"],
"additionalProperties": False,
}
1. config = {"release_name": "dummy product"}:
validate pass, config not changed after validation.
2. config = {"product_name": "dummy product"}:
validate pass, config updated to the following after validation:
config: {"release_name": "dummy product"}
3. config = {"name": "dummy product"}:
validate fail, errror message is "Failed validation in : 'release_name' is a required property",
and warning message is "WARNING: Unrecognized config option: name."
4. config = {"product_name": "dummy product", "release_name": "dummy product"}
validate fail, error message is "Failed validation in : product_name is an alias of release_name, only one can be used."
Signed-off-by: Qixiang Wan <qwan@redhat.com>
For different cases where runroot is used it's now possible to set
custom weight. The usecase for this is to avoid one builder taking too
many tasks. Especially buildinstall is quite resource intensive, so one
builder taking multiple tasks at the same time leads to very slow
compose time.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The optional variant can defined by just adding has_optional=True into
variant xml. In such case it has no comps groups and Pungi would copy
the original file unmodified. This leads to extra packages being pulled
into the optional variant.
In this case the correct solution is to filter the comps and remove all
groups.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
In multiple situations we need to create temporary files or directories
that should not be preserved after compose is finished. Let's add
context managers that ensure these get cleaned up.
This fixes tests leaving garbage around in /tmp.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This adds a new test for Requires(pre) and (post). The general structure
of the test now makes it easy to use the same test scenarios for
different backend.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This adds a new option repoclosure_backend that changes what tool is
used for repoclosure.
Checking build dependencies is currently not supported, as `dnf` does
not have the corresponding option.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of joining the arches as a comma separated string and splitting
it again later. Ultimately we do need the original format to pass to
koji wrapper, but we can produce that value later.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Write file sizes of images in checksum files with comment lines,
checksum files are in BSD-style which supports comments by starting
a line with '#'.
Example:
$ cat RHEL-7.4-20170123.n.4/compose/Server/x86_64/iso/RHEL-Server-7.4-x86_64-20170123.n.4-CHECKSUM
# RHEL-7.4-20170123.n.4-Server-x86_64-dvd1.iso: 3725590528 bytes
# RHEL-7.4-20170123.n.4-Server-x86_64-boot.iso: 377487360 bytes
SHA256 (RHEL-7.4-20170123.n.4-Server-x86_64-dvd1.iso) = fa3de37fe4b859a0285f16ea1123f44f15aec169aea84bf010aa3821bd58fc41
SHA256 (RHEL-7.4-20170123.n.4-Server-x86_64-boot.iso) = 74bf68c54665328adb08b09daf773c67e633b5907e3e2797338ab3c1b58fdf48
(No space at the start of line, because git commit message drops lines
start with '#', added one space to avoid that.)
When there are multiple checksum types specified and checksums are
written to individual files, file size of the image will also be
written to every checksum files.
Fixes: #493
Signed-off-by: Qixiang Wan <qwan@redhat.com>
If gpgkey option is defined in config, set gpgcheck=1 and set
gpgkey=<value> in variant repo files.
Fixes: #487
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Add new key 'repo' to allow specifying multiple repos as the source
repositories. And change 'source_repo_from' to allow specifying multiple
vairant names to use variant repos as source repositories.
Doc of 'source_repo_from' is updated to not mention URL is supported,
though we still support that in code. User should add url of repos in
'repo' key instead of 'source_repo_from'.
Fixes: #508
Signed-off-by: Qixiang Wan <qwan@redhat.com>
When creating unified ISOs, the script will now also create one iso per
architecture containing a repo with debuginfo packages.
There is no switch to turn this off. The images can simply not be
shipped if not wanted.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Add compose.mkdtemp which creates and returns a unique temporary
directory under <compose_topdir>/work/{global,<arch>}/tmp[-<variant>]/
Change tempfile.mkdtemp in code to compose.mkdtemp, so temporary
dirs are created under the compose's workdir, this makes it more
easier to maintain the temporary dirs/files especially when compose
fails with amount of temporary files left there.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
This a standalone script that will look into a compose and create
unified ISO for each architecture. The ISO contains RPM repositories for
all variants that have the arch.
Known issues:
* The filename does not respect settings. This is tricky because the
name could include variant name, which we don't have here (by design
of unified ISO).
* The same is true for volume id.
In order to test the feature without running actual compose, we need to
add essentially a big chunk of compose. Most of the files are empty, as
their content is never accessed.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This makes sure the test configurations will be accepted in real usage.
It also enables us to remove some manual error checking that will be
performed by validator.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The same way live_media and image_build accept additional external repos
or variants list, there is now a `repo` and `repo_from` configuration
key to add these.
Fixes: #486
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This uses the --can-fail option in koji. Failing an optional image will
not abort whole task. If the whole task fails (or there is a problem on
the compose side), we abort unless all arches are optional.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch uses the `--can-fail` option of koji command line. If only
optional arches fail, the task will report as success. Failures on
compose box side are ignored if and only if all architectures are
optional.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Buildinstall only runs for top-level variants. Addons, optionals and
integrated layered products must reuse install tree from their parent,
because otherwise there would be no boot.iso.
Fixes: #472
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Given the way config files can include other files, it is entirely
possible to use the same config file for both Pungi and Distill. Pungi
will however complain about unknown options.
This patch reverts part of c38bb480 and moves deprecation handling back
into schema validation. The validation method then returns a list of
errors and a list of warnings.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Source packages can be excluded while binary packages should still go
into the compose. This patch makes it so that the mapping from binary
packages to source packages contains None in such case. The code is
already capable of handling that. A warning will be emitted for each
binary package without source.
This also allows us to remove some code from `createSourceHashes` that
is now unused.
A test for excluding source package is added as well.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The new sub-command 'installer' is added to support build OSTree
installer image with pungi-make-ostree. It can take an optional argument
'--extra-config' to read some of configurations from a json file. The
content of the json file can contains the configuration which are
supported in OSTree installer phase, the difference is variant UID is
not supported as a repo url in this case. A valid json file can be like
the following:
{
"source_repo_from": "http://www.example.com/repo/workstation/os",
"installpkgs": [
"fedora-productimg-workstation"
],
"add_template": [
"/path/to/installer/template/lorax-configure-repo.tmpl"
],
"add_template_var": [
"ostree_osname=fedora-workstation",
"ostree_ref=fedora/25/x86_64/workstation"
],
"add_arch_template": [
"/path/to/installer/template/lorax-embed-repo.tmpl"
],
"add_arch_template_var": [
"ostree_repo=https://www.example.com/compose/ostree",
"ostree_osname=fedora-workstation",
"ostree_ref=fedora/25/x86_64/workstation"
]
}
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Update pungi-make-ostree to supourt sub-command 'tree', which is just
as the original feature of pungi-make-ostree to compose OSTree tree.
With the change we can add other sub commands later to build other
OSTree artifacts, like the installer image.
Inaddtional to the change, now the the 'tree' command can accept an
optional '--extra-config' parameter to update the original tree
configuration with extra configurations specified in a json file
before composing the OSTree tree.
Example:
pungi-make-ostree tree --repo=/ostree --treefile=/path/to/treefile \
--log-dir=/path/to/log --extra-config=/path/to/extra-config.json
The extra-config file can contains the same configuration as OSTree
phase, the difference is it doesn't understand variant UID as source
repo since it's not ran in the chain of phases. A valid configuration
can be like:
{
"source_repo_from": "http://example.com/repo/x86_64/Server",
"extra_source_repos": [
{
"name": "optional",
"baseurl": "http://example.com/repo/x86_64/optional",
"exclude": "systemd-container",
"gpgcheck": False
},
{
"name": "extra",
"baseurl": "http://example.com/repo/x86_64/extra",
}
],
"keep_original_sources": True
}
The OSTree phase is updated to move out the task of updating treefile,
instead of that, it writes the extra configurations to a json file,
then 'pungi-make-ostree tree' will take it by option '--extra-config'.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
The internal flag in productmd is meant to indicate that a compose is
not meant for publishing. This is potentially useful to allow filtering
in PDC or similar service.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
All tests should print complete diffs on failure, so there is no need to
define this in each test separately.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
