There is no guarantee __del__ will ever be called, and we were leaving a
ton of stuff in /tmp. With this patch we pass the temporary directories
explictly and make sure they are deleted at the end.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
These requires are accessible from a separate attribute, but we want to
handle them the same ways as regular Requires.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This only works in non-greedy mode. When greedy, the same provides can
be linked to multiple packages that should be pulled in.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
With this script it's possible to add additional files into an ISO file.
If the file happens to be ks.cfg, the boot configs are tweaked so that
the kickstart is actually used.
Resolves: #503
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch refactors logic for creating a temporary mount point,
mounting an image, running arbitrary code on it, unmounting the image
and removing the mount point. It immediately uses it in the buildinstall
phase.
Similar mounting is present in product_img phase as well, but due to
different usage pattern it's not changed yet.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Currently if there are multiple ostree installers for the same
architecture, the logfiles all end up in the same location and overwrite
each other.
This patch moves the logs to logs/<arch>/<variant>/ostree_installer-X/
for a unique value of X so that there can be multiple runs even for the
same tree.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When even after retries the unmounting still fails, this patch runs `ls
-lA`, `fuser -vm` and `lsof +D` on the directory to give some idea of
what's blocking it.
Relates: #559
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the image can not be unmounted because the device is busy, we should
retry. There will be increasing pauses between the attempts. At most 10
attempts will be done before giving up.
Fixes: #559
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the checkisomd5 command exits successfully but returns a wrong value,
we should catch and log that. In theory this should be impossible, but
we have seen it in production.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
live_images: additional_repos -> repo
ostree: source_repo_from -> repo_from
extra_source_repos -> repo
ostree_installer: source_repo_from -> repo_from
With the change, the phases have consolidate option names for variant
repos and external repos.
Old option names will continue to work, old names will be converted
to new names after validation automatically if new options are not
specified in config.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
When a property has 'alias' defined, and it's not present in instance,
if the alias property is present, add the property with value from alias
property before remove the alias property from instance.
Examples:
with schema:
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Pungi Configuration",
"type": "object",
"properties": {
"release_name": {"type": "string", "alias": "product_name"},
},
"required": ["release_name"],
"additionalProperties": False,
}
1. config = {"release_name": "dummy product"}:
validate pass, config not changed after validation.
2. config = {"product_name": "dummy product"}:
validate pass, config updated to the following after validation:
config: {"release_name": "dummy product"}
3. config = {"name": "dummy product"}:
validate fail, errror message is "Failed validation in : 'release_name' is a required property",
and warning message is "WARNING: Unrecognized config option: name."
4. config = {"product_name": "dummy product", "release_name": "dummy product"}
validate fail, error message is "Failed validation in : product_name is an alias of release_name, only one can be used."
Signed-off-by: Qixiang Wan <qwan@redhat.com>
For different cases where runroot is used it's now possible to set
custom weight. The usecase for this is to avoid one builder taking too
many tasks. Especially buildinstall is quite resource intensive, so one
builder taking multiple tasks at the same time leads to very slow
compose time.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The optional variant can defined by just adding has_optional=True into
variant xml. In such case it has no comps groups and Pungi would copy
the original file unmodified. This leads to extra packages being pulled
into the optional variant.
In this case the correct solution is to filter the comps and remove all
groups.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
In multiple situations we need to create temporary files or directories
that should not be preserved after compose is finished. Let's add
context managers that ensure these get cleaned up.
This fixes tests leaving garbage around in /tmp.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This adds a new test for Requires(pre) and (post). The general structure
of the test now makes it easy to use the same test scenarios for
different backend.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This adds a new option repoclosure_backend that changes what tool is
used for repoclosure.
Checking build dependencies is currently not supported, as `dnf` does
not have the corresponding option.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of joining the arches as a comma separated string and splitting
it again later. Ultimately we do need the original format to pass to
koji wrapper, but we can produce that value later.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Write file sizes of images in checksum files with comment lines,
checksum files are in BSD-style which supports comments by starting
a line with '#'.
Example:
$ cat RHEL-7.4-20170123.n.4/compose/Server/x86_64/iso/RHEL-Server-7.4-x86_64-20170123.n.4-CHECKSUM
# RHEL-7.4-20170123.n.4-Server-x86_64-dvd1.iso: 3725590528 bytes
# RHEL-7.4-20170123.n.4-Server-x86_64-boot.iso: 377487360 bytes
SHA256 (RHEL-7.4-20170123.n.4-Server-x86_64-dvd1.iso) = fa3de37fe4b859a0285f16ea1123f44f15aec169aea84bf010aa3821bd58fc41
SHA256 (RHEL-7.4-20170123.n.4-Server-x86_64-boot.iso) = 74bf68c54665328adb08b09daf773c67e633b5907e3e2797338ab3c1b58fdf48
(No space at the start of line, because git commit message drops lines
start with '#', added one space to avoid that.)
When there are multiple checksum types specified and checksums are
written to individual files, file size of the image will also be
written to every checksum files.
Fixes: #493
Signed-off-by: Qixiang Wan <qwan@redhat.com>
If gpgkey option is defined in config, set gpgcheck=1 and set
gpgkey=<value> in variant repo files.
Fixes: #487
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Add new key 'repo' to allow specifying multiple repos as the source
repositories. And change 'source_repo_from' to allow specifying multiple
vairant names to use variant repos as source repositories.
Doc of 'source_repo_from' is updated to not mention URL is supported,
though we still support that in code. User should add url of repos in
'repo' key instead of 'source_repo_from'.
Fixes: #508
Signed-off-by: Qixiang Wan <qwan@redhat.com>
When creating unified ISOs, the script will now also create one iso per
architecture containing a repo with debuginfo packages.
There is no switch to turn this off. The images can simply not be
shipped if not wanted.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Add compose.mkdtemp which creates and returns a unique temporary
directory under <compose_topdir>/work/{global,<arch>}/tmp[-<variant>]/
Change tempfile.mkdtemp in code to compose.mkdtemp, so temporary
dirs are created under the compose's workdir, this makes it more
easier to maintain the temporary dirs/files especially when compose
fails with amount of temporary files left there.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
This a standalone script that will look into a compose and create
unified ISO for each architecture. The ISO contains RPM repositories for
all variants that have the arch.
Known issues:
* The filename does not respect settings. This is tricky because the
name could include variant name, which we don't have here (by design
of unified ISO).
* The same is true for volume id.
In order to test the feature without running actual compose, we need to
add essentially a big chunk of compose. Most of the files are empty, as
their content is never accessed.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This makes sure the test configurations will be accepted in real usage.
It also enables us to remove some manual error checking that will be
performed by validator.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
