Instead use the definition from python-six. Once we drop Py 2 support
completely, we'll just swap underscores with dots.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should make all tests pass on both Python 2 and Python 3.
Unittest2 is required on Py 2.6 and Py 3.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This would be useful for modularity. The templates can be added now and
variables set via the existing `lorax_option`.
It's not possible to use custom templates not shipped with lorax, as
passing the path to a random directory is a little bit more tricky.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Currently `None` has to be included to allow using unsigned packages.
ODCS has trouble with including non-string value in the list though, so
we can treat empty string the same way (it's not a valid key ID anyway).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the package name matches, we should take the package either if
architecture matches or the package is noarch.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The path prefix for packages is identical for all architectures. There's
no reason to compute it multiple times and check the directory existence
repeatedly.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When splitting is not possible, we should not print a huge number as
free space on single media. This patch also adds more information so
that the numbers can actually be connected to variant and architecture.
The list of ignored files will only be printed now if not empty.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The name can contains spaces, which would potentially cause problems.
Also this is not consistent with how it works in other phases.
Relates: https://pagure.io/pungi-fedora/pull-request/354
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of the old default value and custom handling introduce new
option `live_images_target` (for consistency with other phases) and use
the usual inheritance rules.
Fixes: https://pagure.io/pungi/issue/749
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the variant is not empty, it buildinstall will run lorax on it and
create some files that would otherwise be overwritten by the ostree
installer.
The validation script is updated to load variants file as it needs a
list of variants to find out if the config is wrong.
Fixes: https://pagure.io/pungi/issue/695
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
`pungi-gather` (the tool that underlies both the `pkgset` and `gather`
phases) contains profiling code that will log statistics about how long
different function calls take. However, pungi-koji did not contain a
way to pass the ``--profiler`` argument to enable this.
This change adds a new configuration option ``gather_profiler`` which,
when set to true, simply passes the argument to `pungi-koji`. Hopefully
this can help shed some light on what is happening in some of our
longer-running composes.
Merges: https://pagure.io/pungi/pull-request/727
Signed-off-by: Ralph Bean <rbean@redhat.com>
If the compose failed, it may not have repos to compute deltas against,
and even if it has them, they were never shipped so no one will have the
older version of the package. We should instead go deeper in history and
pick a successful compose.
Relates: https://pagure.io/pungi/issue/715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The configuration needs to be more granular than a single global option.
With this patch each tree can enable deltas separately.
Fixes: https://pagure.io/pungi/issue/715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It does not make much sense to have deltas for source and debug repos.
No one benefits from it really and it takes a long time.
Relates: https://pagure.io/pungi/issue/715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Pungi write image config file with name of <format>-<name>.cfg, if there
are two or more image configs present for different arches under the same
variant and with same format & name, the config file can be overwritten,
and result in invalid image conf file.
Example:
image_build = {
'^Server$': [
{
'image-build': {
'format': [('qcow2', 'qcow2'),],
'name': 'fedora-guest-image',
'target': 'guest-fedora-26-image',
'version': '26',
'ksurl': "git://git.example.com/ks.git?fedora#HEAD",
'kickstart': "fedora-26-kvm.ks",
'ksversion': 'f26',
'distro': 'fedora-26',
'disk-size': '10',
'arches': ['x86_64'],
'repo': ["http://example.com/linux/fedora/26/Everything/x86_64/os", ]
}
},
{
'image-build': {
'format': [('qcow2', 'qcow2'),],
'name': 'fedora-guest-image',
'target': 'guest-fedora-26-image',
'version': '26',
'ksurl': "git://git.example.com/ks.git?fedora#HEAD",
'kickstart': "fedora-26-kvm.ks",
'ksversion': 'f26',
'distro': 'fedora-26',
'disk-size': '10',
'arches': ['ppc64le'],
}
},
],
}
In this case, config file "qcow2_guest-fedora-26-image.cfg" will be
created for both x86_64 and ppc64le under the same variant dir, and
there is a high chance it will be over-written while Pungi creating the
koji task. We can add arch name(s) in config filename to avoid that.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
In many cases we need to open files as binary to avoid errors on Py3
about writing binary data to file opened in text mode.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Even a failed lorax task can leave behind some in-progress images. Pungi
needs to copy the files into compose/ subdirectory only if the task
finished successfully.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1485021
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It's possible a variant is excluded via tree_variants option and the
section does not match anything. It can be confusing to users why
nothing is happening. This patch lets Pungi log all unmatched patterns.
Fixes: https://pagure.io/pungi/issue/692
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When using repos as gather_source, we should use DNF backend even for
constructing initial package set and to download the packages from
source repos. Without this the repos source would not be usable on
Python 3.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Due to overwriting an existing variable the logs are getting duplicated
line about missing comps packages instead of announcement of gathering
being finished. Rename the variable to fix the problem.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It makes no sense to repeat the same configuration for multiple
architectures. Instead we should just list the architectures as another
key in the mapping. There is an option to specify multiple config dicts.
This preserves full backwards compatibility, the old config format is
still accepted.
Fixes: https://pagure.io/pungi/issue/678
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When a package mentioned in comps is not available in the package set,
print a warning about this. Additionally there is a config option that
allows to turn this warning into a fatal error.
Fixes: https://pagure.io/pungi/issue/50
Fixes: https://pagure.io/pungi/issue/683
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There's no reason for reading the whole log of depsolving into memory
just to split it into lines and process one line at a time.
We can just as well read it in chunks.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Alternatively the call to repoclosure can be turned off. This is
customizable per variant and architecture.
Fixes: https://pagure.io/pungi/issue/676
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of just printing the error directly to stderr, capture the
output and use proper logger. This makes sure the error is included in
the log file and also fixes `--quiet` option which was not properly
honored originally.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Currently, this will fail two lines later when we try to access
`pdc_module['modulemd']` with an unhelpful `TypeError` since
`pdc_module` is `None`.
Signed-off-by: Ralph Bean <rbean@redhat.com>
When there is a noarch subpackage, all compatible debuginfo would be
pulled in, which is not desirable.
Example: Server.x86_64 needs pkg.x86_64 and pkg-data.noarch. We only
want pkg-debuginfo.x86_64, but without this patch even
pkg-debuginfo.i686 would get in.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When there are no groups, we shouldn't try to read comps file (because
it may very well not be there).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of iterating over the images metadata and appending the checksum
to relevant files immediately, we should store them and write only once.
This avoid an issue when the same image is mentioned in the metadata
multiple times. This happens for source images that are listed under
each binary arch.
The unified isos script is updated to use the exact same logic and code.
This also uncovered a problem with the metadata for debuginfo unified
isos: their paths in metadata were incorrect, which lead to missing
checksums.
Fixes: https://pagure.io/pungi/issue/667
Fixes: https://pagure.io/pungi/issue/668
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When creating the final repo, we reuse metadata from arch repo used for
depsolving. This however breaks creating deltas with createrepo_c.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Createrepo expects to be pointed to a directory with the actual RPM
files, not the previous repo. This means that when hashed directories
are used, we need to pass in a lot of directories.
Fixes: https://pagure.io/pungi/issue/344
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The message announcing new ostree commit contains hash of the commit,
the ref it's for, but there is no information about where the repo
actually is.
This patch adds `repo_path` key into the message with URL of the repo
and `local_repo_path` with path to the repo on local filesystem.
Relates: https://pagure.io/pungi/issue/650
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When gather_method is set to nodeps, we should not ignore the comps
group that the method received. Instead it should find out which
packages are in those groups and take them into the compose.
In order for this to be of any reasonable use, the comps file needs to
include all dependencies for the packages.
Fixes: #653
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Prior to this change, if the entire product IDs SCM directory was missing, pungi would crash with an error.
For example, if "ceph-3" was missing from the SCM:
OSError: [Errno 2] No such file or directory: '/tmp/tmpMb9O6r/product_ids/ceph-3'
This occurred even if product_id_allow_missing was set to True.
Make product_id_allow_missing cover this case as well, and gracefully
skip all product IDs.
We now see the following warning in the logs instead:
[WARNING ] No product IDs in {'scm': 'git', 'repo': 'git://example.com/rcm/rcm-metadata.git', 'dir': 'product_ids/ceph-3'}
and the compose succeeds.
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
For each tag we ask Koji about (there might be more than one in
modularity case), we create a log file with list of RPMs and details
about which tag they were pulled from. This makes it easier to find out
where the package is inherited from.
Fixes: #547
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When repos are used as pkgset source, the logs are stored in a file that
should have a better name than `fooo.<arch>.log`.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The Koji task can be restarted and lorax will fail if the output
directory already exists. Let's start the work in runroot by removing
the output directory.
Relates: #641
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a phase is started successfully, it needs to be stopped as well. In
most cases when `stop` is called immediately after `start`, this is not
a problem.
Only when something else happens while a phase is runnning and this
something fails, Pungi will deadlock and never exit. This something
could be another phase or just main thread raising an exception.
Fixes: #625
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
As long as the assertion is not violated, this would not be a problem as
the second argument is not evaluated. However if the condition was
broken, a NameError would be raised instead of AssertionError with a
nice message.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The get_system_release_packages function can never be called without a
variant, so it makes no sense to check for that condition.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
* add comments explaining what's going on
* break too long lines
* simplify the logic where possible
* use with statement to work with files
* remove commented out and unused code
* introduce helpers to reduce code duplication
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of adding a new config option, we can just reuse the existing
`media_checksums` value. If the value is good for image checksums, it
should work for extra files as well.
Relates: #591
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We never need anything more than the ID, so passing around a big
complicated object is not necessary.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The config now uses similar logic what previous commit did for OSTree.
Also we should report error when an unknown generator is used.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Running depsolving with no requested inputs will only lead to a hard to
decipher error. We should instead explicitly tell the user that there is
a problem.
Unit tests are added to add to test this functionality.
Relates: #585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When variants XML lists a group that does not match any known group in
input comps, report a warning. This is not necessarily a problem in
itself, but having this information in the log can help debug problems.
Relates: #585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Config option 'repo' and 'repo_from' are used in several phases, merge
them with one option 'repo'. 'append' in schema is used for appending
the values from deprecated options to 'repo', so it won't break on any
existing config files that have the old options of 'repo_from' and
'source_repo_from' (which is an alias of 'repo_from').
And 'repo' schema is updated to support repo dict as the value or an
item in the values, a repo dict is just a dict contains repo options,
'baseurl' is required in the dict, like:
{"baseurl": "http://example.com/url/to/repo"}
or:
{"baseurl": "Serer"}
currently this is used in ostree phase to support extra repo options
like:
{"baseurl": "Server", "exclude": "systemd-container"}
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Do not create empty skeleton dirs for empty variants which we do for rpm
variants in some phases (some others already have the check):
1. createiso phase
2. extra_files phase
3. gather phase
In addtion to this, compose metadata (composeinfo.json) doesn't include
variant paths which don't exist or are just empty dirs now.
Fixes: #497
Merges: #572
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Calling it gather_backend is similar to repoclosure_backend we already
have. It's also more obvious what it does.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch refactors logic for creating a temporary mount point,
mounting an image, running arbitrary code on it, unmounting the image
and removing the mount point. It immediately uses it in the buildinstall
phase.
Similar mounting is present in product_img phase as well, but due to
different usage pattern it's not changed yet.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Koji expects the value as comma separated list, otherwise it will ignore
it. This makes it possible for the image building to fail on per arch
basis.
Relates: #128
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Currently if there are multiple ostree installers for the same
architecture, the logfiles all end up in the same location and overwrite
each other.
This patch moves the logs to logs/<arch>/<variant>/ostree_installer-X/
for a unique value of X so that there can be multiple runs even for the
same tree.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When even after retries the unmounting still fails, this patch runs `ls
-lA`, `fuser -vm` and `lsof +D` on the directory to give some idea of
what's blocking it.
Relates: #559
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the image can not be unmounted because the device is busy, we should
retry. There will be increasing pauses between the attempts. At most 10
attempts will be done before giving up.
Fixes: #559
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the checkisomd5 command exits successfully but returns a wrong value,
we should catch and log that. In theory this should be impossible, but
we have seen it in production.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If task does not produce the expected number of images, it should fail
at Koji level. Also, this check does not really work when some
architectures are allowed to fail.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
live_images: additional_repos -> repo
ostree: source_repo_from -> repo_from
extra_source_repos -> repo
ostree_installer: source_repo_from -> repo_from
With the change, the phases have consolidate option names for variant
repos and external repos.
Old option names will continue to work, old names will be converted
to new names after validation automatically if new options are not
specified in config.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
This is a follow up of 8bc65a8be5. When
resolving packages that go into optional variant, we only want to skip
filtering groups when working on a top-level variant.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
For different cases where runroot is used it's now possible to set
custom weight. The usecase for this is to avoid one builder taking too
many tasks. Especially buildinstall is quite resource intensive, so one
builder taking multiple tasks at the same time leads to very slow
compose time.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The optional variant can defined by just adding has_optional=True into
variant xml. In such case it has no comps groups and Pungi would copy
the original file unmodified. This leads to extra packages being pulled
into the optional variant.
In this case the correct solution is to filter the comps and remove all
groups.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We call pungi to get the packages from external repos if
pkgset_source=repos. In this case we need to look at additional packages
as well.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Fixes: #528
This adds a new option repoclosure_backend that changes what tool is
used for repoclosure.
Checking build dependencies is currently not supported, as `dnf` does
not have the corresponding option.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of joining the arches as a comma separated string and splitting
it again later. Ultimately we do need the original format to pass to
koji wrapper, but we can produce that value later.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Write file sizes of images in checksum files with comment lines,
checksum files are in BSD-style which supports comments by starting
a line with '#'.
Example:
$ cat RHEL-7.4-20170123.n.4/compose/Server/x86_64/iso/RHEL-Server-7.4-x86_64-20170123.n.4-CHECKSUM
# RHEL-7.4-20170123.n.4-Server-x86_64-dvd1.iso: 3725590528 bytes
# RHEL-7.4-20170123.n.4-Server-x86_64-boot.iso: 377487360 bytes
SHA256 (RHEL-7.4-20170123.n.4-Server-x86_64-dvd1.iso) = fa3de37fe4b859a0285f16ea1123f44f15aec169aea84bf010aa3821bd58fc41
SHA256 (RHEL-7.4-20170123.n.4-Server-x86_64-boot.iso) = 74bf68c54665328adb08b09daf773c67e633b5907e3e2797338ab3c1b58fdf48
(No space at the start of line, because git commit message drops lines
start with '#', added one space to avoid that.)
When there are multiple checksum types specified and checksums are
written to individual files, file size of the image will also be
written to every checksum files.
Fixes: #493
Signed-off-by: Qixiang Wan <qwan@redhat.com>
pungi-make-ostree has to run on the target arch so that rpm
scriptlets can be ran. as a reult we can ask rpm what the
basearch is for the running environment. For notifications
we have to pass in the arch we are running for.
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
If gpgkey option is defined in config, set gpgcheck=1 and set
gpgkey=<value> in variant repo files.
Fixes: #487
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Add new key 'repo' to allow specifying multiple repos as the source
repositories. And change 'source_repo_from' to allow specifying multiple
vairant names to use variant repos as source repositories.
Doc of 'source_repo_from' is updated to not mention URL is supported,
though we still support that in code. User should add url of repos in
'repo' key instead of 'source_repo_from'.
Fixes: #508
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Add compose.mkdtemp which creates and returns a unique temporary
directory under <compose_topdir>/work/{global,<arch>}/tmp[-<variant>]/
Change tempfile.mkdtemp in code to compose.mkdtemp, so temporary
dirs are created under the compose's workdir, this makes it more
easier to maintain the temporary dirs/files especially when compose
fails with amount of temporary files left there.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
This a standalone script that will look into a compose and create
unified ISO for each architecture. The ISO contains RPM repositories for
all variants that have the arch.
Known issues:
* The filename does not respect settings. This is tricky because the
name could include variant name, which we don't have here (by design
of unified ISO).
* The same is true for volume id.
In order to test the feature without running actual compose, we need to
add essentially a big chunk of compose. Most of the files are empty, as
their content is never accessed.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This makes sure the test configurations will be accepted in real usage.
It also enables us to remove some manual error checking that will be
performed by validator.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The same way live_media and image_build accept additional external repos
or variants list, there is now a `repo` and `repo_from` configuration
key to add these.
Fixes: #486
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Also remove the TODO comment from live images phase: the appliances are
already submitted one task per single arch, so this change is not
necessary.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This uses the --can-fail option in koji. Failing an optional image will
not abort whole task. If the whole task fails (or there is a problem on
the compose side), we abort unless all arches are optional.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch uses the `--can-fail` option of koji command line. If only
optional arches fail, the task will report as success. Failures on
compose box side are ignored if and only if all architectures are
optional.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Buildinstall only runs for top-level variants. Addons, optionals and
integrated layered products must reuse install tree from their parent,
because otherwise there would be no boot.iso.
Fixes: #472
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Update pungi-make-ostree to supourt sub-command 'tree', which is just
as the original feature of pungi-make-ostree to compose OSTree tree.
With the change we can add other sub commands later to build other
OSTree artifacts, like the installer image.
Inaddtional to the change, now the the 'tree' command can accept an
optional '--extra-config' parameter to update the original tree
configuration with extra configurations specified in a json file
before composing the OSTree tree.
Example:
pungi-make-ostree tree --repo=/ostree --treefile=/path/to/treefile \
--log-dir=/path/to/log --extra-config=/path/to/extra-config.json
The extra-config file can contains the same configuration as OSTree
phase, the difference is it doesn't understand variant UID as source
repo since it's not ran in the chain of phases. A valid configuration
can be like:
{
"source_repo_from": "http://example.com/repo/x86_64/Server",
"extra_source_repos": [
{
"name": "optional",
"baseurl": "http://example.com/repo/x86_64/optional",
"exclude": "systemd-container",
"gpgcheck": False
},
{
"name": "extra",
"baseurl": "http://example.com/repo/x86_64/extra",
}
],
"keep_original_sources": True
}
The OSTree phase is updated to move out the task of updating treefile,
instead of that, it writes the extra configurations to a json file,
then 'pungi-make-ostree tree' will take it by option '--extra-config'.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Phases createiso, liveimages, image_build, ostree_installer and osbs are
done in parallel, logs from these phases are mixed and and it's not
obvious which log message belongs to which phase. This change adds phase
name in log message for these phases.
The new mixin 'PhaseLoggerMixin' is added to extend a Pungi phase with a
logging logger which copy handlers from compose's logger but with
formatter changed.
Fixes: #58
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Sometimes addtional repos are required to get necessary packages for
composing OSTree repository. For example, RHEL doesn't have an 'Everyting'
variant, so composing OSTree repository from any of the RHEL variants
won't work, addtional source repos need to be enabled to achieve that.
The new option "extra_source_repos" enable the ability of allowing extra
source repos.
And a new option 'keep_original_sources' is introduced to keep the
original repos found in tree config file, if this is enabled, Pungi
will not remove the existing source repos from the tree config file,
just add new repos of "source_repo_from" + "extra_source_repos" to
the existing repos.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
When a file is exported from an RPM in the compose, and there is no
matching RPM in the package set, we want a nice error message.
Fixes: #460
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The `get_variants()` method had a `recursive` argument with default
value of `False. However, this argument had no effect and the method
always returned all variants recursively.
We can just drop the argument. All callers are updated to not supply the
argument. Should any need for getting the top-level variants only arise,
they can be accessed as the `variants` attribute directly on the Compose
object.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The code to search for install tree and repo for image-build and
live-media was only looking at top-level variants. Addons, optional or
integrated layered products could not have been found. This would lead
to error messages such as "There is no variant Server-optional to get
repo from when building live image for Client" even though the variant
exists.
Various tests are updated to exercise this edge case.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Tell lorax to use a specific directory for log files so that we preserve
them despite koji not having any idea about them.
Fixes: #457
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Added new option '--version' to pungi-make-ostree, and this can be
enabled in ostree settings with 'version'. The version string will be
added as versioning metadata if this is specified.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Added new option '--update-summary' to pungi-make-ostree, and this can
be enabled in ostree settings with 'update_summary'. A summary file will
be generated (or re-generated if it was presented in an existing ostree
repo) when it is enabled.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
'mount -o loop' requires root privileges, guestmount from
libguestfs-tools-c package can work without root privileges.
Fixes: #19
Signed-off-by: Qixiang Wan <qwan@redhat.com>
If the media is bootable, we can not split it. Instead we will create an
ISO that is too big and issue a warning (aborting the whole compose
would be too much).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The older buildinstall method fails if it sees ppc64p7 instead of ppc64.
Nothing changes for composes using Lorax.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It really is just a group of independent functions, so we can simplify
it by removing the unused wrapper class. Instead of importing the
wrapper, instantiating it and calling its methods we can import the
module and call its functions directly.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of overloading the `dvd` value, use new value `ostree` that can
be overridden by a config change.
Fixes: https://pagure.io/pungi/issue/418
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
to complement https://pagure.io/koji/pull-request/162 we need to adjust pungi
rpm-ostree uses bublewrap that does not work in mock. --new-chroot to mock
enables the use of systemd-nspawn instead of chroot resulting in working
rpm-ostree again
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
The address is no longer correct. We can just as well simply point to
the web page describing the license.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The same information can be inferred from definitions in variants.xml:
if the variant has no groups defined, we include packages from all
groups. By the same logic we can also include all groups in the comps
file.
The config validation is updated to give a hint on how to remove the
option from the configuration.
Relates: #29
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It does not make sense for this phase to be skipped. If there are any
images, we need to generate the checksums so that writing metadata
works. If there are no images, the phase does not do anything and is
therefore very fast.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch adds a new config option createrepo_use_xz, which when set to
true will cause createrepo to compress sqlite databases with xz. The
default setting is False.
Fixes: #387
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Introduces a new metadata file to track arbitrary files added during the
extra-files phase. This file is placed in the root of each tree and is
called ``extra_files.json``. It is a JSON file containing a single
object, which contains a "header" key with an object describing the
metadata, and a "data" key, which is an array of objects, where each
object represents a file. Each object contains the "file", "checksums",
and "size" keys. "file" is the relative path from the tree root to the
extra file. "checksums" is an object containing one or more checksums,
where the key is the digest type and the value of that key is the hex
digest. Finally, the size is the size of the file in bytes.
For example:
{
"header": {"version": "1.0},
"data": [
{
"file": "GPL",
"checksums": {
"sha256": "8177f97513213526df2cf6184d8ff986c675afb514d4e68a404010521b880643"
},
"size": 18092
},
{
"file": "release-notes/notes.html",
"checksums": {
"sha256": "82b1ba8db522aadf101dca6404235fba179e559b95ea24ff39ee1e5d9a53bdcb"
},
"size": 1120
}
]
}
Signed-off-by: Jeremy Cline <jeremy@jcline.org>
Fixes: #295
If the configuration does not specify version for images or live media,
Pungi will create a default value based on `release_version`. If label
is used for the compose, the milestone from it will be appended to the
version (unless it's RC).
This change is backwards compatible: nothing changes when version is set
in configuration. If the version was missing before, building the
artifacts would fail. With this patch, default values will be supplied.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The schema is written in Python to reduce duplication. When
configuration is loaded, the validation checks if it's correct and fills
in default values.
There is a custom extension to the schema to report deprecated options.
The config dependencies are implemented as a separate pass. While it's
technically possible to express the dependencies in the schema itself,
the error messages are not very helpful and it makes the schema much
harder to read.
Phases no longer define `config_options`. New options should be added to
the schema. Since the default values are populated automatically during
validation, there is no need to duplicate them into the code.
The `pungi-config-validate` script is updated to use the schema and
report errors even for deeply nested fields.
The dependencies are updated: pungi now depends on `python-jsonschema`
(which is already available in Fedora).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Currently the main log only says the phase started and finished. With
this patch each created image will be mentioned by its name, version,
release and arch.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Before the task is started, the output directory is checked and if it
exists and is not empty, the runroot task will be skipped. This is meant
for debugging when restarting the same compose. Under usual
circumstances, the directory will not be created in the first place.
The runroot task will start by removing the output directory. This way,
if koji restarts the task, lorax will not fail.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Originally, the idea was to remove non-blocking images that failed the
check. They were only removed from the manifest, which creates some
confusion as to what is going on. With this patch, a failed check on
non-blocking deliverable will only print an error message.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When ISO is marked as failable and check fails, the compose would still
be aborted. This patch fixes that
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of installing pungi itself in the runroot, we can prepare the
commands to be run on compose box, write the shell script into work/
directory, which is mounted in the chroot, and execute that. This way
there is no business logic in runroot (except for finding lorax
templates).
The main advantage of this approach is that we don't need to pull any
extra dependencies into buildroot.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When package has incompatible arch for a tree, the same error message
could be raised from two different places. This patch adds more
information to the message, so that it is easier to find the actual
problem.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is a breaking change as big part of current failable_deliverables
options will be ignored.
There is no change for buildinstall and creatiso phase.
Failability for artifacts in other phases is now configured per
artifact. It already works correctly for ostree and ostree_installer
phases (even per-arch). For OSBS phase there is currently only a binary
switch as it does not handle multiple arches yet. When it gains that
support, the option should contain list of non-blocking architectures.
For live images, live media and image build phases each config block can
configure list of failable arches. If the list is not empty, it can
fail. Once we have a way to fail only some arches, the config will not
need to change.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Atomic Reactor does not honor this option. In the future we might need
to reintroduce this feature, but given that it does not work in the
current form it is better removed.
Fixes: #348
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
ISO image without MBR and GPT can still be bootable if it has an El
Torito boot catalog. The test phase must accept such images.
This slightly defeats the point of the check: to verify the ISO is
hybrid. However, based on the metadata we have no way to actually tell
if the image is supposed to be hybrid.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Make all imports either use full package name starting with `pungi` or
use explicitly relative import. This will avoid issues when importing a
module that shadows another module on PYTHONPATH.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This lives now in pungi.util. It was copied there long ago and all the
places that reference it reference it in that location.
I think this code in pungi.phases.pkgset is unused and can be removed.
The other function moved to the common module.
Signed-off-by: Ralph Bean <rbean@redhat.com>
Instead of overwriting the same log file, make sure we keep all the logs
for debuginfo, source and binary packages.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch makes it possible to use different style format placeholders.
Instead of the percent encoding it is now possible to use simple curly
braces.
%(foo)s -> {foo}
The old format is still available.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the `<groups></groups>` section is not specified in the variants XML
file, all groups will be used in this variant. The section must be
omitted completely, not just empty. This is (and was) correct according
to the DTD, it just lead to crash before.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Even if gather phase is skipped, it will still generate a rpms.json
metadata file. It will have no payload, but the header structure is
there.
Createrepo phase had to be updated to ignore variants that have no RPMs
listed in metadata.
There is one more fix for writing treeinfo files: if repomd.xml files
are not generated, it will no longer crash.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of running the copy function for all variant.arch pairs
unconditionally, only do it if there is something to do. This makes the
log more understandable.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It will take RPM repo from a variant in this compose and a Dockerfile
from configured git and use it to build an image.
The build images are uploaded to some a Docker registry by OSBS and are
not directly part of compose (because there is no export function).
There is a new metadata file `osbs.json` that has some information that
can be used to find the image.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of iterating over image manifest, loop through all variants and
arches and see if there are any images. This avoids a crash for variants
nested under other variants (layered products, optionals or addons).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Add a documented and tested config options for setting ISO parameters
instead of hardcoding magic values.
Fixes: #256
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch also fixes a bug where packages in hashed directories would
not be recognized as such and would not be placed after other files.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of putting everything to <variant>/<arch>/iso, move images/,
EFI/ and isolinux/ subdirs to <variant>/<arch>/os. The nicely named ISO
image is still in the original location.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
On Python 2.6, it requires the format placeholder to have explicit index
of argument, so using % formatting is easier.
There are a couple places where the method is still used because the
same argument is used twice.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Each variant.arch combination can have multiple ostree repos configured,
so we need to make sure the filesystem paths don't clash.
The paths used now are:
logs/<arch>/<variant>/ostree-<x>/
work/ostree-<x>/config_repo
The x stands for a number identifying the task. It has no relation to
actual contents of the repo.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Make sure the directory exists before calling Koji (because otherwise
the mounting will fail). Update the runroot script to initialize the
repo when there are no files in the target destination.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The performed checks:
* If format is ISO, the file must have correct magic string
* If it's bootable, there must be MBR or GPT
When a check fails on any failable deliverable, it will be logged and
the file removed from metadata (it will still remain on the disk). This
required a change to write the images.json file later (after test
phase).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The config object from Kobo does not allow keys starting with
underscore, so we prefix it with `private_` instead.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is a lot of mock objects needed: we bypass calls to Koji, use a
mock FileCache that does not need valid RPMs on disk and avoid any
multithreading.
The test data in tests/fixtures/tagged-rpms.json comes from Koji. It is
filtered down to only a few packages to make it manageable.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch also fixes the bug where using a preconfigured Koji event
would not actually work and instead the latest event would be used.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The live_images, live_media and image_build phases have same options
that need to be specified on each image. This leads to a lot of
duplication in the config file. This patch adds global settings and
phase-level settings that allow to significantly reduce duplication.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The pungi.global.log should show subvariant for each failed deliverable
(if available). When the compose finishes, there is a new log file in
logs/global/deliverables.json containing details about all deliverables
as triples of (variant, arch, subvariant).
* `required` lists all deliverables that can not fail
* `attempted` lists all failable deliverables that were started
* `failed` is a subset of `attempted` and only contains deliverables
that failed
If the compose fails, the lists may be incomplete.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We can't assume the templates will just be available. This patch adds a
configuration option to point to the git repo with them. It will be
cloned at compose box and relative paths to templates will be made
absolute respective to this clone.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
and pass absolute path to the config into runroot. This could avoid the
problem of not being able to clone the repo in runroot.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
A couple arguments passed from phase to worker threads were not
duplicated. Only one copy is passed now.
A test case was added both for the phase itself and for worker thread as
well.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of running a long command line in the runroot (or locally), move
all that work into a separate script that will be installed. This means
chroot will need to install pungi.
Everything should work as it did before. The only exception to this is
that there is logic to find lorax templates instead of harcoding the
location. This is done using a separate script.
Related: #230Fixes: #231
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This uses a new common function that will ensure consistent logging
without duplicating the functionality all over the place.
A couple tests are updated to verify that correct data is logged.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the koji tasks fails but produces some output (e.g. .discinfo),
copying the artifacts would crash the compose. That should not happen if
the variant/arch is marked as failable.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This phase runs lorax with extra templates in Koji runroot task, links
the boot.iso to proper location in compose directory and adds the
installer iso to image manifest. This phase runs concurrently with live
media etc.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This phase runs the script to make ostree repository in koji runroot
task. It runs right after regular yum repos are created.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Move getting of the precomputed value to a single place in Compose
class. The value now has format `date[.type_suffix].respin`.
Resolves: rhbz#1319924
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
