This patch adds a new config option createrepo_use_xz, which when set to
true will cause createrepo to compress sqlite databases with xz. The
default setting is False.
Fixes: #387
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Introduces a new metadata file to track arbitrary files added during the
extra-files phase. This file is placed in the root of each tree and is
called ``extra_files.json``. It is a JSON file containing a single
object, which contains a "header" key with an object describing the
metadata, and a "data" key, which is an array of objects, where each
object represents a file. Each object contains the "file", "checksums",
and "size" keys. "file" is the relative path from the tree root to the
extra file. "checksums" is an object containing one or more checksums,
where the key is the digest type and the value of that key is the hex
digest. Finally, the size is the size of the file in bytes.
For example:
{
"header": {"version": "1.0},
"data": [
{
"file": "GPL",
"checksums": {
"sha256": "8177f97513213526df2cf6184d8ff986c675afb514d4e68a404010521b880643"
},
"size": 18092
},
{
"file": "release-notes/notes.html",
"checksums": {
"sha256": "82b1ba8db522aadf101dca6404235fba179e559b95ea24ff39ee1e5d9a53bdcb"
},
"size": 1120
}
]
}
Signed-off-by: Jeremy Cline <jeremy@jcline.org>
Fixes: #295
If the configuration does not specify version for images or live media,
Pungi will create a default value based on `release_version`. If label
is used for the compose, the milestone from it will be appended to the
version (unless it's RC).
This change is backwards compatible: nothing changes when version is set
in configuration. If the version was missing before, building the
artifacts would fail. With this patch, default values will be supplied.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The schema is written in Python to reduce duplication. When
configuration is loaded, the validation checks if it's correct and fills
in default values.
There is a custom extension to the schema to report deprecated options.
The config dependencies are implemented as a separate pass. While it's
technically possible to express the dependencies in the schema itself,
the error messages are not very helpful and it makes the schema much
harder to read.
Phases no longer define `config_options`. New options should be added to
the schema. Since the default values are populated automatically during
validation, there is no need to duplicate them into the code.
The `pungi-config-validate` script is updated to use the schema and
report errors even for deeply nested fields.
The dependencies are updated: pungi now depends on `python-jsonschema`
(which is already available in Fedora).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It uses bash specific features, so it should not claim /bin/sh in
shebang. We also want to use `set -e` to catch possible errors instead
of claiming success every time.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When release for an image is specified as explicit `None`, we can
generate the value based on compose label. For example for `Alpha-1.2`
the release would be `1.2` instead of the date based one.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Before the task is started, the output directory is checked and if it
exists and is not empty, the runroot task will be skipped. This is meant
for debugging when restarting the same compose. Under usual
circumstances, the directory will not be created in the first place.
The runroot task will start by removing the output directory. This way,
if koji restarts the task, lorax will not fail.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of installing pungi itself in the runroot, we can prepare the
commands to be run on compose box, write the shell script into work/
directory, which is mounted in the chroot, and execute that. This way
there is no business logic in runroot (except for finding lorax
templates).
The main advantage of this approach is that we don't need to pull any
extra dependencies into buildroot.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is a breaking change as big part of current failable_deliverables
options will be ignored.
There is no change for buildinstall and creatiso phase.
Failability for artifacts in other phases is now configured per
artifact. It already works correctly for ostree and ostree_installer
phases (even per-arch). For OSBS phase there is currently only a binary
switch as it does not handle multiple arches yet. When it gains that
support, the option should contain list of non-blocking architectures.
For live images, live media and image build phases each config block can
configure list of failable arches. If the list is not empty, it can
fail. Once we have a way to fail only some arches, the config will not
need to change.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Atomic Reactor does not honor this option. In the future we might need
to reintroduce this feature, but given that it does not work in the
current form it is better removed.
Fixes: #348
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
ISO image without MBR and GPT can still be bootable if it has an El
Torito boot catalog. The test phase must accept such images.
This slightly defeats the point of the check: to verify the ISO is
hybrid. However, based on the metadata we have no way to actually tell
if the image is supposed to be hybrid.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch makes it possible to use different style format placeholders.
Instead of the percent encoding it is now possible to use simple curly
braces.
%(foo)s -> {foo}
The old format is still available.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the `<groups></groups>` section is not specified in the variants XML
file, all groups will be used in this variant. The section must be
omitted completely, not just empty. This is (and was) correct according
to the DTD, it just lead to crash before.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of running the copy function for all variant.arch pairs
unconditionally, only do it if there is something to do. This makes the
log more understandable.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It will take RPM repo from a variant in this compose and a Dockerfile
from configured git and use it to build an image.
The build images are uploaded to some a Docker registry by OSBS and are
not directly part of compose (because there is no export function).
There is a new metadata file `osbs.json` that has some information that
can be used to find the image.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of iterating over image manifest, loop through all variants and
arches and see if there are any images. This avoids a crash for variants
nested under other variants (layered products, optionals or addons).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is for layered products that have a variant mapped
to multiple variants in a base product, for example:
* Foo-Tools (id: FooTools, uid: Foo-Tools, name: Tools)
* Bar-Tools (id: BarTools, uid: Bar-Tools, name: Tools)
Requires productmd >= 1.2.
Signed-off-by: Daniel Mach <dmach@redhat.com>
Add a documented and tested config options for setting ISO parameters
instead of hardcoding magic values.
Fixes: #256
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch also fixes a bug where packages in hashed directories would
not be recognized as such and would not be placed after other files.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of putting everything to <variant>/<arch>/iso, move images/,
EFI/ and isolinux/ subdirs to <variant>/<arch>/os. The nicely named ISO
image is still in the original location.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of having a separate config option, just use the koji profile.
According to release notes, this should have already been done.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The module backports features to Python 2.6 and 2.7. If it is available,
the tests will use it. If it is not available, it will fall back to
regular unittest. On Python 2.7, the tests pass anyway. On Python 2.6,
there are failures with Python 2.6.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
On Python 2.6, it requires the format placeholder to have explicit index
of argument, so using % formatting is easier.
There are a couple places where the method is still used because the
same argument is used twice.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This tests writing and reading them.
Also, it makes sure the description for layered products is correct:
there was a missing space.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Each variant.arch combination can have multiple ostree repos configured,
so we need to make sure the filesystem paths don't clash.
The paths used now are:
logs/<arch>/<variant>/ostree-<x>/
work/ostree-<x>/config_repo
The x stands for a number identifying the task. It has no relation to
actual contents of the repo.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Use a mock instead of relying on the real module in tests. This fixes
test failures on RHEL-7 (caused by different configuration).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Make sure the directory exists before calling Koji (because otherwise
the mounting will fail). Update the runroot script to initialize the
repo when there are no files in the target destination.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The performed checks:
* If format is ISO, the file must have correct magic string
* If it's bootable, there must be MBR or GPT
When a check fails on any failable deliverable, it will be logged and
the file removed from metadata (it will still remain on the disk). This
required a change to write the images.json file later (after test
phase).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
rhbz#1331317 when we refactored how we make dvds in
df400002d8 we lost the ability to boot
the dvd as a disk image.
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
There is a lot of mock objects needed: we bypass calls to Koji, use a
mock FileCache that does not need valid RPMs on disk and avoid any
multithreading.
The test data in tests/fixtures/tagged-rpms.json comes from Koji. It is
filtered down to only a few packages to make it manageable.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch also fixes the bug where using a preconfigured Koji event
would not actually work and instead the latest event would be used.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The live_images, live_media and image_build phases have same options
that need to be specified on each image. This leads to a lot of
duplication in the config file. This patch adds global settings and
phase-level settings that allow to significantly reduce duplication.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of spawning `cp x/* y` there is now Python code to the same
thing. This should help with debugging if something fails as the
traceback will be more informative (rather than saying a command
failed). As another benefit the tests get much simpler.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The pungi.global.log should show subvariant for each failed deliverable
(if available). When the compose finishes, there is a new log file in
logs/global/deliverables.json containing details about all deliverables
as triples of (variant, arch, subvariant).
* `required` lists all deliverables that can not fail
* `attempted` lists all failable deliverables that were started
* `failed` is a subset of `attempted` and only contains deliverables
that failed
If the compose fails, the lists may be incomplete.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We can't assume the templates will just be available. This patch adds a
configuration option to point to the git repo with them. It will be
cloned at compose box and relative paths to templates will be made
absolute respective to this clone.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
and pass absolute path to the config into runroot. This could avoid the
problem of not being able to clone the repo in runroot.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
A couple arguments passed from phase to worker threads were not
duplicated. Only one copy is passed now.
A test case was added both for the phase itself and for worker thread as
well.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of running a long command line in the runroot (or locally), move
all that work into a separate script that will be installed. This means
chroot will need to install pungi.
Everything should work as it did before. The only exception to this is
that there is logic to find lorax templates instead of harcoding the
location. This is done using a separate script.
Related: #230Fixes: #231
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If umask is set to something too high (>0022), a warning will be
printed. It does not abort the compose though.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This causes a race condition: umask is process-wide setting, so any file
created by another thread while the umask is set to 0 will have wrong
permissions. This also removes the possible problem of not resetting the
umask if exception is raised.
Fixes: #239
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There already were config options tree_arches and tree_variants, but the
filtering was done at the compose level and there was not interaction
between the two options. This led to problems when a variant would have
all its arches filtered out.
This patch moves the filtering to the variant loader. It adds better
logging, so whenever a variant is filtered (for any reason), it will be
explicitly stated in the logs.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This uses a new common function that will ensure consistent logging
without duplicating the functionality all over the place.
A couple tests are updated to verify that correct data is logged.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the koji tasks fails but produces some output (e.g. .discinfo),
copying the artifacts would crash the compose. That should not happen if
the variant/arch is marked as failable.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This phase runs lorax with extra templates in Koji runroot task, links
the boot.iso to proper location in compose directory and adds the
installer iso to image manifest. This phase runs concurrently with live
media etc.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This phase runs the script to make ostree repository in koji runroot
task. It runs right after regular yum repos are created.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is a wrapper over ostree and rpm-ostree. It is intended to be run
in either Koji or Mock chroot.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Move getting of the precomputed value to a single place in Compose
class. The value now has format `date[.type_suffix].respin`.
Resolves: rhbz#1319924
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Print the subvariant together with variant UID and arches. This should
make the global log easier to understand.
The output of koji command should now be split into multiple files in
logs/{arch}/livemedia-{variant}-{subvariant}.{arch}.log instead of
having them all in one file.
The tests were updated to use more sensible value for the subvariant
field.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should generate the same volume id in buildinstall and createiso
phases, with isolinux.cfg being consistent as well.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Check once before iterating through the variants. This greatly
simplifies the tests as each function now has one less code path.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When creating the comps repo, use a separate log file for each variant
instead of overwriting the same global file again and again.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The configuration can specify a list of variants for which the original
comps file will be copied without any modification.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It is used only in createiso and productimg phases.
For productimg, it needs to be present only when the compose is bootable
and productimg phase is explicitly enabled.
For createiso, it is only needed if runroot is not enabled.
Additionally, if we detect pungi running on arch for which syslinux is
not available, a warning is printed, but the compose is allowed to
continue (and possibly crash later).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the configuration specifically requests no jigdos, there is no point
in checking for the binary existence.
This is not 100% reliable. The jigdo option defaults to True, so if the
option is not specified the binary is required even if there are no
images configured.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch adds configuration option to change disc type used in file
name. So far this can only be changed for link to images/boot.iso.
Resolves: #109
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This was already implemented as part of pkgset phase. It is now moved to
the util module and covered with tests.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
With this patch Pungi should be more tolerant of network failures when
running a blocking command (creating live media or live images).
If the connection drops and the output indicates network problems, Pungi
will try to watch the task with `koji watch-task`. This will be retried
until it finishes (successfully or with some other failure). There is an
increasing timeout after each retry. Currently the maximum number of
retries is not limited.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
After config was modified to work with sections, the resolving broke.
This patch fixes it and adds a test to catch this problem in the future.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The log file started with createiso-, which is utterly confusing. With
this patch, the logs are named with liveimage- prefix.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There were asserts that actually did not test anything. They should have
checked that koji is called with correct arguments.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Extracting extensions from the file name is not reliable as there is no
way to determine where extensions start. There can very well be a
version.release separated by dot. To bypass this, just use hardcoded
list of possible formats.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
* Add option to keep filenames generated by Koji.
* Put results of spin-appliance into image dir
* On failure images are no longer deleted.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
With this patch, you can specify a command for
signing of koji builds. For example:
signing_key_password_file = '~/file_with_password_for_key_fedora-24'
signing_key_id = '81b46521'
signing_command = '~/git/releng/scripts/sigulsign_unsigned.py -vv --password=%(signing_key_password)s fedora-24'
'signing_key_password_file' is a path to a file which contains
a password that will be formatted into 'signing_command' string
via '%(signing_key_password)s' string format syntax (if used).
Because pungi config is usualy stored in git and part of compose
logs we don't want password to be included directly in the config.
Note: If '-' is used instead of a filename, then you will be asked
for the password interactivelly right after pungi starts.
'signing_key_id' is ID of the key that will be used for the signing.
This ID will be used when crafting koji paths to signed files
(kojipkgs.fedoraproject.org/packages/NAME/VER/REL/data/signed/KEYID/..).
'signing_command' a command that will be run with a build as a single
argument. This command mustn't require any user interaction.
If you need to pass a password for a signing key to the command,
do this via command line option of the command with use of string
formatting syntax '%(signing_key_password)s' (see details
about 'signing_key_password_file').
Signed-off-by: Tomáš Mlčoch <tmlcoch@redhat.com>
The variants.xml file can list a variant with is_empty="true" and no
groups. If such variant is found, not package gathering will be run for
it, and no repos will be created.
This only makes sense for a variant that will have some other
deliverables like live media or images.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch updates tests so that they pass, and fixes a couple places in
image build that did not work correctly with the nested configuration.
The documentation is reformatted so that it horizontal scrolling is less
likely.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If Koji fails runroot task for some reason, the output will most likely
not have the required format and will crash Pungi.
Pagure: #140
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch fixes how logs are stored if lorax is used as buildinstall
method. The logs for each variant are in a separate file now. If failure
is allowed, the global log will now show why it failed.
A couple tests are added as well.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
A missing input value was causing tests to not check the expected
condition (even though they were still passing).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This phase builds live media in Koji using the Live Media Creator. It
runs in parallel with current live images, create ISO and image build
phases.
The documentation is updated to explain how to configure this.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch adds support for live media creator in Koji. The intended
workflow is to create a command , run it and finally collect built
artifacts.
get_live_media_cmd()
run_blocking_cmd()
get_image_paths()
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The methods mentioning image build are generic and can work for other
task types.
get_image_build_paths -> get_image_paths
run_create_image_cmd -> run_blocking_cmd
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the release is explicitly set to None, generate a value from date
and respin. The documentation is updated to explain how it works.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is now a single option `multilib`, that maps variants and arches
to multilib methods. This replaces old `multilib_methods` option.
Multilib arches are implicitly deduced instead of using the
`multilib_arches` option.
The test compose is updated to only enable multilib on Server and its
addons.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
