There is a lot of mock objects needed: we bypass calls to Koji, use a
mock FileCache that does not need valid RPMs on disk and avoid any
multithreading.
The test data in tests/fixtures/tagged-rpms.json comes from Koji. It is
filtered down to only a few packages to make it manageable.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch also fixes the bug where using a preconfigured Koji event
would not actually work and instead the latest event would be used.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The live_images, live_media and image_build phases have same options
that need to be specified on each image. This leads to a lot of
duplication in the config file. This patch adds global settings and
phase-level settings that allow to significantly reduce duplication.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of spawning `cp x/* y` there is now Python code to the same
thing. This should help with debugging if something fails as the
traceback will be more informative (rather than saying a command
failed). As another benefit the tests get much simpler.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The pungi.global.log should show subvariant for each failed deliverable
(if available). When the compose finishes, there is a new log file in
logs/global/deliverables.json containing details about all deliverables
as triples of (variant, arch, subvariant).
* `required` lists all deliverables that can not fail
* `attempted` lists all failable deliverables that were started
* `failed` is a subset of `attempted` and only contains deliverables
that failed
If the compose fails, the lists may be incomplete.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We can't assume the templates will just be available. This patch adds a
configuration option to point to the git repo with them. It will be
cloned at compose box and relative paths to templates will be made
absolute respective to this clone.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
and pass absolute path to the config into runroot. This could avoid the
problem of not being able to clone the repo in runroot.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
A couple arguments passed from phase to worker threads were not
duplicated. Only one copy is passed now.
A test case was added both for the phase itself and for worker thread as
well.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of running a long command line in the runroot (or locally), move
all that work into a separate script that will be installed. This means
chroot will need to install pungi.
Everything should work as it did before. The only exception to this is
that there is logic to find lorax templates instead of harcoding the
location. This is done using a separate script.
Related: #230Fixes: #231
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If umask is set to something too high (>0022), a warning will be
printed. It does not abort the compose though.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This causes a race condition: umask is process-wide setting, so any file
created by another thread while the umask is set to 0 will have wrong
permissions. This also removes the possible problem of not resetting the
umask if exception is raised.
Fixes: #239
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There already were config options tree_arches and tree_variants, but the
filtering was done at the compose level and there was not interaction
between the two options. This led to problems when a variant would have
all its arches filtered out.
This patch moves the filtering to the variant loader. It adds better
logging, so whenever a variant is filtered (for any reason), it will be
explicitly stated in the logs.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This uses a new common function that will ensure consistent logging
without duplicating the functionality all over the place.
A couple tests are updated to verify that correct data is logged.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the koji tasks fails but produces some output (e.g. .discinfo),
copying the artifacts would crash the compose. That should not happen if
the variant/arch is marked as failable.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This phase runs lorax with extra templates in Koji runroot task, links
the boot.iso to proper location in compose directory and adds the
installer iso to image manifest. This phase runs concurrently with live
media etc.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This phase runs the script to make ostree repository in koji runroot
task. It runs right after regular yum repos are created.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is a wrapper over ostree and rpm-ostree. It is intended to be run
in either Koji or Mock chroot.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Move getting of the precomputed value to a single place in Compose
class. The value now has format `date[.type_suffix].respin`.
Resolves: rhbz#1319924
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Print the subvariant together with variant UID and arches. This should
make the global log easier to understand.
The output of koji command should now be split into multiple files in
logs/{arch}/livemedia-{variant}-{subvariant}.{arch}.log instead of
having them all in one file.
The tests were updated to use more sensible value for the subvariant
field.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should generate the same volume id in buildinstall and createiso
phases, with isolinux.cfg being consistent as well.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Check once before iterating through the variants. This greatly
simplifies the tests as each function now has one less code path.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When creating the comps repo, use a separate log file for each variant
instead of overwriting the same global file again and again.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The configuration can specify a list of variants for which the original
comps file will be copied without any modification.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It is used only in createiso and productimg phases.
For productimg, it needs to be present only when the compose is bootable
and productimg phase is explicitly enabled.
For createiso, it is only needed if runroot is not enabled.
Additionally, if we detect pungi running on arch for which syslinux is
not available, a warning is printed, but the compose is allowed to
continue (and possibly crash later).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the configuration specifically requests no jigdos, there is no point
in checking for the binary existence.
This is not 100% reliable. The jigdo option defaults to True, so if the
option is not specified the binary is required even if there are no
images configured.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch adds configuration option to change disc type used in file
name. So far this can only be changed for link to images/boot.iso.
Resolves: #109
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This was already implemented as part of pkgset phase. It is now moved to
the util module and covered with tests.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
With this patch Pungi should be more tolerant of network failures when
running a blocking command (creating live media or live images).
If the connection drops and the output indicates network problems, Pungi
will try to watch the task with `koji watch-task`. This will be retried
until it finishes (successfully or with some other failure). There is an
increasing timeout after each retry. Currently the maximum number of
retries is not limited.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
After config was modified to work with sections, the resolving broke.
This patch fixes it and adds a test to catch this problem in the future.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The log file started with createiso-, which is utterly confusing. With
this patch, the logs are named with liveimage- prefix.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There were asserts that actually did not test anything. They should have
checked that koji is called with correct arguments.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Extracting extensions from the file name is not reliable as there is no
way to determine where extensions start. There can very well be a
version.release separated by dot. To bypass this, just use hardcoded
list of possible formats.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
* Add option to keep filenames generated by Koji.
* Put results of spin-appliance into image dir
* On failure images are no longer deleted.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
With this patch, you can specify a command for
signing of koji builds. For example:
signing_key_password_file = '~/file_with_password_for_key_fedora-24'
signing_key_id = '81b46521'
signing_command = '~/git/releng/scripts/sigulsign_unsigned.py -vv --password=%(signing_key_password)s fedora-24'
'signing_key_password_file' is a path to a file which contains
a password that will be formatted into 'signing_command' string
via '%(signing_key_password)s' string format syntax (if used).
Because pungi config is usualy stored in git and part of compose
logs we don't want password to be included directly in the config.
Note: If '-' is used instead of a filename, then you will be asked
for the password interactivelly right after pungi starts.
'signing_key_id' is ID of the key that will be used for the signing.
This ID will be used when crafting koji paths to signed files
(kojipkgs.fedoraproject.org/packages/NAME/VER/REL/data/signed/KEYID/..).
'signing_command' a command that will be run with a build as a single
argument. This command mustn't require any user interaction.
If you need to pass a password for a signing key to the command,
do this via command line option of the command with use of string
formatting syntax '%(signing_key_password)s' (see details
about 'signing_key_password_file').
Signed-off-by: Tomáš Mlčoch <tmlcoch@redhat.com>
The variants.xml file can list a variant with is_empty="true" and no
groups. If such variant is found, not package gathering will be run for
it, and no repos will be created.
This only makes sense for a variant that will have some other
deliverables like live media or images.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch updates tests so that they pass, and fixes a couple places in
image build that did not work correctly with the nested configuration.
The documentation is reformatted so that it horizontal scrolling is less
likely.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If Koji fails runroot task for some reason, the output will most likely
not have the required format and will crash Pungi.
Pagure: #140
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch fixes how logs are stored if lorax is used as buildinstall
method. The logs for each variant are in a separate file now. If failure
is allowed, the global log will now show why it failed.
A couple tests are added as well.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
A missing input value was causing tests to not check the expected
condition (even though they were still passing).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This phase builds live media in Koji using the Live Media Creator. It
runs in parallel with current live images, create ISO and image build
phases.
The documentation is updated to explain how to configure this.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch adds support for live media creator in Koji. The intended
workflow is to create a command , run it and finally collect built
artifacts.
get_live_media_cmd()
run_blocking_cmd()
get_image_paths()
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The methods mentioning image build are generic and can work for other
task types.
get_image_build_paths -> get_image_paths
run_create_image_cmd -> run_blocking_cmd
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the release is explicitly set to None, generate a value from date
and respin. The documentation is updated to explain how it works.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is now a single option `multilib`, that maps variants and arches
to multilib methods. This replaces old `multilib_methods` option.
Multilib arches are implicitly deduced instead of using the
`multilib_arches` option.
The test compose is updated to only enable multilib on Server and its
addons.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch removes some duplicated variables that get passed on to the
build thread. It also moves creation of the command for generating image
manifest closer to where it is used. Finally it adds tests for the
thread.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch adds option for adding repositories from other variants to
the koji task for building images.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When compose is finished successfully, and there are some failed
deliverables, modify the final status to FINISHED_INCOMPLETE and log
what failed for which variants/arches.
This means the failures are logged twice, first time immediately after
it failed, second time in the summary at the end.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The configuration can now specify image-build as a deliverable that can
fail but not abort the whole compose.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The config can now additionally specify other variants whose repos will
be passed on to koji. The previous way of specifying extra repos is
still available, as is the automatic adding of repo for current variant.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Given a list of arches, koji can build multiple images in one go
(automatically starting children tasks for each one).
This causes a bunch of changes:
* The configuration no longer allows changing config based on
architecture, only variants are allowed. It is however possible to
filter which arches are used for the building in the variant.
* The configuration files for koji image-build are stored in
work/image-build/$variant (not split based on arch).
This patch also changes the option name that is passed to koji
image-build: the repos should be specified under key `repo` (without the
trailing slash).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is a new configuration option that allows listing what can fail
without aborting the whole compose. So far, only buildinstall, createiso
and liveimages phases react to this option.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The computation of image name has been moved to a separate function.
This allowed simplification in how work dir for isos is computed, but
the result is not changed.
The live image phase has some special casing for names of RPM wrapped
ISOs. This is moved to the actual phase. Since this is (and has been)
undocumented, there should not be many users of this special case.
The documentation is updated to describe how image names and volume ids
are determined and how the process can be customized.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the image build configuration specifies kickstart URL as a HEAD of
a git repo, pungi now figures out what the actual hash of that commit is
and uses that hash instead. This might make logs clearer and should
prevent potential problems if someone pushes to that repo during
composing.
Documentation is updated to mention this.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The config file can now specify options for lorax per variant and arch.
This is needed for Fedora to set x86_64 images as bootable on mac.
The old config option `buildinstall_upgrade_image` was deprecated, as
the same can be specified via new `lorax_options`.
Documentation and tests are updated.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Lorax fails when run on an existing non-empty directory. This patch runs
it in separate subdirectiories base on variant. Running with
buildinstall should not be changed at all.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The buildinstall phase now starts lorax for each variant separately.
Running with buildinstall should not be affected in any way.
Only variants of type=variant are considered. The side-effect of this is
that if an architecture is only used by variants of other types, lorax
will not be called at all.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The lorax wrapper class now understands the --installpkgs argument and
can use it to pass multiple package names to the command.
There is a simple test to make sure the commands includes all specified
options. A bug is fixed where the bug URL would not be correctly
included.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The test only checks that commands are created with correct arguments
and that a proper number of threads is started. There is no validation
for what the threads are actually doing.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch modifies how checksums are stored - it uses BSD-style
checksums.
The filename with the checksum can now be customized depending on actual
compose run and metadata. This required adding another option to the
checksumming phase. Documentation is updated and includes example for
creating names used in Fedora.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
On case-insensitive filesystems it is not such a good idea to have
directories that only differ in case. Packages should be always split
into lowercased directories.
The test data is modified to include some packages starting with
uppercase letters. The example in code can be verified by running
`nosetests --with-doctest`.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The messaging is not really part of compose settings. It is an
infrastructure part. As such, it should really be set up as part of
pungi invocation, not compose configuration.
The documentation is updated to reflect this. Some updates to the
documentation are done as well: listing messages about ISOs and minor
formatting updates.
The test_compose.sh script can now accept additional command line
options and pass them on to pungi-koji to simplify testing.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of hardcoding /usr/bin/python in shebangs, use /usr/bin/env.
This allows Pungi to work with dependencies installed in virtualenv.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
With this patch, Pungi can invoke an arbitrary command on various
moments of the compose process. The invoked command can the decide on
what message to send (and using what messaging system).
The actual command is specified in the config file.
There is a script provided that sends the messages via fedmsg.
The documentation is updated to have details about the new config option
as well as the interface for the messaging script.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The phase goes through all images declared in image manifest, computes
their checksums, stores them in appropriate files and updates the
manifest so that it includes the actual checksums.
The documentation contains details about new configuration options.
The test suite now needs Python's mock package.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
CHANGE: Rename product_* config options to release_* for consistency with productmd.
ACTION: Rename product_name, product_short, product_version, product_is_layered to release_* in config files.
Rename //variant/product to //variant/release in variants XML.
CHANGE: createrepo_c is config option now defaults to True.
ACTION: You can safely remove 'createrepo_c = True' from config files.
Set 'createrepo_c = False' if you need legacy createrepo.
CHANGE: createrepo_checksum config option is now mandatory.
ACTION: Add 'createrepo_checksum = "sha256"' (or "sha") to config files.
Previous test data was insufficient for proper testing.
Test compose runs and depsolving tests require precisely
set NVRs, dependencies, sub-packages, etc.
Using rpmfluff for these would be an overkill, it's better
to create RPMs directly from specs.
Lubomír Sedlář
Dennis Gilmore
Tomas Mlcoch
Daniel Mach
Adam Miller
jkeating@harpoon.lab.boston.redhat.com