Print the subvariant together with variant UID and arches. This should
make the global log easier to understand.
The output of koji command should now be split into multiple files in
logs/{arch}/livemedia-{variant}-{subvariant}.{arch}.log instead of
having them all in one file.
The tests were updated to use more sensible value for the subvariant
field.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should generate the same volume id in buildinstall and createiso
phases, with isolinux.cfg being consistent as well.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Dennis added a 'subvariant' property to the Docker image config,
but we weren't using it, we were just hard-coding the variant
as the subvariant...and doing it wrong, as well, using the
variant object rather than its uid property.
This fixes both problems: we'll now use the explicitly set
image config 'subvariant' property if there is one, otherwise
we use the variant.uid as the subvariant.
Note this is a high priority fix, as it broke F24 compose today.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This adds the new 'subvariant' property to all images produced by Pungi.
For most image types this is the same as the variant, for now. For
live_images and livemedia_phase, the image config may specify
'subvariant' and the value will be used if present, otherwise the
variant name will be used.
We also now synthesize the Koji image 'name' for live and appliance
images if the image config doesn't specify it, and use a name that
complies better with the naming policy for ARM disk images (partially
resolving https://pagure.io/pungi-fedora/issue/7 ).
This is completely untested as yet, just bashing stuff out on a Sunday,
will work more on it tomorrow.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Check once before iterating through the variants. This greatly
simplifies the tests as each function now has one less code path.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When creating the comps repo, use a separate log file for each variant
instead of overwriting the same global file again and again.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The configuration can specify a list of variants for which the original
comps file will be copied without any modification.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch adds configuration option to change disc type used in file
name. So far this can only be changed for link to images/boot.iso.
Resolves: #109
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This was already implemented as part of pkgset phase. It is now moved to
the util module and covered with tests.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch tries to simplify the code by:
* breaking up too long lines
* simplifying the loop to start createrepo jobs
* removing duplication
* general clean up
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
As a side effect, the non-existing repositories are no longer passed to
global repoclosure.
Fixes: #196
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
After config was modified to work with sections, the resolving broke.
This patch fixes it and adds a test to catch this problem in the future.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The log file started with createiso-, which is utterly confusing. With
this patch, the logs are named with liveimage- prefix.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Extracting extensions from the file name is not reliable as there is no
way to determine where extensions start. There can very well be a
version.release separated by dot. To bypass this, just use hardcoded
list of possible formats.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
* Add option to keep filenames generated by Koji.
* Put results of spin-appliance into image dir
* On failure images are no longer deleted.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
With this patch, you can specify a command for
signing of koji builds. For example:
signing_key_password_file = '~/file_with_password_for_key_fedora-24'
signing_key_id = '81b46521'
signing_command = '~/git/releng/scripts/sigulsign_unsigned.py -vv --password=%(signing_key_password)s fedora-24'
'signing_key_password_file' is a path to a file which contains
a password that will be formatted into 'signing_command' string
via '%(signing_key_password)s' string format syntax (if used).
Because pungi config is usualy stored in git and part of compose
logs we don't want password to be included directly in the config.
Note: If '-' is used instead of a filename, then you will be asked
for the password interactivelly right after pungi starts.
'signing_key_id' is ID of the key that will be used for the signing.
This ID will be used when crafting koji paths to signed files
(kojipkgs.fedoraproject.org/packages/NAME/VER/REL/data/signed/KEYID/..).
'signing_command' a command that will be run with a build as a single
argument. This command mustn't require any user interaction.
If you need to pass a password for a signing key to the command,
do this via command line option of the command with use of string
formatting syntax '%(signing_key_password)s' (see details
about 'signing_key_password_file').
Signed-off-by: Tomáš Mlčoch <tmlcoch@redhat.com>
koji for livemedia acts different to image builds and does
not translate $arch to x86_64, i386, etc so we need to just pass
in $basearch so that yum/dnf will translate it for us
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
The variants.xml file can list a variant with is_empty="true" and no
groups. If such variant is found, not package gathering will be run for
it, and no repos will be created.
This only makes sense for a variant that will have some other
deliverables like live media or images.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Since multiple images are created in a single Koji task, the path to log
file contains a list of arches. These used to be separated with comma.
For consistency with live-media they are now separated by dashes.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
KojiWrapper returns the task id as an integer already (if at all). There
is no need to parse it again.
Pagure: #148
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch updates tests so that they pass, and fixes a couple places in
image build that did not work correctly with the nested configuration.
The documentation is reformatted so that it horizontal scrolling is less
likely.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch fixes how logs are stored if lorax is used as buildinstall
method. The logs for each variant are in a separate file now. If failure
is allowed, the global log will now show why it failed.
A couple tests are added as well.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This phase builds live media in Koji using the Live Media Creator. It
runs in parallel with current live images, create ISO and image build
phases.
The documentation is updated to explain how to configure this.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The methods mentioning image build are generic and can work for other
task types.
get_image_build_paths -> get_image_paths
run_create_image_cmd -> run_blocking_cmd
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the release is explicitly set to None, generate a value from date
and respin. The documentation is updated to explain how it works.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is now a single option `multilib`, that maps variants and arches
to multilib methods. This replaces old `multilib_methods` option.
Multilib arches are implicitly deduced instead of using the
`multilib_arches` option.
The test compose is updated to only enable multilib on Server and its
addons.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch removes some duplicated variables that get passed on to the
build thread. It also moves creation of the command for generating image
manifest closer to where it is used. Finally it adds tests for the
thread.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch adds option for adding repositories from other variants to
the koji task for building images.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Fedora variants needs both general fedora-release and
fedora-release-$variant packages installed. This patch disables the code
that would throw fedora-release away (assuming correct config is set).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The configuration can now specify image-build as a deliverable that can
fail but not abort the whole compose.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
* Move getting arches to separate method.
* Modify git url only if there is going to be a build. This is not very
likely to happen, but could save a bit of time if it really does
happen.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The config can now additionally specify other variants whose repos will
be passed on to koji. The previous way of specifying extra repos is
still available, as is the automatic adding of repo for current variant.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Given a list of arches, koji can build multiple images in one go
(automatically starting children tasks for each one).
This causes a bunch of changes:
* The configuration no longer allows changing config based on
architecture, only variants are allowed. It is however possible to
filter which arches are used for the building in the variant.
* The configuration files for koji image-build are stored in
work/image-build/$variant (not split based on arch).
This patch also changes the option name that is passed to koji
image-build: the repos should be specified under key `repo` (without the
trailing slash).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The code modifies the config in place, so it only works in the first
iteration. The fix is to create a copy which can be safely modified and
next iteration will still see the original data.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
First, this fixes the crash when volume id requires variant uid, but it
is not specified. This happened in buildinstall phase.
When lorax is used, it is run once for each variant, so we can actually
specify the variant and get the correct variant.
When using buildinstall, no volume id will be generated and pungi will
possibly crash later on.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is a new configuration option that allows listing what can fail
without aborting the whole compose. So far, only buildinstall, createiso
and liveimages phases react to this option.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The computation of image name has been moved to a separate function.
This allowed simplification in how work dir for isos is computed, but
the result is not changed.
The live image phase has some special casing for names of RPM wrapped
ISOs. This is moved to the actual phase. Since this is (and has been)
undocumented, there should not be many users of this special case.
The documentation is updated to describe how image names and volume ids
are determined and how the process can be customized.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the image build configuration specifies kickstart URL as a HEAD of
a git repo, pungi now figures out what the actual hash of that commit is
and uses that hash instead. This might make logs clearer and should
prevent potential problems if someone pushes to that repo during
composing.
Documentation is updated to mention this.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The config file can now specify options for lorax per variant and arch.
This is needed for Fedora to set x86_64 images as bootable on mac.
The old config option `buildinstall_upgrade_image` was deprecated, as
the same can be specified via new `lorax_options`.
Documentation and tests are updated.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Lorax fails when run on an existing non-empty directory. This patch runs
it in separate subdirectiories base on variant. Running with
buildinstall should not be changed at all.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The buildinstall phase now starts lorax for each variant separately.
Running with buildinstall should not be affected in any way.
Only variants of type=variant are considered. The side-effect of this is
that if an architecture is only used by variants of other types, lorax
will not be called at all.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch modifies how checksums are stored - it uses BSD-style
checksums.
The filename with the checksum can now be customized depending on actual
compose run and metadata. This required adding another option to the
checksumming phase. Documentation is updated and includes example for
creating names used in Fedora.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
On case-insensitive filesystems it is not such a good idea to have
directories that only differ in case. Packages should be always split
into lowercased directories.
The test data is modified to include some packages starting with
uppercase letters. The example in code can be verified by running
`nosetests --with-doctest`.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
getting the commands together to run the iso creation process was
incorrectly moved when jigdo creation was made optional, patch fixes
https://pagure.io/pungi/issue/77
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
a workaround has been put in place on os's that need it. dnf errors
when a package is listed that is not available. breaking rawhide
composes.
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
When phase starts, a message is sent with list of images that are going
to be built. As each images is finished (or fails), another message
announces that status.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
With this patch, Pungi can invoke an arbitrary command on various
moments of the compose process. The invoked command can the decide on
what message to send (and using what messaging system).
The actual command is specified in the config file.
There is a script provided that sends the messages via fedmsg.
The documentation is updated to have details about the new config option
as well as the interface for the messaging script.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The phase goes through all images declared in image manifest, computes
their checksums, stores them in appropriate files and updates the
manifest so that it includes the actual checksums.
The documentation contains details about new configuration options.
The test suite now needs Python's mock package.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
These phases were computing checksums for images:
* buildinstall
* createiso
* image_build
* live_images
* product_img
In each phase the checksummed thing would ultimately end-up in image
manifest.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The checksums are still generated in the ImageBuild and CreatISO phases,
but collecting of them into single file is moved from pungi-koji script
into a separate phase.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Add add_template, add_template_var, add_arch_template, and
add_arch_template_var to the call for lorax.get_lorax_cmd for the Atomic
ISO composes.
Remove duplicate if statement for add_template_var in lorax wrapper
CHANGE: Rename product_* config options to release_* for consistency with productmd.
ACTION: Rename product_name, product_short, product_version, product_is_layered to release_* in config files.
Rename //variant/product to //variant/release in variants XML.
Lubomír Sedlář
Adam Williamson
Dennis Gilmore
Dennis Gilmore
Lubomír Sedlář
Tomas Mlcoch
Lubos Kocman
Jon Disnard
Tomas Kopecek
Adam Miller
Daniel Mach