Commit Graph

1052 Commits

Author SHA1 Message Date
Lubomír Sedlář
3baa386ed4 [image-build] Fix failable tests
A missing input value was causing tests to not check the expected
condition (even though they were still passing).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-08 10:54:24 +01:00
Dennis Gilmore
2b897ec6ea Merge #135 Add live media support 2016-02-02 10:15:56 +00:00
Dennis Gilmore
a2ea84934b Merge #133 media_split: add logger support. Helps with debugging space issues on dvd media 2016-02-02 10:11:46 +00:00
Lubomír Sedlář
439622d576 [live-media] Add live media phase
This phase builds live media in Koji using the Live Media Creator. It
runs in parallel with current live images, create ISO and image build
phases.

The documentation is updated to explain how to configure this.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-01 13:28:49 +01:00
Lubomír Sedlář
a5a0f3d69f [koji-wrapper] Add support for spin-livemedia
This patch adds support for live media creator in Koji. The intended
workflow is to create a command , run it and finally collect built
artifacts.

    get_live_media_cmd()
    run_blocking_cmd()
    get_image_paths()

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-29 09:02:02 +01:00
Lubomír Sedlář
ae30c07553 [koji-wrapper] Use more descriptive method names
The methods mentioning image build are generic and can work for other
task types.

get_image_build_paths -> get_image_paths
run_create_image_cmd -> run_blocking_cmd

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-29 09:02:02 +01:00
Lubomír Sedlář
d85f00818d [image-build] Remove dead code
There are no mounts when running image-build tasks in Koji. This looks
like a copy-paste error.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-29 09:02:02 +01:00
Lubos Kocman
f0a8899ea7 media_split: add logger support. Helps with debugging space issues on dvd media
Signed-off-by: Lubos Kocman <lkocman@redhat.com>
2016-01-26 13:02:14 +00:00
Lubomír Sedlář
a698c7cc0a [image-build] Allow running image build scratch tasks
Issue #130

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-26 12:47:59 +01:00
Lubomír Sedlář
dc4d502f2a [image-build] Allow dynamic release for images
When the release is explicitly set to None, generate a value from date
and respin. The documentation is updated to explain how it works.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-21 14:32:29 +01:00
Dennis Gilmore
39ce2556c3 4.0.4 release
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2016-01-20 09:41:18 -06:00
Dennis Gilmore
c77e78c4f1 Merge #123 Live images: add repo from another variant 2016-01-20 15:37:36 +00:00
Dennis Gilmore
09aebd3db9 Merge #125 [image-build] Stop creating wrong arch dirs 2016-01-20 15:37:27 +00:00
Lubomír Sedlář
8b4ad1ea02 Toggle multilib per variant
There is now a single option `multilib`, that maps variants and arches
to multilib methods. This replaces old `multilib_methods` option.

Multilib arches are implicitly deduced instead of using the
`multilib_arches` option.

The test compose is updated to only enable multilib on Server and its
addons.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-20 12:53:08 +01:00
Lubomír Sedlář
6d7fff5d1b [live-images] Code cleanup
This patch removes some duplicated variables that get passed on to the
build thread. It also moves creation of the command for generating image
manifest closer to where it is used. Finally it adds tests for the
thread.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-19 08:26:59 +01:00
Lubomír Sedlář
b16699ddfc [live-images] Add documentation
Add basic documentation for live-images phases. So far, there are no
examples, just a listing of accepted options.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-19 08:26:59 +01:00
Lubomír Sedlář
9059852ab5 [live-images] Add repos from other variants
This patch adds option for adding repositories from other variants to
the koji task for building images.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-19 08:26:59 +01:00
Lubomír Sedlář
e016015cf3 [image-build] Stop creating wrong arch dirs
There is no such arch as $arch or %(arch)s.

Pagure: #124
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-19 08:24:02 +01:00
Lubomír Sedlář
6e0f6ee73e Enable identifying variants in exception traces
When pungi crashes, the logs will often contain useless description of
a variant. With this patch, the traceback will have details on the
variant.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-15 08:41:08 +01:00
Lubomír Sedlář
5f0675dd66 Store which deliverables failed
When compose is finished successfully, and there are some failed
deliverables, modify the final status to FINISHED_INCOMPLETE and log
what failed for which variants/arches.

This means the failures are logged twice, first time immediately after
it failed, second time in the summary at the end.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-15 08:41:08 +01:00
Lubos Kocman
7ec409a236 scm.py: use git clone instead git archive for http(s)://
Signed-off-by: Lubos Kocman <lkocman@redhat.com>
2016-01-14 06:42:57 -05:00
Lubomír Sedlář
f3f0ce0d35 Fix filtering of system release packages
The code was inconsistent with the documentation: the logic was
inverted.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-13 08:52:06 +01:00
Dennis Gilmore
5f7f23ed96 Merge #114 Use install tree/repo from another variant for image build 2016-01-12 17:56:38 +00:00
Lubomír Sedlář
b78878ac69 Make system release package filtering optional
Fedora variants needs both general fedora-release and
fedora-release-$variant packages installed. This patch disables the code
that would throw fedora-release away (assuming correct config is set).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-12 17:04:37 +01:00
Lubomír Sedlář
5688f1cbae [image-build] Optionally do not break whole compose
The configuration can now specify image-build as a deliverable that can
fail but not abort the whole compose.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-12 08:39:30 +01:00
Lubomír Sedlář
af11bebf1b [image-build] Refactoring
* Move getting arches to separate method.
 * Modify git url only if there is going to be a build. This is not very
   likely to happen, but could save a bit of time if it really does
   happen.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 16:42:05 +01:00
Lubomír Sedlář
d2ea1dd288 [image-build] Use repo from another variant
The config can now additionally specify other variants whose repos will
be passed on to koji. The previous way of specifying extra repos is
still available, as is the automatic adding of repo for current variant.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 16:42:05 +01:00
Lubomír Sedlář
7f815d80f5 [image-build] Take install tree from another variant
Add a configuration option to use install tree from another variant.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 16:42:05 +01:00
Lubomír Sedlář
195b13d434 Add missing formats to volumeid and image name
Also, when the format uses something unrecognized, die with more
descriptive error message.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 16:28:10 +01:00
Lubomír Sedlář
9c1418eb0a [image-build] Use single koji task per variant
Given a list of arches, koji can build multiple images in one go
(automatically starting children tasks for each one).

This causes a bunch of changes:
 * The configuration no longer allows changing config based on
   architecture, only variants are allowed. It is however possible to
   filter which arches are used for the building in the variant.
 * The configuration files for koji image-build are stored in
   work/image-build/$variant (not split based on arch).

This patch also changes the option name that is passed to koji
image-build: the repos should be specified under key `repo` (without the
trailing slash).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-08 14:33:54 +01:00
Lubomír Sedlář
88bb6ac3d7 Fix image-build modifying config
The code modifies the config in place, so it only works in the first
iteration. The fix is to create a copy which can be safely modified and
next iteration will still see the original data.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-06 16:53:00 +01:00
Lubomír Sedlář
a1384b60f4 Fix missing checksums in .treeinfo
The images listed in the file are obtained from .treeinfo files created
during buildinstall phase. However, the files are not stored directly in
buildinstall dir `work/$arch/buildinstall/` but in variant subdirectory
`work/$arch/buildinstall/$variant`.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-06 13:36:44 +01:00
Lubomír Sedlář
770a81b76c Don't crash on generating volid without variant
First, this fixes the crash when volume id requires variant uid, but it
is not specified. This happened in buildinstall phase.

When lorax is used, it is run once for each variant, so we can actually
specify the variant and get the correct variant.

When using buildinstall, no volume id will be generated and pungi will
possibly crash later on.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-04 19:27:11 +01:00
Dennis Gilmore
1af17a5262 Merge #99 Add option to specify non-failing stuff 2015-12-17 18:18:12 +00:00
Lubomír Sedlář
0222d4dc6d Add repo from current compose
This gets added together with the any repos from configuration.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-17 12:15:17 -06:00
Lubomír Sedlář
7a965bcb83 Fix getting compose topdir in CreateImage build thread
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-17 12:15:13 -06:00
Lubomír Sedlář
a6b673dbdd Add option to specify non-failing stuff
There is a new configuration option that allows listing what can fail
without aborting the whole compose. So far, only buildinstall, createiso
and liveimages phases react to this option.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-17 11:15:47 +01:00
Lubomír Sedlář
0e237db5f6 Allow customizing image name and volume id
The computation of image name has been moved to a separate function.
This allowed simplification in how work dir for isos is computed, but
the result is not changed.

The live image phase has some special casing for names of RPM wrapped
ISOs. This is moved to the actual phase. Since this is (and has been)
undocumented, there should not be many users of this special case.

The documentation is updated to describe how image names and volume ids
are determined and how the process can be customized.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-14 14:39:21 +01:00
Lubomír Sedlář
719ec458f4 Fix notifier tests
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-14 14:35:00 +01:00
Ralph Bean
f552eae2bf Publish a url instead of a file path.
Signed-off-by: Ralph Bean <rbean@redhat.com>
2015-12-09 17:07:43 -05:00
Ralph Bean
df0304095d Add 'topdir' to all fedmsg/notifier messages.
This will be useful for fedmsg recipients so they can construct URLs
back to the compose.  They may need to know the compose topdir to
distinguish between https://kojipkgs.fedoraproject.org/compose/rawhide/
and https://kojipkgs.fedoraproject.org/compose/f24/

Signed-off-by: Ralph Bean <rbean@redhat.com>
2015-12-09 14:58:20 -05:00
Dennis Gilmore
10a90f97b4 Merge #75 Start of development guide 2015-12-08 17:41:00 +00:00
Dennis Gilmore
81e90eb780 Merge #88 Resolve HEAD in ksurl to actual hash 2015-12-08 17:37:09 +00:00
Dennis Gilmore
f5862deea7 Merge #87 Add support for customizing lorax options 2015-12-08 17:36:18 +00:00
Ralph Bean
95761c69e1 Update fedmsg notification hook to use appropriate config.
Signed-off-by: Ralph Bean <rbean@redhat.com>
2015-12-08 12:30:16 -05:00
Dennis Gilmore
4b374278e4 we need to ensure that we send all the tasks to koji on the correct arch
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2015-12-08 09:04:32 -06:00
Lubomír Sedlář
fe95221f10 Resolve HEAD in ksurl to actual hash
When the image build configuration specifies kickstart URL as a HEAD of
a git repo, pungi now figures out what the actual hash of that commit is
and uses that hash instead. This might make logs clearer and should
prevent potential problems if someone pushes to that repo during
composing.

Documentation is updated to mention this.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-08 14:29:18 +01:00
Lubomír Sedlář
716b2d521a Add support for customizing lorax options
The config file can now specify options for lorax per variant and arch.
This is needed for Fedora to set x86_64 images as bootable on mac.

The old config option `buildinstall_upgrade_image` was deprecated, as
the same can be specified via new `lorax_options`.

Documentation and tests are updated.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-08 12:47:21 +01:00
Lubomír Sedlář
4a96e8e313 Run lorax in separate dirs for each variant
Lorax fails when run on an existing non-empty directory. This patch runs
it in separate subdirectiories base on variant. Running with
buildinstall should not be changed at all.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-07 09:45:33 +01:00
Dennis Gilmore
bdb1bcb35c Merge #84 Allow specifying --installpkgs for lorax 2015-12-04 19:41:28 +00:00