Commit Graph

945 Commits

Author SHA1 Message Date
Lubomír Sedlář ae30c07553 [koji-wrapper] Use more descriptive method names
The methods mentioning image build are generic and can work for other
task types.

get_image_build_paths -> get_image_paths
run_create_image_cmd -> run_blocking_cmd

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-29 09:02:02 +01:00
Lubomír Sedlář d85f00818d [image-build] Remove dead code
There are no mounts when running image-build tasks in Koji. This looks
like a copy-paste error.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-29 09:02:02 +01:00
Lubomír Sedlář dc4d502f2a [image-build] Allow dynamic release for images
When the release is explicitly set to None, generate a value from date
and respin. The documentation is updated to explain how it works.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-21 14:32:29 +01:00
Dennis Gilmore 39ce2556c3 4.0.4 release
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2016-01-20 09:41:18 -06:00
Dennis Gilmore c77e78c4f1 Merge #123 `Live images: add repo from another variant` 2016-01-20 15:37:36 +00:00
Dennis Gilmore 09aebd3db9 Merge #125 `[image-build] Stop creating wrong arch dirs` 2016-01-20 15:37:27 +00:00
Lubomír Sedlář 8b4ad1ea02 Toggle multilib per variant
There is now a single option `multilib`, that maps variants and arches
to multilib methods. This replaces old `multilib_methods` option.

Multilib arches are implicitly deduced instead of using the
`multilib_arches` option.

The test compose is updated to only enable multilib on Server and its
addons.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-20 12:53:08 +01:00
Lubomír Sedlář 6d7fff5d1b [live-images] Code cleanup
This patch removes some duplicated variables that get passed on to the
build thread. It also moves creation of the command for generating image
manifest closer to where it is used. Finally it adds tests for the
thread.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-19 08:26:59 +01:00
Lubomír Sedlář b16699ddfc [live-images] Add documentation
Add basic documentation for live-images phases. So far, there are no
examples, just a listing of accepted options.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-19 08:26:59 +01:00
Lubomír Sedlář 9059852ab5 [live-images] Add repos from other variants
This patch adds option for adding repositories from other variants to
the koji task for building images.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-19 08:26:59 +01:00
Lubomír Sedlář e016015cf3 [image-build] Stop creating wrong arch dirs
There is no such arch as $arch or %(arch)s.

Pagure: #124
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-19 08:24:02 +01:00
Lubomír Sedlář 6e0f6ee73e Enable identifying variants in exception traces
When pungi crashes, the logs will often contain useless description of
a variant. With this patch, the traceback will have details on the
variant.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-15 08:41:08 +01:00
Lubomír Sedlář 5f0675dd66 Store which deliverables failed
When compose is finished successfully, and there are some failed
deliverables, modify the final status to FINISHED_INCOMPLETE and log
what failed for which variants/arches.

This means the failures are logged twice, first time immediately after
it failed, second time in the summary at the end.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-15 08:41:08 +01:00
Lubos Kocman 7ec409a236 scm.py: use git clone instead git archive for http(s)://
Signed-off-by: Lubos Kocman <lkocman@redhat.com>
2016-01-14 06:42:57 -05:00
Lubomír Sedlář f3f0ce0d35 Fix filtering of system release packages
The code was inconsistent with the documentation: the logic was
inverted.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-13 08:52:06 +01:00
Dennis Gilmore 5f7f23ed96 Merge #114 `Use install tree/repo from another variant for image build` 2016-01-12 17:56:38 +00:00
Lubomír Sedlář b78878ac69 Make system release package filtering optional
Fedora variants needs both general fedora-release and
fedora-release-$variant packages installed. This patch disables the code
that would throw fedora-release away (assuming correct config is set).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-12 17:04:37 +01:00
Lubomír Sedlář 5688f1cbae [image-build] Optionally do not break whole compose
The configuration can now specify image-build as a deliverable that can
fail but not abort the whole compose.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-12 08:39:30 +01:00
Lubomír Sedlář af11bebf1b [image-build] Refactoring
* Move getting arches to separate method.
 * Modify git url only if there is going to be a build. This is not very
   likely to happen, but could save a bit of time if it really does
   happen.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 16:42:05 +01:00
Lubomír Sedlář d2ea1dd288 [image-build] Use repo from another variant
The config can now additionally specify other variants whose repos will
be passed on to koji. The previous way of specifying extra repos is
still available, as is the automatic adding of repo for current variant.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 16:42:05 +01:00
Lubomír Sedlář 7f815d80f5 [image-build] Take install tree from another variant
Add a configuration option to use install tree from another variant.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 16:42:05 +01:00
Lubomír Sedlář 195b13d434 Add missing formats to volumeid and image name
Also, when the format uses something unrecognized, die with more
descriptive error message.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 16:28:10 +01:00
Lubomír Sedlář 9c1418eb0a [image-build] Use single koji task per variant
Given a list of arches, koji can build multiple images in one go
(automatically starting children tasks for each one).

This causes a bunch of changes:
 * The configuration no longer allows changing config based on
   architecture, only variants are allowed. It is however possible to
   filter which arches are used for the building in the variant.
 * The configuration files for koji image-build are stored in
   work/image-build/$variant (not split based on arch).

This patch also changes the option name that is passed to koji
image-build: the repos should be specified under key `repo` (without the
trailing slash).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-08 14:33:54 +01:00
Lubomír Sedlář 88bb6ac3d7 Fix image-build modifying config
The code modifies the config in place, so it only works in the first
iteration. The fix is to create a copy which can be safely modified and
next iteration will still see the original data.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-06 16:53:00 +01:00
Lubomír Sedlář a1384b60f4 Fix missing checksums in .treeinfo
The images listed in the file are obtained from .treeinfo files created
during buildinstall phase. However, the files are not stored directly in
buildinstall dir `work/$arch/buildinstall/` but in variant subdirectory
`work/$arch/buildinstall/$variant`.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-06 13:36:44 +01:00
Lubomír Sedlář 770a81b76c Don't crash on generating volid without variant
First, this fixes the crash when volume id requires variant uid, but it
is not specified. This happened in buildinstall phase.

When lorax is used, it is run once for each variant, so we can actually
specify the variant and get the correct variant.

When using buildinstall, no volume id will be generated and pungi will
possibly crash later on.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-04 19:27:11 +01:00
Dennis Gilmore 1af17a5262 Merge #99 `Add option to specify non-failing stuff` 2015-12-17 18:18:12 +00:00
Lubomír Sedlář 0222d4dc6d Add repo from current compose
This gets added together with the any repos from configuration.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-17 12:15:17 -06:00
Lubomír Sedlář 7a965bcb83 Fix getting compose topdir in CreateImage build thread
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-17 12:15:13 -06:00
Lubomír Sedlář a6b673dbdd Add option to specify non-failing stuff
There is a new configuration option that allows listing what can fail
without aborting the whole compose. So far, only buildinstall, createiso
and liveimages phases react to this option.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-17 11:15:47 +01:00
Lubomír Sedlář 0e237db5f6 Allow customizing image name and volume id
The computation of image name has been moved to a separate function.
This allowed simplification in how work dir for isos is computed, but
the result is not changed.

The live image phase has some special casing for names of RPM wrapped
ISOs. This is moved to the actual phase. Since this is (and has been)
undocumented, there should not be many users of this special case.

The documentation is updated to describe how image names and volume ids
are determined and how the process can be customized.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-14 14:39:21 +01:00
Lubomír Sedlář 719ec458f4 Fix notifier tests
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-14 14:35:00 +01:00
Ralph Bean f552eae2bf Publish a url instead of a file path.
Signed-off-by: Ralph Bean <rbean@redhat.com>
2015-12-09 17:07:43 -05:00
Ralph Bean df0304095d Add 'topdir' to all fedmsg/notifier messages.
This will be useful for fedmsg recipients so they can construct URLs
back to the compose.  They may need to know the compose topdir to
distinguish between https://kojipkgs.fedoraproject.org/compose/rawhide/
and https://kojipkgs.fedoraproject.org/compose/f24/

Signed-off-by: Ralph Bean <rbean@redhat.com>
2015-12-09 14:58:20 -05:00
Dennis Gilmore 10a90f97b4 Merge #75 `Start of development guide` 2015-12-08 17:41:00 +00:00
Dennis Gilmore 81e90eb780 Merge #88 `Resolve HEAD in ksurl to actual hash` 2015-12-08 17:37:09 +00:00
Dennis Gilmore f5862deea7 Merge #87 `Add support for customizing lorax options` 2015-12-08 17:36:18 +00:00
Ralph Bean 95761c69e1 Update fedmsg notification hook to use appropriate config.
Signed-off-by: Ralph Bean <rbean@redhat.com>
2015-12-08 12:30:16 -05:00
Dennis Gilmore 4b374278e4 we need to ensure that we send all the tasks to koji on the correct arch
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2015-12-08 09:04:32 -06:00
Lubomír Sedlář fe95221f10 Resolve HEAD in ksurl to actual hash
When the image build configuration specifies kickstart URL as a HEAD of
a git repo, pungi now figures out what the actual hash of that commit is
and uses that hash instead. This might make logs clearer and should
prevent potential problems if someone pushes to that repo during
composing.

Documentation is updated to mention this.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-08 14:29:18 +01:00
Lubomír Sedlář 716b2d521a Add support for customizing lorax options
The config file can now specify options for lorax per variant and arch.
This is needed for Fedora to set x86_64 images as bootable on mac.

The old config option `buildinstall_upgrade_image` was deprecated, as
the same can be specified via new `lorax_options`.

Documentation and tests are updated.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-08 12:47:21 +01:00
Lubomír Sedlář 4a96e8e313 Run lorax in separate dirs for each variant
Lorax fails when run on an existing non-empty directory. This patch runs
it in separate subdirectiories base on variant. Running with
buildinstall should not be changed at all.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-07 09:45:33 +01:00
Dennis Gilmore bdb1bcb35c Merge #84 `Allow specifying --installpkgs for lorax` 2015-12-04 19:41:28 +00:00
Dennis Gilmore 4a26b99a24 Merge #83 `Fix recently discovered bugs` 2015-12-04 19:39:49 +00:00
Dennis Gilmore 31fe85038b Merge #82 `indentation fixs correcting dvd creation` 2015-12-04 19:39:05 +00:00
Dennis Gilmore 77aca8a00a Merge #69 `Move messaging into cli options and simplify it` 2015-12-04 19:38:07 +00:00
Lubomír Sedlář cdec198b6d Start lorax for each variant separately
The buildinstall phase now starts lorax for each variant separately.
Running with buildinstall should not be affected in any way.

Only variants of type=variant are considered. The side-effect of this is
that if an architecture is only used by variants of other types, lorax
will not be called at all.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-03 15:19:12 +01:00
Lubomír Sedlář 94f519d01c Update lorax wrapper to use --installpkgs
The lorax wrapper class now understands the --installpkgs argument and
can use it to pass multiple package names to the command.

There is a simple test to make sure the commands includes all specified
options. A bug is fixed where the bug URL would not be correctly
included.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-03 15:19:12 +01:00
Lubomír Sedlář 791448cc32 Allow specifying which packages to install in variants xml
This patch updates the variants DTD and parsing to allow specifying
multiple build install packages for each variant.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-03 15:19:12 +01:00
Lubomír Sedlář cfda99e6fe Add basic tests for buildinstall phase
The test only checks that commands are created with correct arguments
and that a proper number of threads is started. There is no validation
for what the threads are actually doing.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-03 15:19:12 +01:00