Commit Graph

1190 Commits

Author SHA1 Message Date
Dennis Gilmore
09aebd3db9 Merge #125 [image-build] Stop creating wrong arch dirs 2016-01-20 15:37:27 +00:00
Lubomír Sedlář
8b4ad1ea02 Toggle multilib per variant
There is now a single option `multilib`, that maps variants and arches
to multilib methods. This replaces old `multilib_methods` option.

Multilib arches are implicitly deduced instead of using the
`multilib_arches` option.

The test compose is updated to only enable multilib on Server and its
addons.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-20 12:53:08 +01:00
Lubomír Sedlář
6d7fff5d1b [live-images] Code cleanup
This patch removes some duplicated variables that get passed on to the
build thread. It also moves creation of the command for generating image
manifest closer to where it is used. Finally it adds tests for the
thread.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-19 08:26:59 +01:00
Lubomír Sedlář
b16699ddfc [live-images] Add documentation
Add basic documentation for live-images phases. So far, there are no
examples, just a listing of accepted options.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-19 08:26:59 +01:00
Lubomír Sedlář
9059852ab5 [live-images] Add repos from other variants
This patch adds option for adding repositories from other variants to
the koji task for building images.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-19 08:26:59 +01:00
Lubomír Sedlář
e016015cf3 [image-build] Stop creating wrong arch dirs
There is no such arch as $arch or %(arch)s.

Pagure: #124
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-19 08:24:02 +01:00
Lubomír Sedlář
6e0f6ee73e Enable identifying variants in exception traces
When pungi crashes, the logs will often contain useless description of
a variant. With this patch, the traceback will have details on the
variant.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-15 08:41:08 +01:00
Lubomír Sedlář
5f0675dd66 Store which deliverables failed
When compose is finished successfully, and there are some failed
deliverables, modify the final status to FINISHED_INCOMPLETE and log
what failed for which variants/arches.

This means the failures are logged twice, first time immediately after
it failed, second time in the summary at the end.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-15 08:41:08 +01:00
Lubos Kocman
7ec409a236 scm.py: use git clone instead git archive for http(s)://
Signed-off-by: Lubos Kocman <lkocman@redhat.com>
2016-01-14 06:42:57 -05:00
Lubomír Sedlář
f3f0ce0d35 Fix filtering of system release packages
The code was inconsistent with the documentation: the logic was
inverted.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-13 08:52:06 +01:00
Dennis Gilmore
5f7f23ed96 Merge #114 Use install tree/repo from another variant for image build 2016-01-12 17:56:38 +00:00
Lubomír Sedlář
b78878ac69 Make system release package filtering optional
Fedora variants needs both general fedora-release and
fedora-release-$variant packages installed. This patch disables the code
that would throw fedora-release away (assuming correct config is set).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-12 17:04:37 +01:00
Lubomír Sedlář
5688f1cbae [image-build] Optionally do not break whole compose
The configuration can now specify image-build as a deliverable that can
fail but not abort the whole compose.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-12 08:39:30 +01:00
Lubomír Sedlář
af11bebf1b [image-build] Refactoring
* Move getting arches to separate method.
 * Modify git url only if there is going to be a build. This is not very
   likely to happen, but could save a bit of time if it really does
   happen.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 16:42:05 +01:00
Lubomír Sedlář
d2ea1dd288 [image-build] Use repo from another variant
The config can now additionally specify other variants whose repos will
be passed on to koji. The previous way of specifying extra repos is
still available, as is the automatic adding of repo for current variant.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 16:42:05 +01:00
Lubomír Sedlář
7f815d80f5 [image-build] Take install tree from another variant
Add a configuration option to use install tree from another variant.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 16:42:05 +01:00
Lubomír Sedlář
195b13d434 Add missing formats to volumeid and image name
Also, when the format uses something unrecognized, die with more
descriptive error message.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 16:28:10 +01:00
Lubomír Sedlář
9c1418eb0a [image-build] Use single koji task per variant
Given a list of arches, koji can build multiple images in one go
(automatically starting children tasks for each one).

This causes a bunch of changes:
 * The configuration no longer allows changing config based on
   architecture, only variants are allowed. It is however possible to
   filter which arches are used for the building in the variant.
 * The configuration files for koji image-build are stored in
   work/image-build/$variant (not split based on arch).

This patch also changes the option name that is passed to koji
image-build: the repos should be specified under key `repo` (without the
trailing slash).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-08 14:33:54 +01:00
Lubomír Sedlář
88bb6ac3d7 Fix image-build modifying config
The code modifies the config in place, so it only works in the first
iteration. The fix is to create a copy which can be safely modified and
next iteration will still see the original data.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-06 16:53:00 +01:00
Lubomír Sedlář
a1384b60f4 Fix missing checksums in .treeinfo
The images listed in the file are obtained from .treeinfo files created
during buildinstall phase. However, the files are not stored directly in
buildinstall dir `work/$arch/buildinstall/` but in variant subdirectory
`work/$arch/buildinstall/$variant`.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-06 13:36:44 +01:00
Lubomír Sedlář
770a81b76c Don't crash on generating volid without variant
First, this fixes the crash when volume id requires variant uid, but it
is not specified. This happened in buildinstall phase.

When lorax is used, it is run once for each variant, so we can actually
specify the variant and get the correct variant.

When using buildinstall, no volume id will be generated and pungi will
possibly crash later on.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-04 19:27:11 +01:00
Dennis Gilmore
1af17a5262 Merge #99 Add option to specify non-failing stuff 2015-12-17 18:18:12 +00:00
Lubomír Sedlář
0222d4dc6d Add repo from current compose
This gets added together with the any repos from configuration.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-17 12:15:17 -06:00
Lubomír Sedlář
7a965bcb83 Fix getting compose topdir in CreateImage build thread
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-17 12:15:13 -06:00
Lubomír Sedlář
a6b673dbdd Add option to specify non-failing stuff
There is a new configuration option that allows listing what can fail
without aborting the whole compose. So far, only buildinstall, createiso
and liveimages phases react to this option.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-17 11:15:47 +01:00
Lubomír Sedlář
0e237db5f6 Allow customizing image name and volume id
The computation of image name has been moved to a separate function.
This allowed simplification in how work dir for isos is computed, but
the result is not changed.

The live image phase has some special casing for names of RPM wrapped
ISOs. This is moved to the actual phase. Since this is (and has been)
undocumented, there should not be many users of this special case.

The documentation is updated to describe how image names and volume ids
are determined and how the process can be customized.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-14 14:39:21 +01:00
Lubomír Sedlář
719ec458f4 Fix notifier tests
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-14 14:35:00 +01:00
Ralph Bean
f552eae2bf Publish a url instead of a file path.
Signed-off-by: Ralph Bean <rbean@redhat.com>
2015-12-09 17:07:43 -05:00
Ralph Bean
df0304095d Add 'topdir' to all fedmsg/notifier messages.
This will be useful for fedmsg recipients so they can construct URLs
back to the compose.  They may need to know the compose topdir to
distinguish between https://kojipkgs.fedoraproject.org/compose/rawhide/
and https://kojipkgs.fedoraproject.org/compose/f24/

Signed-off-by: Ralph Bean <rbean@redhat.com>
2015-12-09 14:58:20 -05:00
Dennis Gilmore
10a90f97b4 Merge #75 Start of development guide 2015-12-08 17:41:00 +00:00
Dennis Gilmore
81e90eb780 Merge #88 Resolve HEAD in ksurl to actual hash 2015-12-08 17:37:09 +00:00
Dennis Gilmore
f5862deea7 Merge #87 Add support for customizing lorax options 2015-12-08 17:36:18 +00:00
Ralph Bean
95761c69e1 Update fedmsg notification hook to use appropriate config.
Signed-off-by: Ralph Bean <rbean@redhat.com>
2015-12-08 12:30:16 -05:00
Dennis Gilmore
4b374278e4 we need to ensure that we send all the tasks to koji on the correct arch
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2015-12-08 09:04:32 -06:00
Lubomír Sedlář
fe95221f10 Resolve HEAD in ksurl to actual hash
When the image build configuration specifies kickstart URL as a HEAD of
a git repo, pungi now figures out what the actual hash of that commit is
and uses that hash instead. This might make logs clearer and should
prevent potential problems if someone pushes to that repo during
composing.

Documentation is updated to mention this.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-08 14:29:18 +01:00
Lubomír Sedlář
716b2d521a Add support for customizing lorax options
The config file can now specify options for lorax per variant and arch.
This is needed for Fedora to set x86_64 images as bootable on mac.

The old config option `buildinstall_upgrade_image` was deprecated, as
the same can be specified via new `lorax_options`.

Documentation and tests are updated.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-08 12:47:21 +01:00
Lubomír Sedlář
4a96e8e313 Run lorax in separate dirs for each variant
Lorax fails when run on an existing non-empty directory. This patch runs
it in separate subdirectiories base on variant. Running with
buildinstall should not be changed at all.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-07 09:45:33 +01:00
Dennis Gilmore
bdb1bcb35c Merge #84 Allow specifying --installpkgs for lorax 2015-12-04 19:41:28 +00:00
Dennis Gilmore
4a26b99a24 Merge #83 Fix recently discovered bugs 2015-12-04 19:39:49 +00:00
Dennis Gilmore
31fe85038b Merge #82 indentation fixs correcting dvd creation 2015-12-04 19:39:05 +00:00
Dennis Gilmore
77aca8a00a Merge #69 Move messaging into cli options and simplify it 2015-12-04 19:38:07 +00:00
Lubomír Sedlář
cdec198b6d Start lorax for each variant separately
The buildinstall phase now starts lorax for each variant separately.
Running with buildinstall should not be affected in any way.

Only variants of type=variant are considered. The side-effect of this is
that if an architecture is only used by variants of other types, lorax
will not be called at all.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-03 15:19:12 +01:00
Lubomír Sedlář
94f519d01c Update lorax wrapper to use --installpkgs
The lorax wrapper class now understands the --installpkgs argument and
can use it to pass multiple package names to the command.

There is a simple test to make sure the commands includes all specified
options. A bug is fixed where the bug URL would not be correctly
included.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-03 15:19:12 +01:00
Lubomír Sedlář
791448cc32 Allow specifying which packages to install in variants xml
This patch updates the variants DTD and parsing to allow specifying
multiple build install packages for each variant.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-03 15:19:12 +01:00
Lubomír Sedlář
cfda99e6fe Add basic tests for buildinstall phase
The test only checks that commands are created with correct arguments
and that a proper number of threads is started. There is no validation
for what the threads are actually doing.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-03 15:19:12 +01:00
Lubomír Sedlář
b6d9b5632e Fix generating checksum files
This patch modifies how checksums are stored - it uses BSD-style
checksums.

The filename with the checksum can now be customized depending on actual
compose run and metadata. This required adding another option to the
checksumming phase. Documentation is updated and includes example for
creating names used in Fedora.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-03 10:54:09 +01:00
Lubomír Sedlář
539736a11e Use lowercase hashed directories
On case-insensitive filesystems it is not such a good idea to have
directories that only differ in case. Packages should be always split
into lowercased directories.

The test data is modified to include some packages starting with
uppercase letters. The example in code can be verified by running
`nosetests --with-doctest`.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-03 09:08:15 +01:00
Dennis Gilmore
290f8f6540 indentation fixs correcting dvd creation
getting the commands together to run the iso creation process was
incorrectly moved when jigdo creation was made optional, patch fixes
https://pagure.io/pungi/issue/77

Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2015-12-02 13:00:06 -06:00
Dennis Gilmore
e9699b7327 remove glibc32 from the runroot tasks
a workaround has been put in place on os's that need it. dnf errors
when a package is listed that is not available. breaking rawhide
composes.

Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2015-12-02 08:51:18 -06:00
Dennis Gilmore
a2ace9e05c fix up the pungi-fedmesg-notification script name
fix up spec so that the package builds
2015-12-02 08:50:32 -06:00