Commit Graph

44 Commits

Author SHA1 Message Date
Lubomír Sedlář
23c454ff67 buildinstall: Make output world readable
This makes it possible to run a compose as non-root user, plus removes
the need for workarounds to publish the results directly.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-17 10:12:52 +02:00
Lubomír Sedlář
e3de4dcccf Report failed failable deliverables as errors
It does not abort the compose, but it's still an error. It should be
marked as such in the log. This will allow easier searching for the
failures.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-17 09:31:50 +02:00
Lubomír Sedlář
ea0964eeef buildinstall: Use metadata if skipped
If the phase is skipped, it could mean that we are doing a debug run and
we don't want to mess up the .treefile by missing arch specific images.
The other alternative is that the phase was really skipped, in which
case there will be no files generated and we already handle that fine.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-23 10:42:33 +02:00
Lubomír Sedlář
15ccd309fa Remove comps from arch repo
The comps could potentially be different in different variants, so
instead we can create the comps repo for every variant separately and
use two repos instead of one (packages in one repository, comps in
another one).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-12 13:52:36 +02:00
Lubomír Sedlář
d9e2101b08 buildinstall: Add extra repos
A new configuration option is added that allows users to point lorax to
extra repositories. This can be handy if some tools to create the
bootable image are not part of the product itself.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
JIRA: COMPOSE-2253
2018-04-05 07:56:20 +02:00
Lubomír Sedlář
95bb147015 buildinstall: Add option to disable it
Fixes: https://pagure.io/pungi/issue/854
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-13 15:54:52 +01:00
Ondrej Nosek
59c162d46f Remove strace from buildinstall runroot
The package is not really needed.

Relates: https://pagure.io/pungi/issue/799

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2017-12-11 17:36:28 +01:00
Jan Kaluza
953fb4c54c buildinstall: Allow using external dire for runroot task
A new `buildinstall_topdir` option allows using buildinstall even when
the compose is created on a different volume that Koji is using.

The files are created in this external directory and then copies into
the usual location.

Merges: https://pagure.io/pungi/pull-request/807
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2017-11-30 10:55:52 +01:00
Lubomír Sedlář
ed22e07ef9 Port to Python 3
This should make all tests pass on both Python 2 and Python 3.

Unittest2 is required on Py 2.6 and Py 3.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-24 15:15:31 +02:00
Lubomír Sedlář
be39dc3caf buildinstall: Expose template arguments for lorax
This would be useful for modularity. The templates can be added now and
variables set via the existing `lorax_option`.

It's not possible to use custom templates not shipped with lorax, as
passing the path to a random directory is a little bit more tricky.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-23 16:08:09 +02:00
Lubomír Sedlář
fcbc3ed4ae buildinstall: No copy if task fails
Even a failed lorax task can leave behind some in-progress images. Pungi
needs to copy the files into compose/ subdirectory only if the task
finished successfully.

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1485021
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-28 13:43:32 +02:00
Lubomír Sedlář
05a666fb3b buildinstall: Move tweaking configs into a function
This will allow us to more easily test the code and also reuse it in
other places.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-21 07:55:19 +01:00
Lubomír Sedlář
4ea1916a87 Enable customizing runroot task weight
For different cases where runroot is used it's now possible to set
custom weight. The usecase for this is to avoid one builder taking too
many tasks. Especially buildinstall is quite resource intensive, so one
builder taking multiple tasks at the same time leads to very slow
compose time.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-23 12:44:27 +01:00
Qixiang Wan
6fbf1e8f59 Create temporary dirs under compose's workdir
Add compose.mkdtemp which creates and returns a unique temporary
directory under <compose_topdir>/work/{global,<arch>}/tmp[-<variant>]/

Change tempfile.mkdtemp in code to compose.mkdtemp, so temporary
dirs are created under the compose's workdir, this makes it more
easier to maintain the temporary dirs/files especially when compose
fails with amount of temporary files left there.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-01-11 15:25:47 +08:00
Qixiang Wan
80fa723b1d Include phase name in log for some phases
Phases createiso, liveimages, image_build, ostree_installer and osbs are
done in parallel, logs from these phases are mixed and and it's not
obvious which log message belongs to which phase. This change adds phase
name in log message for these phases.

The new mixin 'PhaseLoggerMixin' is added to extend a Pungi phase with a
logging logger which copy handlers from compose's logger but with
formatter changed.

Fixes: #58
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-11-23 16:47:22 +08:00
Dennis Gilmore
d4a78a1553 Merge #459 Fix building images and live media for addons 2016-11-15 21:00:50 +00:00
Lubomír Sedlář
13871b64fb compose: Drop unused argument
The `get_variants()` method had a `recursive` argument with default
value of `False. However, this argument had no effect and the method
always returned all variants recursively.

We can just drop the argument. All callers are updated to not supply the
argument. Should any need for getting the top-level variants only arise,
they can be accessed as the `variants` attribute directly on the Compose
object.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 13:39:01 +01:00
Lubomír Sedlář
9a72ea8f6d lorax-wrapper: Put all log files into compose logs
Tell lorax to use a specific directory for log files so that we preserve
them despite koji not having any idea about them.

Fixes: #457
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 09:37:00 +01:00
Patrick Uiterwijk
59dbd9d35f Remove executable permissions on test scripts
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2016-10-25 10:42:16 +00:00
Lubomír Sedlář
e58c78fd2f iso-wrapper: Remove the class
It really is just a group of independent functions, so we can simplify
it by removing the unused wrapper class. Instead of importing the
wrapper, instantiating it and calling its methods we can import the
module and call its functions directly.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-14 09:14:51 +02:00
Lubomír Sedlář
f9a6c8418f Add JSON Schema for configuration
The schema is written in Python to reduce duplication. When
configuration is loaded, the validation checks if it's correct and fills
in default values.

There is a custom extension to the schema to report deprecated options.

The config dependencies are implemented as a separate pass. While it's
technically possible to express the dependencies in the schema itself,
the error messages are not very helpful and it makes the schema much
harder to read.

Phases no longer define `config_options`. New options should be added to
the schema. Since the default values are populated automatically during
validation, there is no need to duplicate them into the code.

The `pungi-config-validate` script is updated to use the schema and
report errors even for deeply nested fields.

The dependencies are updated: pungi now depends on `python-jsonschema`
(which is already available in Fedora).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-01 10:56:15 +02:00
Lubomír Sedlář
b862fc5a50 [buildinstall] Fix cleaning output dir
Before the task is started, the output directory is checked and if it
exists and is not empty, the runroot task will be skipped. This is meant
for debugging when restarting the same compose. Under usual
circumstances, the directory will not be created in the first place.

The runroot task will start by removing the output directory. This way,
if koji restarts the task, lorax will not fail.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-18 09:11:30 +02:00
Lubomír Sedlář
463088d580 Refactor failables
This is a breaking change as big part of current failable_deliverables
options will be ignored.

There is no change for buildinstall and creatiso phase.

Failability for artifacts in other phases is now configured per
artifact. It already works correctly for ostree and ostree_installer
phases (even per-arch). For OSBS phase there is currently only a binary
switch as it does not handle multiple arches yet. When it gains that
support, the option should contain list of non-blocking architectures.

For live images, live media and image build phases each config block can
configure list of failable arches. If the list is not empty, it can
fail. Once we have a way to fail only some arches, the config will not
need to change.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-07-27 13:06:01 +02:00
Lubomír Sedlář
d5512fd6ff Use unittest2 if available
The module backports features to Python 2.6 and 2.7. If it is available,
the tests will use it. If it is not available, it will fall back to
regular unittest. On Python 2.7, the tests pass anyway. On Python 2.6,
there are failures with Python 2.6.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-25 13:41:49 +02:00
Lubomír Sedlář
9ab9aaf8d9 The message attribute on exception is deprecated
One should rather use str(exception) to get the text message.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-25 13:28:56 +02:00
Lubomír Sedlář
98f7ef739e Refactor checking for failable deliverables
This uses a new common function that will ensure consistent logging
without duplicating the functionality all over the place.

A couple tests are updated to verify that correct data is logged.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-24 10:12:20 +01:00
Lubomír Sedlář
bb0267bc7c [buildinstall] Do not crash on failure
When the koji tasks fails but produces some output (e.g. .discinfo),
copying the artifacts would crash the compose. That should not happen if
the variant/arch is marked as failable.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-24 09:17:38 +01:00
Lubomír Sedlář
aa38eb1fd7 [buildinstall] Use customized dvd disc type
This should generate the same volume id in buildinstall and createiso
phases, with isolinux.cfg being consistent as well.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-14 08:20:43 +01:00
Lubomír Sedlář
0c9ad96a31 [buildinstall] Add customizing disc type
This patch adds configuration option to change disc type used in file
name. So far this can only be changed for link to images/boot.iso.

Resolves: #109
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-08 13:04:28 +01:00
Lubomír Sedlář
ad104f994a [buildinstall] Rename method to not mention symlinks
It is either hardlinking or copying, so the name is quite misleading.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-08 11:26:14 +01:00
Dennis Gilmore
8670452b6b Merge #189 [buildinstall] Use -dvd- in volume ids instead of -boot- 2016-02-26 13:53:43 +00:00
Lubomír Sedlář
452da86649 [buildinstall] Use -dvd- in volume ids instead of -boot-
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-26 11:42:41 +01:00
Lubomír Sedlář
636ac79186 [buildinstall] Hardlink boot isos
Instead of creating a symlink, try to hardlink the image, and copy it if
hardlinking fails.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-26 10:58:03 +01:00
Lubomír Sedlář
48979a4979 [tests] Fix wrong checks in buildinstall tests
There were asserts that actually did not test anything. They should have
checked that koji is called with correct arguments.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-23 15:30:49 +01:00
Lubomír Sedlář
c61abac137 [tests] Use temporary files for buildinstall
Instead of mocking open just write to the temporary file and then check
its content.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-23 15:30:49 +01:00
Lubomír Sedlář
e9292fc942 [tests] Dummy compose is no longer private
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-22 19:11:04 +01:00
Lubomír Sedlář
385b52041c [tests] Move buildinstall tests to new infrastructure
This test case now uses the same mocks as other tests, it only restricts
the variants used.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-22 19:11:04 +01:00
Lubomír Sedlář
1b6688d22c [buildinstall] Don't run lorax for empty variants
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-18 16:12:44 +01:00
Lubomír Sedlář
f57665f963 [buildinstall] Improve logging
This patch fixes how logs are stored if lorax is used as buildinstall
method. The logs for each variant are in a separate file now. If failure
is allowed, the global log will now show why it failed.

A couple tests are added as well.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-08 13:17:08 +01:00
Lubomír Sedlář
0e237db5f6 Allow customizing image name and volume id
The computation of image name has been moved to a separate function.
This allowed simplification in how work dir for isos is computed, but
the result is not changed.

The live image phase has some special casing for names of RPM wrapped
ISOs. This is moved to the actual phase. Since this is (and has been)
undocumented, there should not be many users of this special case.

The documentation is updated to describe how image names and volume ids
are determined and how the process can be customized.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-14 14:39:21 +01:00
Lubomír Sedlář
716b2d521a Add support for customizing lorax options
The config file can now specify options for lorax per variant and arch.
This is needed for Fedora to set x86_64 images as bootable on mac.

The old config option `buildinstall_upgrade_image` was deprecated, as
the same can be specified via new `lorax_options`.

Documentation and tests are updated.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-08 12:47:21 +01:00
Lubomír Sedlář
4a96e8e313 Run lorax in separate dirs for each variant
Lorax fails when run on an existing non-empty directory. This patch runs
it in separate subdirectiories base on variant. Running with
buildinstall should not be changed at all.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-07 09:45:33 +01:00
Lubomír Sedlář
cdec198b6d Start lorax for each variant separately
The buildinstall phase now starts lorax for each variant separately.
Running with buildinstall should not be affected in any way.

Only variants of type=variant are considered. The side-effect of this is
that if an architecture is only used by variants of other types, lorax
will not be called at all.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-03 15:19:12 +01:00
Lubomír Sedlář
cfda99e6fe Add basic tests for buildinstall phase
The test only checks that commands are created with correct arguments
and that a proper number of threads is started. There is no validation
for what the threads are actually doing.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-03 15:19:12 +01:00