This makes it possible to run a compose as non-root user, plus removes
the need for workarounds to publish the results directly.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It does not abort the compose, but it's still an error. It should be
marked as such in the log. This will allow easier searching for the
failures.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the phase is skipped, it could mean that we are doing a debug run and
we don't want to mess up the .treefile by missing arch specific images.
The other alternative is that the phase was really skipped, in which
case there will be no files generated and we already handle that fine.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The comps could potentially be different in different variants, so
instead we can create the comps repo for every variant separately and
use two repos instead of one (packages in one repository, comps in
another one).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
A new configuration option is added that allows users to point lorax to
extra repositories. This can be handy if some tools to create the
bootable image are not part of the product itself.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
JIRA: COMPOSE-2253
A new `buildinstall_topdir` option allows using buildinstall even when
the compose is created on a different volume that Koji is using.
The files are created in this external directory and then copies into
the usual location.
Merges: https://pagure.io/pungi/pull-request/807
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
This should make all tests pass on both Python 2 and Python 3.
Unittest2 is required on Py 2.6 and Py 3.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This would be useful for modularity. The templates can be added now and
variables set via the existing `lorax_option`.
It's not possible to use custom templates not shipped with lorax, as
passing the path to a random directory is a little bit more tricky.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Even a failed lorax task can leave behind some in-progress images. Pungi
needs to copy the files into compose/ subdirectory only if the task
finished successfully.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1485021
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
For different cases where runroot is used it's now possible to set
custom weight. The usecase for this is to avoid one builder taking too
many tasks. Especially buildinstall is quite resource intensive, so one
builder taking multiple tasks at the same time leads to very slow
compose time.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Add compose.mkdtemp which creates and returns a unique temporary
directory under <compose_topdir>/work/{global,<arch>}/tmp[-<variant>]/
Change tempfile.mkdtemp in code to compose.mkdtemp, so temporary
dirs are created under the compose's workdir, this makes it more
easier to maintain the temporary dirs/files especially when compose
fails with amount of temporary files left there.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Phases createiso, liveimages, image_build, ostree_installer and osbs are
done in parallel, logs from these phases are mixed and and it's not
obvious which log message belongs to which phase. This change adds phase
name in log message for these phases.
The new mixin 'PhaseLoggerMixin' is added to extend a Pungi phase with a
logging logger which copy handlers from compose's logger but with
formatter changed.
Fixes: #58
Signed-off-by: Qixiang Wan <qwan@redhat.com>
The `get_variants()` method had a `recursive` argument with default
value of `False. However, this argument had no effect and the method
always returned all variants recursively.
We can just drop the argument. All callers are updated to not supply the
argument. Should any need for getting the top-level variants only arise,
they can be accessed as the `variants` attribute directly on the Compose
object.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Tell lorax to use a specific directory for log files so that we preserve
them despite koji not having any idea about them.
Fixes: #457
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It really is just a group of independent functions, so we can simplify
it by removing the unused wrapper class. Instead of importing the
wrapper, instantiating it and calling its methods we can import the
module and call its functions directly.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The schema is written in Python to reduce duplication. When
configuration is loaded, the validation checks if it's correct and fills
in default values.
There is a custom extension to the schema to report deprecated options.
The config dependencies are implemented as a separate pass. While it's
technically possible to express the dependencies in the schema itself,
the error messages are not very helpful and it makes the schema much
harder to read.
Phases no longer define `config_options`. New options should be added to
the schema. Since the default values are populated automatically during
validation, there is no need to duplicate them into the code.
The `pungi-config-validate` script is updated to use the schema and
report errors even for deeply nested fields.
The dependencies are updated: pungi now depends on `python-jsonschema`
(which is already available in Fedora).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Before the task is started, the output directory is checked and if it
exists and is not empty, the runroot task will be skipped. This is meant
for debugging when restarting the same compose. Under usual
circumstances, the directory will not be created in the first place.
The runroot task will start by removing the output directory. This way,
if koji restarts the task, lorax will not fail.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is a breaking change as big part of current failable_deliverables
options will be ignored.
There is no change for buildinstall and creatiso phase.
Failability for artifacts in other phases is now configured per
artifact. It already works correctly for ostree and ostree_installer
phases (even per-arch). For OSBS phase there is currently only a binary
switch as it does not handle multiple arches yet. When it gains that
support, the option should contain list of non-blocking architectures.
For live images, live media and image build phases each config block can
configure list of failable arches. If the list is not empty, it can
fail. Once we have a way to fail only some arches, the config will not
need to change.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The module backports features to Python 2.6 and 2.7. If it is available,
the tests will use it. If it is not available, it will fall back to
regular unittest. On Python 2.7, the tests pass anyway. On Python 2.6,
there are failures with Python 2.6.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This uses a new common function that will ensure consistent logging
without duplicating the functionality all over the place.
A couple tests are updated to verify that correct data is logged.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the koji tasks fails but produces some output (e.g. .discinfo),
copying the artifacts would crash the compose. That should not happen if
the variant/arch is marked as failable.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should generate the same volume id in buildinstall and createiso
phases, with isolinux.cfg being consistent as well.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch adds configuration option to change disc type used in file
name. So far this can only be changed for link to images/boot.iso.
Resolves: #109
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There were asserts that actually did not test anything. They should have
checked that koji is called with correct arguments.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch fixes how logs are stored if lorax is used as buildinstall
method. The logs for each variant are in a separate file now. If failure
is allowed, the global log will now show why it failed.
A couple tests are added as well.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The computation of image name has been moved to a separate function.
This allowed simplification in how work dir for isos is computed, but
the result is not changed.
The live image phase has some special casing for names of RPM wrapped
ISOs. This is moved to the actual phase. Since this is (and has been)
undocumented, there should not be many users of this special case.
The documentation is updated to describe how image names and volume ids
are determined and how the process can be customized.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The config file can now specify options for lorax per variant and arch.
This is needed for Fedora to set x86_64 images as bootable on mac.
The old config option `buildinstall_upgrade_image` was deprecated, as
the same can be specified via new `lorax_options`.
Documentation and tests are updated.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Lorax fails when run on an existing non-empty directory. This patch runs
it in separate subdirectiories base on variant. Running with
buildinstall should not be changed at all.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The buildinstall phase now starts lorax for each variant separately.
Running with buildinstall should not be affected in any way.
Only variants of type=variant are considered. The side-effect of this is
that if an architecture is only used by variants of other types, lorax
will not be called at all.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The test only checks that commands are created with correct arguments
and that a proper number of threads is started. There is no validation
for what the threads are actually doing.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
