This patch adds a new config option createrepo_use_xz, which when set to
true will cause createrepo to compress sqlite databases with xz. The
default setting is False.
Fixes: #387
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of using the _doChecksum function, it now uses a function from
kobo which has nicer API. If there is an error, the message now includes
more details.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Introduces a new metadata file to track arbitrary files added during the
extra-files phase. This file is placed in the root of each tree and is
called ``extra_files.json``. It is a JSON file containing a single
object, which contains a "header" key with an object describing the
metadata, and a "data" key, which is an array of objects, where each
object represents a file. Each object contains the "file", "checksums",
and "size" keys. "file" is the relative path from the tree root to the
extra file. "checksums" is an object containing one or more checksums,
where the key is the digest type and the value of that key is the hex
digest. Finally, the size is the size of the file in bytes.
For example:
{
"header": {"version": "1.0},
"data": [
{
"file": "GPL",
"checksums": {
"sha256": "8177f97513213526df2cf6184d8ff986c675afb514d4e68a404010521b880643"
},
"size": 18092
},
{
"file": "release-notes/notes.html",
"checksums": {
"sha256": "82b1ba8db522aadf101dca6404235fba179e559b95ea24ff39ee1e5d9a53bdcb"
},
"size": 1120
}
]
}
Signed-off-by: Jeremy Cline <jeremy@jcline.org>
Fixes: #295
If the configuration does not specify version for images or live media,
Pungi will create a default value based on `release_version`. If label
is used for the compose, the milestone from it will be appended to the
version (unless it's RC).
This change is backwards compatible: nothing changes when version is set
in configuration. If the version was missing before, building the
artifacts would fail. With this patch, default values will be supplied.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The schema is written in Python to reduce duplication. When
configuration is loaded, the validation checks if it's correct and fills
in default values.
There is a custom extension to the schema to report deprecated options.
The config dependencies are implemented as a separate pass. While it's
technically possible to express the dependencies in the schema itself,
the error messages are not very helpful and it makes the schema much
harder to read.
Phases no longer define `config_options`. New options should be added to
the schema. Since the default values are populated automatically during
validation, there is no need to duplicate them into the code.
The `pungi-config-validate` script is updated to use the schema and
report errors even for deeply nested fields.
The dependencies are updated: pungi now depends on `python-jsonschema`
(which is already available in Fedora).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Currently the main log only says the phase started and finished. With
this patch each created image will be mentioned by its name, version,
release and arch.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When release for an image is specified as explicit `None`, we can
generate the value based on compose label. For example for `Alpha-1.2`
the release would be `1.2` instead of the date based one.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Before the task is started, the output directory is checked and if it
exists and is not empty, the runroot task will be skipped. This is meant
for debugging when restarting the same compose. Under usual
circumstances, the directory will not be created in the first place.
The runroot task will start by removing the output directory. This way,
if koji restarts the task, lorax will not fail.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Originally, the idea was to remove non-blocking images that failed the
check. They were only removed from the manifest, which creates some
confusion as to what is going on. With this patch, a failed check on
non-blocking deliverable will only print an error message.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When ISO is marked as failable and check fails, the compose would still
be aborted. This patch fixes that
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should help with debugging by providing better information on which
Pungi version created the compose. In development, the version will show
output of git describe, in production it asks which version is installed
in site-packages/. The egg-info directory must be installed for this to
work.
It is no longer necessary to synchronize version in `setup.py` with
`pungi/__init__.py`.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of installing pungi itself in the runroot, we can prepare the
commands to be run on compose box, write the shell script into work/
directory, which is mounted in the chroot, and execute that. This way
there is no business logic in runroot (except for finding lorax
templates).
The main advantage of this approach is that we don't need to pull any
extra dependencies into buildroot.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When package has incompatible arch for a tree, the same error message
could be raised from two different places. This patch adds more
information to the message, so that it is easier to find the actual
problem.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is a breaking change as big part of current failable_deliverables
options will be ignored.
There is no change for buildinstall and creatiso phase.
Failability for artifacts in other phases is now configured per
artifact. It already works correctly for ostree and ostree_installer
phases (even per-arch). For OSBS phase there is currently only a binary
switch as it does not handle multiple arches yet. When it gains that
support, the option should contain list of non-blocking architectures.
For live images, live media and image build phases each config block can
configure list of failable arches. If the list is not empty, it can
fail. Once we have a way to fail only some arches, the config will not
need to change.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Atomic Reactor does not honor this option. In the future we might need
to reintroduce this feature, but given that it does not work in the
current form it is better removed.
Fixes: #348
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
ISO image without MBR and GPT can still be bootable if it has an El
Torito boot catalog. The test phase must accept such images.
This slightly defeats the point of the check: to verify the ISO is
hybrid. However, based on the metadata we have no way to actually tell
if the image is supposed to be hybrid.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Make all imports either use full package name starting with `pungi` or
use explicitly relative import. This will avoid issues when importing a
module that shadows another module on PYTHONPATH.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This lives now in pungi.util. It was copied there long ago and all the
places that reference it reference it in that location.
I think this code in pungi.phases.pkgset is unused and can be removed.
The other function moved to the common module.
Signed-off-by: Ralph Bean <rbean@redhat.com>
Instead of overwriting the same log file, make sure we keep all the logs
for debuginfo, source and binary packages.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch makes it possible to use different style format placeholders.
Instead of the percent encoding it is now possible to use simple curly
braces.
%(foo)s -> {foo}
The old format is still available.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the `<groups></groups>` section is not specified in the variants XML
file, all groups will be used in this variant. The section must be
omitted completely, not just empty. This is (and was) correct according
to the DTD, it just lead to crash before.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Even if gather phase is skipped, it will still generate a rpms.json
metadata file. It will have no payload, but the header structure is
there.
Createrepo phase had to be updated to ignore variants that have no RPMs
listed in metadata.
There is one more fix for writing treeinfo files: if repomd.xml files
are not generated, it will no longer crash.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This will avoid confusion when the file can not be found, but previous
export from remote location worked well. In such case the log said:
Exporting from SCM...
Boom, no files found...
This commit changes it into:
Exporting from SCM...
Exporting file from current working dir...
Boom, no files found...
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of running the copy function for all variant.arch pairs
unconditionally, only do it if there is something to do. This makes the
log more understandable.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It will take RPM repo from a variant in this compose and a Dockerfile
from configured git and use it to build an image.
The build images are uploaded to some a Docker registry by OSBS and are
not directly part of compose (because there is no export function).
There is a new metadata file `osbs.json` that has some information that
can be used to find the image.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should fix the issue with only printing information about automatic
toggling of `supported` flag to standard output and not to a file.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of iterating over image manifest, loop through all variants and
arches and see if there are any images. This avoids a crash for variants
nested under other variants (layered products, optionals or addons).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is for layered products that have a variant mapped
to multiple variants in a base product, for example:
* Foo-Tools (id: FooTools, uid: Foo-Tools, name: Tools)
* Bar-Tools (id: BarTools, uid: Bar-Tools, name: Tools)
Requires productmd >= 1.2.
Signed-off-by: Daniel Mach <dmach@redhat.com>
Add a documented and tested config options for setting ISO parameters
instead of hardcoding magic values.
Fixes: #256
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch also fixes a bug where packages in hashed directories would
not be recognized as such and would not be placed after other files.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Some parts can be written in a more straight-forward way. This also
fixes the issue with not raising error on adding to big file.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of putting everything to <variant>/<arch>/iso, move images/,
EFI/ and isolinux/ subdirs to <variant>/<arch>/os. The nicely named ISO
image is still in the original location.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of having a separate config option, just use the koji profile.
According to release notes, this should have already been done.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
On Python 2.6, it requires the format placeholder to have explicit index
of argument, so using % formatting is easier.
There are a couple places where the method is still used because the
same argument is used twice.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This commit also removes the ability to generate timestamp. It must now
always be passed explicitly. The feature was not used anywhere and it
actually did not work.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This tests writing and reading them.
Also, it makes sure the description for layered products is correct:
there was a missing space.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Each variant.arch combination can have multiple ostree repos configured,
so we need to make sure the filesystem paths don't clash.
The paths used now are:
logs/<arch>/<variant>/ostree-<x>/
work/ostree-<x>/config_repo
The x stands for a number identifying the task. It has no relation to
actual contents of the repo.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Make sure the directory exists before calling Koji (because otherwise
the mounting will fail). Update the runroot script to initialize the
repo when there are no files in the target destination.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The performed checks:
* If format is ISO, the file must have correct magic string
* If it's bootable, there must be MBR or GPT
When a check fails on any failable deliverable, it will be logged and
the file removed from metadata (it will still remain on the disk). This
required a change to write the images.json file later (after test
phase).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The config object from Kobo does not allow keys starting with
underscore, so we prefix it with `private_` instead.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
rhbz#1331317 when we refactored how we make dvds in
df400002d8 we lost the ability to boot
the dvd as a disk image.
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
There is a lot of mock objects needed: we bypass calls to Koji, use a
mock FileCache that does not need valid RPMs on disk and avoid any
multithreading.
The test data in tests/fixtures/tagged-rpms.json comes from Koji. It is
filtered down to only a few packages to make it manageable.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch also fixes the bug where using a preconfigured Koji event
would not actually work and instead the latest event would be used.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The live_images, live_media and image_build phases have same options
that need to be specified on each image. This leads to a lot of
duplication in the config file. This patch adds global settings and
phase-level settings that allow to significantly reduce duplication.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It should have raised NotImplementedError, but we better remove it
altogher since we are missing export_dir anyway.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is a sure way to have it removed, without having to duplicate the
logic in all places where it is used.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Lubomír Sedlář
Patrick Uiterwijk
Pat Riehecky
Jeremy Cline
Dennis Gilmore
Lubomír Sedlář
Ralph Bean
Qixiang Wan
Daniel Mach