Add new key 'repo' to allow specifying multiple repos as the source
repositories. And change 'source_repo_from' to allow specifying multiple
vairant names to use variant repos as source repositories.
Doc of 'source_repo_from' is updated to not mention URL is supported,
though we still support that in code. User should add url of repos in
'repo' key instead of 'source_repo_from'.
Fixes: #508
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Add compose.mkdtemp which creates and returns a unique temporary
directory under <compose_topdir>/work/{global,<arch>}/tmp[-<variant>]/
Change tempfile.mkdtemp in code to compose.mkdtemp, so temporary
dirs are created under the compose's workdir, this makes it more
easier to maintain the temporary dirs/files especially when compose
fails with amount of temporary files left there.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Productmd does validation on the parsed value. There is a long list of
rules to populate `short` name in the productmd code.
There is a shim in pungi to make sure the short is populated if no rule
matches. After renaming product to release, the shim stopped working.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This a standalone script that will look into a compose and create
unified ISO for each architecture. The ISO contains RPM repositories for
all variants that have the arch.
Known issues:
* The filename does not respect settings. This is tricky because the
name could include variant name, which we don't have here (by design
of unified ISO).
* The same is true for volume id.
In order to test the feature without running actual compose, we need to
add essentially a big chunk of compose. Most of the files are empty, as
their content is never accessed.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This makes sure the test configurations will be accepted in real usage.
It also enables us to remove some manual error checking that will be
performed by validator.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The same way live_media and image_build accept additional external repos
or variants list, there is now a `repo` and `repo_from` configuration
key to add these.
Fixes: #486
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Using getattr so we also work with versions of koji that do not
read principal and keytab from the config file.
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
Also remove the TODO comment from live images phase: the appliances are
already submitted one task per single arch, so this change is not
necessary.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This uses the --can-fail option in koji. Failing an optional image will
not abort whole task. If the whole task fails (or there is a problem on
the compose side), we abort unless all arches are optional.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch uses the `--can-fail` option of koji command line. If only
optional arches fail, the task will report as success. Failures on
compose box side are ignored if and only if all architectures are
optional.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Buildinstall only runs for top-level variants. Addons, optionals and
integrated layered products must reuse install tree from their parent,
because otherwise there would be no boot.iso.
Fixes: #472
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Given the way config files can include other files, it is entirely
possible to use the same config file for both Pungi and Distill. Pungi
will however complain about unknown options.
This patch reverts part of c38bb480 and moves deprecation handling back
into schema validation. The validation method then returns a list of
errors and a list of warnings.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When running with this option, if a debuginfo package is included in the
final package set, there would be a crash due to missing SRPM. To
replicate this issue, just use the argument with input kickstart listing
a single package which has debuginfo.
This is not an issue for Pungi 4, as it never uses the option in the
first place.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Source packages can be excluded while binary packages should still go
into the compose. This patch makes it so that the mapping from binary
packages to source packages contains None in such case. The code is
already capable of handling that. A warning will be emitted for each
binary package without source.
This also allows us to remove some code from `createSourceHashes` that
is now unused.
A test for excluding source package is added as well.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The new sub-command 'installer' is added to support build OSTree
installer image with pungi-make-ostree. It can take an optional argument
'--extra-config' to read some of configurations from a json file. The
content of the json file can contains the configuration which are
supported in OSTree installer phase, the difference is variant UID is
not supported as a repo url in this case. A valid json file can be like
the following:
{
"source_repo_from": "http://www.example.com/repo/workstation/os",
"installpkgs": [
"fedora-productimg-workstation"
],
"add_template": [
"/path/to/installer/template/lorax-configure-repo.tmpl"
],
"add_template_var": [
"ostree_osname=fedora-workstation",
"ostree_ref=fedora/25/x86_64/workstation"
],
"add_arch_template": [
"/path/to/installer/template/lorax-embed-repo.tmpl"
],
"add_arch_template_var": [
"ostree_repo=https://www.example.com/compose/ostree",
"ostree_osname=fedora-workstation",
"ostree_ref=fedora/25/x86_64/workstation"
]
}
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Update pungi-make-ostree to supourt sub-command 'tree', which is just
as the original feature of pungi-make-ostree to compose OSTree tree.
With the change we can add other sub commands later to build other
OSTree artifacts, like the installer image.
Inaddtional to the change, now the the 'tree' command can accept an
optional '--extra-config' parameter to update the original tree
configuration with extra configurations specified in a json file
before composing the OSTree tree.
Example:
pungi-make-ostree tree --repo=/ostree --treefile=/path/to/treefile \
--log-dir=/path/to/log --extra-config=/path/to/extra-config.json
The extra-config file can contains the same configuration as OSTree
phase, the difference is it doesn't understand variant UID as source
repo since it's not ran in the chain of phases. A valid configuration
can be like:
{
"source_repo_from": "http://example.com/repo/x86_64/Server",
"extra_source_repos": [
{
"name": "optional",
"baseurl": "http://example.com/repo/x86_64/optional",
"exclude": "systemd-container",
"gpgcheck": False
},
{
"name": "extra",
"baseurl": "http://example.com/repo/x86_64/extra",
}
],
"keep_original_sources": True
}
The OSTree phase is updated to move out the task of updating treefile,
instead of that, it writes the extra configurations to a json file,
then 'pungi-make-ostree tree' will take it by option '--extra-config'.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
The internal flag in productmd is meant to indicate that a compose is
not meant for publishing. This is potentially useful to allow filtering
in PDC or similar service.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Phases createiso, liveimages, image_build, ostree_installer and osbs are
done in parallel, logs from these phases are mixed and and it's not
obvious which log message belongs to which phase. This change adds phase
name in log message for these phases.
The new mixin 'PhaseLoggerMixin' is added to extend a Pungi phase with a
logging logger which copy handlers from compose's logger but with
formatter changed.
Fixes: #58
Signed-off-by: Qixiang Wan <qwan@redhat.com>
The check for arch compatibility should not be performed if the
debuginfo package is noarch. Such packages should be included
unconditionally.
Fixes: #450
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the media is bootable, we can not split it. The limit is not used in
that case and we may overflow it with an warning message. The warning
should correctly mention what is going on instead of printing a
non-sensical message about free space on media being some huge number.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Sometimes addtional repos are required to get necessary packages for
composing OSTree repository. For example, RHEL doesn't have an 'Everyting'
variant, so composing OSTree repository from any of the RHEL variants
won't work, addtional source repos need to be enabled to achieve that.
The new option "extra_source_repos" enable the ability of allowing extra
source repos.
And a new option 'keep_original_sources' is introduced to keep the
original repos found in tree config file, if this is enabled, Pungi
will not remove the existing source repos from the tree config file,
just add new repos of "source_repo_from" + "extra_source_repos" to
the existing repos.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Instead of spawning a separate process for each test, move the code to
run in the main process. This gives us correct coverage information and
makes the tests a lot faster.
The input is still provided in a kickstart file. The output also goes
into a log file that is later parsed.
There is some extra work needed with logging setup to make the logs go
to standard output so that nose can correctly capture them.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The Pungi object reuses the same logger for all its instances. Every
time a new instance is created, a stream handler would be added. This
means that a second instance will print everything twice. When running
tests, tens of instances would be created.
This patch only adds the handler if there are no handlers configured
yet.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch changes the format a little and makes sure the output goes to
the same logger as the rest of the output. The output no longer contains
control characters.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When a file is exported from an RPM in the compose, and there is no
matching RPM in the package set, we want a nice error message.
Fixes: #460
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The `get_variants()` method had a `recursive` argument with default
value of `False. However, this argument had no effect and the method
always returned all variants recursively.
We can just drop the argument. All callers are updated to not supply the
argument. Should any need for getting the top-level variants only arise,
they can be accessed as the `variants` attribute directly on the Compose
object.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The code to search for install tree and repo for image-build and
live-media was only looking at top-level variants. Addons, optional or
integrated layered products could not have been found. This would lead
to error messages such as "There is no variant Server-optional to get
repo from when building live image for Client" even though the variant
exists.
Various tests are updated to exercise this edge case.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Tell lorax to use a specific directory for log files so that we preserve
them despite koji not having any idea about them.
Fixes: #457
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When running from git, the files should be found relative to the python
module, not executable itself. This change makes it possible to load the
files when running tests.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the input explicitly lists a package as multilib, we should not
automatically add native version just because of fulltree.
The tests for this use case are now enabled and passing.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Added new option '--version' to pungi-make-ostree, and this can be
enabled in ostree settings with 'version'. The version string will be
added as versioning metadata if this is specified.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Added new option '--update-summary' to pungi-make-ostree, and this can
be enabled in ostree settings with 'update_summary'. A summary file will
be generated (or re-generated if it was presented in an existing ostree
repo) when it is enabled.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
The createrepo package is needed always, but depending on configuration
we should also look for createrepo_c.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
'mount -o loop' requires root privileges, guestmount from
libguestfs-tools-c package can work without root privileges.
Fixes: #19
Signed-off-by: Qixiang Wan <qwan@redhat.com>
These options are in fact removed and have no effect anymore. This patch
changes the validation to print a warning that the option was removed
and what should be done instead. It no longer stops the whole compose.
The validation script still rejects configuration files with these
removed keys.
This change means we no longer check these removals with the JSON schema
(as that makes it hard to determine where exactly the problem is).
Fixes: #438
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Each variant section should contain a path to a directory with packages
and to repodata. There was code to do this, but due to a typo it did not
actually work.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is tricky as this early in the process we don't know the compose
ID. The new message gives the full command line that was called.
Relates: #439
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the media is bootable, we can not split it. Instead we will create an
ISO that is too big and issue a warning (aborting the whole compose
would be too much).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Lorax/buildinstall produce .treeinfo file that is cloned into the
compose dir. However since lorax runs separately for each arch, the
files are nested in a subdirectory. With old buildinstall method, this
causes the file to not be found and copied.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When creating an ISO for a layered product, the name of the integrated
product should be included in the file name.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The older buildinstall method fails if it sees ppc64p7 instead of ppc64.
Nothing changes for composes using Lorax.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It really is just a group of independent functions, so we can simplify
it by removing the unused wrapper class. Instead of importing the
wrapper, instantiating it and calling its methods we can import the
module and call its functions directly.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the mapping in configuration specifies incorrect regular expression
to match regular expressions, we should raise an error immediately and
not wait until the part of config is actually used.
This patch does not cover `live_media`, `image_build` and `osbs`
sections, as they use plain dicts and not the list of tuples format.
Fixes: #424
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of overloading the `dvd` value, use new value `ostree` that can
be overridden by a config change.
Fixes: https://pagure.io/pungi/issue/418
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If user sets `branch` to `None`, it should behave as if the branch is
not set at all. Nagging them about removing it is not helping anything.
Fixes: #415
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
to complement https://pagure.io/koji/pull-request/162 we need to adjust pungi
rpm-ostree uses bublewrap that does not work in mock. --new-chroot to mock
enables the use of systemd-nspawn instead of chroot resulting in working
rpm-ostree again
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
If the path in `translate_paths` config ends with a slash, we would
create public path with double slash.
Fixes: https://pagure.io/pungi/issue/408
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Some modules can be executed as a sort-of test. However, the files do
not have executable bit set, so there is no need for them to have
shebangs. If someone wants to call them directly, they should do so via
python.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The address is no longer correct. We can just as well simply point to
the web page describing the license.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The same information can be inferred from definitions in variants.xml:
if the variant has no groups defined, we include packages from all
groups. By the same logic we can also include all groups in the comps
file.
The config validation is updated to give a hint on how to remove the
option from the configuration.
Relates: #29
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When a variant has optional, it is possible to explictly list it in the
XML file and add extra groups. However, the original DTD did not allow
multiple variants with `id="optional"` as `id` attribute has to be
globally unique. This patch changes its type to `CDATA`, which has less
restrictions. This also means we can no longer define `<ref>` as
`IDREF` and instead check the existence of referenced variant in parser.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It does not make sense for this phase to be skipped. If there are any
images, we need to generate the checksums so that writing metadata
works. If there are no images, the phase does not do anything and is
therefore very fast.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There are big parts of code that can not be reached. Other parts expose
options that are not used anywhere. To keep things simple, all of that
is removed in this patch.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch adds a new config option createrepo_use_xz, which when set to
true will cause createrepo to compress sqlite databases with xz. The
default setting is False.
Fixes: #387
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of using the _doChecksum function, it now uses a function from
kobo which has nicer API. If there is an error, the message now includes
more details.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Introduces a new metadata file to track arbitrary files added during the
extra-files phase. This file is placed in the root of each tree and is
called ``extra_files.json``. It is a JSON file containing a single
object, which contains a "header" key with an object describing the
metadata, and a "data" key, which is an array of objects, where each
object represents a file. Each object contains the "file", "checksums",
and "size" keys. "file" is the relative path from the tree root to the
extra file. "checksums" is an object containing one or more checksums,
where the key is the digest type and the value of that key is the hex
digest. Finally, the size is the size of the file in bytes.
For example:
{
"header": {"version": "1.0},
"data": [
{
"file": "GPL",
"checksums": {
"sha256": "8177f97513213526df2cf6184d8ff986c675afb514d4e68a404010521b880643"
},
"size": 18092
},
{
"file": "release-notes/notes.html",
"checksums": {
"sha256": "82b1ba8db522aadf101dca6404235fba179e559b95ea24ff39ee1e5d9a53bdcb"
},
"size": 1120
}
]
}
Signed-off-by: Jeremy Cline <jeremy@jcline.org>
Fixes: #295
