When cherry picking packages from Koji tag, we need to make sure that
for every binary package we always have a corresponding source package.
Even if it does not go into the compose, we need it to get values for
Exclusive or Exclude Arch tags.
This means we need to process the binary packages first and only then
look at source ones. Instead of sorting a potentially very long list,
let's just iterate twice.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The list of RPMs in a module can contain multilib packages, but they are
never included because we only ask for compatible arches without
multilib. For x86_64 that only adds noarch, but not i686.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It allows specify what ref we want this compose to commit to.
New parameter 'ostree_ref' overrides the default value from the treefile json.
Relates: https://pagure.io/pungi/issue/777
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
A new `buildinstall_topdir` option allows using buildinstall even when
the compose is created on a different volume that Koji is using.
The files are created in this external directory and then copies into
the usual location.
Merges: https://pagure.io/pungi/pull-request/807
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
If buildinstall is supposed to run, we can't pick only some packages
from the tag. The temporary repo is used as a source for lorax. There
could very well be packages not meant to be shipped in the compose that
are still essential for building the bootable images.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We need to use `repository`, not `os_tree` path. For any non-addon
variant they are the same, but for addons the original path does not
work as it points to the parent really.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Without this there is almost no information on what's happening. This
patch should provide basic debugging capabilities.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This would make sure that e.g. "updates" composes don't try to use "updates-testing" as an
old_compose_path, which would create practically useless deltarpms and for no repodata
reuse at all.
Signed-off-by: Patrick Uiterwijk <patrick@puiterwijk.org>
The file extension in configuration is only used to tell Pungi which
files from the task results should be downloaded. The user has to get it
right or the phase will fail. Each format has a single valid suffix.
Pungi should not require users to specify the suffix, since it can just
as well just know the right value.
The old configuration will continue working, only the extension will be
ignored.
Fixes: https://pagure.io/pungi/issue/753
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
To be more precise, new createrepo parameter "workers" was customized and
new default value was set. This fixes issue #752.
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
Instead use the definition from python-six. Once we drop Py 2 support
completely, we'll just swap underscores with dots.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should make all tests pass on both Python 2 and Python 3.
Unittest2 is required on Py 2.6 and Py 3.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This would be useful for modularity. The templates can be added now and
variables set via the existing `lorax_option`.
It's not possible to use custom templates not shipped with lorax, as
passing the path to a random directory is a little bit more tricky.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Currently `None` has to be included to allow using unsigned packages.
ODCS has trouble with including non-string value in the list though, so
we can treat empty string the same way (it's not a valid key ID anyway).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the package name matches, we should take the package either if
architecture matches or the package is noarch.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The path prefix for packages is identical for all architectures. There's
no reason to compute it multiple times and check the directory existence
repeatedly.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When splitting is not possible, we should not print a huge number as
free space on single media. This patch also adds more information so
that the numbers can actually be connected to variant and architecture.
The list of ignored files will only be printed now if not empty.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The name can contains spaces, which would potentially cause problems.
Also this is not consistent with how it works in other phases.
Relates: https://pagure.io/pungi-fedora/pull-request/354
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of the old default value and custom handling introduce new
option `live_images_target` (for consistency with other phases) and use
the usual inheritance rules.
Fixes: https://pagure.io/pungi/issue/749
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the variant is not empty, it buildinstall will run lorax on it and
create some files that would otherwise be overwritten by the ostree
installer.
The validation script is updated to load variants file as it needs a
list of variants to find out if the config is wrong.
Fixes: https://pagure.io/pungi/issue/695
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
`pungi-gather` (the tool that underlies both the `pkgset` and `gather`
phases) contains profiling code that will log statistics about how long
different function calls take. However, pungi-koji did not contain a
way to pass the ``--profiler`` argument to enable this.
This change adds a new configuration option ``gather_profiler`` which,
when set to true, simply passes the argument to `pungi-koji`. Hopefully
this can help shed some light on what is happening in some of our
longer-running composes.
Merges: https://pagure.io/pungi/pull-request/727
Signed-off-by: Ralph Bean <rbean@redhat.com>
If the compose failed, it may not have repos to compute deltas against,
and even if it has them, they were never shipped so no one will have the
older version of the package. We should instead go deeper in history and
pick a successful compose.
Relates: https://pagure.io/pungi/issue/715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The configuration needs to be more granular than a single global option.
With this patch each tree can enable deltas separately.
Fixes: https://pagure.io/pungi/issue/715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It does not make much sense to have deltas for source and debug repos.
No one benefits from it really and it takes a long time.
Relates: https://pagure.io/pungi/issue/715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Pungi write image config file with name of <format>-<name>.cfg, if there
are two or more image configs present for different arches under the same
variant and with same format & name, the config file can be overwritten,
and result in invalid image conf file.
Example:
image_build = {
'^Server$': [
{
'image-build': {
'format': [('qcow2', 'qcow2'),],
'name': 'fedora-guest-image',
'target': 'guest-fedora-26-image',
'version': '26',
'ksurl': "git://git.example.com/ks.git?fedora#HEAD",
'kickstart': "fedora-26-kvm.ks",
'ksversion': 'f26',
'distro': 'fedora-26',
'disk-size': '10',
'arches': ['x86_64'],
'repo': ["http://example.com/linux/fedora/26/Everything/x86_64/os", ]
}
},
{
'image-build': {
'format': [('qcow2', 'qcow2'),],
'name': 'fedora-guest-image',
'target': 'guest-fedora-26-image',
'version': '26',
'ksurl': "git://git.example.com/ks.git?fedora#HEAD",
'kickstart': "fedora-26-kvm.ks",
'ksversion': 'f26',
'distro': 'fedora-26',
'disk-size': '10',
'arches': ['ppc64le'],
}
},
],
}
In this case, config file "qcow2_guest-fedora-26-image.cfg" will be
created for both x86_64 and ppc64le under the same variant dir, and
there is a high chance it will be over-written while Pungi creating the
koji task. We can add arch name(s) in config filename to avoid that.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
In many cases we need to open files as binary to avoid errors on Py3
about writing binary data to file opened in text mode.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Even a failed lorax task can leave behind some in-progress images. Pungi
needs to copy the files into compose/ subdirectory only if the task
finished successfully.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1485021
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It's possible a variant is excluded via tree_variants option and the
section does not match anything. It can be confusing to users why
nothing is happening. This patch lets Pungi log all unmatched patterns.
Fixes: https://pagure.io/pungi/issue/692
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When using repos as gather_source, we should use DNF backend even for
constructing initial package set and to download the packages from
source repos. Without this the repos source would not be usable on
Python 3.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Due to overwriting an existing variable the logs are getting duplicated
line about missing comps packages instead of announcement of gathering
being finished. Rename the variable to fix the problem.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It makes no sense to repeat the same configuration for multiple
architectures. Instead we should just list the architectures as another
key in the mapping. There is an option to specify multiple config dicts.
This preserves full backwards compatibility, the old config format is
still accepted.
Fixes: https://pagure.io/pungi/issue/678
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When a package mentioned in comps is not available in the package set,
print a warning about this. Additionally there is a config option that
allows to turn this warning into a fatal error.
Fixes: https://pagure.io/pungi/issue/50
Fixes: https://pagure.io/pungi/issue/683
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There's no reason for reading the whole log of depsolving into memory
just to split it into lines and process one line at a time.
We can just as well read it in chunks.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Alternatively the call to repoclosure can be turned off. This is
customizable per variant and architecture.
Fixes: https://pagure.io/pungi/issue/676
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Lubomír Sedlář
Ondrej Nosek
Jan Kaluza
Patrick Uiterwijk
Dong Wang
Ralph Bean
Qixiang Wan