Timestamps in the main log are currently printed in local time. This
patch adds UTC offset to the top of the log so it's possible to find out
exactly when events happen.
Fixes: https://pagure.io/pungi/issue/710
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Save the config files which are imported by the following syntax:
from some_config import *
Note: This requires kobo >= 0.6
Signed-off-by: Qixiang Wan <qwan@redhat.com>
If the variant is not empty, it buildinstall will run lorax on it and
create some files that would otherwise be overwritten by the ostree
installer.
The validation script is updated to load variants file as it needs a
list of variants to find out if the config is wrong.
Fixes: https://pagure.io/pungi/issue/695
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
`pungi-gather` (the tool that underlies both the `pkgset` and `gather`
phases) contains profiling code that will log statistics about how long
different function calls take. However, pungi-koji did not contain a
way to pass the ``--profiler`` argument to enable this.
This change adds a new configuration option ``gather_profiler`` which,
when set to true, simply passes the argument to `pungi-koji`. Hopefully
this can help shed some light on what is happening in some of our
longer-running composes.
Merges: https://pagure.io/pungi/pull-request/727
Signed-off-by: Ralph Bean <rbean@redhat.com>
If the compose failed, it may not have repos to compute deltas against,
and even if it has them, they were never shipped so no one will have the
older version of the package. We should instead go deeper in history and
pick a successful compose.
Relates: https://pagure.io/pungi/issue/715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The configuration needs to be more granular than a single global option.
With this patch each tree can enable deltas separately.
Fixes: https://pagure.io/pungi/issue/715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It does not make much sense to have deltas for source and debug repos.
No one benefits from it really and it takes a long time.
Relates: https://pagure.io/pungi/issue/715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Pungi write image config file with name of <format>-<name>.cfg, if there
are two or more image configs present for different arches under the same
variant and with same format & name, the config file can be overwritten,
and result in invalid image conf file.
Example:
image_build = {
'^Server$': [
{
'image-build': {
'format': [('qcow2', 'qcow2'),],
'name': 'fedora-guest-image',
'target': 'guest-fedora-26-image',
'version': '26',
'ksurl': "git://git.example.com/ks.git?fedora#HEAD",
'kickstart': "fedora-26-kvm.ks",
'ksversion': 'f26',
'distro': 'fedora-26',
'disk-size': '10',
'arches': ['x86_64'],
'repo': ["http://example.com/linux/fedora/26/Everything/x86_64/os", ]
}
},
{
'image-build': {
'format': [('qcow2', 'qcow2'),],
'name': 'fedora-guest-image',
'target': 'guest-fedora-26-image',
'version': '26',
'ksurl': "git://git.example.com/ks.git?fedora#HEAD",
'kickstart': "fedora-26-kvm.ks",
'ksversion': 'f26',
'distro': 'fedora-26',
'disk-size': '10',
'arches': ['ppc64le'],
}
},
],
}
In this case, config file "qcow2_guest-fedora-26-image.cfg" will be
created for both x86_64 and ppc64le under the same variant dir, and
there is a high chance it will be over-written while Pungi creating the
koji task. We can add arch name(s) in config filename to avoid that.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
In many cases we need to open files as binary to avoid errors on Py3
about writing binary data to file opened in text mode.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Even a failed lorax task can leave behind some in-progress images. Pungi
needs to copy the files into compose/ subdirectory only if the task
finished successfully.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1485021
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The config can change the default compose type. This can still be
overwritten by a CLI argument. A `--production` option is now added to
CLI (because that was the default before).
Fixes: https://pagure.io/pungi/issue/694
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
In once case we can completely drop try-except statements: it only
logs the exception details but that will be logged again anyway.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is updated in all files for consistency, even the modules that will
never be ported to Py 3 completely due to dependency on Yum.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If we have a custom server CA certificate, it needs to be generally
available, and not just used when logging in so that SSL verification
works.
Signed-off-by: Owen W. Taylor <otaylor@fishsoup.net>
It's possible a variant is excluded via tree_variants option and the
section does not match anything. It can be confusing to users why
nothing is happening. This patch lets Pungi log all unmatched patterns.
Fixes: https://pagure.io/pungi/issue/692
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When using repos as gather_source, we should use DNF backend even for
constructing initial package set and to download the packages from
source repos. Without this the repos source would not be usable on
Python 3.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Due to overwriting an existing variable the logs are getting duplicated
line about missing comps packages instead of announcement of gathering
being finished. Rename the variable to fix the problem.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is needed for correct trimming of addons and optional. A package
pulled in as a dependency but that matches something on fulltree exclude
list should have this flag. It will then be moved from addon to base
variant and therefore excluded from optional.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We can't reliably tell user what system packages are missing as the name
might be different on different systems. Addiotionally there's no reason
why not rely on the packaging to be correct.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It makes no sense to repeat the same configuration for multiple
architectures. Instead we should just list the architectures as another
key in the mapping. There is an option to specify multiple config dicts.
This preserves full backwards compatibility, the old config format is
still accepted.
Fixes: https://pagure.io/pungi/issue/678
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch adds a helper function for easy creation of a structure where
either X or list of Xs is accepted.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When a file should be obtained from a git repository, allow running an
arbitrary command (like `make`) after clone but before copying the files
out. This only works for the Git backend.
The downside is that a clone is needed and we can no longer use `git
archive` to speed things up.
Fixes: https://pagure.io/pungi/issue/5
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The paths need to be absolute so that subprocesses started during the
compose with a modified cwd will still work.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When a package mentioned in comps is not available in the package set,
print a warning about this. Additionally there is a config option that
allows to turn this warning into a fatal error.
Fixes: https://pagure.io/pungi/issue/50
Fixes: https://pagure.io/pungi/issue/683
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There's no reason for reading the whole log of depsolving into memory
just to split it into lines and process one line at a time.
We can just as well read it in chunks.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Alternatively the call to repoclosure can be turned off. This is
customizable per variant and architecture.
Fixes: https://pagure.io/pungi/issue/676
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of just printing the error directly to stderr, capture the
output and use proper logger. This makes sure the error is included in
the log file and also fixes `--quiet` option which was not properly
honored originally.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
With this patch it is possible to run the test compose script from the
top level directory. This makes it slightly easier to use.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Currently, this will fail two lines later when we try to access
`pdc_module['modulemd']` with an unhelpful `TypeError` since
`pdc_module` is `None`.
Signed-off-by: Ralph Bean <rbean@redhat.com>
Instead of adding images to metadata and then creating hardlinks in a
separate step, do it immediately while we still have accurate
information about what variants it should be linked to.
Fixes: https://pagure.io/pungi/issue/670
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The rpmUtils module is provided yum-utils package, which is only
available for Python 2. There is no replacement for the functionality in
DNF.
There is a proposal to add this functionality to rpm itself, but it's
not really moving forward very much:
https://bugzilla.redhat.com/show_bug.cgi?id=1072972
As a short term solution let's copy the needed parts of rpmUtils.arch
module directly to pungi code base.
Fixes: https://pagure.io/pungi/issue/533
Signed-off-by: Qixiang Wan <qwan@redhat.com>
These packages should behave like regular debuginfo packages (at least
for now).
Fixes: https://pagure.io/pungi/issue/684
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When there is a noarch subpackage, all compatible debuginfo would be
pulled in, which is not desirable.
Example: Server.x86_64 needs pkg.x86_64 and pkg-data.noarch. We only
want pkg-debuginfo.x86_64, but without this patch even
pkg-debuginfo.i686 would get in.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When there are no groups, we shouldn't try to read comps file (because
it may very well not be there).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
