If the config uses SCM dicts that include branch or tag names, they will
be resolved to specific commit ids.
It goes through the caching resolver. The main motivation for that is to
correctly support the --offline flag. It's highly unlikely there will be
two scm_dicts in the config with the same repo.
JIRA: COMPOSE-3279
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This can be useful for archiving configuration to freeze the koji
package set to a particular event.
JIRA: COMPOSE-3278
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The default is the original behaviour. On F30+ a new option should be
added to config to make it work.
Over time as users move to this option (which requires a new enough
version of lorax), the default should be switched and then the option
removed.
Resolves: https://pagure.io/pungi/issue/1126
Merges: https://pagure.io/pungi/pull-request/1128
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch allows the configuration to express maximum expected size for
ISOs created in createiso and extra_isos phases. If the image is larger
than this limit, a warning is emitted in test phase. The compose itself
is not affected in any way.
JIRA: COMPOSE-2824
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch does not do any actual pushing. It will only extract data
about push targets from the main configuration and store it together
with exact Koji NVR in a well-defined location, and also send the data
to message bus for another service to handle.
JIRA: COMPOSE-3228
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We already make this possible for the ostree installer, but it was
missing from the traditional one. The default behaviour is to let lorax
decide, but if user knows better, they can overwrite in configuration.
JIRA: COMPOSE-3188
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of multiple places handling the same thing duplicating the
logic, it's better to do it once upfront. This allows easy caching of
the results.
Additional advantage of this approach is that the config dump will
include resolved URLs. The original reference will still be available in
the copy of the original config.
JIRA: COMPOSE-3065
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
They are not always wanted, so let's not include them by default.
There's a new option to include the same files that extra files phases
uses, or alternatively they can be configured specifically and put into
the variant subdirectory.
JIRA: COMPOSE-3084
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Up to now it was possible to change the pattern for all images, but
there are use-cases where different variants might want different names.
For example there could be one main variant that should only have
product name in the ISO filename, but addons should still be marked with
variant name.
JIRA: COMPOSE-3041
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The initial implementation is very coarse. It enables it for all
variants and all modules. That is not always wanted.
With this patch, the config file has to explicitly list the devel
modules for each variant that should have it. The variant must be
configured also to include the non-devel module (but the module may be
in lookaside so it won't be included).
We now include module metadata in the internal lookaside repo, so that
this whole thing works if one variant is built on top of another.
JIRA: COMPOSE-3034
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
By default nothing should change. This patch adds a new config option
that enables all this new craziness:
* it turns of applying module filters at pkgset level
* it creates new modules and adds them to the compose
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This PR adds new pkgset_koji_builds configuration option.
This option allows setting list of extra Koji build NVRs which will be
included in a compose. This is useful in two cases:
a) It allows generating standard composes with few packages update to
certain version to test how the compose behaves when the package is
updated for real.
b) It allows generating compose consisting only from particular builds
when pkgset_koji_tag = '' or None. This is useful when one want to
regenerate the compose with packages which are not tagged in single Koji
tag. This is very useful for ODCS when reproducing old composes.
Merges: https://pagure.io/pungi/pull-request/1049
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
This patch changes the behaviour when both module tag and NSV?C? is
specified. The NSVC are used as a whitelist and only matching modules
will be included in the compose.
Additionally this patch adds filtering based on inheritance: when
finding the latest module for each N:S combination, only the top tag in
which the module is tagged is used. Even if a newer build is available
somewhere deeper in the inheritance, it's not going to be used.
Example inheritance and tagged modules
f29-compose (foo:1:2018:cafe)
└─ f29-candidate (foo:1:2019:cafe)
The compose will use 2018 version, because it's in the topmost tag.
JIRA: COMPOSE-2685
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the same tag is queried with the same event, Pungi can cache the
response and call the API again. Particularly for small composes this
can save up significant amount of time.
Merges: https://pagure.io/pungi/pull-request/1022
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
This could be used to enable zchunk generation, which can require up to
4 different options. Instead of hardcoding every single one, let's just
allow more direct access to the executed command.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch adds a new gather method called `hybrid`, which uses a `fus`
binary, which must exist somewhere on the `$PATH`. It will call it
multiple times to add multilib packages.
The solver can handle packages, modules and comps groups as input.
However comps groups are never passed in. Pungi will expand it to a list
of packages to avoid issues with comps handling in fus. It ignores
optional packages, and if the group mentions a package that does not
exist, nothing else from the group is included.
Multilib is also handled outside of fus. Pungi will run it, parse the
packages from output, determines multilib packages and adds them as
input. Then it runs the solver again. This is done until nothing new is
added. Usually two passes should be enough.
Source packages and debuginfo are added as a final step. All debuginfo
packages from any included source are added. If the source or debuginfo
package is included in any lookaside repo, it will be skipped.
The tool expects to get a platform stream that should be provided for
modules to depend on. Pungi looks into the modules and gets the platform
from there. If there are more requests, an error is raised.
There is some missing functionality and options that are ignored.
Particularly these are:
* gather_fulltree
* gather_selfhosting
* greedy_method
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a file has multiple hard links, genisoimage will put the wrong number
on the ISO. This patch can work around it by copying hard-linked files
into a temporary staging directory.
JIRA: COMPOSE-2610
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
PDC is deprecated in upstream. The usecase for getting list of modules
by NS, NSV or NSVC can however be satisfied by querying modules imported
into Koji.
This makes it possible to deprecate PDC configuration.
Merges: https://pagure.io/pungi/pull-request/985
Signed-off-by: Martin Curlej <mcurlej@redhat.com>
The `add_noarch` option of `get_valid_arches` is broken and doesn't
really do anything (noarch is always present in the result).
This causes packages that have ExclusiveArch including noarch to
actually not be excluded. They should be.
Changing this globally could have a very big impact. Therefore we can
hide it behind a configuration option so that it's opt-in.
JIRA: COMPOSE-2457
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is an optimization for Yum. DNF does not care at all.
The behaviour is configurable, but the default depends on gather
backend, as that is what users should be using to consume the packages
from the repo.
Fixes: https://pagure.io/pungi/issue/951
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It checks config file value 'skip_phases' for valid phases names.
Also checks command-line attribute 'skip-phase' of 'bin/pungi-koji'.
JIRA: COMPOSE-2493
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
With one compose combining traditional and modular content there might
be different requirements for tag inheritance. This patch adds a new
option that controls whether builds in modular tags should be inherited.
It defaults to False, which is the right option for current MBS
behavior.
JIRA: COMPOSE-2148
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
A new configuration option is added that allows users to point lorax to
extra repositories. This can be handy if some tools to create the
bootable image are not part of the product itself.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
JIRA: COMPOSE-2253
OstreeInstaller phase will be moved to a different timeslot
and therefore needs different repo not to depend on Gather
phase which runs at the same time.
Related: https://pagure.io/pungi/issue/778
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
With this patch the gather_source option is no longer used. Instead, all
sources are always used. If they return at least some input packages,
then a configured method is used and the returned lists of packages from
all sources are merged.
The method used for gathering can be configured for each variant and
gather source separately.
Additional packages are only added to the comps source.
Each gathering step is logged separately. All the logs are preserved for
later inspection.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There are valid use cases for not specifying this option: specifically a
modular compose will get the tags to use from modules listed in the
variants file.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Followup from discussion in: https://pagure.io/pungi/issue/811
It's likely now that for Fedora Atomic Host we'll use this, to work
around other issues, after we fix the FAW change detection.
Signed-off-by: Colin Walters <walters@verbum.org>
This essentially forces everyone to install cvs even though they most
likely don't need it. There is no easy way to check what is actually
going to be needed, so let's just put the responsibility of decoding a
potential error message to users.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is no place that would possibly call it, so we can drop the check
and function for generating cli.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When configured to use createrepo_c, we should also use modifyrepo_c.
That allows us to relax the check for createrepo package.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It allows specify what ref we want this compose to commit to.
New parameter 'ostree_ref' overrides the default value from the treefile json.
Relates: https://pagure.io/pungi/issue/777
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
A new `buildinstall_topdir` option allows using buildinstall even when
the compose is created on a different volume that Koji is using.
The files are created in this external directory and then copies into
the usual location.
Merges: https://pagure.io/pungi/pull-request/807
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
Options that are currently marked as deprecated do not have any effect
anymore (other than printing warning). We should remove them and update
the message so that we can mark options as deprecated even when they
still work.
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
This would make sure that e.g. "updates" composes don't try to use "updates-testing" as an
old_compose_path, which would create practically useless deltarpms and for no repodata
reuse at all.
Signed-off-by: Patrick Uiterwijk <patrick@puiterwijk.org>
The file extension in configuration is only used to tell Pungi which
files from the task results should be downloaded. The user has to get it
right or the phase will fail. Each format has a single valid suffix.
Pungi should not require users to specify the suffix, since it can just
as well just know the right value.
The old configuration will continue working, only the extension will be
ignored.
Fixes: https://pagure.io/pungi/issue/753
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Lubomír Sedlář
Jan Kaluza
Martin Curlej
Patrick Uiterwijk
Ondrej Nosek
Petr Šabata
Patrick Uiterwijk
Colin Walters