Translate the using configured patterns, and give OSBS a repo file with
that URL.
JIRA: COMPOSE-3290
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a repo or install tree is specified as an absolute path on the local
filesystem, we should either translate it using the configured mappings,
or if no mapping matches, it should be return unchanged. A variant name
can not start with a slash, so attempting that translation does not make
much sense.
JIRA: COMPOSE-3290
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The default is the original behaviour. On F30+ a new option should be
added to config to make it work.
Over time as users move to this option (which requires a new enough
version of lorax), the default should be switched and then the option
removed.
Resolves: https://pagure.io/pungi/issue/1126
Merges: https://pagure.io/pungi/pull-request/1128
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch allows the configuration to express maximum expected size for
ISOs created in createiso and extra_isos phases. If the image is larger
than this limit, a warning is emitted in test phase. The compose itself
is not affected in any way.
JIRA: COMPOSE-2824
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch does not do any actual pushing. It will only extract data
about push targets from the main configuration and store it together
with exact Koji NVR in a well-defined location, and also send the data
to message bus for another service to handle.
JIRA: COMPOSE-3228
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
For binary packages the filters are handled at the depsolver level.
However sources and debuginfo is added later in the process, so the
filters have to be explicitly applied.
JIRA: COMPOSE-3114
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the hybrid solver is used in a situation where there are modules in
lookaside repo, but not in the compose itself, it will fail to detect
any platform. Since we are already opening the module repodata, we can
retrieve platforms from all modules in there as well.
If there are conflicts (e.g. multiple modules depending on different
platforms), an error will be reported.
JIRA: COMPOSE-3277
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If there are variants that depend on another, they should be processed
in order to make sure packages from the base variant are linked first.
That way the srpm cache is populated and any package in layered variant
but with source in base will have access to correct epoch information.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If there are multiple matching configuration blocks, we should take the
value that is defined in the last one, but only if it actually is there.
That's how it works for other options.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is really an oversight. The list of packages from prepopulate list
should be treated the same as if the packagees were coming from comps or
additional packages.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should tell lorax what arch to use and avoids fragile detection
based on contents of source repo.
It uses the same logic buildinstall phase uses to get the buildarch.
Related: https://pagure.io/teamsilverblue/issue/67
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It may make sense to break a big compose into smaller chunks that can be
done independently. This script allows describing the smaller parts,
runs them with correct dependencies and arranges the result to look like
a single big compose.
All parts use the same koji event, that is either obtained from Koji, or
from command line argument.
JIRA: COMPOSE-2654
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We already make this possible for the ostree installer, but it was
missing from the traditional one. The default behaviour is to let lorax
decide, but if user knows better, they can overwrite in configuration.
JIRA: COMPOSE-3188
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If configuration requests a different ref than what treefile has, a
wrong message can be sent.
If the treefile is in JSON, the local copy will be updated in-place and
it works. However with YAML the updated version is still written as JSON
and thus we are still sending the original value.
Fixes: https://pagure.io/pungi/issue/1092
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of multiple places handling the same thing duplicating the
logic, it's better to do it once upfront. This allows easy caching of
the results.
Additional advantage of this approach is that the config dump will
include resolved URLs. The original reference will still be available in
the copy of the original config.
JIRA: COMPOSE-3065
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should be on top level of the ISO, and list files added
specifically to the ISO. If there's anything inherited from one some
variant, the files will be listed in metadata in the variant directory.
JIRA: COMPOSE-3069
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The files created in runroot task are owned by root by default (because
that's who is running the processes to create them). Making the results
world readable allows the compose to work, but it still can be difficult
to clean up old composes if they contain random files owned by root.
Fixes: https://pagure.io/pungi/issue/1039
JIRA: COMPOSE-2906
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Currently the `copy_all` function resolves all symlinks and crashes on
broken symlinks. Instead it should copy symlinks as symlinks.
Fixes: https://pagure.io/pungi/issue/1084
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
yaml.load is equally powerful as python pickles, and we don't
need that level of power for the ostree yaml files.
Better safe than sorry.
Signed-off-by: Patrick Uiterwijk <patrick@puiterwijk.org>
They are not always wanted, so let's not include them by default.
There's a new option to include the same files that extra files phases
uses, or alternatively they can be configured specifically and put into
the variant subdirectory.
JIRA: COMPOSE-3084
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Up to now it was possible to change the pattern for all images, but
there are use-cases where different variants might want different names.
For example there could be one main variant that should only have
product name in the ISO filename, but addons should still be marked with
variant name.
JIRA: COMPOSE-3041
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The environment in comps for a variant can refer to groups in parent
variant (either for addons, or because of other configuration). We
should not remove the groups in this case.
This requires changes in two places:
* teaching `comps_filter` about groups that should not be removed
* fixing writing comps so that it does not actually change the data as
well
JIRA: COMPOSE-2612
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The initial implementation is very coarse. It enables it for all
variants and all modules. That is not always wanted.
With this patch, the config file has to explicitly list the devel
modules for each variant that should have it. The variant must be
configured also to include the non-devel module (but the module may be
in lookaside so it won't be included).
We now include module metadata in the internal lookaside repo, so that
this whole thing works if one variant is built on top of another.
JIRA: COMPOSE-3034
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Since 1.18 productmd will make the list available for consumers. If
possible, we should use it, and fall back to hardcoded list.
JIRA: COMPOSE-3044
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the configuration lists `*` in `additional_packages`, it has a
special meaning. If it's passed to fus directly, it will use it to match
all modules and RPMs that are not masked by a package available in some
default stream. Neither is good. We don't want it to match modules, and
we want even the masked packages.
The fix is to expand the wildcard to a list of NVRs and give that to
fus. It should include the package even if it is masked.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is no repoclosure that correctly understands modules. The best
thing we can offer is the errors reported by the depsolver.
JIRA: COMPOSE-2321
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a multilib package fails to be added (broken dependencies, or it
doesn't exist at all), we want to track it and eventually stop the
iterations.
Merges: https://pagure.io/pungi/pull-request/1071
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Stream can contain dash sign and when MBS imports such NSVC to Koji, the
dash is replaced with underscore. The current Pungi code does not
respect that and tries to use the stream from Koji directly, which
results in wrong stream being using in some Pungi internal data.
In this PR, the NSVC is taken from module metadata section of CG Koji
build, which contains real stream including the dashes.
Merges: https://pagure.io/pungi/pull-request/1072
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
Without this we don't have artifacts from all modules when the first
modular variant is solved. That makes some modular packages appear as
bare.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If there are two variants with different module sets, the missing
modulemd from variant B causeing depsolving in variant A to consider the
packages as non-modular. That is wrong.
Related: https://bugzilla.redhat.com/show_bug.cgi?id=1640125
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of having one giant command line, write the long list into a
file and pass that to the solver.
The items on the input list are sorted for easy processing.
JIRA: COMPOSE-3012
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We can't drop the whole package set once the first solver iteration
finishes, because it runs for each arch separately and we need the data
for each of them. We can however delete the arch specific portion.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the file is generated, Pungi will modify it at the end of
buildinstall phase. We need to point ISO creation to the modified file,
otherwise the checksum in .treeinfo will be incorrect.
It's sufficient to just update the graft point to correct file, copying
is not needed.
JIRA: COMPOSE-2976
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The function loads existing treeinfo, removes reference to boot.iso and
adds [media] section. This is the basic tweak that should happen for all
ISOs. Additional changes depend on the actual content.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If we want to include the -devel modules, the original modules should be
present for the solver. However if we put them in both local and
lookaside repos, fus will get confused. Let's not include modules that
are in lookaside in the module repo created for the solving job.
Even if the modular packages are present in the local repo, they are
identical to the ones in lookaside, and so they should not make it into
the result anyway.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It's possible we ask to include module X, but it's in lookaside and as
such it should not be in the output. Therefore we need to remove it from
the variant.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
By default nothing should change. This patch adds a new config option
that enables all this new craziness:
* it turns of applying module filters at pkgset level
* it creates new modules and adds them to the compose
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The module has same S:V:C, but the name is suffixed with `-devel`. The
module should contain all packages from the module koji tag that were
not included in the actual module.
The devel module has the same dependencies as the regular module, but
also additionally depends on the original module. The API and profiles
are cleared in the new module.
In the metadata it shows the same koji tag.
The test if package goes to the module is refactored to a function to
make work with the negated case a bit easier.
There may be unneeded multilib packages in the -devel module, because
there might be buildtime dependencies between things that we don't see.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Clear API and profiles
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This works around a bug where fus prioritizes first repo and thus could
include packages even if they were in lookaside.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
A repo does not necessarily have the location_base attribute if the
packages are next to the repodata. This can easily happen for lookaside
repos.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is a sorted dict implementation in Productmd used to achieve the
exact same thing as Pungi does here. No need for an extra dependency.
While we're at it, we can also sort the imports.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
repodata/repomd.xml include timestamps of all the other repodata files.
Even when those files are created reproducibly, they have current
modification time. In general case this is a good thing (ease checking
if repodata cache is up to date). But in case of composing installation
image, it breaks reproducibility.
Avoid this by instructing createrepo to set mtime and revision to
$SOURCE_DATE_EPOCH.
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
OrderedDict used by default by ConfigParser isn't enough because order
of entries being added may not be deterministic (depends on directory
list order). To solve this problem, use SortedDict as a base.
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
If not set explicitly, isohybrid choose it randomly, which harm
reproducibility.
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
xorriso make the image reproducible (given the same input files),
including support for SOURCE_DATE_EPOCH in various metadata.
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
The main variant is already available, this patch adds information about
additional variants that are included in the image.
JIRA: COMPOSE-2917
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The pool is created in multiple places, and the process is always to
create an instance and add workers to it. Let's abstract the loop in a
method.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There's an environment variable that can suppress printing of PID for
every debug message. It provides us with no real information and only
makes the log larger.
JIRA: COMPOSE-2973
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Only local paths are supported currently. As such, `file://` can be
stripped, and for anything else we should raise an exception.
JIRA: COMPOSE-2996
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This PR adds new pkgset_koji_builds configuration option.
This option allows setting list of extra Koji build NVRs which will be
included in a compose. This is useful in two cases:
a) It allows generating standard composes with few packages update to
certain version to test how the compose behaves when the package is
updated for real.
b) It allows generating compose consisting only from particular builds
when pkgset_koji_tag = '' or None. This is useful when one want to
regenerate the compose with packages which are not tagged in single Koji
tag. This is very useful for ODCS when reproducing old composes.
Merges: https://pagure.io/pungi/pull-request/1049
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
There are three different cases:
* we expect commitid and it's there
* we expect commitid and it's missing
* we don't expect commitid
This patch helps differentiate between the second two. In former one we
should report an error and mark the phase as failed. The latter is
perfectly fine and no error should be reported
Fixes: https://pagure.io/pungi/issue/1046
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There should not be multilib arches when checking if noarch package is
compatible with current arch. Otherwise we might be excluding a package
from x86_64 just because it does not work on i686.
JIRA: COMPOSE-2885
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The multilib library expects us to ask about the candidate package for
another arch (should I include this as multilib?) and not the current
arch (should I include other arches for this package?).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There can be multiple modules with the same name and stream. They should
all have the same version, but will have different contexts. Fus takes
only N:S as input, but should pull in all matching modules. We just need
to give it correct data in the repo.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch changes the behaviour when both module tag and NSV?C? is
specified. The NSVC are used as a whitelist and only matching modules
will be included in the compose.
Additionally this patch adds filtering based on inheritance: when
finding the latest module for each N:S combination, only the top tag in
which the module is tagged is used. Even if a newer build is available
somewhere deeper in the inheritance, it's not going to be used.
Example inheritance and tagged modules
f29-compose (foo:1:2018:cafe)
└─ f29-candidate (foo:1:2019:cafe)
The compose will use 2018 version, because it's in the topmost tag.
JIRA: COMPOSE-2685
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
In the search result, we should ignore any module build that finished
after the event that we are working with.
Fixes: https://pagure.io/pungi/issue/999
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the same tag is queried with the same event, Pungi can cache the
response and call the API again. Particularly for small composes this
can save up significant amount of time.
Merges: https://pagure.io/pungi/pull-request/1022
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
If we're running in debug mode and the file is already present, it
should not be modified. This means that in order to rerun the actual
gather phase the file needs to be manually deleted first. However the
much more common use is to skip gather phase (because only images should
be re-run). In that case the manifest will be preserved correctly.
JIRA: COMPOSE-2756
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the configuration specifically excludes a variant from buildinstall,
but does not also disable ISO creation, we should just create a
non-bootable ISO instead of reporting a warning.
JIRA: COMPOSE-2887
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the modulemd contains a whitelist of packages (under buildopts), it
means the packages are possibly renamed, and we need to check that list
instead of components.rpms.
Multilib does not really work now, anything with non-native arch is
skipped.
JIRA: RCM-38019
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We should not lose the flag and version. The name is not sufficient to
fully determine the problem.
The log information printed in `pungi/gather.py` is parsed by regex in
`pungi/wrappers/pungi.py`. It handles the dependency with spaces and
versions fine.
JIRA: COMPOSE-2880
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It does not like the flags kwargs for re.sub(), which should not really
be needed, since \W is case insensitive anyway.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
On some systems we need to use python3, in other places it can be
another version. Instead of guessing, let's look at shebang line in
lorax executable and use the same.
JIRA: COMPOSE-2852
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This could be used to enable zchunk generation, which can require up to
4 different options. Instead of hardcoding every single one, let's just
allow more direct access to the executed command.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Fus returns also RPMs in modules, but until latest version it only
worked if the package was in the same repo as the metadata. This changed
in latest version and now Pungi does not need to expand the list
anymore.
JIRA: COMPOSE-2779
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This was not configurable for users, and the default was always used,
which meant no escaping. Might as well just remove the dead code.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This will be used by Anaconda to consume multiple repos for
installation.
JIRA: RCM-36970
JIRA: COMPOSE-2753
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Fixup
If the data needs to be split into multiple ISOs, we need to make sure
the paths are unique for each image. Otherwise all files will be copied
into the same directory, and once the first image is finished, the whole
staging dir is deleting. That obviously breaks the tasks that are still
in progress.
JIRA: COMPOSE-2610
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The check if a variant is hybrid (or modular only) currently only looks
at comps groups. However it's possible there will be no comps groups,
but packages will be listed explicitly in config as additional_packages.
Relates: RCM-37979
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This only affects Yum backend, and allows to mention packages in comps
groups before they become available. This means a compose can be set up
ahead of all packages being available, and it will simply produce empty
repos.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Comps file specifies a pattern for some packages. If that package is
installed, all packages matching the pattern are added as well. This can
be added to fus as another pass.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Using `-J` each path component can only be 64 characters long. This is
not sufficient for some packages. Adding `--joliet-long` increases the
limit to 103.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1605103
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The configuration for modules has changed.
Lookaside handling has changed, and there are now test failures. This
is a not a bug in DNF, so we need to fix it on our side.
Relates: https://bugzilla.redhat.com/show_bug.cgi?id=1603123
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When getting list of initial packages, only run the source and do
nothing else. Additional package, system-release etc. will be added only
to comps.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This makes it possible to run a compose as non-root user, plus removes
the need for workarounds to publish the results directly.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The files are generally owned by root. If the compose is running as
root, this will still create files owned by root. If it's running as
non-priviledged user, it will crash. With this patch it will work.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It does not abort the compose, but it's still an error. It should be
marked as such in the log. This will allow easier searching for the
failures.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch adds a new gather method called `hybrid`, which uses a `fus`
binary, which must exist somewhere on the `$PATH`. It will call it
multiple times to add multilib packages.
The solver can handle packages, modules and comps groups as input.
However comps groups are never passed in. Pungi will expand it to a list
of packages to avoid issues with comps handling in fus. It ignores
optional packages, and if the group mentions a package that does not
exist, nothing else from the group is included.
Multilib is also handled outside of fus. Pungi will run it, parse the
packages from output, determines multilib packages and adds them as
input. Then it runs the solver again. This is done until nothing new is
added. Usually two passes should be enough.
Source packages and debuginfo are added as a final step. All debuginfo
packages from any included source are added. If the source or debuginfo
package is included in any lookaside repo, it will be skipped.
The tool expects to get a platform stream that should be provided for
modules to depend on. Pungi looks into the modules and gets the platform
from there. If there are more requests, an error is raised.
There is some missing functionality and options that are ignored.
Particularly these are:
* gather_fulltree
* gather_selfhosting
* greedy_method
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There's no point in checking for a layered release first. If there are
no repos, the loop will simply not execute even once. If there are
lookasides configured, we want to use them no matter if the release is
layered or not.
Also the log is updated to include the actual command for easier
debugging next time.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
And include variant in repo file name. The whole path is unique already,
but not the filename itself.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We can specify !VERSION_FROM_VERSION in version field during
image_build to expand it to correct release number without any label
information.
Also implemented !RELEASE_FROM_DATE_RESPIN to provide correct
release number. This helps to keep Atomic Host media files name
produced by image_build during bodhi updates compose run
consistent with nightly run.
Fixes: https://pagure.io/pungi/issue/987
Merges: https://pagure.io/pungi/pull-request/995
Signed-off-by: Sinny Kumari <sinny@redhat.com>
Debug packages are usually installed by exact NVR, and having two repos
providing the same module (but with different packages) is confusing for
the tooling.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There are now two places where we need to do this, so we can simplify
the logic of finding and filtering them.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a module says to filter a package out, we can do it immediately when
getting the build information from Koji.
This avoids a possible problem of something pulling the module package
in as a dependency, but it should also make the package set slightly
smaller.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a package name contains leading or trailing whitespace, it will
eventually lead to issues: pungi will try to include that group, but
since it does not exist, the packages will not make it in.
The root cause is hard to find. Better report an error immediately.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a file has multiple hard links, genisoimage will put the wrong number
on the ISO. This patch can work around it by copying hard-linked files
into a temporary staging directory.
JIRA: COMPOSE-2610
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
PDC is deprecated in upstream. The usecase for getting list of modules
by NS, NSV or NSVC can however be satisfied by querying modules imported
into Koji.
This makes it possible to deprecate PDC configuration.
Merges: https://pagure.io/pungi/pull-request/985
Signed-off-by: Martin Curlej <mcurlej@redhat.com>
If there is leading or trailing whitespace in a comps group name, it
will not be included in the compose and there will even be no error
message. Whitespace on module name results in a failure.
To avoid these errors, validating the variants file will now also check
that there is no whitespace in significant places, and abort the compose
if there a problem.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The task can take a while to finish, and it's possible for the
authenticated session to expire in the mean time. Watching the task will
work, because that happens by spawning `koji watch-task` as subprocess.
We can work around this by creating a fresh unauthenticated session for
getting the task results.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This fixes a case where extra packages are pulled in.
The scenario is:
* there's a lookaside repo which contains group G which has package P
* we want to pull group G into the compose, but our definition of G
does not contain P
* lookaside repo does not contain package P
* current package set has P
DNF depsolver will then merge the two definitions and try to get all the
packages. For most cases this is not a problem, since the package is in
the lookaside repo and will not be pulled into the compose. But in the
example above since P is not in lookaside, Pungi will put it into
current compose.
This is also ugly in the depsolving log says it includes package P
because it was in input. But checking comps file does not show it.
The result of this change is that some packages might disappear from
current composes. This can only happen if there is a lookaside which
defines groups with similar IDs, which should be very rare. In cases
where this would be a problem the fix is to explicitly add wanted
packages either to comps or to additional packages.
Fixes: https://pagure.io/pungi/issue/978
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
After cloning the repository with defaults, open each file and try to
check if there is more than one definition for the same module. It's not
a problem for the compose process, but consumers of the compose would
get confused and possibly explode. Better alert people early.
Conceptually this should be part of the test phase, but that would mean
waiting for the compose to finish before reporting the error. The
earlier the better.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the compose is not using comps, we can't pass the comps to lorax,
since it doesn't exist and it would cause a crash.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
During the createrepo phase for Modular variants, this will now
interrogate the repodata from the "work" repositories for the set of
licenses in use by each of the RPMs in a module and add those to the
metadata to be written out into the final repodata location.
Merges: https://pagure.io/pungi/pull-request/968
Signed-off-by: Stephen Gallagher <sgallagh@redhat.com>
When multiple composes are chained, they should reuse the same event.
However it is tricky as the value would have to be passed by hand. This
patch makes it possible to read the value from another compose (the
first one in the chain).
JIRA: COMPOSE-2571
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The Modulemd.copy() method has been available since libmodulemd 1.1
and is much faster than dumping to a string and parsing it again.
Signed-off-by: Stephen Gallagher <sgallagh@redhat.com>
Merges: https://pagure.io/pungi/pull-request/965
The commands in runroot run as root every time. If they create files
that are not readable to other users, the reset of compose could have
problems with it if it does not run as root too. Particularly updates
composes in Bodhi run under apache user.
Relates: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
All the components are there already separately, but having the full NVR
should simplify searching the metadata with grep.
JIRA: COMPOSE-2519
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The `add_noarch` option of `get_valid_arches` is broken and doesn't
really do anything (noarch is always present in the result).
This causes packages that have ExclusiveArch including noarch to
actually not be excluded. They should be.
Changing this globally could have a very big impact. Therefore we can
hide it behind a configuration option so that it's opt-in.
JIRA: COMPOSE-2457
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is an optimization for Yum. DNF does not care at all.
The behaviour is configurable, but the default depends on gather
backend, as that is what users should be using to consume the packages
from the repo.
Fixes: https://pagure.io/pungi/issue/951
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It should not be needed there, since the repo is empty anyway. A test is
added for the variant specific comps repo.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is used for mapping basearch to binary arch.
One use for this is when running depsolving to tell yum/dnf what arch to
work with. With this change it will get i686 instead of athlon when
working on i386 basearch.
The other use case is finding arches compatible with given basearch. The
change will remove athlon from the list for i386.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should give us better error reporting. The `copy_all` function
should preserve permissions on all files.
Relates: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It does not exist on Python 3. Converting the exception to string works
identically.
The validate methods on many phases are simplified by not calling the
parent (which does not do anything).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It checks config file value 'skip_phases' for valid phases names.
Also checks command-line attribute 'skip-phase' of 'bin/pungi-koji'.
JIRA: COMPOSE-2493
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
This way some parts of the code will be reusable. This should have no
effects on the outcome, the tests still pass without any changes needed.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
For variants that contain all packages (Fedora's Everything) we don't
want to lose any environments.
Fixes: https://pagure.io/pungi/issue/940
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is really no need to write out megabytes of logs that are not
really interesting. This should also help the parallelization. With the
verbose log createrepo fills the output buffer and needs to wait for the
busy python program to read it first.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The files created in koji runroot will be owned by root. If the compose
is done under different user, there could be a problem with copying the
files preserving the owner. Let's just copy them without that.
Fixes: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the ISO is meant to be bootable but lorax fails, there's no point in
creating the ISO as it will not behave as expected.
Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1574585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the parent task is successful, there can still be failed child tasks
for failable arches. We need to log those and potentially mark the
compose as incomplete.
Fixes: https://pagure.io/pungi/issue/874
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
With one compose combining traditional and modular content there might
be different requirements for tag inheritance. This patch adds a new
option that controls whether builds in modular tags should be inherited.
It defaults to False, which is the right option for current MBS
behavior.
JIRA: COMPOSE-2148
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The code checked with `assert` that there is only one module matching
given NSV. In actual package that would not do anything and we would
silently pick the first value.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
On stdout it gets mixed with listing of gathered packages. This has no
effect in a real pungi-koji run, where both streams are merged into a
single file which is then parsed, but in manual debugging runs it's a
little obnoxious.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This speeds up the compose quite a bit and there is no need for the
database anyway.
Merges: https://pagure.io/pungi/pull-request/922
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
If the phase is skipped, it could mean that we are doing a debug run and
we don't want to mess up the .treefile by missing arch specific images.
The other alternative is that the phase was really skipped, in which
case there will be no files generated and we already handle that fine.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a package is in lookaside, we don't really need a debuginfo nor
source for it. Definitely we should not pull them from the package set
even if there is some suitable package. Therefore we should include them
in the output but marked as lookaside.
Fixes: https://pagure.io/pungi/issue/915
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The comps could potentially be different in different variants, so
instead we can create the comps repo for every variant separately and
use two repos instead of one (packages in one repository, comps in
another one).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The log file used to contain a dump of the configuration, which could
potentially be very large. This patch removes data that is directly
visible in the input kickstart file. Instead it just displays the number
of items.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the compose configuration includes the module_defaults_dir (an
scm_dict), clone the directory, read the module defaults contained
therein and include relevant defaults in the combined modulemd file.
Only defaults for modules present in the variant are included.
This requires libmodulemd 1.2.0+.
Merges: https://pagure.io/pungi/pull-request/891
Signed-off-by: Petr Šabata <contyk@redhat.com>
Even for Everything we want to filter the comps file to make sure we
remove the stuff that is not compatible with current arch. All groups
are still preserved in that case.
This allows us to do the filtering once in init phase than just use the
prepared file in comps source.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a variant contains both modules and comps groups, we need to include
builds from the compose tag in the package whitelist. However only
packages that are not already provided by any module should be added.
JIRA: COMPOSE-2435
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
All use cases that are actually used by pungi-koji are tested. There is
missing coverage for
* keeping only items with matching arch
* not reindenting the file
These aren't currently used and should be removed in the future, but
there may be other tools depending on the comps_filter executable.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The code should not live directly in the executable, that makes it very
hard to test.
Other than the move there is no functional change.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We can't rely on the UID to be correctly joined with colons. There may
be historical data that still uses dashes.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is no need to create the directory in work/, as it will get
deleted immediately. Let's move it to /tmp and use the context manager
to clean it up.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
A new configuration option is added that allows users to point lorax to
extra repositories. This can be handy if some tools to create the
bootable image are not part of the product itself.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
JIRA: COMPOSE-2253
When ref is not modified via pungi config, we read it from the treefile
and substitute in basearch.
When pungi is configured to replace it, it modifies the treefile and
then used the value from config to avoid parsing the file. This however
did not substitute the basearch value.
We can simply use one code path for getting the value. This will work
for both cases.
Fixes: https://pagure.io/pungi/issue/866
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
