This was a workaround to make some packages from the global repo
invisible for depsolving. This is now handled by packages being in
different repos. We can select which repos are enabled at which point.
This achieves the same result, but much faster.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
With this patch, there should be a separate package set for each tag
that is consumed.
Generally each module will create a separate package set, with the
exception of -devel modules that will be in the same set as their
non-devel version.
Variants no longer need to keep their own package set objects. Instead
they now include a set of package set names that should be used for the
variant. This can replace the whitelist mechanism of deps gather method.
JIRA: COMPOSE-3620
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Historically each variant had a list of modules. This is no longer
needed and can be dropped. We can also stop logging the modulemd since
we know it was retrieved from Koji and not modified locally.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It is possible the user set a command to run in cloned Git repository,
but that command can ask for additional input. However Pungi will
capture all output, so if there is a prompt, it will never be shown.
In order to prevent confusion ("Did this hang?"), let's send empty
string to stdin of the program. That will cause any possible read to see
EOF immediately, which should cause an error that will then be reported
by Pungi to the user.
It is still possible the program will wait for input if it reads
directly from TTY. However in such case the prompt should hopefully also
be sent to TTY directly, so that possible confusion should be cleared.
JIRA: COMPOSE-3598
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The directory with logs should have updated owner and permission as
well as the actual output. This patch lays foundation for that by
allowing multiple paths to be specified.
JIRA: COMPOSE-3545
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Python 3.8 no longer sorts attributes automatically, which is causing
some of the tests to fail. The easiest fix is to update the code to make
sure sorting is in place.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1698514
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If it fails, we can't really tell if it's a transient error or just too
old git client. Fall back to full clone immediately and retry there.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Before this patch, there were two code paths: either getting the only
the wanted content by calling git-archive, or cloning the repository and
copying the files.
Both these approaches have the downside of not allowing retriving
content from a specific git commit.
The workaround is to create a new empty repo (in the location to which
we cloned previously), fetching the specific commit to there and then
checking it out.
This supports any commit and works identically for any protocol. The
downside is that all files in that commit will be downloaded. It should
be no worse than the git-clone path, but can result in bigger transfers
than git-archive.
Unfortunately this is only supported with git 2.5+. On older version
fetch will fail with no error message (tested with 1.8.3). This can be
used to fall back to full clone. This fallback is clearly suboptimal in
terms of data transfer, but it should work reliably.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The default is the original behaviour. On F30+ a new option should be
added to config to make it work.
Over time as users move to this option (which requires a new enough
version of lorax), the default should be switched and then the option
removed.
Resolves: https://pagure.io/pungi/issue/1126
Merges: https://pagure.io/pungi/pull-request/1128
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The files created in runroot task are owned by root by default (because
that's who is running the processes to create them). Making the results
world readable allows the compose to work, but it still can be difficult
to clean up old composes if they contain random files owned by root.
Fixes: https://pagure.io/pungi/issue/1039
JIRA: COMPOSE-2906
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The environment in comps for a variant can refer to groups in parent
variant (either for addons, or because of other configuration). We
should not remove the groups in this case.
This requires changes in two places:
* teaching `comps_filter` about groups that should not be removed
* fixing writing comps so that it does not actually change the data as
well
JIRA: COMPOSE-2612
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The initial implementation is very coarse. It enables it for all
variants and all modules. That is not always wanted.
With this patch, the config file has to explicitly list the devel
modules for each variant that should have it. The variant must be
configured also to include the non-devel module (but the module may be
in lookaside so it won't be included).
We now include module metadata in the internal lookaside repo, so that
this whole thing works if one variant is built on top of another.
JIRA: COMPOSE-3034
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is no repoclosure that correctly understands modules. The best
thing we can offer is the errors reported by the depsolver.
JIRA: COMPOSE-2321
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of having one giant command line, write the long list into a
file and pass that to the solver.
The items on the input list are sorted for easy processing.
JIRA: COMPOSE-3012
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It's possible we ask to include module X, but it's in lookaside and as
such it should not be in the output. Therefore we need to remove it from
the variant.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This works around a bug where fus prioritizes first repo and thus could
include packages even if they were in lookaside.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Only local paths are supported currently. As such, `file://` can be
stripped, and for anything else we should raise an exception.
JIRA: COMPOSE-2996
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This PR adds new pkgset_koji_builds configuration option.
This option allows setting list of extra Koji build NVRs which will be
included in a compose. This is useful in two cases:
a) It allows generating standard composes with few packages update to
certain version to test how the compose behaves when the package is
updated for real.
b) It allows generating compose consisting only from particular builds
when pkgset_koji_tag = '' or None. This is useful when one want to
regenerate the compose with packages which are not tagged in single Koji
tag. This is very useful for ODCS when reproducing old composes.
Merges: https://pagure.io/pungi/pull-request/1049
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
This could be used to enable zchunk generation, which can require up to
4 different options. Instead of hardcoding every single one, let's just
allow more direct access to the executed command.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Fus returns also RPMs in modules, but until latest version it only
worked if the package was in the same repo as the metadata. This changed
in latest version and now Pungi does not need to expand the list
anymore.
JIRA: COMPOSE-2779
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch adds a new gather method called `hybrid`, which uses a `fus`
binary, which must exist somewhere on the `$PATH`. It will call it
multiple times to add multilib packages.
The solver can handle packages, modules and comps groups as input.
However comps groups are never passed in. Pungi will expand it to a list
of packages to avoid issues with comps handling in fus. It ignores
optional packages, and if the group mentions a package that does not
exist, nothing else from the group is included.
Multilib is also handled outside of fus. Pungi will run it, parse the
packages from output, determines multilib packages and adds them as
input. Then it runs the solver again. This is done until nothing new is
added. Usually two passes should be enough.
Source packages and debuginfo are added as a final step. All debuginfo
packages from any included source are added. If the source or debuginfo
package is included in any lookaside repo, it will be skipped.
The tool expects to get a platform stream that should be provided for
modules to depend on. Pungi looks into the modules and gets the platform
from there. If there are more requests, an error is raised.
There is some missing functionality and options that are ignored.
Particularly these are:
* gather_fulltree
* gather_selfhosting
* greedy_method
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a package name contains leading or trailing whitespace, it will
eventually lead to issues: pungi will try to include that group, but
since it does not exist, the packages will not make it in.
The root cause is hard to find. Better report an error immediately.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If there is leading or trailing whitespace in a comps group name, it
will not be included in the compose and there will even be no error
message. Whitespace on module name results in a failure.
To avoid these errors, validating the variants file will now also check
that there is no whitespace in significant places, and abort the compose
if there a problem.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The commands in runroot run as root every time. If they create files
that are not readable to other users, the reset of compose could have
problems with it if it does not run as root too. Particularly updates
composes in Bodhi run under apache user.
Relates: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is really no need to write out megabytes of logs that are not
really interesting. This should also help the parallelization. With the
verbose log createrepo fills the output buffer and needs to wait for the
busy python program to read it first.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the parent task is successful, there can still be failed child tasks
for failable arches. We need to log those and potentially mark the
compose as incomplete.
Fixes: https://pagure.io/pungi/issue/874
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
All use cases that are actually used by pungi-koji are tested. There is
missing coverage for
* keeping only items with matching arch
* not reindenting the file
These aren't currently used and should be removed in the future, but
there may be other tools depending on the comps_filter executable.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The code should not live directly in the executable, that makes it very
hard to test.
Other than the move there is no functional change.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When allowing empty list of modules, the check for variant tags got
broken, causing Everything to no longer have an associated list of
allowed packages.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Relates: https://pagure.io/pungi/issue/862
