This was not configurable for users, and the default was always used,
which meant no escaping. Might as well just remove the dead code.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This will be used by Anaconda to consume multiple repos for
installation.
JIRA: RCM-36970
JIRA: COMPOSE-2753
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Fixup
If the data needs to be split into multiple ISOs, we need to make sure
the paths are unique for each image. Otherwise all files will be copied
into the same directory, and once the first image is finished, the whole
staging dir is deleting. That obviously breaks the tasks that are still
in progress.
JIRA: COMPOSE-2610
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The check if a variant is hybrid (or modular only) currently only looks
at comps groups. However it's possible there will be no comps groups,
but packages will be listed explicitly in config as additional_packages.
Relates: RCM-37979
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Comps file specifies a pattern for some packages. If that package is
installed, all packages matching the pattern are added as well. This can
be added to fus as another pass.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When getting list of initial packages, only run the source and do
nothing else. Additional package, system-release etc. will be added only
to comps.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This makes it possible to run a compose as non-root user, plus removes
the need for workarounds to publish the results directly.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The files are generally owned by root. If the compose is running as
root, this will still create files owned by root. If it's running as
non-priviledged user, it will crash. With this patch it will work.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch adds a new gather method called `hybrid`, which uses a `fus`
binary, which must exist somewhere on the `$PATH`. It will call it
multiple times to add multilib packages.
The solver can handle packages, modules and comps groups as input.
However comps groups are never passed in. Pungi will expand it to a list
of packages to avoid issues with comps handling in fus. It ignores
optional packages, and if the group mentions a package that does not
exist, nothing else from the group is included.
Multilib is also handled outside of fus. Pungi will run it, parse the
packages from output, determines multilib packages and adds them as
input. Then it runs the solver again. This is done until nothing new is
added. Usually two passes should be enough.
Source packages and debuginfo are added as a final step. All debuginfo
packages from any included source are added. If the source or debuginfo
package is included in any lookaside repo, it will be skipped.
The tool expects to get a platform stream that should be provided for
modules to depend on. Pungi looks into the modules and gets the platform
from there. If there are more requests, an error is raised.
There is some missing functionality and options that are ignored.
Particularly these are:
* gather_fulltree
* gather_selfhosting
* greedy_method
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There's no point in checking for a layered release first. If there are
no repos, the loop will simply not execute even once. If there are
lookasides configured, we want to use them no matter if the release is
layered or not.
Also the log is updated to include the actual command for easier
debugging next time.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
And include variant in repo file name. The whole path is unique already,
but not the filename itself.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We can specify !VERSION_FROM_VERSION in version field during
image_build to expand it to correct release number without any label
information.
Also implemented !RELEASE_FROM_DATE_RESPIN to provide correct
release number. This helps to keep Atomic Host media files name
produced by image_build during bodhi updates compose run
consistent with nightly run.
Fixes: https://pagure.io/pungi/issue/987
Merges: https://pagure.io/pungi/pull-request/995
Signed-off-by: Sinny Kumari <sinny@redhat.com>
Debug packages are usually installed by exact NVR, and having two repos
providing the same module (but with different packages) is confusing for
the tooling.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There are now two places where we need to do this, so we can simplify
the logic of finding and filtering them.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a module says to filter a package out, we can do it immediately when
getting the build information from Koji.
This avoids a possible problem of something pulling the module package
in as a dependency, but it should also make the package set slightly
smaller.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a package name contains leading or trailing whitespace, it will
eventually lead to issues: pungi will try to include that group, but
since it does not exist, the packages will not make it in.
The root cause is hard to find. Better report an error immediately.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a file has multiple hard links, genisoimage will put the wrong number
on the ISO. This patch can work around it by copying hard-linked files
into a temporary staging directory.
JIRA: COMPOSE-2610
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
PDC is deprecated in upstream. The usecase for getting list of modules
by NS, NSV or NSVC can however be satisfied by querying modules imported
into Koji.
This makes it possible to deprecate PDC configuration.
Merges: https://pagure.io/pungi/pull-request/985
Signed-off-by: Martin Curlej <mcurlej@redhat.com>
The task can take a while to finish, and it's possible for the
authenticated session to expire in the mean time. Watching the task will
work, because that happens by spawning `koji watch-task` as subprocess.
We can work around this by creating a fresh unauthenticated session for
getting the task results.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
After cloning the repository with defaults, open each file and try to
check if there is more than one definition for the same module. It's not
a problem for the compose process, but consumers of the compose would
get confused and possibly explode. Better alert people early.
Conceptually this should be part of the test phase, but that would mean
waiting for the compose to finish before reporting the error. The
earlier the better.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the compose is not using comps, we can't pass the comps to lorax,
since it doesn't exist and it would cause a crash.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
During the createrepo phase for Modular variants, this will now
interrogate the repodata from the "work" repositories for the set of
licenses in use by each of the RPMs in a module and add those to the
metadata to be written out into the final repodata location.
Merges: https://pagure.io/pungi/pull-request/968
Signed-off-by: Stephen Gallagher <sgallagh@redhat.com>
The Modulemd.copy() method has been available since libmodulemd 1.1
and is much faster than dumping to a string and parsing it again.
Signed-off-by: Stephen Gallagher <sgallagh@redhat.com>
Merges: https://pagure.io/pungi/pull-request/965
The commands in runroot run as root every time. If they create files
that are not readable to other users, the reset of compose could have
problems with it if it does not run as root too. Particularly updates
composes in Bodhi run under apache user.
Relates: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
All the components are there already separately, but having the full NVR
should simplify searching the metadata with grep.
JIRA: COMPOSE-2519
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The `add_noarch` option of `get_valid_arches` is broken and doesn't
really do anything (noarch is always present in the result).
This causes packages that have ExclusiveArch including noarch to
actually not be excluded. They should be.
Changing this globally could have a very big impact. Therefore we can
hide it behind a configuration option so that it's opt-in.
JIRA: COMPOSE-2457
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is an optimization for Yum. DNF does not care at all.
The behaviour is configurable, but the default depends on gather
backend, as that is what users should be using to consume the packages
from the repo.
Fixes: https://pagure.io/pungi/issue/951
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It should not be needed there, since the repo is empty anyway. A test is
added for the variant specific comps repo.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should give us better error reporting. The `copy_all` function
should preserve permissions on all files.
Relates: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It does not exist on Python 3. Converting the exception to string works
identically.
The validate methods on many phases are simplified by not calling the
parent (which does not do anything).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It checks config file value 'skip_phases' for valid phases names.
Also checks command-line attribute 'skip-phase' of 'bin/pungi-koji'.
JIRA: COMPOSE-2493
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
This way some parts of the code will be reusable. This should have no
effects on the outcome, the tests still pass without any changes needed.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
For variants that contain all packages (Fedora's Everything) we don't
want to lose any environments.
Fixes: https://pagure.io/pungi/issue/940
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The files created in koji runroot will be owned by root. If the compose
is done under different user, there could be a problem with copying the
files preserving the owner. Let's just copy them without that.
Fixes: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the ISO is meant to be bootable but lorax fails, there's no point in
creating the ISO as it will not behave as expected.
Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1574585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the parent task is successful, there can still be failed child tasks
for failable arches. We need to log those and potentially mark the
compose as incomplete.
Fixes: https://pagure.io/pungi/issue/874
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
With one compose combining traditional and modular content there might
be different requirements for tag inheritance. This patch adds a new
option that controls whether builds in modular tags should be inherited.
It defaults to False, which is the right option for current MBS
behavior.
JIRA: COMPOSE-2148
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The code checked with `assert` that there is only one module matching
given NSV. In actual package that would not do anything and we would
silently pick the first value.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Lubomír Sedlář
Jan Kaluza
Sinny Kumari
Martin Curlej
Stephen Gallagher
Patrick Uiterwijk
Patrick Uiterwijk
Ondrej Nosek