The module has same S:V:C, but the name is suffixed with `-devel`. The
module should contain all packages from the module koji tag that were
not included in the actual module.
The devel module has the same dependencies as the regular module, but
also additionally depends on the original module. The API and profiles
are cleared in the new module.
In the metadata it shows the same koji tag.
The test if package goes to the module is refactored to a function to
make work with the negated case a bit easier.
There may be unneeded multilib packages in the -devel module, because
there might be buildtime dependencies between things that we don't see.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Clear API and profiles
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This works around a bug where fus prioritizes first repo and thus could
include packages even if they were in lookaside.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The main variant is already available, this patch adds information about
additional variants that are included in the image.
JIRA: COMPOSE-2917
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There's an environment variable that can suppress printing of PID for
every debug message. It provides us with no real information and only
makes the log larger.
JIRA: COMPOSE-2973
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Only local paths are supported currently. As such, `file://` can be
stripped, and for anything else we should raise an exception.
JIRA: COMPOSE-2996
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This PR adds new pkgset_koji_builds configuration option.
This option allows setting list of extra Koji build NVRs which will be
included in a compose. This is useful in two cases:
a) It allows generating standard composes with few packages update to
certain version to test how the compose behaves when the package is
updated for real.
b) It allows generating compose consisting only from particular builds
when pkgset_koji_tag = '' or None. This is useful when one want to
regenerate the compose with packages which are not tagged in single Koji
tag. This is very useful for ODCS when reproducing old composes.
Merges: https://pagure.io/pungi/pull-request/1049
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
There are three different cases:
* we expect commitid and it's there
* we expect commitid and it's missing
* we don't expect commitid
This patch helps differentiate between the second two. In former one we
should report an error and mark the phase as failed. The latter is
perfectly fine and no error should be reported
Fixes: https://pagure.io/pungi/issue/1046
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There can be multiple modules with the same name and stream. They should
all have the same version, but will have different contexts. Fus takes
only N:S as input, but should pull in all matching modules. We just need
to give it correct data in the repo.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch changes the behaviour when both module tag and NSV?C? is
specified. The NSVC are used as a whitelist and only matching modules
will be included in the compose.
Additionally this patch adds filtering based on inheritance: when
finding the latest module for each N:S combination, only the top tag in
which the module is tagged is used. Even if a newer build is available
somewhere deeper in the inheritance, it's not going to be used.
Example inheritance and tagged modules
f29-compose (foo:1:2018:cafe)
└─ f29-candidate (foo:1:2019:cafe)
The compose will use 2018 version, because it's in the topmost tag.
JIRA: COMPOSE-2685
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
In the search result, we should ignore any module build that finished
after the event that we are working with.
Fixes: https://pagure.io/pungi/issue/999
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the same tag is queried with the same event, Pungi can cache the
response and call the API again. Particularly for small composes this
can save up significant amount of time.
Merges: https://pagure.io/pungi/pull-request/1022
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
If we're running in debug mode and the file is already present, it
should not be modified. This means that in order to rerun the actual
gather phase the file needs to be manually deleted first. However the
much more common use is to skip gather phase (because only images should
be re-run). In that case the manifest will be preserved correctly.
JIRA: COMPOSE-2756
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the configuration specifically excludes a variant from buildinstall,
but does not also disable ISO creation, we should just create a
non-bootable ISO instead of reporting a warning.
JIRA: COMPOSE-2887
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The tests create empty directories in current working directory, but
they never clean them up. Instead we can switch to a temporary location
that will be cleaned up automatically.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The file is not being included in the source tarball, which breaks
internal jenkins and will eventually cause the test to fail at RPM build
time.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
On some systems we need to use python3, in other places it can be
another version. Instead of guessing, let's look at shebang line in
lorax executable and use the same.
JIRA: COMPOSE-2852
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This could be used to enable zchunk generation, which can require up to
4 different options. Instead of hardcoding every single one, let's just
allow more direct access to the executed command.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Fus returns also RPMs in modules, but until latest version it only
worked if the package was in the same repo as the metadata. This changed
in latest version and now Pungi does not need to expand the list
anymore.
JIRA: COMPOSE-2779
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This was not configurable for users, and the default was always used,
which meant no escaping. Might as well just remove the dead code.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This will be used by Anaconda to consume multiple repos for
installation.
JIRA: RCM-36970
JIRA: COMPOSE-2753
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Fixup
Comps file specifies a pattern for some packages. If that package is
installed, all packages matching the pattern are added as well. This can
be added to fus as another pass.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The configuration for modules has changed.
Lookaside handling has changed, and there are now test failures. This
is a not a bug in DNF, so we need to fix it on our side.
Relates: https://bugzilla.redhat.com/show_bug.cgi?id=1603123
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This makes it possible to run a compose as non-root user, plus removes
the need for workarounds to publish the results directly.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It does not abort the compose, but it's still an error. It should be
marked as such in the log. This will allow easier searching for the
failures.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch adds a new gather method called `hybrid`, which uses a `fus`
binary, which must exist somewhere on the `$PATH`. It will call it
multiple times to add multilib packages.
The solver can handle packages, modules and comps groups as input.
However comps groups are never passed in. Pungi will expand it to a list
of packages to avoid issues with comps handling in fus. It ignores
optional packages, and if the group mentions a package that does not
exist, nothing else from the group is included.
Multilib is also handled outside of fus. Pungi will run it, parse the
packages from output, determines multilib packages and adds them as
input. Then it runs the solver again. This is done until nothing new is
added. Usually two passes should be enough.
Source packages and debuginfo are added as a final step. All debuginfo
packages from any included source are added. If the source or debuginfo
package is included in any lookaside repo, it will be skipped.
The tool expects to get a platform stream that should be provided for
modules to depend on. Pungi looks into the modules and gets the platform
from there. If there are more requests, an error is raised.
There is some missing functionality and options that are ignored.
Particularly these are:
* gather_fulltree
* gather_selfhosting
* greedy_method
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
And include variant in repo file name. The whole path is unique already,
but not the filename itself.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We can specify !VERSION_FROM_VERSION in version field during
image_build to expand it to correct release number without any label
information.
Also implemented !RELEASE_FROM_DATE_RESPIN to provide correct
release number. This helps to keep Atomic Host media files name
produced by image_build during bodhi updates compose run
consistent with nightly run.
Fixes: https://pagure.io/pungi/issue/987
Merges: https://pagure.io/pungi/pull-request/995
Signed-off-by: Sinny Kumari <sinny@redhat.com>
If a module says to filter a package out, we can do it immediately when
getting the build information from Koji.
This avoids a possible problem of something pulling the module package
in as a dependency, but it should also make the package set slightly
smaller.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a package name contains leading or trailing whitespace, it will
eventually lead to issues: pungi will try to include that group, but
since it does not exist, the packages will not make it in.
The root cause is hard to find. Better report an error immediately.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a file has multiple hard links, genisoimage will put the wrong number
on the ISO. This patch can work around it by copying hard-linked files
into a temporary staging directory.
JIRA: COMPOSE-2610
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
PDC is deprecated in upstream. The usecase for getting list of modules
by NS, NSV or NSVC can however be satisfied by querying modules imported
into Koji.
This makes it possible to deprecate PDC configuration.
Merges: https://pagure.io/pungi/pull-request/985
Signed-off-by: Martin Curlej <mcurlej@redhat.com>
If there is leading or trailing whitespace in a comps group name, it
will not be included in the compose and there will even be no error
message. Whitespace on module name results in a failure.
To avoid these errors, validating the variants file will now also check
that there is no whitespace in significant places, and abort the compose
if there a problem.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
After cloning the repository with defaults, open each file and try to
check if there is more than one definition for the same module. It's not
a problem for the compose process, but consumers of the compose would
get confused and possibly explode. Better alert people early.
Conceptually this should be part of the test phase, but that would mean
waiting for the compose to finish before reporting the error. The
earlier the better.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the compose is not using comps, we can't pass the comps to lorax,
since it doesn't exist and it would cause a crash.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
