Modular variants can either be specified by a list of modules, or by a
list of Koji tags. In terms of comps preprocessing there should not be
any difference between the two.
Resolves: https://pagure.io/pungi/issue/1640
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Rather than tracking this directly in OSBS phase, move this into Compose
object, which will allow access to this from multiple phases.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
New `buildinstall.metadata` file is created once the buildinstall
phase is done. This file contains:
- list of lorax command line arguments.
- list of RPMs installed in the buildinstall buildroot.
- list of RPMs installed in the resulting boot.iso.
This file is checked in the next compose run to find out if
the result of buildinstall phase from the previous compose
can be reused. Following is checked:
- lorax commandline arguments are the same (except of expected
differences).
- The NVRAs of RPMs in the runroot_tag are the same as the ones
installed in the old buildinstall buildroot.
- The NVRAs of RPMs installed in the boot.iso are the same as
the ones in package sets in the current compose.
By its implementation, this reuse strategy is used only if
pungi_buildinstall Koji plugin is used.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
Add tests for buildinstall reuse and buildinstall_allow_reuse option.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
This should make it possible to only import the library only when it's
really needed.
DNF does not work with libmodulemd v2. If we import libmodulemd2 and
then dnf, the program will just hang forever. We only need DNF in
pungi-gather, where libmodulemd is not needed, and also where we do need
libmodulemd, we don't have DNF.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
With this patch, there should be a separate package set for each tag
that is consumed.
Generally each module will create a separate package set, with the
exception of -devel modules that will be in the same set as their
non-devel version.
Variants no longer need to keep their own package set objects. Instead
they now include a set of package set names that should be used for the
variant. This can replace the whitelist mechanism of deps gather method.
JIRA: COMPOSE-3620
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This was already discouraged to not be used, and is a bad idea in
current setup anyway. Removing this can simplify the code.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Historically each variant had a list of modules. This is no longer
needed and can be dropped. We can also stop logging the modulemd since
we know it was retrieved from Koji and not modified locally.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This reverts commit ac15f21135.
It is still needed if nodeps gather method is used. It simply returns
all packages listed in all modules.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1708661
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch allows the configuration to express maximum expected size for
ISOs created in createiso and extra_isos phases. If the image is larger
than this limit, a warning is emitted in test phase. The compose itself
is not affected in any way.
JIRA: COMPOSE-2824
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of multiple places handling the same thing duplicating the
logic, it's better to do it once upfront. This allows easy caching of
the results.
Additional advantage of this approach is that the config dump will
include resolved URLs. The original reference will still be available in
the copy of the original config.
JIRA: COMPOSE-3065
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should be on top level of the ISO, and list files added
specifically to the ISO. If there's anything inherited from one some
variant, the files will be listed in metadata in the variant directory.
JIRA: COMPOSE-3069
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The initial implementation is very coarse. It enables it for all
variants and all modules. That is not always wanted.
With this patch, the config file has to explicitly list the devel
modules for each variant that should have it. The variant must be
configured also to include the non-devel module (but the module may be
in lookaside so it won't be included).
We now include module metadata in the internal lookaside repo, so that
this whole thing works if one variant is built on top of another.
JIRA: COMPOSE-3034
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of having one giant command line, write the long list into a
file and pass that to the solver.
The items on the input list are sorted for easy processing.
JIRA: COMPOSE-3012
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The function loads existing treeinfo, removes reference to boot.iso and
adds [media] section. This is the basic tweak that should happen for all
ISOs. Additional changes depend on the actual content.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The module has same S:V:C, but the name is suffixed with `-devel`. The
module should contain all packages from the module koji tag that were
not included in the actual module.
The devel module has the same dependencies as the regular module, but
also additionally depends on the original module. The API and profiles
are cleared in the new module.
In the metadata it shows the same koji tag.
The test if package goes to the module is refactored to a function to
make work with the negated case a bit easier.
There may be unneeded multilib packages in the -devel module, because
there might be buildtime dependencies between things that we don't see.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Clear API and profiles
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the same tag is queried with the same event, Pungi can cache the
response and call the API again. Particularly for small composes this
can save up significant amount of time.
Merges: https://pagure.io/pungi/pull-request/1022
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
This will be used by Anaconda to consume multiple repos for
installation.
JIRA: RCM-36970
JIRA: COMPOSE-2753
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Fixup
This is an optimization for Yum. DNF does not care at all.
The behaviour is configurable, but the default depends on gather
backend, as that is what users should be using to consume the packages
from the repo.
Fixes: https://pagure.io/pungi/issue/951
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should indicate that it's a modular variant, but there is no
modular content yet. We don't want to treat that as Everything.
The end result will be an empty repository.
Fixes: https://pagure.io/pungi/issue/871
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If we have a package set for the variant (which happens if there are
modules), include a list of all NEVRAs in the pungi kickstart.
This can be used to make sure only packages from correct tag get into
the compose. If two packages with same name but different version get
into the compose, this can help get even older version into a particular
variant.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
With this patch the gather_source option is no longer used. Instead, all
sources are always used. If they return at least some input packages,
then a configured method is used and the returned lists of packages from
all sources are merged.
The method used for gathering can be configured for each variant and
gather source separately.
Additional packages are only added to the comps source.
Each gathering step is logged separately. All the logs are preserved for
later inspection.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should make all tests pass on both Python 2 and Python 3.
Unittest2 is required on Py 2.6 and Py 3.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
In many cases we need to open files as binary to avoid errors on Py3
about writing binary data to file opened in text mode.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Alternatively the call to repoclosure can be turned off. This is
customizable per variant and architecture.
Fixes: https://pagure.io/pungi/issue/676
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of iterating over the images metadata and appending the checksum
to relevant files immediately, we should store them and write only once.
This avoid an issue when the same image is mentioned in the metadata
multiple times. This happens for source images that are listed under
each binary arch.
The unified isos script is updated to use the exact same logic and code.
This also uncovered a problem with the metadata for debuginfo unified
isos: their paths in metadata were incorrect, which lead to missing
checksums.
Fixes: https://pagure.io/pungi/issue/667
Fixes: https://pagure.io/pungi/issue/668
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
