For 'yum' backend, only cache dirs following repoclosure-$COMPOSE_ID-$variant.$arch
name convention are created, e.g. repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64
But for 'dnf' backend, the dir name looks like
repoclosure-$COMPOSE_ID-$variant.$arch-$suffix and there are other files
created, e.g.
repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64-df9fe164317e314e
repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64-filenames.solvx
repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64.solv
JIRA: COMPOSE-2565
Signed-off-by: Haibo Lin <hlin@redhat.com>
If --lookaside option passed to repoclosure command, extra cache dir
will be created and it should be deleted too.
JIRA: COMPOSE-2565
Signed-off-by: Haibo Lin <hlin@redhat.com>
Both pkgset sources use the same logic to create per-arch repos. There
is no reason to have that code in both places.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The data parsed from variants.xml uses a different format that what we
added in `_add_module_to_variant`. This leads to crashes later.
JIRA: COMPOSE-3746
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This was already discouraged to not be used, and is a bad idea in
current setup anyway. Removing this can simplify the code.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of testing its return value. Future version of libmodulemd will
raise an exception instead of returning empty data.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When libmodulemd is available, but the compose contains no modules, we
don't want to add the modular metadata anywhere. This patch actually
avoids a crash as `ModuleIndex.dump_to_string()` will return `None` if
there is no data.
JIRA: COMPOSE-3662
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Sometimes it's practical not just warn when ISO is larger than expected,
but to also abort the compose.
JIRA: COMPOSE-3658
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We can avoid parsing source modulemd information since we can get the
same information from the Koji build info.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Historically each variant had a list of modules. This is no longer
needed and can be dropped. We can also stop logging the modulemd since
we know it was retrieved from Koji and not modified locally.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
On Python 3 it is not possible to sort str and None or RpmWrapper.
First convert everything to strings and then sort it. The sorting is
really to simplify diffing the files, so exact order does not have to be
preserved.
Fixes: https://pagure.io/pungi/issue/1227
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This also cleans up the runroot method detection code to not rely on the
now removed option.
JIRA: COMPOSE-2634
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the package set repo contains any modular package, the module
metadata is added there as well.
This is needed to accomodate change in DNF that refuses to work with
repo with modular packages if the metadata is not there. This DNF change
can cause issues in buildinstall phase.
Related: https://bugzilla.redhat.com/show_bug.cgi?id=1623128
The hybrid solver is modified to not create a separate repo with the
module metadata anymore, since it will be available in the repo with
packages. This also allows us to drop code to look into lookaside repos.
We still need to iterate over local modules in order to find out what
platform should be used.
JIRA: COMPOSE-3621
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch reuses the existing createrepo_num_threads options to limit
maximum number of parallel createrepo processes.
Fixes: https://pagure.io/pungi/issue/955
JIRA: COMPOSE-2575
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Higher protocols should be more efficient in terms of performance and
storage size. Since we don't really care about interoperability with
different python version, we can safely go to the highest version.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Since there can be multiple tags, the check must be done once for all of
them at the same time. Otherwise any module found only in some and not
all tags would raise this error.
The code builds a set of all existing patterns and then removes items
from it. If there is something left once all tags are processed, it
means such patterns were not matched by anything.
JIRA: COMPOSE-3609
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the package set is loaded from a pickle, it will not contain this
key, and thus it can not be again written down into a new pickle. While
this is not a common use case, it's still better not to crash in that
case.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We need to update the source modulemd to match what we want. For most
modules the name will be correct already, but in case of devel modules
the original name will still be present there.
Fixes: COMPOSE-3596
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is fairly similar to a package only being built for particular
arches.
Fixes: https://pagure.io/pungi/issue/1198
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It's created in runroot, the owner and permissions should be updated so
that anyone can read it and user running compose can delete the files.
JIRA: COMPOSE-3545
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When using lorax, this directory will contain subdirectories for
individual variants. Those are created in runroot and their ownership
and permissions are fixed there. However the top level dir was only
created and not updated.
JIRA: COMPOSE-3545
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This reverts commit ac15f21135.
It is still needed if nodeps gather method is used. It simply returns
all packages listed in all modules.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1708661
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of validating both variants exist, let's just check the
existence of only the variant that is being used as a lookaside.
If the configuration says Foo depends on Bar, the error is reported only
if Foo exists but Bar does not. Any other situation is silently ignored.
JIRA: COMPOSE-3393
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of running the copy from the main script explicitly, make it
part of the thread.
This should make things very slightly faster, and the code is much
simpler.
Fixes: https://pagure.io/pungi/issue/959
JIRA: COMPOSE-2604
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should be mostly IO intensive operation, so running multiple
commands at the same time should save us a tiny bit of time.
Fixes: https://pagure.io/pungi/issue/981
JIRA: COMPOSE-2646
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The prefix checking only works if there are no streams that would share
prefixes. Let's instead check the value as a whole. There is extra
complexity from the fact that version and context may not be specified.
The stream as specified in input is processed to replace dashes (`-`)
with underscores (`_`) to match how the builds are imported into Koji.
JIRA: COMPOSE-3547
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
For modular tags we only include packages that are included in the
module. Originally the filter was taking package names from
rpms.artifacts section of MMD.
This however does not work correctly, as there can easily be module
which lists foo.src but does not want to include foo.x86_64 or other
arches.
This patch fixes this particular problem by included arch in the set of
packages to be kept.
JIRA: COMPOSE-3543
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The scheme for generating versions has changed multiple times. MBS is
careful to only modify them so that they always compare correctly. This
only works though if the versions are treated as numbers.
This should be safe in that non-numbers should never be encountered as
module version. Libmodulemd internally stores the version as int (or
some version of int).
JIRA: COMPOSE-3540
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Hardlink command can be installed in /usr/sbin, where it is not visible
to non-priviledged users. They can still run it, but don't have it in
their PATH.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Even if we want to break hardlinks from Koji volume, there may be files
that can be hardlinked and save some space on the media.
JIRA: COMPOSE-3482
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
