Originally the list of solvables for fus was growing with each iteration
and nothing was ever removed. That later changed so that fus iterations
are only done on newly added stuff. It's great for performance, but
means that the last log is not a superset of all others.
To get all dependency problems we need to look into all log files, not
just the last one.
JIRA: COMPOSE-3964
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When running repoclosure as root user, it will use other dir instead of
the one returned by getCacheDir().
For yum, with --tempcache option could let the cache dir returned by
getCacheDir() always be used.
For dnf, there's no such an option and we have to handle it specially.
JIRA: COMPOSE-3922
Signed-off-by: Haibo Lin <hlin@redhat.com>
When probing lookasides for platform definition, we need to make sure it
works for repos specified as HTTP urls. Createrepo doesn't seem to
automatically download the repodata, so we have to help it.
JIRA: COMPOSE-3958
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Each depsolved tree will be using its own cache for fus. This should
still allow for faster loading of metadata after first iteration, but
should prevent errors from using cached files meant for another variant
or architecture. The cache is deleted after the last iteration.
JIRA: COMPOSE-3959
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is basically collecting all individual extra_files.json and putting
their content into a single location in
compose/metadata/extra_files.json. The file format is part of productmd
1.23.
JIRA: COMPOSE-3831
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
A lorax template used for the ostree-installer might need an additional
package dependency (e.g., flatpak to embed a flatpak repository) - add
a config key 'extra_runroot_pkgs' to the ostree installer configuration
to allow supplementing the set of packages installed into the runroot.
Signed-off-by: Owen W. Taylor <otaylor@fishsoup.net>
It was needed to provide assertItemsEqual method. Starting with Python
3.2, there's assertCountEqual that does the same thing. Six provides a
helper that will dispatch to the existing method. With this change,
unittest 2 is only needed on Python 2.6 to backport the method.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Until now, the behaviour was that all debuginfo from a build would be
included if at least one package with the same arch was included.
This resulted in many debuginfo packages being included even though
their corresponding package was not present.
With this patch, we will always pull in debugsource, but foo-debuginfo
will only be included if foo is included for the same arch. As a
consequence, it is necessary to resolve dependencies of debuginfo
packages. There are cases where foo-debuginfo needs foo-debuginfo-common
for example.
This change means that DNF and YUM backends are no longer identical in
the output. The tests where this is demonstrated are duplicated and
their results are modified.
JIRA: COMPOSE-3823
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Make dummy-bash -> dummy-glibc dependency require archful. This avoids
potential race condition where order of dependency processing can result in
different packages being pulled in. The tests where this could happen are
updated.
Make dummy-glibc-debuginfo depend on dummy-glibc-debuginfo-common.
The filenames for the repo no longer include hash, and sqlite databases are not
generated.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should make it possible to only import the library only when it's
really needed.
DNF does not work with libmodulemd v2. If we import libmodulemd2 and
then dnf, the program will just hang forever. We only need DNF in
pungi-gather, where libmodulemd is not needed, and also where we do need
libmodulemd, we don't have DNF.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Add a configuration option to enable skipping some modules found in the
configured tag.
Fixes: https://pagure.io/pungi/issue/1260
JIRA: COMPOSE-3794
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch adds a new config option. This is expected to be a name of
subdirectory in the repo with module defaults. If supplied, overrides
from that location are loaded every time defaults are loaded.
This raises the minimal required version of libmodulemd to 2.8.0
JIRA: COMPOSE-3828
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There will be a new log file logs/global/excluding-arch.global.log
Fixes: https://pagure.io/pungi/issue/1251
Signed-off-by: Haibo Lin <hlin@redhat.com>
This was a workaround to make some packages from the global repo
invisible for depsolving. This is now handled by packages being in
different repos. We can select which repos are enabled at which point.
This achieves the same result, but much faster.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The repo was used to speed up creating lookaside repo from a variant.
This uses a similar approach as createrepo phase: selecting the last
available package set and using that data.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Simply use all existing package set repos as input for the runroot
task. The command line gets a bit long, but the actual behaviour should
remain the same.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is no longer a single repo with all packages. This means that the
metadata has to be loaded from another location.
When taking packages from Koji, we can assume that the non-modular
package tag will be processed last. The repo for this tag will be used.
This has better chance of being useful than using a random module.
For repo sources, there is only one package set anyway, so this change
makes no difference.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
With this patch, there should be a separate package set for each tag
that is consumed.
Generally each module will create a separate package set, with the
exception of -devel modules that will be in the same set as their
non-devel version.
Variants no longer need to keep their own package set objects. Instead
they now include a set of package set names that should be used for the
variant. This can replace the whitelist mechanism of deps gather method.
JIRA: COMPOSE-3620
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Once a package set repo is written to disk, let's use this object that
connects the repository path with the mapping of packages.
This change also makes it explicit where the dependency on package set
repos are. In the original codebase, any part of code could generate a
path to the repo, even if that repo has not yet been written.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This name will serve as an identifier for the group of packages.
For Koji package sets, it should the name of the tag from which the
packages come. For package sets based on repos a dummy constant name is
used.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This opens up a path to having multiple package sets in the compose. The
pkgset phase now creates a list of them (although at this time there is
always a single item in that list).
Any consumer of the package sets objects is updated to handle a list.
Generally this means an extra loop.
JIRA: COMPOSE-3620
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
For 'yum' backend, only cache dirs following repoclosure-$COMPOSE_ID-$variant.$arch
name convention are created, e.g. repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64
But for 'dnf' backend, the dir name looks like
repoclosure-$COMPOSE_ID-$variant.$arch-$suffix and there are other files
created, e.g.
repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64-df9fe164317e314e
repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64-filenames.solvx
repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64.solv
JIRA: COMPOSE-2565
Signed-off-by: Haibo Lin <hlin@redhat.com>
Both pkgset sources use the same logic to create per-arch repos. There
is no reason to have that code in both places.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The data parsed from variants.xml uses a different format that what we
added in `_add_module_to_variant`. This leads to crashes later.
JIRA: COMPOSE-3746
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This was already discouraged to not be used, and is a bad idea in
current setup anyway. Removing this can simplify the code.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Sometimes it's practical not just warn when ISO is larger than expected,
but to also abort the compose.
JIRA: COMPOSE-3658
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We can avoid parsing source modulemd information since we can get the
same information from the Koji build info.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Historically each variant had a list of modules. This is no longer
needed and can be dropped. We can also stop logging the modulemd since
we know it was retrieved from Koji and not modified locally.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
On Python 3 it is not possible to sort str and None or RpmWrapper.
First convert everything to strings and then sort it. The sorting is
really to simplify diffing the files, so exact order does not have to be
preserved.
Fixes: https://pagure.io/pungi/issue/1227
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This also cleans up the runroot method detection code to not rely on the
now removed option.
JIRA: COMPOSE-2634
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the package set repo contains any modular package, the module
metadata is added there as well.
This is needed to accomodate change in DNF that refuses to work with
repo with modular packages if the metadata is not there. This DNF change
can cause issues in buildinstall phase.
Related: https://bugzilla.redhat.com/show_bug.cgi?id=1623128
The hybrid solver is modified to not create a separate repo with the
module metadata anymore, since it will be available in the repo with
packages. This also allows us to drop code to look into lookaside repos.
We still need to iterate over local modules in order to find out what
platform should be used.
JIRA: COMPOSE-3621
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the validation or dumping script is given some options, they should
only be removed if they are not valid. We have to remove the invalid
ones, otherwise that would cause a warning about unknown options.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When trying to validate a template that should later be filled in with
`pungi-config-dump`, there will be errors about undefined variables.
These are meant to be set when the template is populated.
This patch adds support for `-e`, `--define` argument to the validation
script that can be used to suppress these errors.
Alternatively a JSON file is read from the directory with config file
that can contain values for the variables.
The `--define` option is changed in both validation and dumping to allow
empty string as an accepted value.
JIRA: COMPOSE-3599
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch reuses the existing createrepo_num_threads options to limit
maximum number of parallel createrepo processes.
Fixes: https://pagure.io/pungi/issue/955
JIRA: COMPOSE-2575
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Starting tests just to run mock functions slows the tests down for no
good reason. Let's instead mock the runner and run the dummy tasks
serially.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Higher protocols should be more efficient in terms of performance and
storage size. Since we don't really care about interoperability with
different python version, we can safely go to the highest version.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Since there can be multiple tags, the check must be done once for all of
them at the same time. Otherwise any module found only in some and not
all tags would raise this error.
The code builds a set of all existing patterns and then removes items
from it. If there is something left once all tags are processed, it
means such patterns were not matched by anything.
JIRA: COMPOSE-3609
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
For scm dict resolving the return value should be git ref (or source
branch for offline mode).
JIRA: COMPOSE-3614
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We need to update the source modulemd to match what we want. For most
modules the name will be correct already, but in case of devel modules
the original name will still be present there.
Fixes: COMPOSE-3596
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch fixes some issues with inconsistent use of whitespace. It
only modifies lines that do not contain any code to not break git blame
too much.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It's created in runroot, the owner and permissions should be updated so
that anyone can read it and user running compose can delete the files.
JIRA: COMPOSE-3545
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The directory with logs should have updated owner and permission as
well as the actual output. This patch lays foundation for that by
allowing multiple paths to be specified.
JIRA: COMPOSE-3545
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This reverts commit ac15f21135.
It is still needed if nodeps gather method is used. It simply returns
all packages listed in all modules.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1708661
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of validating both variants exist, let's just check the
existence of only the variant that is being used as a lookaside.
If the configuration says Foo depends on Bar, the error is reported only
if Foo exists but Bar does not. Any other situation is silently ignored.
JIRA: COMPOSE-3393
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of guessing based on a good enough name, we can use
`sys.executable` and get path to current python.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The script is run as a standalone process. Anything printed to stderr
breaks the test.
Since Python 3.8, we are getting warnings about invalid escape sequences
in some modules that are imported but not owned by us.
This patch should silence the warnings.
Relates: https://bugzilla.redhat.com/show_bug.cgi?id=1698514
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
- Pass the runroot_tag to init command in OpenSSH Runroot method.
This is needed for the init command as a source for initial packages
for the buildroot.
- Rename the "runroot_ssh_init_command" to "runroot_ssh_init_template"
to make it consistent with the rest of "runroot_ssh_*" options.
- Add missing "runroot_ssh_*" options to checks.py.
- Use chmod/chown to `output_dir` in OpenSSH Runroot method the same way
as it is used in Koji runroot method to make the runroot output readable
for Pungi user.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
Instead of running the copy from the main script explicitly, make it
part of the thread.
This should make things very slightly faster, and the code is much
simpler.
Fixes: https://pagure.io/pungi/issue/959
JIRA: COMPOSE-2604
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the repo contains the same name under multiple directories, make the
resolving work by filtering only to refs/heads and refs/tags.
Fixes: https://pagure.io/pungi/issue/1180
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The prefix checking only works if there are no streams that would share
prefixes. Let's instead check the value as a whole. There is extra
complexity from the fact that version and context may not be specified.
The stream as specified in input is processed to replace dashes (`-`)
with underscores (`_`) to match how the builds are imported into Koji.
JIRA: COMPOSE-3547
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This adds few new config options which are well described in the
configuration documentation. Please refer to it for more information.
Merges: https://pagure.io/pungi/pull-request/1170
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
Embedding the registry configuration into OSBS config itself is
simple, but makes it impossible to reuse the same configuration for
multiple different composes.
A nice example is a nightly pushing images to a testing registry, and
production compose building the same images but pushing to staging
location. The original design requires duplication of all the
configuration just because registries are different.
With this option, the push information is stored in a separate option as
a mapping from NVR patterns to arbitrary data. The patterns are used to
match finished builds to registry.
The old configuration is marked as deprecated in code and will
eventually be removed. The deprecation handling in config validation
does not allow emitting warnings for nested values.
JIRA: COMPOSE-3394
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The parts in multi compose are now named reasonably.
The package set repos are selected only once and not per variant. We
can't use repos with packages for specific arch only as that would
require more transformations during generating test data.
Lookasides for ResilientStorage part are added to point to Server part.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Tests are not passing in Fedora builds with this code, since it was
deleted in DNF. I think it's not actually doing anything, so we should
be able to drop it as well.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
A module build can create packages that are tagged in the content tag,
but should not be included in the module. Originally Pungi didn't know
what exactly the module contains and so it needed to apply filters to
exclude stuff that was definitely out.
With getting the final MMD from Koji, we can actually make this a bit
more strict by only keeping packages that we know we need.
When processing each content tag, we can put into package set only
packages that are included in some module using that tag. This should
work with -devel modules as well. Both the regular and -devel modules
will contribute to the set and thus all packages will go to the package
set.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This source does not really return anything useful. It was necessary to
process the source modulemd to fill in list of RPMs. Since we now get
the final files from Koji, this is not needed anymore and the source can
be dropped.
This change requires a lot of tweaks for test.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The file we get from Koji has all important bits already filled in.
There is no need to add anything.
This patch also stops filtering the artifacts to only contain packages
that are actually in the repo. Thus debuginfo and source packages will
appear there.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of loading the "source" modulemd, always get the final file for
each architecture from Koji.
Logging the downloaded files locally is no longer necessary, when
debugging a problem we can find the files in the respective builds in
Koji.
Tests are updated to work with new code, and an obsolete test is
removed.
JIRA: COMPOSE-3147
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Sometimes the release version can be more specific than what should be
exposed to users of the boot iso.
JIRA: COMPOSE-3295
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This adds new `Runroot` class and new `runroot_method` option which makes
it possible to choose between two currently available runroot methods:
- Local
- Koji
The main goal of this commit is to make it possible to add new runroot
methods in the future and this is the first step in that direction.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
