This patch reuses the existing createrepo_num_threads options to limit
maximum number of parallel createrepo processes.
Fixes: https://pagure.io/pungi/issue/955
JIRA: COMPOSE-2575
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Starting tests just to run mock functions slows the tests down for no
good reason. Let's instead mock the runner and run the dummy tasks
serially.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Higher protocols should be more efficient in terms of performance and
storage size. Since we don't really care about interoperability with
different python version, we can safely go to the highest version.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Since there can be multiple tags, the check must be done once for all of
them at the same time. Otherwise any module found only in some and not
all tags would raise this error.
The code builds a set of all existing patterns and then removes items
from it. If there is something left once all tags are processed, it
means such patterns were not matched by anything.
JIRA: COMPOSE-3609
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
For scm dict resolving the return value should be git ref (or source
branch for offline mode).
JIRA: COMPOSE-3614
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the package set is loaded from a pickle, it will not contain this
key, and thus it can not be again written down into a new pickle. While
this is not a common use case, it's still better not to crash in that
case.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We need to update the source modulemd to match what we want. For most
modules the name will be correct already, but in case of devel modules
the original name will still be present there.
Fixes: COMPOSE-3596
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch fixes some issues with inconsistent use of whitespace. It
only modifies lines that do not contain any code to not break git blame
too much.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Clone the main config, all referenced parts and included variants or
comps files.
This requires a new argument: destination directory to which to write
the copied files.
JIRA: COMPOSE-3289
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is fairly similar to a package only being built for particular
arches.
Fixes: https://pagure.io/pungi/issue/1198
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It's created in runroot, the owner and permissions should be updated so
that anyone can read it and user running compose can delete the files.
JIRA: COMPOSE-3545
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The directory with logs should have updated owner and permission as
well as the actual output. This patch lays foundation for that by
allowing multiple paths to be specified.
JIRA: COMPOSE-3545
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When using lorax, this directory will contain subdirectories for
individual variants. Those are created in runroot and their ownership
and permissions are fixed there. However the top level dir was only
created and not updated.
JIRA: COMPOSE-3545
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Add Hygon CPU vendor ID("HygonGenuine") detection and optimize arch to
"athlon" in x86 and "amd64" in x86_64.
Merges: https://pagure.io/pungi/pull-request/1196
Signed-off-by: fanjinke <fanjinke@hygon.cn>
This reverts commit ac15f21135.
It is still needed if nodeps gather method is used. It simply returns
all packages listed in all modules.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1708661
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
For example when a variant does not have any debuginfo packages, the
metadata will contain path to a repository, but it will be missing the
package path despite the (empty) directory being present on the
filesystem.
We should really only skip missing directories.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of validating both variants exist, let's just check the
existence of only the variant that is being used as a lookaside.
If the configuration says Foo depends on Bar, the error is reported only
if Foo exists but Bar does not. Any other situation is silently ignored.
JIRA: COMPOSE-3393
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of guessing based on a good enough name, we can use
`sys.executable` and get path to current python.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The script is run as a standalone process. Anything printed to stderr
breaks the test.
Since Python 3.8, we are getting warnings about invalid escape sequences
in some modules that are imported but not owned by us.
This patch should silence the warnings.
Relates: https://bugzilla.redhat.com/show_bug.cgi?id=1698514
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
- Pass the runroot_tag to init command in OpenSSH Runroot method.
This is needed for the init command as a source for initial packages
for the buildroot.
- Rename the "runroot_ssh_init_command" to "runroot_ssh_init_template"
to make it consistent with the rest of "runroot_ssh_*" options.
- Add missing "runroot_ssh_*" options to checks.py.
- Use chmod/chown to `output_dir` in OpenSSH Runroot method the same way
as it is used in Koji runroot method to make the runroot output readable
for Pungi user.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
Instead of running the copy from the main script explicitly, make it
part of the thread.
This should make things very slightly faster, and the code is much
simpler.
Fixes: https://pagure.io/pungi/issue/959
JIRA: COMPOSE-2604
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should be mostly IO intensive operation, so running multiple
commands at the same time should save us a tiny bit of time.
Fixes: https://pagure.io/pungi/issue/981
JIRA: COMPOSE-2646
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the repo contains the same name under multiple directories, make the
resolving work by filtering only to refs/heads and refs/tags.
Fixes: https://pagure.io/pungi/issue/1180
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If user configures branch as explicit None, we want to default to HEAD
(which is most likely refs/heads/master in git).
The original code was getting branch as None, which lead to wrong
resolver being used and the repo url being used as branch.
Fixes: https://pagure.io/pungi/issue/1181
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The prefix checking only works if there are no streams that would share
prefixes. Let's instead check the value as a whole. There is extra
complexity from the fact that version and context may not be specified.
The stream as specified in input is processed to replace dashes (`-`)
with underscores (`_`) to match how the builds are imported into Koji.
JIRA: COMPOSE-3547
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
For modular tags we only include packages that are included in the
module. Originally the filter was taking package names from
rpms.artifacts section of MMD.
This however does not work correctly, as there can easily be module
which lists foo.src but does not want to include foo.x86_64 or other
arches.
This patch fixes this particular problem by included arch in the set of
packages to be kept.
JIRA: COMPOSE-3543
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The scheme for generating versions has changed multiple times. MBS is
careful to only modify them so that they always compare correctly. This
only works though if the versions are treated as numbers.
This should be safe in that non-numbers should never be encountered as
module version. Libmodulemd internally stores the version as int (or
some version of int).
JIRA: COMPOSE-3540
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Hardlink command can be installed in /usr/sbin, where it is not visible
to non-priviledged users. They can still run it, but don't have it in
their PATH.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Even if we want to break hardlinks from Koji volume, there may be files
that can be hardlinked and save some space on the media.
JIRA: COMPOSE-3482
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Python 3.8 no longer sorts attributes automatically, which is causing
some of the tests to fail. The easiest fix is to update the code to make
sure sorting is in place.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1698514
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This adds few new config options which are well described in the
configuration documentation. Please refer to it for more information.
Merges: https://pagure.io/pungi/pull-request/1170
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
Embedding the registry configuration into OSBS config itself is
simple, but makes it impossible to reuse the same configuration for
multiple different composes.
A nice example is a nightly pushing images to a testing registry, and
production compose building the same images but pushing to staging
location. The original design requires duplication of all the
configuration just because registries are different.
With this option, the push information is stored in a separate option as
a mapping from NVR patterns to arbitrary data. The patterns are used to
match finished builds to registry.
The old configuration is marked as deprecated in code and will
eventually be removed. The deprecation handling in config validation
does not allow emitting warnings for nested values.
JIRA: COMPOSE-3394
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
