Clone the main config, all referenced parts and included variants or
comps files.
This requires a new argument: destination directory to which to write
the copied files.
JIRA: COMPOSE-3289
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is fairly similar to a package only being built for particular
arches.
Fixes: https://pagure.io/pungi/issue/1198
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It's created in runroot, the owner and permissions should be updated so
that anyone can read it and user running compose can delete the files.
JIRA: COMPOSE-3545
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The directory with logs should have updated owner and permission as
well as the actual output. This patch lays foundation for that by
allowing multiple paths to be specified.
JIRA: COMPOSE-3545
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When using lorax, this directory will contain subdirectories for
individual variants. Those are created in runroot and their ownership
and permissions are fixed there. However the top level dir was only
created and not updated.
JIRA: COMPOSE-3545
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Add Hygon CPU vendor ID("HygonGenuine") detection and optimize arch to
"athlon" in x86 and "amd64" in x86_64.
Merges: https://pagure.io/pungi/pull-request/1196
Signed-off-by: fanjinke <fanjinke@hygon.cn>
This reverts commit ac15f21135.
It is still needed if nodeps gather method is used. It simply returns
all packages listed in all modules.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1708661
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
For example when a variant does not have any debuginfo packages, the
metadata will contain path to a repository, but it will be missing the
package path despite the (empty) directory being present on the
filesystem.
We should really only skip missing directories.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of validating both variants exist, let's just check the
existence of only the variant that is being used as a lookaside.
If the configuration says Foo depends on Bar, the error is reported only
if Foo exists but Bar does not. Any other situation is silently ignored.
JIRA: COMPOSE-3393
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of guessing based on a good enough name, we can use
`sys.executable` and get path to current python.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The script is run as a standalone process. Anything printed to stderr
breaks the test.
Since Python 3.8, we are getting warnings about invalid escape sequences
in some modules that are imported but not owned by us.
This patch should silence the warnings.
Relates: https://bugzilla.redhat.com/show_bug.cgi?id=1698514
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
- Pass the runroot_tag to init command in OpenSSH Runroot method.
This is needed for the init command as a source for initial packages
for the buildroot.
- Rename the "runroot_ssh_init_command" to "runroot_ssh_init_template"
to make it consistent with the rest of "runroot_ssh_*" options.
- Add missing "runroot_ssh_*" options to checks.py.
- Use chmod/chown to `output_dir` in OpenSSH Runroot method the same way
as it is used in Koji runroot method to make the runroot output readable
for Pungi user.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
Instead of running the copy from the main script explicitly, make it
part of the thread.
This should make things very slightly faster, and the code is much
simpler.
Fixes: https://pagure.io/pungi/issue/959
JIRA: COMPOSE-2604
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should be mostly IO intensive operation, so running multiple
commands at the same time should save us a tiny bit of time.
Fixes: https://pagure.io/pungi/issue/981
JIRA: COMPOSE-2646
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the repo contains the same name under multiple directories, make the
resolving work by filtering only to refs/heads and refs/tags.
Fixes: https://pagure.io/pungi/issue/1180
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If user configures branch as explicit None, we want to default to HEAD
(which is most likely refs/heads/master in git).
The original code was getting branch as None, which lead to wrong
resolver being used and the repo url being used as branch.
Fixes: https://pagure.io/pungi/issue/1181
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The prefix checking only works if there are no streams that would share
prefixes. Let's instead check the value as a whole. There is extra
complexity from the fact that version and context may not be specified.
The stream as specified in input is processed to replace dashes (`-`)
with underscores (`_`) to match how the builds are imported into Koji.
JIRA: COMPOSE-3547
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
For modular tags we only include packages that are included in the
module. Originally the filter was taking package names from
rpms.artifacts section of MMD.
This however does not work correctly, as there can easily be module
which lists foo.src but does not want to include foo.x86_64 or other
arches.
This patch fixes this particular problem by included arch in the set of
packages to be kept.
JIRA: COMPOSE-3543
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The scheme for generating versions has changed multiple times. MBS is
careful to only modify them so that they always compare correctly. This
only works though if the versions are treated as numbers.
This should be safe in that non-numbers should never be encountered as
module version. Libmodulemd internally stores the version as int (or
some version of int).
JIRA: COMPOSE-3540
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Hardlink command can be installed in /usr/sbin, where it is not visible
to non-priviledged users. They can still run it, but don't have it in
their PATH.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Even if we want to break hardlinks from Koji volume, there may be files
that can be hardlinked and save some space on the media.
JIRA: COMPOSE-3482
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Python 3.8 no longer sorts attributes automatically, which is causing
some of the tests to fail. The easiest fix is to update the code to make
sure sorting is in place.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1698514
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This adds few new config options which are well described in the
configuration documentation. Please refer to it for more information.
Merges: https://pagure.io/pungi/pull-request/1170
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
Embedding the registry configuration into OSBS config itself is
simple, but makes it impossible to reuse the same configuration for
multiple different composes.
A nice example is a nightly pushing images to a testing registry, and
production compose building the same images but pushing to staging
location. The original design requires duplication of all the
configuration just because registries are different.
With this option, the push information is stored in a separate option as
a mapping from NVR patterns to arbitrary data. The patterns are used to
match finished builds to registry.
The old configuration is marked as deprecated in code and will
eventually be removed. The deprecation handling in config validation
does not allow emitting warnings for nested values.
JIRA: COMPOSE-3394
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Lockfile and dict.sorted are only used in pungi.gather module, which
does not work with Python 3 due to dependency on yum. There's no point
in always pulling in these two libraries there.
This change fixes issues with python dependencies generator that puts
python3.6dist(lockfile) into RPM requires.
For builds on Python 2.6 these libraries are actually needed, but there
are no generators on RHEL 6. So this change can only break development
on RHEL 6. And if someone needs to do that, they have bigger problems...
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This can be used to fill particular values to a predefined template
without editing any file disk.
JIRA: COMPOSE-3316
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The parts in multi compose are now named reasonably.
The package set repos are selected only once and not per variant. We
can't use repos with packages for specific arch only as that would
require more transformations during generating test data.
Lookasides for ResilientStorage part are added to point to Server part.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Use a method to add repos that will apply $arch and $basearch
substitution automatically. Yum backend only applies $basearch, so if
compatibility is needed, that one should be used.
Drop code for handling mirrorlist, since Pungi does not ever use it, and
being used externally is not really supported.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Tests are not passing in Fedora builds with this code, since it was
deleted in DNF. I think it's not actually doing anything, so we should
be able to drop it as well.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When a phase is started or stopped, add a line to the to output. This
should help users keep track of what is happening in case the part takes
a long time to run.
JIRA: COMPOSE-3287
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
A module build can create packages that are tagged in the content tag,
but should not be included in the module. Originally Pungi didn't know
what exactly the module contains and so it needed to apply filters to
exclude stuff that was definitely out.
With getting the final MMD from Koji, we can actually make this a bit
more strict by only keeping packages that we know we need.
When processing each content tag, we can put into package set only
packages that are included in some module using that tag. This should
work with -devel modules as well. Both the regular and -devel modules
will contribute to the set and thus all packages will go to the package
set.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This source does not really return anything useful. It was necessary to
process the source modulemd to fill in list of RPMs. Since we now get
the final files from Koji, this is not needed anymore and the source can
be dropped.
This change requires a lot of tweaks for test.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The file we get from Koji has all important bits already filled in.
There is no need to add anything.
This patch also stops filtering the artifacts to only contain packages
that are actually in the repo. Thus debuginfo and source packages will
appear there.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of loading the "source" modulemd, always get the final file for
each architecture from Koji.
Logging the downloaded files locally is no longer necessary, when
debugging a problem we can find the files in the respective builds in
Koji.
Tests are updated to work with new code, and an obsolete test is
removed.
JIRA: COMPOSE-3147
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
