Since there can be multiple tags, the check must be done once for all of
them at the same time. Otherwise any module found only in some and not
all tags would raise this error.
The code builds a set of all existing patterns and then removes items
from it. If there is something left once all tags are processed, it
means such patterns were not matched by anything.
JIRA: COMPOSE-3609
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
For scm dict resolving the return value should be git ref (or source
branch for offline mode).
JIRA: COMPOSE-3614
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We need to update the source modulemd to match what we want. For most
modules the name will be correct already, but in case of devel modules
the original name will still be present there.
Fixes: COMPOSE-3596
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch fixes some issues with inconsistent use of whitespace. It
only modifies lines that do not contain any code to not break git blame
too much.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It's created in runroot, the owner and permissions should be updated so
that anyone can read it and user running compose can delete the files.
JIRA: COMPOSE-3545
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The directory with logs should have updated owner and permission as
well as the actual output. This patch lays foundation for that by
allowing multiple paths to be specified.
JIRA: COMPOSE-3545
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This reverts commit ac15f21135.
It is still needed if nodeps gather method is used. It simply returns
all packages listed in all modules.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1708661
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of validating both variants exist, let's just check the
existence of only the variant that is being used as a lookaside.
If the configuration says Foo depends on Bar, the error is reported only
if Foo exists but Bar does not. Any other situation is silently ignored.
JIRA: COMPOSE-3393
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of guessing based on a good enough name, we can use
`sys.executable` and get path to current python.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The script is run as a standalone process. Anything printed to stderr
breaks the test.
Since Python 3.8, we are getting warnings about invalid escape sequences
in some modules that are imported but not owned by us.
This patch should silence the warnings.
Relates: https://bugzilla.redhat.com/show_bug.cgi?id=1698514
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
- Pass the runroot_tag to init command in OpenSSH Runroot method.
This is needed for the init command as a source for initial packages
for the buildroot.
- Rename the "runroot_ssh_init_command" to "runroot_ssh_init_template"
to make it consistent with the rest of "runroot_ssh_*" options.
- Add missing "runroot_ssh_*" options to checks.py.
- Use chmod/chown to `output_dir` in OpenSSH Runroot method the same way
as it is used in Koji runroot method to make the runroot output readable
for Pungi user.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
Instead of running the copy from the main script explicitly, make it
part of the thread.
This should make things very slightly faster, and the code is much
simpler.
Fixes: https://pagure.io/pungi/issue/959
JIRA: COMPOSE-2604
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the repo contains the same name under multiple directories, make the
resolving work by filtering only to refs/heads and refs/tags.
Fixes: https://pagure.io/pungi/issue/1180
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The prefix checking only works if there are no streams that would share
prefixes. Let's instead check the value as a whole. There is extra
complexity from the fact that version and context may not be specified.
The stream as specified in input is processed to replace dashes (`-`)
with underscores (`_`) to match how the builds are imported into Koji.
JIRA: COMPOSE-3547
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This adds few new config options which are well described in the
configuration documentation. Please refer to it for more information.
Merges: https://pagure.io/pungi/pull-request/1170
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
Embedding the registry configuration into OSBS config itself is
simple, but makes it impossible to reuse the same configuration for
multiple different composes.
A nice example is a nightly pushing images to a testing registry, and
production compose building the same images but pushing to staging
location. The original design requires duplication of all the
configuration just because registries are different.
With this option, the push information is stored in a separate option as
a mapping from NVR patterns to arbitrary data. The patterns are used to
match finished builds to registry.
The old configuration is marked as deprecated in code and will
eventually be removed. The deprecation handling in config validation
does not allow emitting warnings for nested values.
JIRA: COMPOSE-3394
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The parts in multi compose are now named reasonably.
The package set repos are selected only once and not per variant. We
can't use repos with packages for specific arch only as that would
require more transformations during generating test data.
Lookasides for ResilientStorage part are added to point to Server part.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Tests are not passing in Fedora builds with this code, since it was
deleted in DNF. I think it's not actually doing anything, so we should
be able to drop it as well.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
A module build can create packages that are tagged in the content tag,
but should not be included in the module. Originally Pungi didn't know
what exactly the module contains and so it needed to apply filters to
exclude stuff that was definitely out.
With getting the final MMD from Koji, we can actually make this a bit
more strict by only keeping packages that we know we need.
When processing each content tag, we can put into package set only
packages that are included in some module using that tag. This should
work with -devel modules as well. Both the regular and -devel modules
will contribute to the set and thus all packages will go to the package
set.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This source does not really return anything useful. It was necessary to
process the source modulemd to fill in list of RPMs. Since we now get
the final files from Koji, this is not needed anymore and the source can
be dropped.
This change requires a lot of tweaks for test.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The file we get from Koji has all important bits already filled in.
There is no need to add anything.
This patch also stops filtering the artifacts to only contain packages
that are actually in the repo. Thus debuginfo and source packages will
appear there.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of loading the "source" modulemd, always get the final file for
each architecture from Koji.
Logging the downloaded files locally is no longer necessary, when
debugging a problem we can find the files in the respective builds in
Koji.
Tests are updated to work with new code, and an obsolete test is
removed.
JIRA: COMPOSE-3147
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Sometimes the release version can be more specific than what should be
exposed to users of the boot iso.
JIRA: COMPOSE-3295
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This adds new `Runroot` class and new `runroot_method` option which makes
it possible to choose between two currently available runroot methods:
- Local
- Koji
The main goal of this commit is to make it possible to add new runroot
methods in the future and this is the first step in that direction.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
Run arbitrary commands before and after the compose.
The example config is updated to generate latest symlink with a
post-compose script. The pre compose script runs always, post compose
runs only if the compose is not doomed.
JIRA: COMPOSE-3288
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the configuration sets keytab path and principal, run kinit with
custom cache file, and delete the file at the end of the run.
JIRA: COMPOSE-3288
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Behavior before this change: debuginfo was added based on source RPMs.
Pungi would get all debuginfo for all architectures that include at
least one binary package. This had a consequence that if foo-x.x86_64
and foo-x.i686 from the same built were included, foo-y-debuginfo.i686
would be included despite foo-y only being present for x86_64.
The patch changes this to work on binary package level: for each
included package `x`, check if there is `x-debuginfo` or
`x-debugsource`, and include them. Packages added in this way have to go
through one iteration of the solver, to account for cases such as
`x-libs-debuginfo` depending on `x-debuginfo` (with only `x` starting
this chain).
To make it slightly faster, the invocation of fus is changed to only
process newly added package in each iteration. It should have no effect
on the results, and subsequent iterations are much faster if they don't
process the same stuff again and again.
JIRA: COMPOSE-3247
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
SOURCE_DATE_EPOCH is preferred over time.time() call in metadata
generation, which makes time.time() mock ineffective. Set
SOURCE_DATE_EPOCH instead of mocking time.time(). This additionally test
if SOURCE_DATE_EPOCH is really used.
Adjust expected output, as SOURCE_DATE_EPOCH is defined without
fractional component.
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Split out a smaller function that receives a git repo URL and ref and
returns the commit id. The caching resolver class is modified to use
this second function if branch is given to it.
The new function checks if the ref received already looks like a commit
ID, and if so it does not attempt to do anything with it.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Before this patch, there were two code paths: either getting the only
the wanted content by calling git-archive, or cloning the repository and
copying the files.
Both these approaches have the downside of not allowing retriving
content from a specific git commit.
The workaround is to create a new empty repo (in the location to which
we cloned previously), fetching the specific commit to there and then
checking it out.
This supports any commit and works identically for any protocol. The
downside is that all files in that commit will be downloaded. It should
be no worse than the git-clone path, but can result in bigger transfers
than git-archive.
Unfortunately this is only supported with git 2.5+. On older version
fetch will fail with no error message (tested with 1.8.3). This can be
used to fall back to full clone. This fallback is clearly suboptimal in
terms of data transfer, but it should work reliably.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Translate the using configured patterns, and give OSBS a repo file with
that URL.
JIRA: COMPOSE-3290
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a repo or install tree is specified as an absolute path on the local
filesystem, we should either translate it using the configured mappings,
or if no mapping matches, it should be return unchanged. A variant name
can not start with a slash, so attempting that translation does not make
much sense.
JIRA: COMPOSE-3290
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The default is the original behaviour. On F30+ a new option should be
added to config to make it work.
Over time as users move to this option (which requires a new enough
version of lorax), the default should be switched and then the option
removed.
Resolves: https://pagure.io/pungi/issue/1126
Merges: https://pagure.io/pungi/pull-request/1128
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This will avoid possible conflicts. The names for parts are chosen by
users, and if we decide to add more options in the future, this can
become handy.
JIRA: COMPOSE-3285
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch allows the configuration to express maximum expected size for
ISOs created in createiso and extra_isos phases. If the image is larger
than this limit, a warning is emitted in test phase. The compose itself
is not affected in any way.
JIRA: COMPOSE-2824
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch does not do any actual pushing. It will only extract data
about push targets from the main configuration and store it together
with exact Koji NVR in a well-defined location, and also send the data
to message bus for another service to handle.
JIRA: COMPOSE-3228
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
For binary packages the filters are handled at the depsolver level.
However sources and debuginfo is added later in the process, so the
filters have to be explicitly applied.
JIRA: COMPOSE-3114
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If there are multiple matching configuration blocks, we should take the
value that is defined in the last one, but only if it actually is there.
That's how it works for other options.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is really an oversight. The list of packages from prepopulate list
should be treated the same as if the packagees were coming from comps or
additional packages.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should tell lorax what arch to use and avoids fragile detection
based on contents of source repo.
It uses the same logic buildinstall phase uses to get the buildarch.
Related: https://pagure.io/teamsilverblue/issue/67
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It may make sense to break a big compose into smaller chunks that can be
done independently. This script allows describing the smaller parts,
runs them with correct dependencies and arranges the result to look like
a single big compose.
All parts use the same koji event, that is either obtained from Koji, or
from command line argument.
JIRA: COMPOSE-2654
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We already make this possible for the ostree installer, but it was
missing from the traditional one. The default behaviour is to let lorax
decide, but if user knows better, they can overwrite in configuration.
JIRA: COMPOSE-3188
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of multiple places handling the same thing duplicating the
logic, it's better to do it once upfront. This allows easy caching of
the results.
Additional advantage of this approach is that the config dump will
include resolved URLs. The original reference will still be available in
the copy of the original config.
JIRA: COMPOSE-3065
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should be on top level of the ISO, and list files added
specifically to the ISO. If there's anything inherited from one some
variant, the files will be listed in metadata in the variant directory.
JIRA: COMPOSE-3069
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The files created in runroot task are owned by root by default (because
that's who is running the processes to create them). Making the results
world readable allows the compose to work, but it still can be difficult
to clean up old composes if they contain random files owned by root.
Fixes: https://pagure.io/pungi/issue/1039
JIRA: COMPOSE-2906
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Currently the `copy_all` function resolves all symlinks and crashes on
broken symlinks. Instead it should copy symlinks as symlinks.
Fixes: https://pagure.io/pungi/issue/1084
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
yaml.load is equally powerful as python pickles, and we don't
need that level of power for the ostree yaml files.
Better safe than sorry.
Signed-off-by: Patrick Uiterwijk <patrick@puiterwijk.org>
They are not always wanted, so let's not include them by default.
There's a new option to include the same files that extra files phases
uses, or alternatively they can be configured specifically and put into
the variant subdirectory.
JIRA: COMPOSE-3084
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Up to now it was possible to change the pattern for all images, but
there are use-cases where different variants might want different names.
For example there could be one main variant that should only have
product name in the ISO filename, but addons should still be marked with
variant name.
JIRA: COMPOSE-3041
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The environment in comps for a variant can refer to groups in parent
variant (either for addons, or because of other configuration). We
should not remove the groups in this case.
This requires changes in two places:
* teaching `comps_filter` about groups that should not be removed
* fixing writing comps so that it does not actually change the data as
well
JIRA: COMPOSE-2612
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The initial implementation is very coarse. It enables it for all
variants and all modules. That is not always wanted.
With this patch, the config file has to explicitly list the devel
modules for each variant that should have it. The variant must be
configured also to include the non-devel module (but the module may be
in lookaside so it won't be included).
We now include module metadata in the internal lookaside repo, so that
this whole thing works if one variant is built on top of another.
JIRA: COMPOSE-3034
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the configuration lists `*` in `additional_packages`, it has a
special meaning. If it's passed to fus directly, it will use it to match
all modules and RPMs that are not masked by a package available in some
default stream. Neither is good. We don't want it to match modules, and
we want even the masked packages.
The fix is to expand the wildcard to a list of NVRs and give that to
fus. It should include the package even if it is masked.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is no repoclosure that correctly understands modules. The best
thing we can offer is the errors reported by the depsolver.
JIRA: COMPOSE-2321
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Stream can contain dash sign and when MBS imports such NSVC to Koji, the
dash is replaced with underscore. The current Pungi code does not
respect that and tries to use the stream from Koji directly, which
results in wrong stream being using in some Pungi internal data.
In this PR, the NSVC is taken from module metadata section of CG Koji
build, which contains real stream including the dashes.
Merges: https://pagure.io/pungi/pull-request/1072
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
Without this we don't have artifacts from all modules when the first
modular variant is solved. That makes some modular packages appear as
bare.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If there are two variants with different module sets, the missing
modulemd from variant B causeing depsolving in variant A to consider the
packages as non-modular. That is wrong.
Related: https://bugzilla.redhat.com/show_bug.cgi?id=1640125
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of having one giant command line, write the long list into a
file and pass that to the solver.
The items on the input list are sorted for easy processing.
JIRA: COMPOSE-3012
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the file is generated, Pungi will modify it at the end of
buildinstall phase. We need to point ISO creation to the modified file,
otherwise the checksum in .treeinfo will be incorrect.
It's sufficient to just update the graft point to correct file, copying
is not needed.
JIRA: COMPOSE-2976
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The function loads existing treeinfo, removes reference to boot.iso and
adds [media] section. This is the basic tweak that should happen for all
ISOs. Additional changes depend on the actual content.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It's possible we ask to include module X, but it's in lookaside and as
such it should not be in the output. Therefore we need to remove it from
the variant.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
By default nothing should change. This patch adds a new config option
that enables all this new craziness:
* it turns of applying module filters at pkgset level
* it creates new modules and adds them to the compose
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The module has same S:V:C, but the name is suffixed with `-devel`. The
module should contain all packages from the module koji tag that were
not included in the actual module.
The devel module has the same dependencies as the regular module, but
also additionally depends on the original module. The API and profiles
are cleared in the new module.
In the metadata it shows the same koji tag.
The test if package goes to the module is refactored to a function to
make work with the negated case a bit easier.
There may be unneeded multilib packages in the -devel module, because
there might be buildtime dependencies between things that we don't see.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Clear API and profiles
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This works around a bug where fus prioritizes first repo and thus could
include packages even if they were in lookaside.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The main variant is already available, this patch adds information about
additional variants that are included in the image.
JIRA: COMPOSE-2917
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There's an environment variable that can suppress printing of PID for
every debug message. It provides us with no real information and only
makes the log larger.
JIRA: COMPOSE-2973
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Only local paths are supported currently. As such, `file://` can be
stripped, and for anything else we should raise an exception.
JIRA: COMPOSE-2996
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This PR adds new pkgset_koji_builds configuration option.
This option allows setting list of extra Koji build NVRs which will be
included in a compose. This is useful in two cases:
a) It allows generating standard composes with few packages update to
certain version to test how the compose behaves when the package is
updated for real.
b) It allows generating compose consisting only from particular builds
when pkgset_koji_tag = '' or None. This is useful when one want to
regenerate the compose with packages which are not tagged in single Koji
tag. This is very useful for ODCS when reproducing old composes.
Merges: https://pagure.io/pungi/pull-request/1049
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
There are three different cases:
* we expect commitid and it's there
* we expect commitid and it's missing
* we don't expect commitid
This patch helps differentiate between the second two. In former one we
should report an error and mark the phase as failed. The latter is
perfectly fine and no error should be reported
Fixes: https://pagure.io/pungi/issue/1046
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There can be multiple modules with the same name and stream. They should
all have the same version, but will have different contexts. Fus takes
only N:S as input, but should pull in all matching modules. We just need
to give it correct data in the repo.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch changes the behaviour when both module tag and NSV?C? is
specified. The NSVC are used as a whitelist and only matching modules
will be included in the compose.
Additionally this patch adds filtering based on inheritance: when
finding the latest module for each N:S combination, only the top tag in
which the module is tagged is used. Even if a newer build is available
somewhere deeper in the inheritance, it's not going to be used.
Example inheritance and tagged modules
f29-compose (foo:1:2018:cafe)
└─ f29-candidate (foo:1:2019:cafe)
The compose will use 2018 version, because it's in the topmost tag.
JIRA: COMPOSE-2685
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
In the search result, we should ignore any module build that finished
after the event that we are working with.
Fixes: https://pagure.io/pungi/issue/999
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the same tag is queried with the same event, Pungi can cache the
response and call the API again. Particularly for small composes this
can save up significant amount of time.
Merges: https://pagure.io/pungi/pull-request/1022
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
If we're running in debug mode and the file is already present, it
should not be modified. This means that in order to rerun the actual
gather phase the file needs to be manually deleted first. However the
much more common use is to skip gather phase (because only images should
be re-run). In that case the manifest will be preserved correctly.
JIRA: COMPOSE-2756
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
