I've analysed multiple nigthly composes and often the only difference
in the configuration between two nightly composes is different
`product_id` commit hash.
The `product_id` is used in later `createrepo` phase and does not
influence the gather phase at all. I therefore think it can be
whitelisted in gather phase reuse code.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
GitWrapper._clone crashes occasionally but unfortunately
the root cause is not found yet. We need more info for debugging.
JIRA: COMPOSE-4219
Signed-off-by: Haibo Lin <hlin@redhat.com>
The `re.match` function already anchors the pattern at the start of the
string, but allows for other characters to continue after match.
This is causing problems with packages like `elfutils-debuginfod-client`
which are not debuginfo.
Let's be safe and explicitly anchor both start and end.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When modules are used, there are lot of small package sets. These
package sets have usually less than 500 packages. The createrepo
part of `MaterializedPackageSet.create` executed for such small
set of packages takes around 1 second. Most of this time
the createrepo_c runs in single thread. It does the initialization,
it writes the XML files, ...
The parts of createrepo which can be run in parallel and therefore
would use all the CPUs are quite small for very small package sets.
This commit therefore executes multiple threads with
`MaterializedPackageSet.create` for these very small package sets.
This saves around 40 seconds from pkgset phase for RHEL compose.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
This change adds an option which allows users to specify
the number of version components (i.e. values between dots)
The default behavior is preserved, and is equivalent to using
'-1' as the value.
- Negative values remove items from the end.
- Positive values specify the count of of components
- Zero will remove the version field entirely from the symlink
- When nonzero, at least one version component will appear
Merges: https://pagure.io/pungi/pull-request/1361
Signed-off-by: Lon Hohberger <lhh@redhat.com>
- Get also requires/provides of RPMs in package set.
- Store the results of gather phase as pickle file.
- Reuse old gather phase results in case Pungi configuration
did not change, the "names" of RPMs in global package set
did not change and their requires/provides did not change.
- Add `gather_allow_reuse` option to enable this feature.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
Add gather_allow_reuse, add more tests and better handling of gather_lookaside_repos.
It seems the other phases executed in Weaver phase does not need
extra_files output, so it can be run in parallel to them.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
The current code calls `find_old_compose` followed by multiple `os.path.*`
calls to find out if particular file exists in the old compose. This
duplicates code a lot and makes it harder to read.
In this commit, new `Compose.old_compose_path` is introduced and
used instead of direct calls of `find_old_compose`.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
ODCS creates symlinks to real directories containing the composes.
The directory structure is similar to following one:
- `./nightly/Fedora-Rawhide-20200304.n.0` -> `../odcs-3`
- `./nightly/Fedora-Rawhide-20200305.n.0` -> `../odcs-4`
- `./nightly/latest-Fedora-Rawhide` -> `../odcs-5`
The current Pungi code to search for old composes skips symlinks
and therefore old ODCS composes are not found.
This commit removes this check and therefore symlinks are allowed
when searching for old compose.
I think this check existed to prevent using `latest-*` symlink as
source for the compose. But this is not possible, because the
code checks that the old compose directory name has certain pattern
constructed from release_short, release_version, ... The `latest-*`
symlink definitely does not match this pattern.
I also executed test compose and it worked as expected.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
If the signing machinery crashes, we don't want to wait forever. This
way at least we can shout for help periodically.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Fixes: https://pagure.io/pungi/issue/1350
JIRA: COMPOSE-4166
If Koji pungi-buildinstall is used, then the buildinstall results are
stored in the `output_dir` dir, but in "results" and "logs" subdirectories.
We need to move them to final_output_dir.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
When running with `--quiet`, the warning should not be printed. It would
confuse tools that expect only compose path to be printed.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It is useful to actually check how the schema changed after applying
--schema-override. This commit changes --dump-schema in a way that
--schema-override is taken into account and dumped schema contains
the changes done using the --schema-override.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
E231 missing whitespace after ','
E265 block comment should start with '# '
E266 too many leading '#' for block comment
E302 expected 2 blank lines, found 1
E501 line too long (115 > 88 characters)
E713 test for membership should be 'not in'
E722 do not use bare 'except'
F812 list comprehension redefines 'g' from line 1499
F821 undefined name 'cmp'
F841 local variable 'ex' is assigned to but never used
JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
Some composes might need extra validation to ensure they are following
certain strict rules - for example containing only signed packages or
packages only from particular Koji tag.
There is currently no way how to check that Pungi configuration fulfills
these extra requirements.
This commit adds new `--schema-override` option to
`pungi-config-validate` script which allows caller to specify path to
JSON schema overriding the default JSON schema and therefore limitting
it further.
For exmaple, to limit the `pkgset_source` to `koji`, one can use
following JSON schema override:
```
{
"properties": {
"pkgset_source": {
"enum": ["koji"]
}
}
}
```
It is possible to use `--schema-override` multiple times to apply
multiple schema overrides.
Merges: https://pagure.io/pungi/pull-request/1341
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
When partial cleanup messes up the guestfs cache, the call to guestmount
will fail. To fix that, let's check if there is a problem first and
clean up everything if needed.
Relates: https://bugzilla.redhat.com/show_bug.cgi?id=1771976
JIRA: COMPOSE-3932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Newer versions of Python report an error when a file is not closed.
Let's avoid it by using with statement.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We would like to start generating the buildinstall phase using the safer
Koji Pungi Buildinstall plugin and stop the direct use of Runroot plugin.
The plugin so far exists only as PR for Koji:
https://pagure.io/koji/pull-request/1939
This commit adds support for this plugin when `lorax_use_koji_plugin`
is set to `True`.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
There was an attempt to achieve this by customizing --version and
--release arguments for Lorax, but Pungi does not take the [general] and
[release] sections from its .treeinfo. Instead it was always using
release version.
The value from this new option will be put into .treeinfo and used for
--version and --release arguments in Lorax (unless explicitly defined in
lorax_options to another value).
JIRA: COMPOSE-4029
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
