The change allows for setting the parameters as described below to Lorax.
Lorax, a program called during the buildInstall phase, creates the SquashFS
during the buildInstall phase.
The Squash filesystem is present both on the DVD and the BOOT.ISO.
squashfs_only --- (str) passes --squashfs-only option.
configuration_file --- (str or scm_dict) passes -c option to Lorax.
The final goal of this change is to allow for optimization of
the installation medium size.
This pull request is related to the Fedora change proposal, which is available
at this location:
https://fedoraproject.org/wiki/Category:Changes/OptimizeSquashFS
See the change proposal for more information about the benefits of higher
compression ratio.
Jira: RHELCMP-693
Signed-off-by: Bohdan Khomutskyi <bkhomuts@redhat.com>
This uses a previous change that added ability to retry and simply
extends the conditions when it fires.
JIRA: RHELCMP-1863
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Reusing old arch repo may fail for reasons such as arch not
available in old compose or unexpected error when copying data
from old compose.
JIRA: RHELCMP-994
Signed-off-by: Haibo Lin <hlin@redhat.com>
Logs of DepsolveError will be printed in the main log.
Fixes: https://pagure.io/pungi/issue/1399
JIRA: RHELCMP-955
Signed-off-by: Haibo Lin <hlin@redhat.com>
This patch updates the documentation to match the actual behavior, and
adds a configuration option to remove the leading prefix.
The extra wildcard is causing problems when there are two variants in
the compose and one UID is a suffix of the other (e.g. DevTools and
Tools), since multiple files will match the shorter name and an error
will be reported.
JIRA: RHELCMP-1086
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is a workaround for modularity design issues and DNF bugs. If there
were gaps in contexts, DNF has trouble handling the upgrades. Thus we
may need to add module metadata for older versions of previously
released module streams and add the missing contexts.
JIRA: RHELCMP-982
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This commits changes `ostree` and `ostree_installer` phases
so they can run with Koji Pungi plugin instead of the plain runroot.
It is similar to `buildinstall` phase running with Koji plugin.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
If the input packages contain e.g. `ansible`, and there's
`ansible-runner-service` in the package set, we don't want to pull it
in.
JIRA: RHELCMP-446
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
New `buildinstall.metadata` file is created once the buildinstall
phase is done. This file contains:
- list of lorax command line arguments.
- list of RPMs installed in the buildinstall buildroot.
- list of RPMs installed in the resulting boot.iso.
This file is checked in the next compose run to find out if
the result of buildinstall phase from the previous compose
can be reused. Following is checked:
- lorax commandline arguments are the same (except of expected
differences).
- The NVRAs of RPMs in the runroot_tag are the same as the ones
installed in the old buildinstall buildroot.
- The NVRAs of RPMs installed in the boot.iso are the same as
the ones in package sets in the current compose.
By its implementation, this reuse strategy is used only if
pungi_buildinstall Koji plugin is used.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
Add tests for buildinstall reuse and buildinstall_allow_reuse option.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
Move repoclosure out from test phase into its own phase and
run parallel with image building phases(osbs, imagebuild, ...)
to speed things up.
JIRA: RHELCMP-8
Signed-off-by: Haibo Lin <hlin@redhat.com>
I've analysed multiple nigthly composes and often the only difference
in the configuration between two nightly composes is different
`product_id` commit hash.
The `product_id` is used in later `createrepo` phase and does not
influence the gather phase at all. I therefore think it can be
whitelisted in gather phase reuse code.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
When modules are used, there are lot of small package sets. These
package sets have usually less than 500 packages. The createrepo
part of `MaterializedPackageSet.create` executed for such small
set of packages takes around 1 second. Most of this time
the createrepo_c runs in single thread. It does the initialization,
it writes the XML files, ...
The parts of createrepo which can be run in parallel and therefore
would use all the CPUs are quite small for very small package sets.
This commit therefore executes multiple threads with
`MaterializedPackageSet.create` for these very small package sets.
This saves around 40 seconds from pkgset phase for RHEL compose.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
- Get also requires/provides of RPMs in package set.
- Store the results of gather phase as pickle file.
- Reuse old gather phase results in case Pungi configuration
did not change, the "names" of RPMs in global package set
did not change and their requires/provides did not change.
- Add `gather_allow_reuse` option to enable this feature.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
Add gather_allow_reuse, add more tests and better handling of gather_lookaside_repos.
The current code calls `find_old_compose` followed by multiple `os.path.*`
calls to find out if particular file exists in the old compose. This
duplicates code a lot and makes it harder to read.
In this commit, new `Compose.old_compose_path` is introduced and
used instead of direct calls of `find_old_compose`.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
If Koji pungi-buildinstall is used, then the buildinstall results are
stored in the `output_dir` dir, but in "results" and "logs" subdirectories.
We need to move them to final_output_dir.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
We would like to start generating the buildinstall phase using the safer
Koji Pungi Buildinstall plugin and stop the direct use of Runroot plugin.
The plugin so far exists only as PR for Koji:
https://pagure.io/koji/pull-request/1939
This commit adds support for this plugin when `lorax_use_koji_plugin`
is set to `True`.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
There was an attempt to achieve this by customizing --version and
--release arguments for Lorax, but Pungi does not take the [general] and
[release] sections from its .treeinfo. Instead it was always using
release version.
The value from this new option will be put into .treeinfo and used for
--version and --release arguments in Lorax (unless explicitly defined in
lorax_options to another value).
JIRA: COMPOSE-4029
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When `link_type = "symlink"` is used, the packages are in fact symlinks
to /mnt/koji. When graft points file is generated, the paths in this graft
points file point to symlinks and therefore symlinks are copied into the
generated ISO file instead of real files.
In this commit, the code to generate the graft points file is changed
so it resolves the symlink to real file stored on /mnt/koji. To make
this code safer, it does such resolving only in case the symlink points
outside of `compose.paths.compose.topdir()`. Therefore you can still
generate ISO file with symlink pointing to file stored within the ISO
file itself, although this is not done currently afaik.
The main reason for this is to be able to generate ISO files even
without hardlinks (which would need read-write access on /mnt/koji)
and without copying all the packages from /mnt/koji to local storage.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
When pungi-gather (or old pungi) download the packages from repos into
work/$arch/, they are linked to work/global/. This was using link_type
configuration option.
However if that is set to symlink, we get a relative symlink in
work/global/ which is later copied under the compose/ directory. Since
it's a relative symlink, it gets broken by this.
The fix is to hardlink the downloaded packages instead in the first
step. Since both the source and destination are in work/ directory, we
know it's the same volume and hardlinks should work. There is a fallback
to copy just to be sure, but it shouldn't ever be used.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The `runroot_method` now accepts `dict` value with phase name as a key
and runroot method as a value. For backward compatibility, the `str`
value is still supported.
The new `global_runroot_method` option has been added which defines
the runroot method in case it is not set in `dict` in the `runroot_method`.
This commit allows running `createiso` phase locally while keeping the other
phases in Koji.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
Theoretically it is possible, and this is the bare minimum of changes
needed to survive such situation. There may be other pitfalls.
Fixes: https://pagure.io/pungi/issue/1309
JIRA: COMPOSE-4016
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Bohdan Khomutskyi
Lubomír Sedlář
Haibo Lin
Jan Kaluza
Buvanesh Kumar