Commit Graph

208 Commits

Author SHA1 Message Date
Bohdan Khomutskyi
b899126b7e Allow squashfs-only and configuration_file in lorax_options
The change allows for setting the parameters as described below to Lorax.
Lorax, a program called during the buildInstall phase, creates the SquashFS
during the buildInstall phase.
The Squash filesystem is present both on the DVD and the BOOT.ISO.

squashfs_only --- (str) passes --squashfs-only option.
configuration_file --- (str or scm_dict) passes -c option to Lorax.

The final goal of this change is to allow for optimization of
the installation medium size.

This pull request is related to the Fedora change proposal, which is available
at this location:
https://fedoraproject.org/wiki/Category:Changes/OptimizeSquashFS
See the change proposal for more information about the benefits of higher
compression ratio.

Jira: RHELCMP-693
Signed-off-by: Bohdan Khomutskyi <bkhomuts@redhat.com>
2020-08-11 09:29:50 +00:00
Haibo Lin
f7167fa3b6 Allow including scratch module builds
JIRA: RHELCMP-439
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-07-17 09:08:36 +08:00
Lubomír Sedlář
54882a0fc4 Fix typos
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-06-24 14:16:42 +02:00
Jan Kaluza
4a15d1351a Allow building compose with scratch builds defined by pkgset_koji_scratch_tasks.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-06-11 12:05:47 +00:00
Lubomír Sedlář
b59bdcea92 createrepo: Allow making productid glob stricter
This patch updates the documentation to match the actual behavior, and
adds a configuration option to remove the leading prefix.

The extra wildcard is causing problems when there are two variants in
the compose and one UID is a suffix of the other (e.g. DevTools and
Tools), since multiple files will match the shorter name and an error
will be reported.

JIRA: RHELCMP-1086
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-06-11 10:36:37 +02:00
Lubomír Sedlář
649ff095c0 docs: Remove outdated note
Fixes: https://pagure.io/pungi/issue/1396
JIRA: RHELCMP-850
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-06-10 14:01:42 +02:00
Lubomír Sedlář
3bb1e3df11 createrepo: Add extra modulemd files to the repo
This is a workaround for modularity design issues and DNF bugs. If there
were gaps in contexts, DNF has trouble handling the upgrades. Thus we
may need to add module metadata for older versions of previously
released module streams and add the missing contexts.

JIRA: RHELCMP-982
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-06-10 09:50:44 +02:00
Haibo Lin
ffb65e8770 docs: update doc for gather_lookaside_repos option
JIRA: RHELCMP-566
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-06-02 14:41:56 +08:00
Jan Kaluza
b8c3ca1abe Allow using Pungi Koji plugin for ostree phases.
This commits changes `ostree` and `ostree_installer` phases
so they can run with Koji Pungi plugin instead of the plain runroot.

It is similar to `buildinstall` phase running with Koji plugin.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-05-26 13:23:58 +00:00
Jan Kaluza
f1eea0b5a6 Allow getting the compose id from CTS (Compose Tracking Service).
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-05-26 13:18:38 +00:00
Ken Dreyer
4ba65710c2 doc: explain sigkey behavior
Explain how Pungi operates on the sigkeys list, and what happens when
there is only one item in the list.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2020-05-18 15:42:37 -06:00
Ken Dreyer
5ed5646bca doc: explain tradeoffs with pkgset_allow_reuse
Explain why someone would want to enable or disable the
pkgset_allow_reuse option.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2020-05-15 12:56:46 -06:00
Ken Dreyer
d3acb0fa9e doc: fix grammar for pkgset_allow_reuse setting
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2020-05-15 12:56:37 -06:00
Haibo Lin
d4efe17328 Support --skip-branding option in lorax
JIRA: RHELCMP-572
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-05-14 11:49:40 +08:00
Haibo Lin
0525768519 Add pkgset_allow_reuse option
JIRA: RHELCMP-492
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-04-28 09:54:36 +08:00
Jan Kaluza
5395af416c Allow reusing old buildinstall phase results.
New `buildinstall.metadata` file is created once the buildinstall
phase is done. This file contains:

- list of lorax command line arguments.
- list of RPMs installed in the buildinstall buildroot.
- list of RPMs installed in the resulting boot.iso.

This file is checked in the next compose run to find out if
the result of buildinstall phase from the previous compose
can be reused. Following is checked:

- lorax commandline arguments are the same (except of expected
  differences).
- The NVRAs of RPMs in the runroot_tag are the same as the ones
  installed in the old buildinstall buildroot.
- The NVRAs of RPMs installed in the boot.iso are the same as
  the ones in package sets in the current compose.

By its implementation, this reuse strategy is used only if
pungi_buildinstall Koji plugin is used.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>

Add tests for buildinstall reuse and buildinstall_allow_reuse option.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-04-17 11:37:13 +00:00
Jan Kaluza
169fa5b453 Allow reusing gather phase results.
- Get also requires/provides of RPMs in package set.
- Store the results of gather phase as pickle file.
- Reuse old gather phase results in case Pungi configuration
  did not change, the "names" of RPMs in global package set
  did not change and their requires/provides did not change.
- Add `gather_allow_reuse` option to enable this feature.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>

Add gather_allow_reuse, add more tests and better handling of gather_lookaside_repos.
2020-03-13 10:17:59 +01:00
Haibo Lin
ff269a8675 gather: Make additional_packages accept debuginfo packages
Now debuginfo packages defined in additional_packages option
could be handled correctly.

JIRA: COMPOSE-4085
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-12 16:38:08 +08:00
Jan Kaluza
3cde5c3a87 Add support for new Pungi Buildinstall Koji plugin.
We would like to start generating the buildinstall phase using the safer
Koji Pungi Buildinstall plugin and stop the direct use of Runroot plugin.

The plugin so far exists only as PR for Koji:
https://pagure.io/koji/pull-request/1939

This commit adds support for this plugin when `lorax_use_koji_plugin`
is set to `True`.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-01-23 16:09:15 +01:00
Haibo Lin
93ed40ad2d Remove invalid parameters from osbs config
JIRA: COMPOSE-4049
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-13 09:05:55 +00:00
Lubomír Sedlář
aa6a213c8d Add a new option for customizing version in .treeinfo
There was an attempt to achieve this by customizing --version and
--release arguments for Lorax, but Pungi does not take the [general] and
[release] sections from its .treeinfo. Instead it was always using
release version.

The value from this new option will be put into .treeinfo and used for
--version and --release arguments in Lorax (unless explicitly defined in
lorax_options to another value).

JIRA: COMPOSE-4029
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-07 10:55:48 +01:00
Jan Kaluza
3750d6795f Update runroot_method documentation.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-01-06 07:47:38 +01:00
Haibo Lin
794d151bef Remove productimg phase
JIRA: COMPOSE-4004
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-03 11:42:38 +00:00
Owen W. Taylor
72bf795bd4 ostree-install: allow configuring additional depenencies for runroot
A lorax template used for the ostree-installer might need an additional
package dependency (e.g., flatpak to embed a flatpak repository) - add
a config key 'extra_runroot_pkgs' to the ostree installer configuration
to allow supplementing the set of packages installed into the runroot.

Signed-off-by: Owen W. Taylor <otaylor@fishsoup.net>
2019-10-10 15:13:38 -04:00
Lubomír Sedlář
c346492df4 buildinstall: Allow customizing dracut arguments
JIRA: COMPOSE-3853
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-09 15:42:10 +02:00
Lubomír Sedlář
21d45eb243 pkgset: Allow filtering modules from Koji tags
Add a configuration option to enable skipping some modules found in the
configured tag.

Fixes: https://pagure.io/pungi/issue/1260
JIRA: COMPOSE-3794
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-24 10:13:00 +02:00
Haibo Lin
3811c0a176 config: Deprecate bootable option
JIRA: COMPOSE-2635
Fixes: https://pagure.io/pungi/issue/976
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-07-22 16:51:38 +08:00
Lubomír Sedlář
3097019338 test: Allow turning strictness off as well
When the value is explicitly set to `False`, the check should not be
strict.

JIRA: COMPOSE-3658
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-16 08:24:18 +02:00
Lubomír Sedlář
62c6d4ddcf test: Option to make size check strict
Sometimes it's practical not just warn when ISO is larger than expected,
but to also abort the compose.

JIRA: COMPOSE-3658
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-15 13:57:50 +02:00
Haibo Lin
c4ed2bf3b2 config: Deprecate runroot option
Fixes: https://pagure.io/pungi/issue/975
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-06-27 18:00:07 +08:00
Haibo Lin
ce066707c1 config: Deprecate release_is_layered option
Fixes: https://pagure.io/pungi/issue/977
Merges: https://pagure.io/pungi/pull-request/1204
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-06-06 13:45:49 +02:00
Jan Kaluza
cfb7b71fca Fix issues in OpenSSH Runroot method found by real tests.
- Pass the runroot_tag to init command in OpenSSH Runroot method.
  This is needed for the init command as a source for initial packages
  for the buildroot.
- Rename the "runroot_ssh_init_command" to "runroot_ssh_init_template"
  to make it consistent with the rest of "runroot_ssh_*" options.
- Add missing "runroot_ssh_*" options to checks.py.
- Use chmod/chown to `output_dir` in OpenSSH Runroot method the same way
  as it is used in Koji runroot method to make the runroot output readable
  for Pungi user.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2019-05-15 07:40:31 +02:00
Jan Kaluza
49d137b444 Extend "openssh" runroot_method to be able to execute "mock"
This adds few new config options which are well described in the
configuration documentation. Please refer to it for more information.

Merges: https://pagure.io/pungi/pull-request/1170
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2019-04-12 11:10:40 +02:00
Lubomír Sedlář
d521711957 osbs: Rework configuration for image pushes
Embedding the registry configuration into OSBS config itself is
simple, but makes it impossible to reuse the same configuration for
multiple different composes.

A nice example is a nightly pushing images to a testing registry, and
production compose building the same images but pushing to staging
location. The original design requires duplication of all the
configuration just because registries are different.

With this option, the push information is stored in a separate option as
a mapping from NVR patterns to arbitrary data. The patterns are used to
match finished builds to registry.

The old configuration is marked as deprecated in code and will
eventually be removed. The deprecation handling in config validation
does not allow emitting warnings for nested values.

JIRA: COMPOSE-3394
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-05 10:28:34 +02:00
Jan Kaluza
959d6979d4 Add "openssh" runroot method
Merges: https://pagure.io/pungi/pull-request/1166
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2019-04-04 14:44:05 +02:00
Lubomír Sedlář
358fdd50ce buildinstall: Allow overwriting version for lorax
Sometimes the release version can be more specific than what should be
exposed to users of the boot iso.

JIRA: COMPOSE-3295
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-22 12:47:16 +01:00
Jan Kaluza
75bb48a882 Create new Runroot class and use it everywhere where runroot task is executed.
This adds new `Runroot` class and new `runroot_method` option which makes
it possible to choose between two currently available runroot methods:

- Local
- Koji

The main goal of this commit is to make it possible to add new runroot
methods in the future and this is the first step in that direction.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2019-03-21 14:15:58 +01:00
Ken Dreyer
b5bef68c60 doc: explain koji_profile
Add more details to the documentation for the koji_profile setting.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2019-03-14 15:53:15 -06:00
Lubomír Sedlář
3efcfede6a image-build: Support repo/install_tree as path
If a repo or install tree is specified as an absolute path on the local
filesystem, we should either translate it using the configured mappings,
or if no mapping matches, it should be return unchanged. A variant name
can not start with a slash, so attempting that translation does not make
much sense.

JIRA: COMPOSE-3290
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-25 08:18:47 +01:00
Lubomír Sedlář
f33973ee65 Make the Apple/HFS compatibility configurable
The default is the original behaviour. On F30+ a new option should be
added to config to make it work.

Over time as users move to this option (which requires a new enough
version of lorax), the default should be switched and then the option
removed.

Resolves: https://pagure.io/pungi/issue/1126
Merges: https://pagure.io/pungi/pull-request/1128
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-21 16:11:01 +01:00
Lubomír Sedlář
6c6d4759f5 isos: Check maximum expected size
This patch allows the configuration to express maximum expected size for
ISOs created in createiso and extra_isos phases. If the image is larger
than this limit, a warning is emitted in test phase. The compose itself
is not affected in any way.

JIRA: COMPOSE-2824
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-12 15:15:30 +01:00
Lubomír Sedlář
7693e562b1 osbs: Process data about pushing images to registries
This patch does not do any actual pushing. It will only extract data
about push targets from the main configuration and store it together
with exact Koji NVR in a well-defined location, and also send the data
to message bus for another service to handle.

JIRA: COMPOSE-3228
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-12 15:12:50 +01:00
Lubomír Sedlář
ba260c24e8 buildinstall: Expose lorax's --rootfs-size argument
We already make this possible for the ostree installer, but it was
missing from the traditional one. The default behaviour is to let lorax
decide, but if user knows better, they can overwrite in configuration.

JIRA: COMPOSE-3188
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-01-07 11:00:43 +01:00
Ken Dreyer
27d5073b98 doc: explain product_id_allow_missing results in detail
Add more details to the documentation for the product_id_allow_missing
setting:

- Pungi expects a product cert for each arch in each variant.

- Pungi will exit with an error by default if a variant+arch's cert is
  missing.

- Pungi will log a warning if product_id_allow_missing is True.

Merges: https://pagure.io/pungi/pull-request/1102
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2019-01-03 10:14:53 +01:00
Ken Dreyer
7c45157a92 doc: describe product_id's output and purpose
Document the "productid" files that Pungi creates when the user
configures the "product_id" setting. Describe where to find the
"productid" files, and how they relate to subscription-manager.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2019-01-02 14:22:21 -05:00
Lubomír Sedlář
dc692bc604 extra_iso: Stop including variant extra files
They are not always wanted, so let's not include them by default.
There's a new option to include the same files that extra files phases
uses, or alternatively they can be configured specifically and put into
the variant subdirectory.

JIRA: COMPOSE-3084
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-09 12:06:25 +01:00
Lubomír Sedlář
fa752eb2b5 Configure image name per variant
Up to now it was possible to change the pattern for all images, but
there are use-cases where different variants might want different names.
For example there could be one main variant that should only have
product name in the ISO filename, but addons should still be marked with
variant name.

JIRA: COMPOSE-3041
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-05 08:48:36 +01:00
Jan Kaluza
37c89dfde6 Add 'pkgset_koji_builds' option to include extra builds in a compose
This PR adds new pkgset_koji_builds configuration option.

This option allows setting list of extra Koji build NVRs which will be
included in a compose. This is useful in two cases:

a) It allows generating standard composes with few packages update to
certain version to test how the compose behaves when the package is
updated for real.

b) It allows generating compose consisting only from particular builds
when pkgset_koji_tag = '' or None. This is useful when one want to
regenerate the compose with packages which are not tagged in single Koji
tag. This is very useful for ODCS when reproducing old composes.

Merges: https://pagure.io/pungi/pull-request/1049
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-09-21 14:06:49 +02:00
Lubomír Sedlář
fbb739ef17 pkgset: Apply whitelist to modules in the tag
This patch changes the behaviour when both module tag and NSV?C? is
specified. The NSVC are used as a whitelist and only matching modules
will be included in the compose.

Additionally this patch adds filtering based on inheritance: when
finding the latest module for each N:S combination, only the top tag in
which the module is tagged is used. Even if a newer build is available
somewhere deeper in the inheritance, it's not going to be used.

Example inheritance and tagged modules

    f29-compose (foo:1:2018:cafe)
    └─ f29-candidate (foo:1:2019:cafe)

The compose will use 2018 version, because it's in the topmost tag.

JIRA: COMPOSE-2685
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-03 12:19:50 +02:00
Jan Kaluza
d2f392fac8 Use dogpile.cache to cache the listTaggedRPMS calls if possible
If the same tag is queried with the same event, Pungi can cache the
response and call the API again. Particularly for small composes this
can save up significant amount of time.

Merges: https://pagure.io/pungi/pull-request/1022
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-08-30 13:21:01 +02:00