Commit Graph

224 Commits

Author SHA1 Message Date
Ozan Unsal
7475d2a3a9 Allow ISO-Level configuration within the config file
In order to enable this feature set "iso_level=<Value from 1 to 4>"
in config file

Jira: RHELCMP-6880

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
2021-10-18 16:32:04 +02:00
Jan Kaluza
195bfbefa4 Allow specifying $COMPOSE_ID in the repo value for osbs phase.
There should be an option for `yum_repourls` to point to static
URL, for example when CTS is used. The idea is that instead of
setting `repo` to `AppStream`, we could use link similar to this one:

`https://cts.localhost/api/1/composes/$COMPOSE_ID/repo/?variant=AppStream`

This would be translated to real static link during the OSBS phase:

`https://cts.localhost/api/1/composes/CentOS-Stream-9-20210803.0/repo/?variant=AppStream`

That way this statis link would appear in the yum_repourls.

Merges: https://pagure.io/pungi/pull-request/1543
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2021-09-09 10:46:33 +02:00
Haibo Lin
66dacb21e0 Add createrepo_enable_cache to configuration doc
JIRA: RHELCMP-5984
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-08-18 11:14:55 +08:00
Ken Dreyer
5a8df7b69c doc: more additional_packages documentation
Contrast the additional_packages setting with the comps_file setting.

Explain what happens when a user lists a package in additional_packages
but Pungi cannot find it.

Give an example of composing all builds in a Koji tag.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2021-08-12 07:01:49 +00:00
Ken Dreyer
6afcfef919 doc: fix typo in additional_packages description
not -> nor

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2021-08-12 07:01:49 +00:00
Ken Dreyer
2a679dcb81 doc: improve signed packages retry docs
Reword the signed_packages_retries and signed_packages_wait
configuration option documentation to use the active voice. This makes
it easier to understand who is doing what in a signing workflow.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2021-08-11 14:07:12 -04:00
Ken Dreyer
01a52447bc doc: explain buildContainer API
Explain how to discover the API documentation about the buildContainer
method, so users can discover more about how "scratch" and "priority"
work.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2021-08-05 12:42:32 -04:00
Lubomír Sedlář
735bfaa0d6 pkgset: Fix meaning of retries
The name brings a different expectation than how it actually worked.
This patch makes the code work similarly to the expectation.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-23 16:01:41 +01:00
Lubomír Sedlář
64897d7d48 pkgset: Add ability to wait for signed packages
If packages are appearing quickly in Koji, and signing them is triggered
by automation, there may be a delay between the package being signed and
compose running. In such case it may be preferable to wait for the
signed copy rather than fail the compose.

JIRA: RHELCMP-3932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-11 15:31:14 +01:00
Lubomír Sedlář
40133074b3 Add image-container phase
This phase runs after image-build and osbuild and can embed an image
into a container.

JIRA: RHELCMP-3820
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-09 14:47:57 +01:00
Ken Dreyer
39b847094a doc: remove default createrepo_checksum value from example
createrepo_checksum already defaults to sha256. Remove this setting from
the documented Minimal Example configuration to make it easier to read.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2021-01-25 08:22:17 +00:00
Haibo Lin
4c88e7dc0e Use xorrisofs for creating ISOs when needed
A new configuration *createiso_use_xorrisofs* is added to determine
which tool to use for creating ISOs.

By default, createiso_use_xorrisofs = False and genisoimage will be used.
When set to True, xorrisofs will be used.

JIRA: RHELCMP-2875
Fixes: https://pagure.io/pungi/issue/1130

Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-11-20 09:47:49 +08:00
Lubomír Sedlář
e3a500ca50 Fix typo in configuration
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-11-16 10:50:01 +01:00
Lubomír Sedlář
a45f4969f3 Add phase for building images with osbuild
This is similar to image-build in terms of what it does, and somewhat
similar to OSBS phase in how it's implemented.

The phase reads configuration, submits the build via XMLRPC call and
waits for the task to finish. Then it downloads the built image and
includes it in the compose metadata.

JIRA: RHELCMP-315
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-11-04 13:04:43 +01:00
Haibo Lin
27a825de48 pkgset: Allow to include extra module builds
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-09-22 13:22:56 +08:00
Lubomír Sedlář
c6312b34d0 doc: Improve description of module defaults dir
JIRA: RHELCMP-812
Resolves: https://pagure.io/pungi/issue/1394
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-09-08 07:31:38 +00:00
Bohdan Khomutskyi
b899126b7e Allow squashfs-only and configuration_file in lorax_options
The change allows for setting the parameters as described below to Lorax.
Lorax, a program called during the buildInstall phase, creates the SquashFS
during the buildInstall phase.
The Squash filesystem is present both on the DVD and the BOOT.ISO.

squashfs_only --- (str) passes --squashfs-only option.
configuration_file --- (str or scm_dict) passes -c option to Lorax.

The final goal of this change is to allow for optimization of
the installation medium size.

This pull request is related to the Fedora change proposal, which is available
at this location:
https://fedoraproject.org/wiki/Category:Changes/OptimizeSquashFS
See the change proposal for more information about the benefits of higher
compression ratio.

Jira: RHELCMP-693
Signed-off-by: Bohdan Khomutskyi <bkhomuts@redhat.com>
2020-08-11 09:29:50 +00:00
Haibo Lin
f7167fa3b6 Allow including scratch module builds
JIRA: RHELCMP-439
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-07-17 09:08:36 +08:00
Lubomír Sedlář
54882a0fc4 Fix typos
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-06-24 14:16:42 +02:00
Jan Kaluza
4a15d1351a Allow building compose with scratch builds defined by pkgset_koji_scratch_tasks.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-06-11 12:05:47 +00:00
Lubomír Sedlář
b59bdcea92 createrepo: Allow making productid glob stricter
This patch updates the documentation to match the actual behavior, and
adds a configuration option to remove the leading prefix.

The extra wildcard is causing problems when there are two variants in
the compose and one UID is a suffix of the other (e.g. DevTools and
Tools), since multiple files will match the shorter name and an error
will be reported.

JIRA: RHELCMP-1086
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-06-11 10:36:37 +02:00
Lubomír Sedlář
649ff095c0 docs: Remove outdated note
Fixes: https://pagure.io/pungi/issue/1396
JIRA: RHELCMP-850
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-06-10 14:01:42 +02:00
Lubomír Sedlář
3bb1e3df11 createrepo: Add extra modulemd files to the repo
This is a workaround for modularity design issues and DNF bugs. If there
were gaps in contexts, DNF has trouble handling the upgrades. Thus we
may need to add module metadata for older versions of previously
released module streams and add the missing contexts.

JIRA: RHELCMP-982
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-06-10 09:50:44 +02:00
Haibo Lin
ffb65e8770 docs: update doc for gather_lookaside_repos option
JIRA: RHELCMP-566
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-06-02 14:41:56 +08:00
Jan Kaluza
b8c3ca1abe Allow using Pungi Koji plugin for ostree phases.
This commits changes `ostree` and `ostree_installer` phases
so they can run with Koji Pungi plugin instead of the plain runroot.

It is similar to `buildinstall` phase running with Koji plugin.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-05-26 13:23:58 +00:00
Jan Kaluza
f1eea0b5a6 Allow getting the compose id from CTS (Compose Tracking Service).
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-05-26 13:18:38 +00:00
Ken Dreyer
4ba65710c2 doc: explain sigkey behavior
Explain how Pungi operates on the sigkeys list, and what happens when
there is only one item in the list.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2020-05-18 15:42:37 -06:00
Ken Dreyer
5ed5646bca doc: explain tradeoffs with pkgset_allow_reuse
Explain why someone would want to enable or disable the
pkgset_allow_reuse option.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2020-05-15 12:56:46 -06:00
Ken Dreyer
d3acb0fa9e doc: fix grammar for pkgset_allow_reuse setting
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2020-05-15 12:56:37 -06:00
Haibo Lin
d4efe17328 Support --skip-branding option in lorax
JIRA: RHELCMP-572
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-05-14 11:49:40 +08:00
Haibo Lin
0525768519 Add pkgset_allow_reuse option
JIRA: RHELCMP-492
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-04-28 09:54:36 +08:00
Jan Kaluza
5395af416c Allow reusing old buildinstall phase results.
New `buildinstall.metadata` file is created once the buildinstall
phase is done. This file contains:

- list of lorax command line arguments.
- list of RPMs installed in the buildinstall buildroot.
- list of RPMs installed in the resulting boot.iso.

This file is checked in the next compose run to find out if
the result of buildinstall phase from the previous compose
can be reused. Following is checked:

- lorax commandline arguments are the same (except of expected
  differences).
- The NVRAs of RPMs in the runroot_tag are the same as the ones
  installed in the old buildinstall buildroot.
- The NVRAs of RPMs installed in the boot.iso are the same as
  the ones in package sets in the current compose.

By its implementation, this reuse strategy is used only if
pungi_buildinstall Koji plugin is used.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>

Add tests for buildinstall reuse and buildinstall_allow_reuse option.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-04-17 11:37:13 +00:00
Jan Kaluza
169fa5b453 Allow reusing gather phase results.
- Get also requires/provides of RPMs in package set.
- Store the results of gather phase as pickle file.
- Reuse old gather phase results in case Pungi configuration
  did not change, the "names" of RPMs in global package set
  did not change and their requires/provides did not change.
- Add `gather_allow_reuse` option to enable this feature.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>

Add gather_allow_reuse, add more tests and better handling of gather_lookaside_repos.
2020-03-13 10:17:59 +01:00
Haibo Lin
ff269a8675 gather: Make additional_packages accept debuginfo packages
Now debuginfo packages defined in additional_packages option
could be handled correctly.

JIRA: COMPOSE-4085
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-12 16:38:08 +08:00
Jan Kaluza
3cde5c3a87 Add support for new Pungi Buildinstall Koji plugin.
We would like to start generating the buildinstall phase using the safer
Koji Pungi Buildinstall plugin and stop the direct use of Runroot plugin.

The plugin so far exists only as PR for Koji:
https://pagure.io/koji/pull-request/1939

This commit adds support for this plugin when `lorax_use_koji_plugin`
is set to `True`.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-01-23 16:09:15 +01:00
Haibo Lin
93ed40ad2d Remove invalid parameters from osbs config
JIRA: COMPOSE-4049
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-13 09:05:55 +00:00
Lubomír Sedlář
aa6a213c8d Add a new option for customizing version in .treeinfo
There was an attempt to achieve this by customizing --version and
--release arguments for Lorax, but Pungi does not take the [general] and
[release] sections from its .treeinfo. Instead it was always using
release version.

The value from this new option will be put into .treeinfo and used for
--version and --release arguments in Lorax (unless explicitly defined in
lorax_options to another value).

JIRA: COMPOSE-4029
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-07 10:55:48 +01:00
Jan Kaluza
3750d6795f Update runroot_method documentation.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-01-06 07:47:38 +01:00
Haibo Lin
794d151bef Remove productimg phase
JIRA: COMPOSE-4004
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-03 11:42:38 +00:00
Owen W. Taylor
72bf795bd4 ostree-install: allow configuring additional depenencies for runroot
A lorax template used for the ostree-installer might need an additional
package dependency (e.g., flatpak to embed a flatpak repository) - add
a config key 'extra_runroot_pkgs' to the ostree installer configuration
to allow supplementing the set of packages installed into the runroot.

Signed-off-by: Owen W. Taylor <otaylor@fishsoup.net>
2019-10-10 15:13:38 -04:00
Lubomír Sedlář
c346492df4 buildinstall: Allow customizing dracut arguments
JIRA: COMPOSE-3853
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-09 15:42:10 +02:00
Lubomír Sedlář
21d45eb243 pkgset: Allow filtering modules from Koji tags
Add a configuration option to enable skipping some modules found in the
configured tag.

Fixes: https://pagure.io/pungi/issue/1260
JIRA: COMPOSE-3794
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-24 10:13:00 +02:00
Haibo Lin
3811c0a176 config: Deprecate bootable option
JIRA: COMPOSE-2635
Fixes: https://pagure.io/pungi/issue/976
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-07-22 16:51:38 +08:00
Lubomír Sedlář
3097019338 test: Allow turning strictness off as well
When the value is explicitly set to `False`, the check should not be
strict.

JIRA: COMPOSE-3658
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-16 08:24:18 +02:00
Lubomír Sedlář
62c6d4ddcf test: Option to make size check strict
Sometimes it's practical not just warn when ISO is larger than expected,
but to also abort the compose.

JIRA: COMPOSE-3658
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-15 13:57:50 +02:00
Haibo Lin
c4ed2bf3b2 config: Deprecate runroot option
Fixes: https://pagure.io/pungi/issue/975
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-06-27 18:00:07 +08:00
Haibo Lin
ce066707c1 config: Deprecate release_is_layered option
Fixes: https://pagure.io/pungi/issue/977
Merges: https://pagure.io/pungi/pull-request/1204
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-06-06 13:45:49 +02:00
Jan Kaluza
cfb7b71fca Fix issues in OpenSSH Runroot method found by real tests.
- Pass the runroot_tag to init command in OpenSSH Runroot method.
  This is needed for the init command as a source for initial packages
  for the buildroot.
- Rename the "runroot_ssh_init_command" to "runroot_ssh_init_template"
  to make it consistent with the rest of "runroot_ssh_*" options.
- Add missing "runroot_ssh_*" options to checks.py.
- Use chmod/chown to `output_dir` in OpenSSH Runroot method the same way
  as it is used in Koji runroot method to make the runroot output readable
  for Pungi user.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2019-05-15 07:40:31 +02:00
Jan Kaluza
49d137b444 Extend "openssh" runroot_method to be able to execute "mock"
This adds few new config options which are well described in the
configuration documentation. Please refer to it for more information.

Merges: https://pagure.io/pungi/pull-request/1170
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2019-04-12 11:10:40 +02:00
Lubomír Sedlář
d521711957 osbs: Rework configuration for image pushes
Embedding the registry configuration into OSBS config itself is
simple, but makes it impossible to reuse the same configuration for
multiple different composes.

A nice example is a nightly pushing images to a testing registry, and
production compose building the same images but pushing to staging
location. The original design requires duplication of all the
configuration just because registries are different.

With this option, the push information is stored in a separate option as
a mapping from NVR patterns to arbitrary data. The patterns are used to
match finished builds to registry.

The old configuration is marked as deprecated in code and will
eventually be removed. The deprecation handling in config validation
does not allow emitting warnings for nested values.

JIRA: COMPOSE-3394
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-05 10:28:34 +02:00