Commit Graph

344 Commits

Author SHA1 Message Date
Jan Kaluza
195bfbefa4 Allow specifying $COMPOSE_ID in the repo value for osbs phase.
There should be an option for `yum_repourls` to point to static
URL, for example when CTS is used. The idea is that instead of
setting `repo` to `AppStream`, we could use link similar to this one:

`https://cts.localhost/api/1/composes/$COMPOSE_ID/repo/?variant=AppStream`

This would be translated to real static link during the OSBS phase:

`https://cts.localhost/api/1/composes/CentOS-Stream-9-20210803.0/repo/?variant=AppStream`

That way this statis link would appear in the yum_repourls.

Merges: https://pagure.io/pungi/pull-request/1543
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2021-09-09 10:46:33 +02:00
Haibo Lin
66dacb21e0 Add createrepo_enable_cache to configuration doc
JIRA: RHELCMP-5984
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-08-18 11:14:55 +08:00
Ken Dreyer
5a8df7b69c doc: more additional_packages documentation
Contrast the additional_packages setting with the comps_file setting.

Explain what happens when a user lists a package in additional_packages
but Pungi cannot find it.

Give an example of composing all builds in a Koji tag.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2021-08-12 07:01:49 +00:00
Ken Dreyer
6afcfef919 doc: fix typo in additional_packages description
not -> nor

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2021-08-12 07:01:49 +00:00
Ken Dreyer
2a679dcb81 doc: improve signed packages retry docs
Reword the signed_packages_retries and signed_packages_wait
configuration option documentation to use the active voice. This makes
it easier to understand who is doing what in a signing workflow.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2021-08-11 14:07:12 -04:00
Ken Dreyer
01a52447bc doc: explain buildContainer API
Explain how to discover the API documentation about the buildContainer
method, so users can discover more about how "scratch" and "priority"
work.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2021-08-05 12:42:32 -04:00
Haibo Lin
cf761633f4 4.2.10 release
JIRA: RHELCMP-6108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-08-04 17:23:19 +08:00
Ondrej Nosek
76d13d0062 4.2.9 release
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2021-04-29 06:59:45 +02:00
Lubomír Sedlář
735bfaa0d6 pkgset: Fix meaning of retries
The name brings a different expectation than how it actually worked.
This patch makes the code work similarly to the expectation.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-23 16:01:41 +01:00
Ondrej Nosek
98359654cf 4.2.8 release
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2021-02-12 15:20:40 +01:00
Lubomír Sedlář
64897d7d48 pkgset: Add ability to wait for signed packages
If packages are appearing quickly in Koji, and signing them is triggered
by automation, there may be a delay between the package being signed and
compose running. In such case it may be preferable to wait for the
signed copy rather than fail the compose.

JIRA: RHELCMP-3932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-11 15:31:14 +01:00
Lubomír Sedlář
40133074b3 Add image-container phase
This phase runs after image-build and osbuild and can embed an image
into a container.

JIRA: RHELCMP-3820
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-09 14:47:57 +01:00
Ken Dreyer
39b847094a doc: remove default createrepo_checksum value from example
createrepo_checksum already defaults to sha256. Remove this setting from
the documented Minimal Example configuration to make it easier to read.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2021-01-25 08:22:17 +00:00
Lubomír Sedlář
62a97c0e1b 4.2.7 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-12-03 14:33:22 +01:00
Lubomír Sedlář
9accf5ecf4 docs: Add osbuild phase to overview diagram
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-11-27 09:56:10 +01:00
Haibo Lin
4c88e7dc0e Use xorrisofs for creating ISOs when needed
A new configuration *createiso_use_xorrisofs* is added to determine
which tool to use for creating ISOs.

By default, createiso_use_xorrisofs = False and genisoimage will be used.
When set to True, xorrisofs will be used.

JIRA: RHELCMP-2875
Fixes: https://pagure.io/pungi/issue/1130

Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-11-20 09:47:49 +08:00
Lubomír Sedlář
e3a500ca50 Fix typo in configuration
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-11-16 10:50:01 +01:00
Bohdan Khomutskyi
4c4c816e70 Pungi 4.2.6 release
Signed-off-by: Bohdan Khomutskyi <bkhomuts@redhat.com>
2020-11-04 12:33:06 +00:00
Lubomír Sedlář
a45f4969f3 Add phase for building images with osbuild
This is similar to image-build in terms of what it does, and somewhat
similar to OSBS phase in how it's implemented.

The phase reads configuration, submits the build via XMLRPC call and
waits for the task to finish. Then it downloads the built image and
includes it in the compose metadata.

JIRA: RHELCMP-315
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-11-04 13:04:43 +01:00
Lubomír Sedlář
327019264f 4.2.5 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-09-25 09:52:31 +02:00
Haibo Lin
27a825de48 pkgset: Allow to include extra module builds
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-09-22 13:22:56 +08:00
Lubomír Sedlář
c6312b34d0 doc: Improve description of module defaults dir
JIRA: RHELCMP-812
Resolves: https://pagure.io/pungi/issue/1394
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-09-08 07:31:38 +00:00
Haibo Lin
29761d1656 4.2.4 release
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-08-17 09:19:10 +08:00
Haibo Lin
c273350fe5 doc: Update doc/contributing.rst
JIRA: RHELCMP-1773
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-08-11 20:36:22 +08:00
Bohdan Khomutskyi
b899126b7e Allow squashfs-only and configuration_file in lorax_options
The change allows for setting the parameters as described below to Lorax.
Lorax, a program called during the buildInstall phase, creates the SquashFS
during the buildInstall phase.
The Squash filesystem is present both on the DVD and the BOOT.ISO.

squashfs_only --- (str) passes --squashfs-only option.
configuration_file --- (str or scm_dict) passes -c option to Lorax.

The final goal of this change is to allow for optimization of
the installation medium size.

This pull request is related to the Fedora change proposal, which is available
at this location:
https://fedoraproject.org/wiki/Category:Changes/OptimizeSquashFS
See the change proposal for more information about the benefits of higher
compression ratio.

Jira: RHELCMP-693
Signed-off-by: Bohdan Khomutskyi <bkhomuts@redhat.com>
2020-08-11 09:29:50 +00:00
Haibo Lin
3c72755814 Use pytest instead of nosetests
Nose has been in maintenance mode for the past several years
and pytest is a good replacement.

Other changes:
- Replace deprecated assertRegexpMatches with assertRegex
- Replace deprecated assertRaisesRegexp with assertRaisesRegex
- Replace deprecated SafeConfigParser with ConfigParser
- Force reinstall pytest and mock in tox virtualenv. This is because
  the globally installed packages may not work as expected(occured in
  jenkins job).

JIRA: RHELCMP-1619
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-07-29 14:57:16 +08:00
Haibo Lin
f7167fa3b6 Allow including scratch module builds
JIRA: RHELCMP-439
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-07-17 09:08:36 +08:00
Lubomír Sedlář
54882a0fc4 Fix typos
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-06-24 14:16:42 +02:00
Lubomír Sedlář
634d30fac5 4.2.3 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-06-11 14:17:49 +02:00
Jan Kaluza
4a15d1351a Allow building compose with scratch builds defined by pkgset_koji_scratch_tasks.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-06-11 12:05:47 +00:00
Lubomír Sedlář
b59bdcea92 createrepo: Allow making productid glob stricter
This patch updates the documentation to match the actual behavior, and
adds a configuration option to remove the leading prefix.

The extra wildcard is causing problems when there are two variants in
the compose and one UID is a suffix of the other (e.g. DevTools and
Tools), since multiple files will match the shorter name and an error
will be reported.

JIRA: RHELCMP-1086
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-06-11 10:36:37 +02:00
Lubomír Sedlář
649ff095c0 docs: Remove outdated note
Fixes: https://pagure.io/pungi/issue/1396
JIRA: RHELCMP-850
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-06-10 14:01:42 +02:00
Lubomír Sedlář
3bb1e3df11 createrepo: Add extra modulemd files to the repo
This is a workaround for modularity design issues and DNF bugs. If there
were gaps in contexts, DNF has trouble handling the upgrades. Thus we
may need to add module metadata for older versions of previously
released module streams and add the missing contexts.

JIRA: RHELCMP-982
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-06-10 09:50:44 +02:00
Haibo Lin
ffb65e8770 docs: update doc for gather_lookaside_repos option
JIRA: RHELCMP-566
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-06-02 14:41:56 +08:00
Jan Kaluza
b8c3ca1abe Allow using Pungi Koji plugin for ostree phases.
This commits changes `ostree` and `ostree_installer` phases
so they can run with Koji Pungi plugin instead of the plain runroot.

It is similar to `buildinstall` phase running with Koji plugin.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-05-26 13:23:58 +00:00
Jan Kaluza
f1eea0b5a6 Allow getting the compose id from CTS (Compose Tracking Service).
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-05-26 13:18:38 +00:00
Ken Dreyer
4ba65710c2 doc: explain sigkey behavior
Explain how Pungi operates on the sigkeys list, and what happens when
there is only one item in the list.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2020-05-18 15:42:37 -06:00
Ken Dreyer
5ed5646bca doc: explain tradeoffs with pkgset_allow_reuse
Explain why someone would want to enable or disable the
pkgset_allow_reuse option.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2020-05-15 12:56:46 -06:00
Ken Dreyer
d3acb0fa9e doc: fix grammar for pkgset_allow_reuse setting
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2020-05-15 12:56:37 -06:00
Haibo Lin
d4efe17328 Support --skip-branding option in lorax
JIRA: RHELCMP-572
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-05-14 11:49:40 +08:00
Haibo Lin
fe4b2dd302 docs: Embed phases.svg directly
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-05-08 17:44:27 +08:00
Haibo Lin
20ba1a7639 Execute image_checksum phase right after the dependent phases
JIRA: RHELCMP-468
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-05-08 16:12:30 +08:00
Haibo Lin
5d9dcf61fb 4.2.2 release
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-04-29 19:25:59 +08:00
Haibo Lin
0525768519 Add pkgset_allow_reuse option
JIRA: RHELCMP-492
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-04-28 09:54:36 +08:00
Jan Kaluza
5395af416c Allow reusing old buildinstall phase results.
New `buildinstall.metadata` file is created once the buildinstall
phase is done. This file contains:

- list of lorax command line arguments.
- list of RPMs installed in the buildinstall buildroot.
- list of RPMs installed in the resulting boot.iso.

This file is checked in the next compose run to find out if
the result of buildinstall phase from the previous compose
can be reused. Following is checked:

- lorax commandline arguments are the same (except of expected
  differences).
- The NVRAs of RPMs in the runroot_tag are the same as the ones
  installed in the old buildinstall buildroot.
- The NVRAs of RPMs installed in the boot.iso are the same as
  the ones in package sets in the current compose.

By its implementation, this reuse strategy is used only if
pungi_buildinstall Koji plugin is used.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>

Add tests for buildinstall reuse and buildinstall_allow_reuse option.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-04-17 11:37:13 +00:00
Haibo Lin
477b43d4e9 Split repoclosure into separate phase
Move repoclosure out from test phase into its own phase and
run parallel with image building phases(osbs, imagebuild, ...)
to speed things up.

JIRA: RHELCMP-8
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-04-14 09:25:57 +08:00
Lubomír Sedlář
4efdacd0a0 4.2.1 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-03-13 12:55:44 +01:00
Jan Kaluza
169fa5b453 Allow reusing gather phase results.
- Get also requires/provides of RPMs in package set.
- Store the results of gather phase as pickle file.
- Reuse old gather phase results in case Pungi configuration
  did not change, the "names" of RPMs in global package set
  did not change and their requires/provides did not change.
- Add `gather_allow_reuse` option to enable this feature.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>

Add gather_allow_reuse, add more tests and better handling of gather_lookaside_repos.
2020-03-13 10:17:59 +01:00
Jan Kaluza
e70ad8aaa5 Run extra_files phase in parallel to other phases.
It seems the other phases executed in Weaver phase does not need
extra_files output, so it can be run in parallel to them.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-03-12 07:58:57 +01:00
Haibo Lin
ff269a8675 gather: Make additional_packages accept debuginfo packages
Now debuginfo packages defined in additional_packages option
could be handled correctly.

JIRA: COMPOSE-4085
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-12 16:38:08 +08:00