Commit Graph

2884 Commits

Author SHA1 Message Date
Lubomír Sedlář
927a0d35ab
iso: Extract volume id with xorriso if available
Pungi can use either genisoimage or xorriso to create ISOs.

It also needed isoinfo utility for querying volume ID from the ISO
image. However, the utility is part of the genisoimage suite of tools.

On systems that no longer provide genisoimage, the image would be
successfully generate with xorriso, but then pungi would fail to extract
the volume id leading to metadata with missing values.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit bc0334cc09)
2024-08-30 13:39:47 +03:00
Adam Williamson
d81ee0f553
De-duplicate log messages for ostree and ostree_container phases
The ostree and ostree_container phases both log messages in the
exact same form, which is rather confusing. This will make it
much clearer which message comes from which phase.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit 5c9e79f535)
2024-08-30 13:39:46 +03:00
Lubomír Sedlář
e601345a38
Handle tracebacks as str or bytes
Kobo 0.36.0 changed how tracebacks are handled. Instead of `bytes`, it
returns a `str`. That makes pungi fail to write it into a file opened as
binary.

Relates: https://github.com/release-engineering/kobo/pull/246
Fixes: https://pagure.io/pungi/issue/1756
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 29c166ab99)
2024-08-30 13:39:45 +03:00
Adam Williamson
1fe075e7e4
ostree/container: add missing --version arg
https://pagure.io/pungi/pull-request/1726 tries to use
`self.args.version`, but the `pungi-make-ostree container`
subcommand does not actually have a `--version` arg, so that is
not going to work. This adds the required arg.

We *could* make it optional by still setting an empty update
dict if it's not specified, I guess, but not sure if that's worth
the effort.

Fixes: https://pagure.io/pungi/issue/1751

Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit 51d58322f2)
2024-08-30 13:39:44 +03:00
Lubomír Sedlář
a8fc1b183b
Block pkgset reuse on module defaults change
JIRA: RHELCMP-13463
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 0ef1c102b8)
2024-08-30 13:39:43 +03:00
Adam Williamson
8f171b81a1
Include task ID in DONE message for OSBS phase
Again, composetracker expects the message in this format.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit b6cfd8c5d4)
2024-08-30 13:39:41 +03:00
Adam Williamson
ee8a56e64d
Various phases: consistent format of failure message
composetracker expects the failure message to be in a specific
form, but some phases weren't using it. They were phrasing it
slightly differently, which throws off composetracker's parsing.
We could extend composetracker to handle both forms, but it seems
simpler to just make all the phases use a consistent form.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit 9f8377abab)
2024-08-30 13:39:40 +03:00
Lubomír Sedlář
2bf6c216bc
Update tests to exercise kiwi specific metadata
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 949add0dac)
2024-08-30 13:39:39 +03:00
Adam Williamson
99a6dfe8ad
Kiwi: translate virtualbox and azure productmd formats
As discussed in
https://pagure.io/releng/failed-composes/issue/6047#comment-899622
the list of 'acceptable' types and formats (in productmd terms)
is locked down in productmd, we cannot just 'declare' new formats
in pungi as we kinda wound up doing by adding these Kiwi
extensions to the EXTENSIONS dict in image_build phase. So
instead, let's return the image_build phase to the way it was,
and add an additional layer of handling in kiwibuild phase for
these awkward cases, which 'translates' the file suffix to a
format productmd knows about already. This is actually how we
would rather behave anyway, because a Kiwi-produced
`vagrant.libvirt.box` file really is the same kind of thing as an
ImageFactory-produced `vagrant-libvirt.box` file; we want them to
have compatible metadata, we don't want them to look like
different things.

Merges: https://pagure.io/pungi/pull-request/1740
Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit 8fb694f000)
2024-08-30 13:39:37 +03:00
Lubomír Sedlář
c63f9f41b6
kiwibuild: Add tests for the basic functionality
Merges: https://pagure.io/pungi/pull-request/1739
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 8a3b64e5b8)
2024-08-30 13:39:36 +03:00
Lubomír Sedlář
ab1960de6d
kiwibuild: Remove repos as dicts
The task needs just URLs, the dics don't bring anything here.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit c80ebb029b)
2024-08-30 13:39:35 +03:00
Lubomír Sedlář
c17b820490
Fix additional image metadata
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit e2ceb48450)
2024-08-30 13:39:33 +03:00
Lubomír Sedlář
36133b71da
Drop kiwibuild_version option
Version in kiwibuild is embedded in the definition file. The option
makes no sense.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 242d7d951f)
2024-08-30 13:39:32 +03:00
Lubomír Sedlář
50b217145c
Update docs with kiwibuild options
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 04d4e1d585)
2024-08-30 13:39:31 +03:00
Adam Williamson
57f2b428d5
kiwibuild: allow setting description scm and path at phase level
Neal wanted this to work - he tried using global_description_scm
and global_description_path in the initial PR - but it wasn't
wired up to work. This should make it possible to set
`kiwibuild_description_scm` and `kiwibuild_description_path`.
It also technically lets you set `global_` for both, since the
`get_config` implementation is very generic, but it doesn't add
it to the checks, so you'd still get an "unrecognized config
option" warning, I think. It seems appropriate to encourage
setting this as a phase-level option rather than a global one
since it seems quite specific to the kiwibuild phase.

Merges: https://pagure.io/pungi/pull-request/1737
Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit e90ffdfd93)
2024-08-30 13:39:29 +03:00
Lubomír Sedlář
3cdc8d0ba7
Use latest Fedora for python 3 test environment
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 0d310fb3b3)
2024-08-30 13:39:28 +03:00
Lubomír Sedlář
07829f2229
Install unittest2 only on python 2
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 5172d7e5eb)
2024-08-30 13:39:26 +03:00
Adam Williamson
bdf06ea038
Fix 'failable' handling for kiwibuild phase
The mechanisms here are a bit subtle and the kiwibuild phase
didn't quite get them right. The arg passed to `util.failable`
is supposed to be a boolean, but kiwibuild was passing it the
list of failable arches (which will always evaluate True).

How this is meant to work is that we only make *the Koji task
as a whole* failable (by passing `True` to `util.failable`) if
*all* the arches in it are failable. If *any* arch in the task
is not failable, the task should not be failable.

We allow a subset of arches to fail by passing the Koji task a
list of `optional_arches`, later. If an arch is 'optional', that
arch failing won't cause the Koji task itself to be considered
failed.

This commit fixes the logic (I hope), renames all the variables
and adds a couple of comments to make it clearer what's going on,
and does a bit of making the code simpler.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit 0d306d4964)
2024-08-30 13:39:25 +03:00
Jeremy Cline
bcab3431e1
image_build: Accept Kiwi extension for Azure VHD images
Kiwi builds for Azure fixed VHD images are suffixed with "vhdfixed"
instead of plain "vhd". Add that to the list of suffixes.

Signed-off-by: Jeremy Cline <jeremycline@microsoft.com>
(cherry picked from commit 1494f203ce)
2024-08-30 13:39:24 +03:00
Adam Williamson
b181b08033
image_build: accept Kiwi vagrant image name format
According to Neal, Vagrant images produced by Kiwi end in e.g.
`vagrant.libvirt.box` and `vagrant.virtualbox.box` - with a
period between `vagrant` and the image type, not a dash as with
oz. We should accept this slightly different format so we can
correctly derive the productmd `type` and `format` for these.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit 93b4b4ae0f)
2024-08-30 13:39:23 +03:00
Lubomír Sedlář
e05b1bcd78
Release 4.6.2
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>

(cherry picked from commit b8e26bfb64)
2024-08-30 13:39:22 +03:00
Tomáš Hozza
a97488721d
Phases/osbuild: support passing 'customizations' for image builds
The osbuild Koji plugin supports passing customizations for an image
build. This is also supported in the Koji CLI plugin. Some teams want to
pass image customizations for images built as part of Pungi composes.
Extend the osbuild phase to support passing customizations in the Pungi
configuration.

Merges: https://pagure.io/pungi/pull-request/1733
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
(cherry picked from commit e738f65458)
2024-08-30 13:39:16 +03:00
Lubomír Sedlář
4d858ef958
dnf: Load filelists for actual solver too
Not just in tests.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 209d308e1c)
2024-08-30 13:39:14 +03:00
Lubomír Sedlář
744b00499d
kiwibuild: Tell Koji which arches are allowed to fail
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit be410d9fd5)
2024-08-30 13:39:13 +03:00
Lubomír Sedlář
583547c6ee
kiwibuild: Update documentation with more details
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 1f819ee08a)
2024-08-30 13:39:11 +03:00
Lubomír Sedlář
f28053eecc
kiwibuild: Add kiwibuild global options
This is already supported by code, just missing in the schema.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit b9d94970b5)
2024-08-30 13:39:10 +03:00
Lubomír Sedlář
a196e9c895
kiwibuild: Process images same as image-build
Getting the images from task is less hacky then matching on filenames.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit b032425f30)
2024-08-30 13:39:08 +03:00
Lubomír Sedlář
a6f6199910
kiwibuild: Add subvariant configuration
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit bcd937d16d)
2024-08-30 13:39:07 +03:00
Lubomír Sedlář
a3dcec5059
kiwibuild: Work around missing arch in build data
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit f0137fd9b9)
2024-08-30 13:39:05 +03:00
Haibo Lin
6aa674fbb3
Support KiwiBuild
Adding kiwibuild phase which is similar to osbuild.

Fixes: https://pagure.io/pungi/issue/1710
Merges: https://pagure.io/pungi/pull-request/1720
JIRA: RHELCMP-13348
Signed-off-by: Haibo Lin <hlin@redhat.com>
(cherry picked from commit 3d630d3e8e)
2024-08-30 13:39:04 +03:00
Timothée Ravier
05d9651eba
ostree/container: Set version in treefile 'automatic-version-prefix'
In the non container path, we're setting the version for the build using
the `--add-metadata-string=version=XYZ` argument passed to `rpm-ostree
compose tree ...`.

The `rpm-ostree compose image` path does not expose this option yet so
modify the treefile directly as we are already doing it to set the
repos used for the compose.

See: https://github.com/coreos/rpm-ostree/issues/4829
See: https://pagure.io/workstation-ostree-config/pull-request/472
Merges: https://pagure.io/pungi/pull-request/1726
Signed-off-by: Timothée Ravier <tim@siosm.fr>
(cherry picked from commit 8412890640)
2024-08-30 13:39:02 +03:00
Lubomír Sedlář
75ab6a14b2
dnf: Explicitly load filelists
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=2264414
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 42befba0b1)
2024-08-30 13:39:01 +03:00
Lubomír Sedlář
533ea641d8
Fix buildinstall reuse with pungi_buildinstall plugin
The keys may not exist anymore. If there's nothing to delete, it's fine.

JIRA: RHELCMP-13464
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 52c2cea0ef)
2024-08-30 13:38:59 +03:00
Lubomír Sedlář
185a53d56b
Fix filters for DNF query
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit d2e9ccefde)
2024-08-30 13:38:58 +03:00
Lubomír Sedlář
305deab9ed
gather-dnf: Support dotarch in filter_packages
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 2c61416423)
2024-08-30 13:38:57 +03:00
Lubomír Sedlář
6af11d5747
gather: Support dotarch notation for debuginfo packages
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 986099f8b5)
2024-08-30 13:38:56 +03:00
Lubomír Sedlář
58f96531c7
Correctly set input and fultree_exclude flags for debuginfo
This only matters for composes that use the functionality for trimming
addon packages from parent variants.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 947ddf0a1a)
2024-08-30 13:38:55 +03:00
Lubomír Sedlář
e570aa7726
4.6.1 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>

(cherry picked from commit e46393263e)
2024-08-30 13:38:53 +03:00
Lubomír Sedlář
d8a553163f
Make python3-mock dependency optional
https://fedoraproject.org/wiki/Changes/RemovePythonMockUsage

Prefer using unittest.mock to a standalone package. The separate
packages should only really be needed on Python 2.7 these days.

The test requirements file is updated to only require mock on old
Python, and the dependency is removed from setup.py to avoid issues
there.

Relates: https://src.fedoraproject.org/rpms/pungi/pull-request/9

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>

(cherry picked from commit ff5a7e6377)
2024-08-30 13:38:43 +03:00
Lubomír Sedlář
a9839d8078
Make latest black happy
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit dd7ecbd5fd)
2024-08-30 13:31:29 +03:00
Lubomír Sedlář
dc05d1fbba
Update tox configuration
The whitelist_externals option has been renamed to allowlist_externals.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit ba613563f6)
2024-08-30 13:31:28 +03:00
Lubomír Sedlář
dc4e8b2fb7
Fix scm tests to not use user configuration
If you configure default branch name in new repos to anything else than
master, there will be failures in tests. The test expects the branch to
be called master, but does not ensure it in any way.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit c8d16e6978)
2024-08-30 13:31:26 +03:00
Lubomír Sedlář
27d055992e
Add workaround for old requests in kojiwrapper
When running with requests<2.18 (i.e. on RHEL 7), streaming responses
are not a context manager and need to be wrapped in contextlib.closing.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 860360629d)
2024-08-30 13:31:25 +03:00
Lubomír Sedlář
34fcd550b6
Use pungi_buildinstall without NFS
The plugin supports two modes of operation:
1. Mount a shared storage volume into the runroot and have the output
   written there.
2. Have the plugin create a tar.gz with the outputs and upload them to
   the hub, from where they can be downloaded.

This patch switches from option 1 to option 2.

This requires all input repositories to be passes in as URLs and not
paths. Once the task finishes, Pungi will download the output archives
and unpack them into the expected locations.

JIRA: RHELCMP-13284
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit f25489d060)
2024-08-30 13:31:24 +03:00
Adam Williamson
4c0059e91b
checks: don't require "repo" in the "ostree" schema
Per @siosm in https://pagure.io/pungi-fedora/pull-request/1227
this option "is deprecated and not needed anymore", so Pungi
should not be requiring it.

Merges: https://pagure.io/pungi/pull-request/1714
Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit 432b0bce04)
2024-08-30 13:31:23 +03:00
Lubomír Sedlář
bb2e32132e
ostree_container: Use unique temporary directory
The config repository is cloned into a path that conflicts with the
regular ostree phase. Let's use a unique name to avoid that problem.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 7e779aa90f)
2024-08-30 13:31:22 +03:00
Lubomír Sedlář
dca3be5861
4.6.0 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>

(cherry picked from commit f4bf0739aa)
2024-08-30 13:31:20 +03:00
Lubomír Sedlář
38ec4ca159
Add ostree container to image metadata
This requires https://github.com/release-engineering/productmd/pull/172

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 119b212241)
2024-08-30 13:30:44 +03:00
Lubomír Sedlář
c589ccb56f
Updates for ostree-container phase
This patch connects the phase into the main script, and adds other
modifications:

* The archive is now stored in the images/ subdirectory in the compose.
* Documentation is updated to correctly mention that variant repos are
  not available.
* Configuration for path and name of the final archive is dropped. There
  are reasonable defaults for this and there's no point in having users
  configure it.
* The extra message for the archive is no longer sent.
* The pungi-make-ostree utility is no longer required in the buildroot.

The pungi-make-ostree utility doesn't do any significant work. It
modifies configuration files (which can happen on the compose host), and
it starts other processes.

This patch changes the ostree-container phase to no longer need the
script in the buildroot. Instead, the utility is called on the compose
host to do the config manipulation and output the needed commands. Those
are then passed into the runroot task.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 081c31238b)
2024-08-30 13:30:42 +03:00
Timothée Ravier
e413955849
Add ostree native container support
Add a new `ostree_container` stage to create ostree native container
images as OCI archives, using rpm-ostree compose image.

See: https://fedoraproject.org/wiki/Changes/OstreeNativeContainerStable
See: https://gitlab.com/CentOS/cloud/issue-tracker/-/issues/1

Fixes: https://pagure.io/pungi/issue/1698
Merges: https://pagure.io/pungi/pull-request/1699

Signed-off-by: Timothée Ravier <tim@siosm.fr>
(cherry picked from commit 95497d2676)
2024-08-30 13:30:41 +03:00