Commit Graph

2979 Commits

Author SHA1 Message Date
Lubomír Sedlář
feffd284a4 Add basic telemetry support
This patch adds support for Opentelemetry. If
OTEL_EXPORTER_OTLP_ENDPOINT env variable is defined, it will send traces
there. Otherwise there is no change.

The whole compose is wrapped in a single span. Nested under that are
spans for operations that involve a remote server.

* Talking to CTS
* Sending API requests to Koji
* Any git repo clone

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>

(cherry picked from commit c15ddbc946cc6a820dfb2f0bbacb72ca118100ba)
2025-09-29 18:27:59 +03:00
Haibo Lin
49a3e6cd12 Reorder ostree and ostree_installer phases
osbuild phase needs to wait for ostree phase in some cases, this patch
makes the various image build phases waiting for ostree phase, it may
introduce some slowdown, but it's still faster than the version before
PR#1790.

JIRA: RHELCMP-14349
Fixes: https://pagure.io/pungi/issue/1816
Signed-off-by: Haibo Lin <hlin@redhat.com>
(cherry picked from commit b3e0b6d7b73c48588b9aacd933f3e0e8ae3506ac)
2025-09-29 18:27:16 +03:00
Lubomír Sedlář
545215da19 Fix test data generation script
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 6a9c8551d2449b4a9581b403562338be765861a5)
2025-09-29 18:27:16 +03:00
Lubomír Sedlář
74ceea10ba extra_isos: Mention all extra files in the manifest
When container-images are downloaded, they would be skipped from the
extra_files.json manifest. This patch fixes that by enumerating all
files rather than relying on the getter to return a list.

JIRA: RHELCMP-14406
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit cb0399238e097c6917ffa847f546ff01fdff7599)
2025-09-29 18:27:16 +03:00
Lubomír Sedlář
64e1c30100 scm: Add retries to container-image download
If all retries fail, let's also log the error output.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit b99bcfb5eebf0d14d284fa0ab1bc2631d9e14ae3)
2025-09-29 18:27:15 +03:00
Lubomír Sedlář
4f53c5257d Release 4.9.0
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit e33ac15d9972f0ad83d4c639e5af6e7fe96aad62)
2025-09-29 18:27:15 +03:00
Haibo Lin
136a02bdbb scm: Fix git clone issue for git+http protocol
`git clone` failed if the URL is specified as git+http.

    git: 'remote-git+http' is not a git command. See 'git --help'.

JIRA: RHELCMP-14340
Signed-off-by: Haibo Lin <hlin@redhat.com>
(cherry picked from commit 1a594e4148c409fc5383fd0a4b0e7ba04d13ec1c)
2025-09-29 18:27:15 +03:00
Lubomír Sedlář
a6e7828033 Make black happy
The latest version seems to want escape sequences written in lowercase.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit fc0de97c5e8e11527858e6d835525ded28d2501e)
2025-09-29 18:27:15 +03:00
Lubomír Sedlář
6891038eb8 buildinstall: Add support for rootfs-type lorax option
JIRA: ENGCMP-5117
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 3c6298ee28b32a41ee458975730f246fd9284f93)
2025-09-29 18:27:15 +03:00
Lubomír Sedlář
dd8d22f0e3 scm: Stop trying to download src arch
This simplifies configuring extra isos to avoid failing on downloading
non-existing images.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit b3a316776e878d56c683c6558316d0c578c65992)
2025-09-29 18:27:15 +03:00
Lubomír Sedlář
cdc275741b extra_isos: Provide arch to extra files getter
The getter is already running once per architecture, it just doesn't
make the information available to the scm wrapper.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 46d6c48e0a03146f05a996a9529cffbcbcc8447c)
2025-09-29 18:27:13 +03:00
Lubomír Sedlář
a034b8b977 Move temporary buildinstall download to work/
The files should always be cleaned up immediately after the archive is
extracted, but we are seeing them being left behind for some reason.

With this page, even if the data is not cleaned up, it will not clog up
/tmp and be eventually deleted together with the compose.

JIRA: RHELCMP-14319
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit e4d1bd4783de28b34ec289d6218205756ee916ad)
2025-09-29 18:27:13 +03:00
Adam Williamson
f3dcb036a5 Protect against decoding errors with subprocess text mode
All these are calling subprocess in 'text mode', where it will
try to decode stdout/stderr using the default encoding (utf-8
for us). If it doesn't decode, subprocess will raise an exception
and kobo doesn't handle it, it just passes it along to us, so
things blow up - see https://pagure.io/releng/issue/12474 . To
avoid this, let's set `errors="replace"`, which tells the decoder
to replace invalid data with ? characters. This way we should get
as much of the output as can be read, and no crashes.

We also replace `universal_newlines=True` with `text=True` as
the latter is shorter, clearer, and what Python 3 subprocess
wants us to use, it considers `universal_newlines` to just be
a backwards-compatibility thing - "The universal_newlines argument
is equivalent to text and is provided for backwards compatibility"

Signed-off-by: Adam Williamson <awilliam@redhat.com>
Merges: https://pagure.io/pungi/pull-request/1812
(cherry picked from commit 2d16a3af004f61cf41e4eb2e5e694bb46a5d3cda)
2025-09-29 18:27:13 +03:00
Adam Williamson
e59566feb2 Revert "Avoid to crash on unicode decoding errors"
This reverts commit 7d8f3b4b9b2cf65967b4d3f8dd249aec2e3cbbf8. It
doesn't really fix the problem. A better fix follows.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 98e3b3f8c410943c6dbeb21ebca9934b60a30f2f)
2025-09-29 18:27:13 +03:00
Lubomír Sedlář
ed0713c572 Download extra files from container registry
This could be useful for handling flatpak applications in the installer.

All of the specified containers are downloaded into a single oci
layout.

JIRA: RHELCMP-14302
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 3d5348a6728b4d01cf8770494902e64c99e21a14)
2025-09-29 18:27:12 +03:00
Haibo Lin
e550458c9f Avoid to crash on unicode decoding errors
As kobo.shortcuts.run can't handle binary output correctly, it causes
pungi-make-ostree crashed when rpm-ostree outputs unexpected characters.

JIRA: RHELCMP-14253
Fixes: https://pagure.io/releng/issue/12474
Signed-off-by: Haibo Lin <hlin@redhat.com>
(cherry picked from commit 7d8f3b4b9b2cf65967b4d3f8dd249aec2e3cbbf8)
2025-09-29 18:27:12 +03:00
Lubomír Sedlář
c2852f7034 Remove python 2.7 dependencies from setup.py
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>

(cherry picked from commit b058f64abe797b78059d084a3af5d9f7e4049d88)
2025-09-29 18:27:12 +03:00
Lubomír Sedlář
6a293639cf util: Drop dead code
These functions were only used in the legacy pungi.gather module that
has since been removed.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 8a36744f02040108cbe7e6d590984b3cf8e53b40)
2025-09-29 18:26:36 +03:00
Adam Williamson
ac7e1e515e Use new container and bootable-container productmd types
In https://github.com/release-engineering/productmd/pull/181 I
added new `bootable-container` and `container` types to
productmd. This makes pungi always use the bootable-container
type for ostree_container images (previously 'ociarchive'), and
default to using the container type for Kiwi-built oci.tar.xz
container images (previously 'docker').

This is a significant change for anything that relies on
productmd/fedfind conventions to 'identify' images, as these
images will now have a different identity. But I think it's a
valuable improvement in their identities. 'ociarchive' never made
any sense as an image 'type' - it's a format - and 'docker'
wasn't a very good type for images that are explicitly OCI
container images, not Docker-native ones. We also can now easily
distinguish between 'regular' container images and ones that are
intended to be bootable.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit 3cb8992d56f2cee8a7cb151253125e30931ccd6d)
2025-09-29 18:26:35 +03:00
Lubomír Sedlář
fddce94704 Directly import mock from unittest
It is not a separate package since Python 3.3

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>

(cherry picked from commit 3987688de6720d951bfeb0b49c364df9738b490b)
2025-09-29 18:26:35 +03:00
Lubomír Sedlář
26959621a6 Release 4.8.0
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 7d512293611eb91d005226974c14a42a1bc44dc1)
2025-09-29 18:23:51 +03:00
Lubomír Sedlář
74db11a836 Remove python 2.7 from tox configuration
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit af10ab312b248052e987ee2c50650e2db85be5e4)
2025-09-29 18:23:45 +03:00
Lubomír Sedlář
e98dd56fce Remove forgotten multilib module for yum
There's no more yum anymore. This was also the only user of the
pathmatch module, which is thus also removed.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 989a9c2565ab9466b55d6edc056973c3080dfeae)
2025-09-29 18:23:44 +03:00
Lubomír Sedlář
4ff13b1993 Drop usage of six
We no longer need to support Python 2, so there's no point in this
compatibility layer.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>

(cherry picked from commit b34de57813187f1781aef733468c9745a144d9af)
2025-09-29 18:23:44 +03:00
Lubomír Sedlář
b044ebdba1 Ensure ostree phase threads are stopped
The ostree phase now runs in parallel with a lot of other stuff. If
there's any error while the phase is running, the compose would be
aborted but the ostree threads wouldn't be stopped automatically. With
the threads left alive, the process would never finish.

This patch makes sure that whatever happens in the other code, we always
stop the ostree phases.

Fixes: https://pagure.io/pungi/issue/1799
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 8558b74d7810dd92144924542a63bae7b1999bd3)
2025-09-29 18:19:15 +03:00
Lubomír Sedlář
f8932bc1f4 scm: Clone git submodules
If the repo contains .gitmodules file, run the commands to clone all
submodules.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 6d1428ab89de6ffa5c18466a469606887a0300b8)
2025-09-29 18:19:14 +03:00
Lubomír Sedlář
755004af02 Drop unittest2
The library is imported if available, but we never build it in any
environment where the package would be installed. It was last used for
RHEL 6 builds.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>

(cherry picked from commit d95d1f59e2ae243ea794c5f5613fef3249b4fad6)
2025-09-29 18:19:14 +03:00
Adam Williamson
567baed60f kiwibuild: extend productmd type/format detection for FEX images
Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit eb4ba5f637153f0037f05981adea8b35fc0f6b25)
2025-09-29 18:16:18 +03:00
Lubomír Sedlář
2e9baeaf51 Remove pungi/gather.py and associated code
This commit completly drops support for Yum as a depsolving/repoclosure
backend.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>

(cherry picked from commit f5702e4c9d0d5d9d31421d3d47200581e41f02bf)
2025-09-29 18:16:18 +03:00
Adam Williamson
4454619be6 Reduce legacy pungi script to gather phase only (#1792)
This reduces the legacy 'pungi' script to only its gather phase,
and removes related stuff in gather.py. The gather phase is used
in the yum path through phases/gather/methods/method_deps.py, so
it cannot be entirely removed until all users of that are gone.
But we can at least get rid of the non-Koji support for creating
install trees, ISOs and repos.

Merges: https://pagure.io/pungi/pull-request/1793
Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit 3bc35a9a271c50ca093b186938eae7cbc1bbd3de)
2025-09-29 18:15:21 +03:00
Adam Williamson
4f69f6c242 pkgset: optimize cache check (saves 20 minutes)
The pkgset phase takes around 35 minutes in current composes.
Around 20 minutes of that is spent creating these per-arch
subsets of the global package set. In a rather roundabout way
(see #1794 ), I figured out that almost all of this time is
spent in this cache check, which is broken for a subtle reason.

Python's `in` keyword works by first attempting to call the
container's magic `__contains__` method. If the container does
not implement `__contains__`, it falls back to iteration - it
tries to iterate over the container until it either hits what
it's looking for, or runs out. (If the container implements
neither, you get an error).

The FileCache instance's `file_cache` is a plain Python dict.
dicts have a very efficient `__contains__` implementation, so
doing `foo in (somedict)` is basically always very fast no matter
how huge the dict is. FileCache itself, though, implements
`__iter__` by returning an iterator over the `file_cache` dict's
keys, but it does *not* implement `__contains__`. So when we do
`foo in self.file_cache`, Python has to iterate over every key
in the dict until it hits foo or runs out. This is massively
slower than `foo in self.file_cache.file_cache`, which uses the
efficient `__contains__` method.

Because these package sets are so huge, and we're looping over
*one* huge set and checking each package from it against the cache
of another, increasingly huge, set, this effect becomes massive.
To make it even worse, I ran a few tests where I added a debug log
if we ever hit the cache, and it looks like we never actually do -
so every check has to iterate through the entire dict.

We could probably remove this entirely, but changing it to check
the dict instead of the FileCache instance makes it just about as
fast as taking it out, so I figured let's go with that in case
there's some unusual scenario in which the cache does work here.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit c8fe99b1aa5a9a9b941b7515cda367d24829dedf)
2025-09-29 18:15:21 +03:00
Lubomír Sedlář
37f9f1fcaf Install dnf4 into test image
The fedora:latest image is now based on 41, and contains dnf5. This is
causing some tests to fail due to failing imports of dnf version 4.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit bc60af794dc54c99bc295c742bbfba03393f7e0f)
2025-09-29 18:15:21 +03:00
Lubomír Sedlář
fdea2c88d9 Update phase diagram
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 83f56da0f17d5b9e400cfcc4678d176ce4b0f5b3)
2025-09-29 18:15:20 +03:00
Adam Williamson
0483f914c4 Don't block main image build phase on ostree_install phase
I did a time map of a Fedora compose today, and noticed that we
spend about an hour waiting for the ostree_install phase to
complete before we start up the compose_images_phase which does
all the other image builds.

This is unnecessary. Nothing else depends on ostree_install; it
should be fine to start up the extra_phase (which contains
compose_images_phase) while the ostree stuff is still running.

This implements that by splitting the ostree phases out of the
essentials_phase which contains the real precursors to the
extra_phase. We start the essentials and ostree phases together,
but only wait for the essentials phase to complete before
kicking off extra_phase, so it can start while the ostree
phase is still running.

One tweak we have to make to accommodate this is to move
image_checksum_phase out of extra_phase, to avoid it potentially
running before all ostree installer images are built. The
checksum phase is quite fast - it takes about five minutes -
and any time benefit of running it in parallel with the osbs and
repoclosure phases seems like it must be smaller than the time
loss of waiting for ostree_install before kicking off extra.

Merges: https://pagure.io/pungi/pull-request/1790
Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit 18bda22fcb842c00a606e5f357aeb9f3d02aa626)
2025-09-29 18:15:20 +03:00
Adam Williamson
a24c6d52ce ostree_container: make filename configurable, include arch
The default base name is probably fine in most cases, but there
are some where we might want to tweak it. We already allow this
for other phases (e.g. the livemedia phase).

Also, we should include the arch in the image filename. Not doing
this doesn't blow up the compose as, while they have identical
filenames, the images for different arches are in different paths,
but it's confusing for people who actually download and use the
images.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit aea8da5225aeb31b4c5dd413f0a31b6ab395a9ac)
2025-09-29 18:15:19 +03:00
Adam Williamson
a0a155ebcd Correct subvariant handling for ostree_container phase
The image metadata construction code allows for subvariant to be
set in the image config dict, but checks.py doesn't expect it;
fix that. Also, when a subvariant is set, use it in the image
name template rather than the variant; otherwise you can't
build more than one subvariant in any variant (they will have
identical names, which isn't alllowed).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit 391a5eaed5198e5ee2941dac4ae43a2fe057eedd)
2025-09-29 18:15:19 +03:00
Adam Williamson
059995a200 image_build: drop .tar.gz as an expected extension for docker
Koji's image-build command has not been capable of producing a
docker image with .tar.gz as its extension since 2015:

https://pagure.io/koji/c/b489f282bee7a008108534404dd2e78efb2256e7?branch=master

as that commit message implies, the files have not actually been
gzip-compressed for even longer:

https://pagure.io/koji/c/82a405c7943192e3bba3340efe7a8d07a0e26b70?branch=master

so there's no point to having this any more. It is causing the
wrong productmd 'type' to be set for GCE cloud images, which *do*
have the .tar.gz extension - because docker appears in this dict
before tar-gz, their type is being set as 'docker' not 'tar-gz'.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit 739062ed3c471e74ba9c5144c4047f67f9fbe8c8)
2025-09-29 18:15:19 +03:00
Adam Williamson
53c273f025 move osbuild/kiwi-specific EXTENSIONS to each phase
The image-build phase's EXTENSIONS dict is meant to exactly
mirror the 'formats' that exist in the context of the command
`koji image-build`, which is driven by this phase. That nice
association was lost, however, by adding a couple of items to it
which exist for the purposes of the osbuild phase (and in the
case of .iso, also the kiwibuild phase), which import this dict
and uses it for image identification.

To make the association 1:1 again and more clearly show what's
going on here, let's move those entries out into the osbuild and
kiwi phases. osbuild now has its own dict which starts out as a
copy of the image-build one before being extended. And let's
update the relevant comments.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit 5338d3098ccd614a8fd32f837a393aed78b471bd)
2025-09-29 18:15:18 +03:00
Lubomír Sedlář
9594954287 Drop compatibility helper for dnf.Package.source_name
The bug that caused the attribute to have wrong value has been fixed in
DNF a long time ago.

Fixes: https://pagure.io/pungi/issue/1786
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 225f04cf43326c136e95356681f1461270673ca6)
2025-09-29 18:15:18 +03:00
Lubomír Sedlář
c586c0b03b kiwibuild: Allow setting metadata type explicitly
It is not possible to reliably detect what the type for an image should
be in the metadata. This commit adds an option for user to explicitly
provide it.

It can only be configured on the specific image, not globally.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit cd2ae81e3c63316997b9617ff2e30e3148af14f2)
2025-09-29 18:15:17 +03:00
Lubomír Sedlář
6576ab9b32 kiwibuild: Fix location and metadata for ISOs
When Kiwi builds an ISO, it is always supposed to be bootable and should
be located in the iso/ subdirectory.

Any other kind of image should still land in images/ and be listed as
not bootable in the metadata.

Relates: https://pagure.io/pungi-fedora/issue/1342
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit d9d21d3cf4eaad5cc7f2959a4abdafed781bb9cf)
2025-09-29 18:15:17 +03:00
Lubomír Sedlář
d93b358959 kiwibuild: Add options for version and repo_releasever
The version follows the same rules as versioning for live media etc.
That means it's always going to be set. The precedence goes like this:

 * image specific option
 * `kiwibuild_version`
 * `global_version`
 * `release_version` or `<release_version>_<label_milestone>`.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit d351773dab7b3aa8e6de82bbe23058b6b3448dd4)
2025-09-29 18:15:17 +03:00
Lubomír Sedlář
d2fc85437b Release 4.10.1
(cherry picked from commit d14925b85c4f0e26eb4b097b6603f3dbc5d00d60)
2025-09-29 18:14:47 +03:00
Lubomír Sedlář
ca0984611b Release 4.10.0
(cherry picked from commit 79c630a8599978b3b073c9fbc17abf7df347bb40)
2025-09-29 18:14:37 +03:00
Fedora Release Engineering
4dd7ecf875 Rebuilt for https://fedoraproject.org/wiki/Fedora_43_Mass_Rebuild
(cherry picked from commit 94ede558b3ac60de921095d23ec8d6ca762087b0)
2025-09-29 18:14:25 +03:00
Miro Hrončok
2f8ce9dbca Remove one generated runtime Requires
It's required by python3-pungi.

(cherry picked from commit 73e8998491ecb630eb31877692fe03b4eceaff3e)
2025-09-29 18:14:09 +03:00
Miro Hrončok
eaaa5a6a0c Remove generated runtime dependencies from BuildRequires
Those are handled by %pyproject_buildrequires

(cherry picked from commit 4af7eaba63f34cd84ee4dbe9e908068a6d89e354)
2025-09-29 18:14:09 +03:00
Lubomír Sedlář
e164c6ed14 Release 4.9.3
Merges: https://src.fedoraproject.org/rpms/pungi/pull-request/11

(cherry picked from commit e0a9959d1f7478a0357f76d8d31c96b9d8cda895)
2025-09-29 18:10:49 +03:00
Python Maint
e33373f74c Rebuilt for Python 3.14
(cherry picked from commit 280b98bf8383ae70aa07938d08c5291f9e872d96)
2025-09-29 18:08:30 +03:00
Lubomír Sedlář
8e5c545c22 Fix tests on Python 3.14
(cherry picked from commit 40a6fe451dab89ed180b89cadd4c32e3a9328700)
2025-09-29 18:08:19 +03:00