Commit Graph

3012 Commits

Author SHA1 Message Date
eabdullin
65986b8eaf Disable hfs by default 2025-09-30 14:07:08 +03:00
Lubomír Sedlář
bc12ed7d69 Release 4.10.1
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 2eb6fe34118c85a1e86e8482b44e61504e7cbf81)
2025-09-29 18:28:14 +03:00
Lubomír Sedlář
834ee63331 osbuild: Handle wsl2 images
The images are imported to Koji with type_name set to wsl. We need to
know about this so that the image is pulled into the compose, and also
translate the type into correct value for productmd.

JIRA: RHELCMP-14724
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 3ff06bc8ea0345a3c96afe6c1f44a83615ed02d0)
2025-09-29 18:28:13 +03:00
Lubomír Sedlář
2fbc4e7bcb repoclosure: Clean up cache for dnf5
If `dnf repoclosure` actually calls dnf5 (which is not easy to tell),
our cache clean up is ineffective as dnf5 uses different locations to
dnf4. Since it's not easy to tell what `dnf` actually is, let's be safe
and iterate over both possibilities.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit a31f4233226d1df85d3f197ecc4b7fd47b827593)
2025-09-29 18:28:13 +03:00
Dominika Vesela
363a28f561 Ignore errors for rmtree after archive extraction
The folder was there and it was empty.
It looks like https://github.com/python/cpython/issues/128076.

Signed-off-by: Dominika Vesela <dhodovsk@redhat.com>
(cherry picked from commit 6569e5726298af8fdb5f6d54cd37b1b3d409de8d)
2025-09-29 18:28:12 +03:00
Simon de Vlieger
05ded4aaa8 imagebuilder: accept manifest_type
The `imagebuilder` phase was missing the `manifest_type` property in its
schema. While pungi (often) guesses correctly for the `manifest_type` it
doesn't do so in the case of ostree installer images; thus it needs to
be allowed.

This was an oversight as the phase implementation already looked for
this value in the configuration.

Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
(cherry picked from commit 0ade83b0e9adc43b4e9fd38d536d647f42126688)
2025-09-29 18:28:12 +03:00
Lubomír Sedlář
f200e493ec Add a telemetry span over image building threads
Currently there is pretty much no structure under the toplevel
run-compose span. It's a mess of random Koji calls. This change should
group spans related to a particular image build.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 661b75c6098182f652a724d3e89b913bac56e579)
2025-09-29 18:28:11 +03:00
Lubomír Sedlář
603527c6cc Add specific exception for skopeo copy
If the command fails on timeout, we can raise more specific exception
instead of the generic RuntimeError. This has no effect on the actual
runtime, but it will be beneficial for tracing as the error output will
be directly surfaced in the telemetry span.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 14e8665283f945a89d642d7da236f8a2ce854c89)
2025-09-29 18:28:11 +03:00
Lubomír Sedlář
d3bc078089 Release 4.10.0
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit c276a5a25abffcc990476267adf42125b87dee94)
2025-09-29 18:28:10 +03:00
Lubomír Sedlář
4d432fd385 Add more tracing to kojiwrapper
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 431cd630e93d8655329a0a1c77d291e7714c12c4)
2025-09-29 18:28:10 +03:00
Simon de Vlieger
85d7d19dc5 phases: implement image-builder
Implement a phase for the `imageBuilderBuild` task that is provided by
the `koji-image-builder` plugin, which schedules tasks to run with
`image-builder`.

This change is part of an accepted change proposal [1] for Fedora to use
`koji-image-builder` to build (some of) its variants.

[1]: https://fedoraproject.org/wiki/Changes/KojiLocalImageBuilder

Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
(cherry picked from commit 69d87c27ff29b128aa8ff1e8aebd278a00d9fed8)
2025-09-29 18:28:09 +03:00
Lubomír Sedlář
84f7766dcf Add a tracing span around call to skopeo inspect
This call can fail and with a span we can get better visibility into
that.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 7b803d620f9951ef4ee1a17eeb5e46b8c9728e47)
2025-09-29 18:28:08 +03:00
Lubomír Sedlář
858c0ab252 Add retries to skopeo inspect calls
These can fail for any transient networking reason. Let's make a few
attempts before giving up.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 2c289729b9b24300f18ca5ca2eb1edd83da63a53)
2025-09-29 18:28:08 +03:00
Lubomír Sedlář
d818c781f9 Release 4.9.4
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit b2f82644e5ccf5b01e21412b7ed0f83a6d898249)
2025-09-29 18:28:07 +03:00
Lubomír Sedlář
2dd2b5c82a otel: Explicitly initialize telemetry provider and tracer
Doing this setup on import is simple, but it has issues if the pungi
code is directly imported into a different process.

Specifically, ODCS may have created its own provider and set things up
as needed, and then imports pungi, which tries to set a new provider.
This is prohibited by the SDK docs, and emits a warning. In reality it
is causing spans to be attributed to a wrong service.

As a side effect, RequestsIntrumentor doesn't start, and so the parent
process will need to do that on its own instead of relying on the side
effect.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit e0a3343a4be96f9e284b752e5c5f31e02883a0a8)
2025-09-29 18:28:07 +03:00
Lubomír Sedlář
65c507e2c0 Release 4.9.3
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 73408f081de82e865e5df8ea28b513ca311509bb)
2025-09-29 18:28:07 +03:00
Lubomír Sedlář
2d48a341a6 Recognize wsl2 images produced by koji
The image type was added to productmd 1.45, so we should also require
that version.

Merges: https://pagure.io/pungi/pull-request/1841
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 668547ed3f8da2528fbeb1b12040182ca02e7f31)
2025-09-29 18:28:05 +03:00
Lubomír Sedlář
384a7a0bee Specify data_files with relative paths
The paths should be relative to sys.prefix, which happens to be /usr in
the RPM world. This change should make installation with
%pyproject_install macro from a generated wheel work correctly.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 1116e1f278ea547cf3bf9265eba70d7a8aa4795e)
2025-09-29 18:28:05 +03:00
Andrew Hills
466e924e6f Crossreference koji_cache from the Koji cache page
Signed-off-by: Andrew Hills <ahills@redhat.com>
(cherry picked from commit 33bb36814903fe2990f1a48e21a7c456d0c16a9e)
2025-09-29 18:28:04 +03:00
Andrew Hills
b0e025cea1 Add documentation for koji_cache configuration
Signed-off-by: Andrew Hills <ahills@redhat.com>
(cherry picked from commit 18ed52d10b35d2a0718588279a4419db8358cbf5)
2025-09-29 18:28:04 +03:00
Lubomír Sedlář
e765db157f linker: Drop ability to link dirs recursively
Nothing in the code base uses this functionality, and the semantins are
not well defined anyway when it comes to symlinks.

Now the tests are failing in Python 3.14 rebuild when hardlinking
symlinks. Rather than trying to fix the unused code, we could just drop
it.

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=2367780
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit ab11e0e4a9e0b70e1b78399931d357f92f3a27aa)
2025-09-29 18:28:03 +03:00
Lubomír Sedlář
37479bbc6a Record exceptions for top level OTel span
If there is an exception in the code, the cli_main function captures it,
saves the traceback and exits the process.

With the original tracing span, the instrumentation never saw the actual
exception, only SystemExit. This meant the span was not recorded as
failed. (Technically python-opentelemetry 1.31.0 does record it, but
that change was reverted in 1.32.0.)

It is somewhat tricky to structure the code so that the exception is
recorded implicitly. The status update to DOOMED must happen inside the
span (in order to propagate it to the trace). Thus a new function is
exported from the tracing module to record the exception explicitly
before it gets discarded and replaced with the exit.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit d3630bfa6f8dc5ccf3dcef9cb4a947d82d7f09b8)
2025-09-29 18:28:03 +03:00
Lubomír Sedlář
5cf13491df Make black happy
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 16eda470c9a0bc2aa94c0637619537fae8de6778)
2025-09-29 18:28:03 +03:00
Lubomír Sedlář
7a4bd62978 Fix jenkins tests
Defining a variable on top level is now causing the pipeline to not
execute anything and just report success.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 912baf119879e8d5b113e64b9a98e169ef3aa516)
2025-09-29 18:28:02 +03:00
Lubomír Sedlář
2c80051446 Release 4.9.2
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 0badc6f3c2d63ee26766f6a21ef2722d3d716840)
2025-09-29 18:28:01 +03:00
Lubomír Sedlář
dcd7e5ff2a Drop compatibility with Koji < 1.32
The 1.32 version with checksum API has been released more than 2 years
ago.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 65336b406c01a99ccc9dabd011cc969b4395e106)
2025-09-29 18:28:01 +03:00
Lubomír Sedlář
fc2cc0073a kiwibuild: Add support for use_buildroot_repo option
This option can be set for a particular image or globally for all
kiwibuild images (with individual overload).

Fixes: https://pagure.io/pungi/issue/1833
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit d91adfd34d9bd97e2cc73a6023924f3f7b73cef4)
2025-09-29 18:28:01 +03:00
Lubomír Sedlář
8c897dda71 gather: Resolve symlinks before linking packages
If we happen to have a symlink to an RPM that should be linked into the
compose, we should first resolve it to the actual path. This avoids a
problem if the symlink is relative, as otherwise Pungi would copy/link
the actual relative symlink, which would break it in the new location.

If the path is not a symlink, resolving the real path should make no
difference.

JIRA: RHELCMP-14504
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 859b968483b75f630bb9c7e15a90767eb0b68d95)
2025-09-29 18:28:01 +03:00
Lubomír Sedlář
027eb45b1d Make requests instrumentation optional
Even if basic otel dependencies are available, this instrumentor is a
separate dependency which may be missing.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 004f357acb0c6907e7cd06a9e6cc6a547e67e070)
2025-09-29 18:28:01 +03:00
Lubomír Sedlář
7bcf9df307 Release 4.9.1
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit e13504296716dbd3900e2860d2aa9dbd4d8af875)
2025-09-29 18:28:01 +03:00
Lubomír Sedlář
0e203553fa util: Fix typo in regex for container digests
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit ab4b221e6465ca387cd4c0c5928f133737cc0905)
2025-09-29 18:28:00 +03:00
Lubomír Sedlář
45c3b1d9b1 Resolve container tags to digests
When the compose is configured to include any container image, it just
followed the provided URL. This is not particularly reproducible. If the
image spec contains a tag, it may point to different images at different
time.

This commit adds a step to validating the configuration that will query
the registry and replace the tag with a digest.

This makes it more reproducible, and also fixes a problem where changing
container image would not stop ISO reuse. There's still a chance of
non-container file changing and not forcing the reuse, but that is not
very common.

JIRA: RHELCMP-14381
JIRA: RHELCMP-14465
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 3ed09991c17c05551ea2d86286a72d13c726439f)
2025-09-29 18:27:59 +03:00
Lubomír Sedlář
4bfbe8afc2 kojiwrapper: Remove unused code
These methods were used in the live_images phase that doesn't exist
anymore.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 252044a9118d0a2ccffc91dde64226a9901a9d6b)
2025-09-29 18:27:59 +03:00
Lubomír Sedlář
feffd284a4 Add basic telemetry support
This patch adds support for Opentelemetry. If
OTEL_EXPORTER_OTLP_ENDPOINT env variable is defined, it will send traces
there. Otherwise there is no change.

The whole compose is wrapped in a single span. Nested under that are
spans for operations that involve a remote server.

* Talking to CTS
* Sending API requests to Koji
* Any git repo clone

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>

(cherry picked from commit c15ddbc946cc6a820dfb2f0bbacb72ca118100ba)
2025-09-29 18:27:59 +03:00
Haibo Lin
49a3e6cd12 Reorder ostree and ostree_installer phases
osbuild phase needs to wait for ostree phase in some cases, this patch
makes the various image build phases waiting for ostree phase, it may
introduce some slowdown, but it's still faster than the version before
PR#1790.

JIRA: RHELCMP-14349
Fixes: https://pagure.io/pungi/issue/1816
Signed-off-by: Haibo Lin <hlin@redhat.com>
(cherry picked from commit b3e0b6d7b73c48588b9aacd933f3e0e8ae3506ac)
2025-09-29 18:27:16 +03:00
Lubomír Sedlář
545215da19 Fix test data generation script
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 6a9c8551d2449b4a9581b403562338be765861a5)
2025-09-29 18:27:16 +03:00
Lubomír Sedlář
74ceea10ba extra_isos: Mention all extra files in the manifest
When container-images are downloaded, they would be skipped from the
extra_files.json manifest. This patch fixes that by enumerating all
files rather than relying on the getter to return a list.

JIRA: RHELCMP-14406
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit cb0399238e097c6917ffa847f546ff01fdff7599)
2025-09-29 18:27:16 +03:00
Lubomír Sedlář
64e1c30100 scm: Add retries to container-image download
If all retries fail, let's also log the error output.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit b99bcfb5eebf0d14d284fa0ab1bc2631d9e14ae3)
2025-09-29 18:27:15 +03:00
Lubomír Sedlář
4f53c5257d Release 4.9.0
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit e33ac15d9972f0ad83d4c639e5af6e7fe96aad62)
2025-09-29 18:27:15 +03:00
Haibo Lin
136a02bdbb scm: Fix git clone issue for git+http protocol
`git clone` failed if the URL is specified as git+http.

    git: 'remote-git+http' is not a git command. See 'git --help'.

JIRA: RHELCMP-14340
Signed-off-by: Haibo Lin <hlin@redhat.com>
(cherry picked from commit 1a594e4148c409fc5383fd0a4b0e7ba04d13ec1c)
2025-09-29 18:27:15 +03:00
Lubomír Sedlář
a6e7828033 Make black happy
The latest version seems to want escape sequences written in lowercase.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit fc0de97c5e8e11527858e6d835525ded28d2501e)
2025-09-29 18:27:15 +03:00
Lubomír Sedlář
6891038eb8 buildinstall: Add support for rootfs-type lorax option
JIRA: ENGCMP-5117
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 3c6298ee28b32a41ee458975730f246fd9284f93)
2025-09-29 18:27:15 +03:00
Lubomír Sedlář
dd8d22f0e3 scm: Stop trying to download src arch
This simplifies configuring extra isos to avoid failing on downloading
non-existing images.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit b3a316776e878d56c683c6558316d0c578c65992)
2025-09-29 18:27:15 +03:00
Lubomír Sedlář
cdc275741b extra_isos: Provide arch to extra files getter
The getter is already running once per architecture, it just doesn't
make the information available to the scm wrapper.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 46d6c48e0a03146f05a996a9529cffbcbcc8447c)
2025-09-29 18:27:13 +03:00
Lubomír Sedlář
a034b8b977 Move temporary buildinstall download to work/
The files should always be cleaned up immediately after the archive is
extracted, but we are seeing them being left behind for some reason.

With this page, even if the data is not cleaned up, it will not clog up
/tmp and be eventually deleted together with the compose.

JIRA: RHELCMP-14319
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit e4d1bd4783de28b34ec289d6218205756ee916ad)
2025-09-29 18:27:13 +03:00
Adam Williamson
f3dcb036a5 Protect against decoding errors with subprocess text mode
All these are calling subprocess in 'text mode', where it will
try to decode stdout/stderr using the default encoding (utf-8
for us). If it doesn't decode, subprocess will raise an exception
and kobo doesn't handle it, it just passes it along to us, so
things blow up - see https://pagure.io/releng/issue/12474 . To
avoid this, let's set `errors="replace"`, which tells the decoder
to replace invalid data with ? characters. This way we should get
as much of the output as can be read, and no crashes.

We also replace `universal_newlines=True` with `text=True` as
the latter is shorter, clearer, and what Python 3 subprocess
wants us to use, it considers `universal_newlines` to just be
a backwards-compatibility thing - "The universal_newlines argument
is equivalent to text and is provided for backwards compatibility"

Signed-off-by: Adam Williamson <awilliam@redhat.com>
Merges: https://pagure.io/pungi/pull-request/1812
(cherry picked from commit 2d16a3af004f61cf41e4eb2e5e694bb46a5d3cda)
2025-09-29 18:27:13 +03:00
Adam Williamson
e59566feb2 Revert "Avoid to crash on unicode decoding errors"
This reverts commit 7d8f3b4b9b2cf65967b4d3f8dd249aec2e3cbbf8. It
doesn't really fix the problem. A better fix follows.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 98e3b3f8c410943c6dbeb21ebca9934b60a30f2f)
2025-09-29 18:27:13 +03:00
Lubomír Sedlář
ed0713c572 Download extra files from container registry
This could be useful for handling flatpak applications in the installer.

All of the specified containers are downloaded into a single oci
layout.

JIRA: RHELCMP-14302
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 3d5348a6728b4d01cf8770494902e64c99e21a14)
2025-09-29 18:27:12 +03:00
Haibo Lin
e550458c9f Avoid to crash on unicode decoding errors
As kobo.shortcuts.run can't handle binary output correctly, it causes
pungi-make-ostree crashed when rpm-ostree outputs unexpected characters.

JIRA: RHELCMP-14253
Fixes: https://pagure.io/releng/issue/12474
Signed-off-by: Haibo Lin <hlin@redhat.com>
(cherry picked from commit 7d8f3b4b9b2cf65967b4d3f8dd249aec2e3cbbf8)
2025-09-29 18:27:12 +03:00
Lubomír Sedlář
c2852f7034 Remove python 2.7 dependencies from setup.py
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>

(cherry picked from commit b058f64abe797b78059d084a3af5d9f7e4049d88)
2025-09-29 18:27:12 +03:00