Doing this setup on import is simple, but it has issues if the pungi
code is directly imported into a different process.
Specifically, ODCS may have created its own provider and set things up
as needed, and then imports pungi, which tries to set a new provider.
This is prohibited by the SDK docs, and emits a warning. In reality it
is causing spans to be attributed to a wrong service.
As a side effect, RequestsIntrumentor doesn't start, and so the parent
process will need to do that on its own instead of relying on the side
effect.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit e0a3343a4be96f9e284b752e5c5f31e02883a0a8)
The image type was added to productmd 1.45, so we should also require
that version.
Merges: https://pagure.io/pungi/pull-request/1841
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 668547ed3f8da2528fbeb1b12040182ca02e7f31)
The paths should be relative to sys.prefix, which happens to be /usr in
the RPM world. This change should make installation with
%pyproject_install macro from a generated wheel work correctly.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 1116e1f278ea547cf3bf9265eba70d7a8aa4795e)
Nothing in the code base uses this functionality, and the semantins are
not well defined anyway when it comes to symlinks.
Now the tests are failing in Python 3.14 rebuild when hardlinking
symlinks. Rather than trying to fix the unused code, we could just drop
it.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=2367780
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit ab11e0e4a9e0b70e1b78399931d357f92f3a27aa)
If there is an exception in the code, the cli_main function captures it,
saves the traceback and exits the process.
With the original tracing span, the instrumentation never saw the actual
exception, only SystemExit. This meant the span was not recorded as
failed. (Technically python-opentelemetry 1.31.0 does record it, but
that change was reverted in 1.32.0.)
It is somewhat tricky to structure the code so that the exception is
recorded implicitly. The status update to DOOMED must happen inside the
span (in order to propagate it to the trace). Thus a new function is
exported from the tracing module to record the exception explicitly
before it gets discarded and replaced with the exit.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit d3630bfa6f8dc5ccf3dcef9cb4a947d82d7f09b8)
Defining a variable on top level is now causing the pipeline to not
execute anything and just report success.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 912baf119879e8d5b113e64b9a98e169ef3aa516)
The 1.32 version with checksum API has been released more than 2 years
ago.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 65336b406c01a99ccc9dabd011cc969b4395e106)
This option can be set for a particular image or globally for all
kiwibuild images (with individual overload).
Fixes: https://pagure.io/pungi/issue/1833
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit d91adfd34d9bd97e2cc73a6023924f3f7b73cef4)
If we happen to have a symlink to an RPM that should be linked into the
compose, we should first resolve it to the actual path. This avoids a
problem if the symlink is relative, as otherwise Pungi would copy/link
the actual relative symlink, which would break it in the new location.
If the path is not a symlink, resolving the real path should make no
difference.
JIRA: RHELCMP-14504
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 859b968483b75f630bb9c7e15a90767eb0b68d95)
Even if basic otel dependencies are available, this instrumentor is a
separate dependency which may be missing.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 004f357acb0c6907e7cd06a9e6cc6a547e67e070)
When the compose is configured to include any container image, it just
followed the provided URL. This is not particularly reproducible. If the
image spec contains a tag, it may point to different images at different
time.
This commit adds a step to validating the configuration that will query
the registry and replace the tag with a digest.
This makes it more reproducible, and also fixes a problem where changing
container image would not stop ISO reuse. There's still a chance of
non-container file changing and not forcing the reuse, but that is not
very common.
JIRA: RHELCMP-14381
JIRA: RHELCMP-14465
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 3ed09991c17c05551ea2d86286a72d13c726439f)
These methods were used in the live_images phase that doesn't exist
anymore.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 252044a9118d0a2ccffc91dde64226a9901a9d6b)
This patch adds support for Opentelemetry. If
OTEL_EXPORTER_OTLP_ENDPOINT env variable is defined, it will send traces
there. Otherwise there is no change.
The whole compose is wrapped in a single span. Nested under that are
spans for operations that involve a remote server.
* Talking to CTS
* Sending API requests to Koji
* Any git repo clone
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit c15ddbc946cc6a820dfb2f0bbacb72ca118100ba)
osbuild phase needs to wait for ostree phase in some cases, this patch
makes the various image build phases waiting for ostree phase, it may
introduce some slowdown, but it's still faster than the version before
PR#1790.
JIRA: RHELCMP-14349
Fixes: https://pagure.io/pungi/issue/1816
Signed-off-by: Haibo Lin <hlin@redhat.com>
(cherry picked from commit b3e0b6d7b73c48588b9aacd933f3e0e8ae3506ac)
When container-images are downloaded, they would be skipped from the
extra_files.json manifest. This patch fixes that by enumerating all
files rather than relying on the getter to return a list.
JIRA: RHELCMP-14406
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit cb0399238e097c6917ffa847f546ff01fdff7599)
If all retries fail, let's also log the error output.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit b99bcfb5eebf0d14d284fa0ab1bc2631d9e14ae3)
`git clone` failed if the URL is specified as git+http.
git: 'remote-git+http' is not a git command. See 'git --help'.
JIRA: RHELCMP-14340
Signed-off-by: Haibo Lin <hlin@redhat.com>
(cherry picked from commit 1a594e4148c409fc5383fd0a4b0e7ba04d13ec1c)
The latest version seems to want escape sequences written in lowercase.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit fc0de97c5e8e11527858e6d835525ded28d2501e)
This simplifies configuring extra isos to avoid failing on downloading
non-existing images.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit b3a316776e878d56c683c6558316d0c578c65992)
The getter is already running once per architecture, it just doesn't
make the information available to the scm wrapper.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 46d6c48e0a03146f05a996a9529cffbcbcc8447c)
The files should always be cleaned up immediately after the archive is
extracted, but we are seeing them being left behind for some reason.
With this page, even if the data is not cleaned up, it will not clog up
/tmp and be eventually deleted together with the compose.
JIRA: RHELCMP-14319
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit e4d1bd4783de28b34ec289d6218205756ee916ad)
All these are calling subprocess in 'text mode', where it will
try to decode stdout/stderr using the default encoding (utf-8
for us). If it doesn't decode, subprocess will raise an exception
and kobo doesn't handle it, it just passes it along to us, so
things blow up - see https://pagure.io/releng/issue/12474 . To
avoid this, let's set `errors="replace"`, which tells the decoder
to replace invalid data with ? characters. This way we should get
as much of the output as can be read, and no crashes.
We also replace `universal_newlines=True` with `text=True` as
the latter is shorter, clearer, and what Python 3 subprocess
wants us to use, it considers `universal_newlines` to just be
a backwards-compatibility thing - "The universal_newlines argument
is equivalent to text and is provided for backwards compatibility"
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Merges: https://pagure.io/pungi/pull-request/1812
(cherry picked from commit 2d16a3af004f61cf41e4eb2e5e694bb46a5d3cda)
This reverts commit 7d8f3b4b9b2cf65967b4d3f8dd249aec2e3cbbf8. It
doesn't really fix the problem. A better fix follows.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 98e3b3f8c410943c6dbeb21ebca9934b60a30f2f)
This could be useful for handling flatpak applications in the installer.
All of the specified containers are downloaded into a single oci
layout.
JIRA: RHELCMP-14302
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 3d5348a6728b4d01cf8770494902e64c99e21a14)
These functions were only used in the legacy pungi.gather module that
has since been removed.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 8a36744f02040108cbe7e6d590984b3cf8e53b40)
In https://github.com/release-engineering/productmd/pull/181 I
added new `bootable-container` and `container` types to
productmd. This makes pungi always use the bootable-container
type for ostree_container images (previously 'ociarchive'), and
default to using the container type for Kiwi-built oci.tar.xz
container images (previously 'docker').
This is a significant change for anything that relies on
productmd/fedfind conventions to 'identify' images, as these
images will now have a different identity. But I think it's a
valuable improvement in their identities. 'ociarchive' never made
any sense as an image 'type' - it's a format - and 'docker'
wasn't a very good type for images that are explicitly OCI
container images, not Docker-native ones. We also can now easily
distinguish between 'regular' container images and ones that are
intended to be bootable.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit 3cb8992d56f2cee8a7cb151253125e30931ccd6d)
It is not a separate package since Python 3.3
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 3987688de6720d951bfeb0b49c364df9738b490b)
There's no more yum anymore. This was also the only user of the
pathmatch module, which is thus also removed.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 989a9c2565ab9466b55d6edc056973c3080dfeae)
We no longer need to support Python 2, so there's no point in this
compatibility layer.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit b34de57813187f1781aef733468c9745a144d9af)
The ostree phase now runs in parallel with a lot of other stuff. If
there's any error while the phase is running, the compose would be
aborted but the ostree threads wouldn't be stopped automatically. With
the threads left alive, the process would never finish.
This patch makes sure that whatever happens in the other code, we always
stop the ostree phases.
Fixes: https://pagure.io/pungi/issue/1799
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 8558b74d7810dd92144924542a63bae7b1999bd3)
If the repo contains .gitmodules file, run the commands to clone all
submodules.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 6d1428ab89de6ffa5c18466a469606887a0300b8)
The library is imported if available, but we never build it in any
environment where the package would be installed. It was last used for
RHEL 6 builds.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit d95d1f59e2ae243ea794c5f5613fef3249b4fad6)
This commit completly drops support for Yum as a depsolving/repoclosure
backend.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit f5702e4c9d0d5d9d31421d3d47200581e41f02bf)
This reduces the legacy 'pungi' script to only its gather phase,
and removes related stuff in gather.py. The gather phase is used
in the yum path through phases/gather/methods/method_deps.py, so
it cannot be entirely removed until all users of that are gone.
But we can at least get rid of the non-Koji support for creating
install trees, ISOs and repos.
Merges: https://pagure.io/pungi/pull-request/1793
Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit 3bc35a9a271c50ca093b186938eae7cbc1bbd3de)
The pkgset phase takes around 35 minutes in current composes.
Around 20 minutes of that is spent creating these per-arch
subsets of the global package set. In a rather roundabout way
(see #1794 ), I figured out that almost all of this time is
spent in this cache check, which is broken for a subtle reason.
Python's `in` keyword works by first attempting to call the
container's magic `__contains__` method. If the container does
not implement `__contains__`, it falls back to iteration - it
tries to iterate over the container until it either hits what
it's looking for, or runs out. (If the container implements
neither, you get an error).
The FileCache instance's `file_cache` is a plain Python dict.
dicts have a very efficient `__contains__` implementation, so
doing `foo in (somedict)` is basically always very fast no matter
how huge the dict is. FileCache itself, though, implements
`__iter__` by returning an iterator over the `file_cache` dict's
keys, but it does *not* implement `__contains__`. So when we do
`foo in self.file_cache`, Python has to iterate over every key
in the dict until it hits foo or runs out. This is massively
slower than `foo in self.file_cache.file_cache`, which uses the
efficient `__contains__` method.
Because these package sets are so huge, and we're looping over
*one* huge set and checking each package from it against the cache
of another, increasingly huge, set, this effect becomes massive.
To make it even worse, I ran a few tests where I added a debug log
if we ever hit the cache, and it looks like we never actually do -
so every check has to iterate through the entire dict.
We could probably remove this entirely, but changing it to check
the dict instead of the FileCache instance makes it just about as
fast as taking it out, so I figured let's go with that in case
there's some unusual scenario in which the cache does work here.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit c8fe99b1aa5a9a9b941b7515cda367d24829dedf)
