Commit Graph

2843 Commits

Author SHA1 Message Date
Lubomír Sedlář
dc4e8b2fb7
Fix scm tests to not use user configuration
If you configure default branch name in new repos to anything else than
master, there will be failures in tests. The test expects the branch to
be called master, but does not ensure it in any way.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit c8d16e6978)
2024-08-30 13:31:26 +03:00
Lubomír Sedlář
27d055992e
Add workaround for old requests in kojiwrapper
When running with requests<2.18 (i.e. on RHEL 7), streaming responses
are not a context manager and need to be wrapped in contextlib.closing.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 860360629d)
2024-08-30 13:31:25 +03:00
Lubomír Sedlář
34fcd550b6
Use pungi_buildinstall without NFS
The plugin supports two modes of operation:
1. Mount a shared storage volume into the runroot and have the output
   written there.
2. Have the plugin create a tar.gz with the outputs and upload them to
   the hub, from where they can be downloaded.

This patch switches from option 1 to option 2.

This requires all input repositories to be passes in as URLs and not
paths. Once the task finishes, Pungi will download the output archives
and unpack them into the expected locations.

JIRA: RHELCMP-13284
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit f25489d060)
2024-08-30 13:31:24 +03:00
Adam Williamson
4c0059e91b
checks: don't require "repo" in the "ostree" schema
Per @siosm in https://pagure.io/pungi-fedora/pull-request/1227
this option "is deprecated and not needed anymore", so Pungi
should not be requiring it.

Merges: https://pagure.io/pungi/pull-request/1714
Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit 432b0bce04)
2024-08-30 13:31:23 +03:00
Lubomír Sedlář
bb2e32132e
ostree_container: Use unique temporary directory
The config repository is cloned into a path that conflicts with the
regular ostree phase. Let's use a unique name to avoid that problem.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 7e779aa90f)
2024-08-30 13:31:22 +03:00
Lubomír Sedlář
dca3be5861
4.6.0 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>

(cherry picked from commit f4bf0739aa)
2024-08-30 13:31:20 +03:00
Lubomír Sedlář
38ec4ca159
Add ostree container to image metadata
This requires https://github.com/release-engineering/productmd/pull/172

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 119b212241)
2024-08-30 13:30:44 +03:00
Lubomír Sedlář
c589ccb56f
Updates for ostree-container phase
This patch connects the phase into the main script, and adds other
modifications:

* The archive is now stored in the images/ subdirectory in the compose.
* Documentation is updated to correctly mention that variant repos are
  not available.
* Configuration for path and name of the final archive is dropped. There
  are reasonable defaults for this and there's no point in having users
  configure it.
* The extra message for the archive is no longer sent.
* The pungi-make-ostree utility is no longer required in the buildroot.

The pungi-make-ostree utility doesn't do any significant work. It
modifies configuration files (which can happen on the compose host), and
it starts other processes.

This patch changes the ostree-container phase to no longer need the
script in the buildroot. Instead, the utility is called on the compose
host to do the config manipulation and output the needed commands. Those
are then passed into the runroot task.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 081c31238b)
2024-08-30 13:30:42 +03:00
Timothée Ravier
e413955849
Add ostree native container support
Add a new `ostree_container` stage to create ostree native container
images as OCI archives, using rpm-ostree compose image.

See: https://fedoraproject.org/wiki/Changes/OstreeNativeContainerStable
See: https://gitlab.com/CentOS/cloud/issue-tracker/-/issues/1

Fixes: https://pagure.io/pungi/issue/1698
Merges: https://pagure.io/pungi/pull-request/1699

Signed-off-by: Timothée Ravier <tim@siosm.fr>
(cherry picked from commit 95497d2676)
2024-08-30 13:30:41 +03:00
Adam Williamson
e70e1841c7
Improve autodetection of productmd image type for osbuild images
I don't love inferring the type from the filename like this -
it's kinda backwards - but it's an improvement on the current
logic (I don't think 'dvd' is ever currently the correct value
here, I don't think osbuild *can* currently build the type of
image that 'dvd' is meant to indicate). I can't immediately see
any better source of data here (we could use the 'name' or
'package_name' from 'build_info', but those are pretty much
just inputs to the filenames anyway).

Types that are possible in productmd but not covered here are
'cd' (never likely to be used again in Fedora at least, not sure
about RHEL), 'dvd-debuginfo' (again not used in Fedora, may be
used in RHEL), 'ec2', 'kvm' (not sure about those), 'netinst'
(this is a synonym for 'boot', we use 'boot' in practice in
Fedora metadata), 'p2v' and 'rescue' (not sure).

Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit aa7fcc1c20)
2024-08-30 13:30:40 +03:00
Lubomír Sedlář
fc86e03e44
pkgset: ignore events for modular content tags
Generally we want all packages to come from particular event.

There are two exceptions: packages configured via `pkgset_koji_builds`
are pulled in by exact NVR and skip event; and modules in
`pkgset_koji_modules` are pulled in by NSVC and also ignore events.

However, the modular content tag did honor event, and could lead to a
crashed compose if the content tag did not exist at the configured
event.

This patch is a slightly too big hammer. It ignores events for all
modules, not just ones configured by explicit NSVC. It's not a huge deal
as the content tags are created before the corresponding module build is
created, and once all rpm builds are tagged into the content tag, MBS
will never change it again.

JIRA: RHELCMP-12765
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit b32c8f3e5e)
2024-08-30 13:30:38 +03:00
Lubomír Sedlář
548441644b
pkgset: Ignore duplicated module builds
If the module tag contains the same module build multiple times (because
it's in multiple tags in the inheritance), Pungi will not process that
correctly and try to include the same NSVC in the compose multiple
times. That leads to a crash.

This patch adds another step to the inheritance filter to ensure the
result contains each module only once.

JIRA: RHELCMP-12768
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 935da7c246)
2024-08-30 13:30:36 +03:00
Aditya Bisoi
ca369df0df
Drop buildinstall method
JIRA: RHELCMP-12388

Signed-off-by: Aditya Bisoi <abisoi@redhat.com>
(cherry picked from commit b513c8cd00)
2024-08-30 13:30:35 +03:00
Lingyan Zhuang
67ae4202c4
Add step to send UMB message
If reuse old ISO finished, send out UMB message.

Signed-off-by: Lingyan Zhuang <lzhuang@redhat.com>
(cherry picked from commit 8cf1d98312)
2024-08-30 13:30:33 +03:00
Timothée Ravier
aba5a7a093
Fix minor Ruff/flake8 warnings
```
pungi/checks.py:575:17: F601 [*] Dictionary key literal `"type"` repeated
pungi/phases/pkgset/pkgsets.py:617:12: E721 Do not compare types, use `isinstance()`
tests/test_pkgset_source_koji.py:241:16: E721 Do not compare types, use `isinstance()`
tests/test_pkgset_source_koji.py:244:16: E721 Do not compare types, use `isinstance()`
tests/test_pkgset_source_koji.py:370:16: E721 Do not compare types, use `isinstance()`
tests/test_pkgset_source_koji.py:374:20: E721 Do not compare types, use `isinstance()`
```

Signed-off-by: Timothée Ravier <tim@siosm.fr>
(cherry picked from commit 2534ddee99)
2024-08-30 13:30:32 +03:00
Simon de Vlieger
323d1c1eb6
osbuild: manifest type in config
Allow the manifest type used to be specified in the pungi configuration
instead of always selecting the manifest type based on the koji output.

Signed-off-by: Simon de Vlieger <cmdr@supakeen.com>
(cherry picked from commit f30a8b4d15)
2024-08-30 13:30:31 +03:00
Lubomír Sedlář
b0964ff555
4.5.1 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>

(cherry picked from commit 3ffb991bac)
2024-08-30 13:30:30 +03:00
Ozan Unsal
79bc4e0c3a
gather_dnf.py: Do not raise error when the downloaded package is exists.
If the packages are pulled from different repos and a package is already
exists in target directory, pungi raises File exists error and breaks. This
behavior can be suspended and skipped if the package is already available.

Merges: https://pagure.io/pungi/pull-request/1696
Signed-off-by: Ozan Unsal <ounsal@redhat.com>
(cherry picked from commit dbc0e531b2)
2024-08-30 13:30:05 +03:00
Lubomír Sedlář
8772ccca23
New upstream release 4.7.0
(cherry picked from commit e0600a2abac9e0e9b8a3b15b51eb44e3cd467bd3)
2024-08-30 13:29:32 +03:00
Fedora Release Engineering
3bb34225a9
Rebuilt for https://fedoraproject.org/wiki/Fedora_41_Mass_Rebuild
(cherry picked from commit 192a8ef731fbc134bf5337dfb3d60ba6c5ad7bd5)
2024-08-30 13:29:32 +03:00
Haibo Lin
daea6cabdf
Upstream release 4.6.3
(cherry picked from commit 9a24cfff1bfccbafde32a4a34805d9d0aeff5650)
2024-08-30 13:29:30 +03:00
Python Maint
35b720e87a
Rebuilt for Python 3.13
(cherry picked from commit 639bb6214433a96a6275817baf893ab4850a3309)
2024-08-30 13:29:29 +03:00
Lubomír Sedlář
5a6ee9f8eb
Bump release over f40-infra build
(cherry picked from commit 1ad8b6fa2edeb91316dd1d1e33a9c234800e28d9)
2024-08-30 13:29:28 +03:00
Lubomír Sedlář
9a64db0485
Require xorriso for bug#2278677
(cherry picked from commit 22214e03b888c9b5f85919815f2825ad176c5370)
2024-08-30 13:29:27 +03:00
Lubomír Sedlář
de7210f69a
Upstream release 4.6.2
(cherry picked from commit f24f577c89647dc80a84bfa76f3055d24ced55a5)
2024-08-30 13:29:05 +03:00
Lubomír Sedlář
24418ef74d
New upstream release 4.6.1
(cherry picked from commit 98b4f26e0972a2bea2d46f2c74c1db94ed087477)
2024-08-30 13:29:03 +03:00
f4765fbe3a
Remove python3-mock dependency
Merges: https://src.fedoraproject.org/rpms/pungi/pull-request/9

(cherry picked from commit 67a11d878b04bd46a0d9fb98036467bca6ffed92)
2024-08-30 13:28:01 +03:00
Fedora Release Engineering
80b9add9f7
Rebuilt for https://fedoraproject.org/wiki/Fedora_40_Mass_Rebuild
(cherry picked from commit 40fd963a495689a2a3a0279760f5a4024e7e5857)
2024-08-30 13:27:24 +03:00
Fedora Release Engineering
b241545ca6
Rebuilt for https://fedoraproject.org/wiki/Fedora_40_Mass_Rebuild
(cherry picked from commit 5cfb290545fdd5b18bb1691218e5e8e732e351e4)
2024-08-30 13:27:00 +03:00
Lubomír Sedlář
2e536228ae
Backport: Stop requiring repo option in ostree phase
(cherry picked from commit 6778cae05afb2b5784a46ed72ee2703785756dde)
2024-08-30 13:26:39 +03:00
Lubomír Sedlář
ff7950b9d1
ostree_container: Use unique temporary directory
(cherry picked from commit 58ca2a86231e53cc329e3e20294853230fabf587)
2024-08-30 13:26:38 +03:00
Lubomír Sedlář
6971624f83
New upstream release 4.6.0
(cherry picked from commit 2b47d8ea021a7b6e694c52fd8d74880f9a6b79a5)
2024-08-30 13:26:11 +03:00
Lubomír Sedlář
b7d371d1c3
Backport patch for explicit setting of osbuild image type
(cherry picked from commit c0bf9a2a78)
2024-08-30 13:25:21 +03:00
bc8c776872
- Method get_remote_file_content is object's method now 2024-05-04 10:43:19 +03:00
91d282708e
- Method get_remote_file_content is object's method now 2023-11-21 09:19:01 +02:00
ccaf31bc87
- Method get_remote_file_content is object's method now 2023-11-21 08:51:05 +02:00
5fe0504265
- Spec's changelog chronology is fixed 2023-11-15 15:14:22 +02:00
d79f163685
- Bump version 2023-11-15 14:49:51 +02:00
793fb23958
- Bump version 2023-11-15 14:02:10 +02:00
65d0c09e97
- Return empty list if a repo doesn't contain any module 2023-11-15 13:17:57 +02:00
0a9e5df66c
- Properly removing tmp files 2023-11-10 21:38:01 +02:00
ae527a2e01
- The unittests are fixed 2023-11-10 18:08:03 +02:00
Aditya Bisoi
4991144a01
4.5.0 release
Signed-off-by: Aditya Bisoi <abisoi@redhat.com>

(cherry picked from commit 4c7611291d (centos_master))
2023-11-10 16:58:03 +02:00
Lubomír Sedlář
68d94ff488
kojiwrapper: Stop being smart about local access
Rather than trying to use local access when it's accessible, let user
make the decision:

 * if koji_cache is configured use it and download stuff
 * if not, fall back to local access

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 0d3cd150bd)
2023-11-10 16:57:53 +02:00
Ozan Unsal
ce45fdc39a
Fix unittest errors
Signed-off-by: Ozan Unsal <ounsal@redhat.com>

(cherry picked from commit aa0aae3d3e (centos_master))
2023-11-10 16:57:51 +02:00
Lubomír Sedlář
b625ccea06
Add integrity checking for builds
When a real build is downloaded, Koji can provide a checksum via API.
This commit adds verification of that checksum.

A mismatch will abort the compose. If Koji doesn't provide a checksum
for the particular sigkey, no checking will happen.

Nothing is still checked for scratch builds and images.

This patch requires Koji 1.32. When talking to an older version, there
is no checking done.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 77f8fa25ad)
2023-11-10 16:55:44 +02:00
Lubomír Sedlář
8eccfc5a03
Add script for cleaning up the cache
Pungi would by default only ever add files to the cache. That would
eventually result in essentially a mirror of the Koji volume.

This patch adds a helper cleanup script. When called, it goes through
files in the cache and deletes anything that is not hardlinked from
elsewhere and with mtime not updated recently.

Cleaning up files that hardlinked from some compose would not save any
space anyway. The mtime check should account for cases like subpackage
being downloaded but not included in any compose. This would avoid it
from being downloaded over and over again.

When a compose fails or is aborted, there can be a stale lock file left
behind in the cache. This script cleans that up too.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>

(cherry picked from commit e6d9f31ef4 (centos_master))
2023-11-10 16:55:43 +02:00
Lubomír Sedlář
f5a0e06af5
Add ability to download images
This patch extends the ability to download files from Koji to image
building phases too.

There is no integrity checking for the downloaded images.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit bf3e9bc53a)
2023-11-10 16:55:20 +02:00
Lubomír Sedlář
f6f54b56ca
Add support for not having koji volume mounted locally
With this patch, Pungi can be configured with a local directory to be
used as a cache for RPMs, and it will download packages from Koji over
HTTP instead of reading them from filesystem directly.

The files from the cache can then be hardlink as usual.

There is locking in place to avoid different composes running at the
same time to step on each other.

This is now supported for RPMs only, be it real builds or scratch
builds.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 631bb01d8f)
2023-11-10 16:55:19 +02:00
Aditya Bisoi
fcee346c7c
Remove repository cloning multiple times
JIRA: RHELCMP-8913
Signed-off-by: Aditya Bisoi <abisoi@redhat.com>
(cherry picked from commit b6296bdfcd)
2023-11-10 16:55:18 +02:00