Commit Graph

949 Commits

Author SHA1 Message Date
Lubomír Sedlář
cb9dede604
kiwibuild: Add support for type, type attr and bundle format
This is a very basic support. Whatever users specify in the new option
will be passed to the koji task.

Related: https://bugzilla.redhat.com/show_bug.cgi?id=2270197
Related: https://pagure.io/koji/pull-request/4157
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit e43cf68f08)
2024-08-30 13:40:50 +03:00
Lubomír Sedlář
ce2c222dc2
createiso: Block reuse if unsigned packages are allowed
We can have a compose with unsigned packages.

By the time the next compose is generated, the packages could have been
signed. However, the new compose would still reuse the ISO with unsigned
copies.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit d546a49299)
2024-08-30 13:40:49 +03:00
Lubomír Sedlář
33bb0ceceb
createiso: Recompute .treeinfo checksums for images
Running xorriso to modify an ISO image can update content of included
images such as images/eltorito.img, unless we explicitly update the
image, which is undesirable (https://pagure.io/pungi/issue/1647).

However, when the file is changed, the checksum changes and .treeinfo no
longer matches.

This patch implements a workaround: once the DVD is written, it looks
for incorrect checksums, recalculates them and updates the .treeinfo on
the DVD. Since only the checksum is changing and the size of the file
remains the same, this seems to help fix the issue.

An additional step for implanting MD5 is needed again, as that gets
erased by the workaround.

JIRA: RHELCMP-13664
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>

(cherry picked from commit 3b2c6ae72a)
2024-08-30 13:40:47 +03:00
Adam Williamson
bd91ef1d10
Remove live_images.py (LiveImagesPhase)
This phase was used to create live images with livecd-creator
and 32-bit ARM images with appliance-creator. We also remove
get_create_image_cmd from the Koji wrapper as it was only used
for this phase, remove associated tests, and remove related
configuration settings and documentation.

Fixes: https://pagure.io/pungi/issue/1753
Merges: https://pagure.io/pungi/pull-request/1774
Signed-off-by: Adam Williamson <awilliam@redhat.com>

(cherry picked from commit 531f0ef389)
2024-08-30 13:40:14 +03:00
Haibo Lin
9d6226b436
pkgset: Avoid adding modules to unavailable arches
If a module is not built for specific arches, pungi will skip adding it
to these arches in pkgset phase.

JIRA: RHELCMP-13625
Signed-off-by: Haibo Lin <hlin@redhat.com>
(cherry picked from commit 627b72597e)
2024-08-30 13:39:48 +03:00
Lubomír Sedlář
927a0d35ab
iso: Extract volume id with xorriso if available
Pungi can use either genisoimage or xorriso to create ISOs.

It also needed isoinfo utility for querying volume ID from the ISO
image. However, the utility is part of the genisoimage suite of tools.

On systems that no longer provide genisoimage, the image would be
successfully generate with xorriso, but then pungi would fail to extract
the volume id leading to metadata with missing values.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit bc0334cc09)
2024-08-30 13:39:47 +03:00
Adam Williamson
ee8a56e64d
Various phases: consistent format of failure message
composetracker expects the failure message to be in a specific
form, but some phases weren't using it. They were phrasing it
slightly differently, which throws off composetracker's parsing.
We could extend composetracker to handle both forms, but it seems
simpler to just make all the phases use a consistent form.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit 9f8377abab)
2024-08-30 13:39:40 +03:00
Lubomír Sedlář
2bf6c216bc
Update tests to exercise kiwi specific metadata
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 949add0dac)
2024-08-30 13:39:39 +03:00
Lubomír Sedlář
c63f9f41b6
kiwibuild: Add tests for the basic functionality
Merges: https://pagure.io/pungi/pull-request/1739
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 8a3b64e5b8)
2024-08-30 13:39:36 +03:00
Lubomír Sedlář
3cdc8d0ba7
Use latest Fedora for python 3 test environment
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 0d310fb3b3)
2024-08-30 13:39:28 +03:00
Tomáš Hozza
a97488721d
Phases/osbuild: support passing 'customizations' for image builds
The osbuild Koji plugin supports passing customizations for an image
build. This is also supported in the Koji CLI plugin. Some teams want to
pass image customizations for images built as part of Pungi composes.
Extend the osbuild phase to support passing customizations in the Pungi
configuration.

Merges: https://pagure.io/pungi/pull-request/1733
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
(cherry picked from commit e738f65458)
2024-08-30 13:39:16 +03:00
Lubomír Sedlář
75ab6a14b2
dnf: Explicitly load filelists
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=2264414
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 42befba0b1)
2024-08-30 13:39:01 +03:00
Lubomír Sedlář
d8a553163f
Make python3-mock dependency optional
https://fedoraproject.org/wiki/Changes/RemovePythonMockUsage

Prefer using unittest.mock to a standalone package. The separate
packages should only really be needed on Python 2.7 these days.

The test requirements file is updated to only require mock on old
Python, and the dependency is removed from setup.py to avoid issues
there.

Relates: https://src.fedoraproject.org/rpms/pungi/pull-request/9

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>

(cherry picked from commit ff5a7e6377)
2024-08-30 13:38:43 +03:00
Lubomír Sedlář
a9839d8078
Make latest black happy
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit dd7ecbd5fd)
2024-08-30 13:31:29 +03:00
Lubomír Sedlář
dc4e8b2fb7
Fix scm tests to not use user configuration
If you configure default branch name in new repos to anything else than
master, there will be failures in tests. The test expects the branch to
be called master, but does not ensure it in any way.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit c8d16e6978)
2024-08-30 13:31:26 +03:00
Lubomír Sedlář
34fcd550b6
Use pungi_buildinstall without NFS
The plugin supports two modes of operation:
1. Mount a shared storage volume into the runroot and have the output
   written there.
2. Have the plugin create a tar.gz with the outputs and upload them to
   the hub, from where they can be downloaded.

This patch switches from option 1 to option 2.

This requires all input repositories to be passes in as URLs and not
paths. Once the task finishes, Pungi will download the output archives
and unpack them into the expected locations.

JIRA: RHELCMP-13284
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit f25489d060)
2024-08-30 13:31:24 +03:00
Lubomír Sedlář
fc86e03e44
pkgset: ignore events for modular content tags
Generally we want all packages to come from particular event.

There are two exceptions: packages configured via `pkgset_koji_builds`
are pulled in by exact NVR and skip event; and modules in
`pkgset_koji_modules` are pulled in by NSVC and also ignore events.

However, the modular content tag did honor event, and could lead to a
crashed compose if the content tag did not exist at the configured
event.

This patch is a slightly too big hammer. It ignores events for all
modules, not just ones configured by explicit NSVC. It's not a huge deal
as the content tags are created before the corresponding module build is
created, and once all rpm builds are tagged into the content tag, MBS
will never change it again.

JIRA: RHELCMP-12765
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit b32c8f3e5e)
2024-08-30 13:30:38 +03:00
Lubomír Sedlář
548441644b
pkgset: Ignore duplicated module builds
If the module tag contains the same module build multiple times (because
it's in multiple tags in the inheritance), Pungi will not process that
correctly and try to include the same NSVC in the compose multiple
times. That leads to a crash.

This patch adds another step to the inheritance filter to ensure the
result contains each module only once.

JIRA: RHELCMP-12768
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 935da7c246)
2024-08-30 13:30:36 +03:00
Aditya Bisoi
ca369df0df
Drop buildinstall method
JIRA: RHELCMP-12388

Signed-off-by: Aditya Bisoi <abisoi@redhat.com>
(cherry picked from commit b513c8cd00)
2024-08-30 13:30:35 +03:00
Timothée Ravier
aba5a7a093
Fix minor Ruff/flake8 warnings
```
pungi/checks.py:575:17: F601 [*] Dictionary key literal `"type"` repeated
pungi/phases/pkgset/pkgsets.py:617:12: E721 Do not compare types, use `isinstance()`
tests/test_pkgset_source_koji.py:241:16: E721 Do not compare types, use `isinstance()`
tests/test_pkgset_source_koji.py:244:16: E721 Do not compare types, use `isinstance()`
tests/test_pkgset_source_koji.py:370:16: E721 Do not compare types, use `isinstance()`
tests/test_pkgset_source_koji.py:374:20: E721 Do not compare types, use `isinstance()`
```

Signed-off-by: Timothée Ravier <tim@siosm.fr>
(cherry picked from commit 2534ddee99)
2024-08-30 13:30:32 +03:00
d79f163685
- Bump version 2023-11-15 14:49:51 +02:00
ae527a2e01
- The unittests are fixed 2023-11-10 18:08:03 +02:00
Lubomír Sedlář
68d94ff488
kojiwrapper: Stop being smart about local access
Rather than trying to use local access when it's accessible, let user
make the decision:

 * if koji_cache is configured use it and download stuff
 * if not, fall back to local access

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 0d3cd150bd)
2023-11-10 16:57:53 +02:00
Ozan Unsal
ce45fdc39a
Fix unittest errors
Signed-off-by: Ozan Unsal <ounsal@redhat.com>

(cherry picked from commit aa0aae3d3e (centos_master))
2023-11-10 16:57:51 +02:00
Aditya Bisoi
fcee346c7c
Remove repository cloning multiple times
JIRA: RHELCMP-8913
Signed-off-by: Aditya Bisoi <abisoi@redhat.com>
(cherry picked from commit b6296bdfcd)
2023-11-10 16:55:18 +02:00
Lubomír Sedlář
f3485410ad
Add support for git-credential-helper
This patch adds an additional field `options` to scm_dict, which can be
used to provide additional information to the backends.

It implements a single new option for GitWrapper. This option allows
setting a custom git credentials wrapper. This can be useful if Pungi
needs to get files from a git repository that requires authentication.

The helper can be as simple as this (assuming the username is already
provided in the url):

    #!/bin/sh
    echo password=i-am-secret

The helper would need to be referenced by an absolute path from the
pungi configuration, or prefixed with ! to have git interpret it as a
shell script and look it up in PATH.

See https://git-scm.com/docs/gitcredentials for more details.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
JIRA: RHELCMP-11808
(cherry picked from commit ada8f4e346)
2023-11-10 16:54:59 +02:00
Haibo Lin
cccfaea14e
Support OIDC Client Credentials authentication to CTS
JIRA: RHELCMP-11324
Signed-off-by: Haibo Lin <hlin@redhat.com>
(cherry picked from commit e4c525ecbf)
2023-11-10 16:54:58 +02:00
Lubomír Sedlář
c8118527ea
Drop pungi-orchestrator code
This was never actually used.

JIRA: RHELCMP-10218
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>

(cherry picked from commit b7adbf8a91 (centos_master))
2023-11-10 16:54:40 +02:00
Lubomír Sedlář
a8ea322907
isos: Ensure proper file ownership and permissions
The genisoimage backend uses the -rational-rock option, which sets uid
and gid to 0, and makes file readable by everyone.

With xorriso this must be done explicitly. Setting ownership is a single
command, but the permissions require a per-file command to not make
files executable where not needed.

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=2203888
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>

(cherry picked from commit 82ae9e86d5 (centos_master))
2023-11-10 16:54:22 +02:00
Lubomír Sedlář
72ddf65e62
comps: Apply arch filtering to environment/optionlist
Let's filter this list too, not just the grouplist tag.

JIRA: RHELCMP-7926
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit d3f0701e01)
2023-11-10 16:51:52 +02:00
Lubomír Sedlář
48366177cc
pkgset: Stop reuse if configuration changed
When options controlling excluding arches change, it should break reuse.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit a71c8e23be)
2023-11-10 16:51:38 +02:00
Lubomír Sedlář
4cb8671fe4
Allow disabling inheriting ExcludeArch to noarch packages
Copying ExcludeArch/ExclusiveArch from source rpm to noarch is an easy
option to block shipping that particular noarch package from a certain
architecture. However, there is no way to bypass it, and it is rather
confusing and not discoverable.

An alternative way to remove an unwanted package is to use the good old
`filter_packages`, which has enough granularity to remove pretty much
anything from anywhere. The only downside is that it requires a change
in configuration, so it can't be done by a packager directly from a spec
file.

When we decide to break backwards compatibility, this option should be
removed and the entire ExcludeArch/ExclusiveArch inheritance removed
completely.

JIRA: ENGCMP-2606
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit ab508c1511)
2023-11-10 16:51:37 +02:00
Lubomír Sedlář
f9190d1fd1
Fix black complaints
These are newly detected by black 23.1.0.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 826169af7c)
2023-11-10 16:50:38 +02:00
Lubomír Sedlář
41048f60b7
Update to work with latest mock
The `called_once` attribute now raises an exception. Switch to
`assert_called_once` method. Also replace `assertTrue(x.called)` with
`x.assert_called()`.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 51628a974d)
2023-11-10 16:50:34 +02:00
8b11bb81af AL-5220: Investigate why CL9 can't built on the new nebula
- Exclude the packages for using in a build
2023-07-24 18:26:51 +03:00
soksanichenko
6592735aec ALBS-987: Generate i686 and dev repositories with pungi on building new distr. version automatically
- Unittests are fixed
2023-03-30 14:05:47 +03:00
soksanichenko
943fd8e77d ALBS-987: Generate i686 and dev repositories with pungi on building new distr. version automatically
- Script `create extra repo` is fixed
- Unittests are fixed
2023-03-30 12:52:51 +03:00
Lubomír Sedlář
6aabfc9285 osbuild: test passing of rich repos from configuration
Test that "rich" repositories defined as dicts in the configuration
stay as dicts in the arguments passed to the osbuild phase.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
(cherry picked from commit 8be0d84f8a)
2023-03-17 11:58:11 +02:00
Tomáš Hozza
9e014fed6a osbuild: support specifying package_sets for repos
The `koji-osbuild` plugin supports additional formats for the `repo`
property since v4 [1]. Specifically, a repo can be specified as a
dictionary with `baseurl` key and `package_sets` list containing
specific package set names, that the repository should be used for.

Extend the configuration schema to reflect the plugin change.
Extend the documentation to cover the new repository format.
Extend an existing unit test to specify additional repository using the
added format.

[1] https://github.com/osbuild/koji-osbuild/pull/82

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
(cherry picked from commit 8f0906be53)
2023-03-17 11:58:11 +02:00
Tomáš Hozza
7ccb1d4849 osbuild: don't use util.get_repo_urls()
Don't use `util.get_repo_urls()` to resolve provided repositories, but
implement osbuild-specific variant of the function named
`_get_repo_urls(). The reason is that the function from `utils`
transforms repositories defined as dicts to strings, which is
undesired for osbuild. The requirement for osbuild is to preserve the
dict as is, just to resolve the string in `baseurl` to the actual
repository URL.

Add a unit test covering the newly added function. It is inspired by a
similar test from `test_util.py`.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
(cherry picked from commit e3072c3d5f)
2023-03-17 11:58:11 +02:00
Lubomír Sedlář
46216b4f17 Speed up tests by 30 seconds
The retry test for CTS doesn't actually need to wait. Let's mock the
sleep function.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit df6664098d)
2023-03-17 11:58:11 +02:00
Lubomír Sedlář
d17e578645 Report errors from CTS
If the service returns a status code indicating a user error, report
that and do not retry.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit dd8c1002d4)
2023-03-17 11:58:11 +02:00
soksanichenko
d7b173cae5 ALBS-1040: Investigate why Pungi doesn't put modules packages into the final repos
- The unitttest is fixed
2023-03-14 18:43:14 +02:00
soksanichenko
fa4640f03e ALBS-1040: Investigate why Pungi doesn't put modules packages into the final repos
- Refactoring
- KojiMock extracts all modules which are suitable for the variant's arches
2023-03-14 18:25:21 +02:00
soksanichenko
b61e59d676 - Use unittest.mock instead external mock 2022-11-11 15:32:00 +02:00
soksanichenko
9bb5550d36 ALBS-732 2022-11-09 21:01:30 +02:00
soksanichenko
364ed6c3af - kojimock is added to pungi.phases.gather._make_lookaside_repo#prefixes
- unittests are fixed
2022-11-09 20:56:56 +02:00
soksanichenko
b14e85324c - Fix unittests 2022-11-08 14:57:52 +02:00
soksanichenko
5a19ad2258 - Fix unittests 2022-11-08 12:47:14 +02:00
soksanichenko
ee9c9a74e6 - Fix unittests 2022-11-07 23:55:26 +02:00