Commit Graph

2655 Commits

Author SHA1 Message Date
Haibo Lin
e4c525ecbf Support OIDC Client Credentials authentication to CTS
JIRA: RHELCMP-11324
Signed-off-by: Haibo Lin <hlin@redhat.com>
2023-06-28 15:49:08 +08:00
Lubomír Sedlář
091d228219 4.4.0 release
JIRA: RHELCMP-11764
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2023-06-06 15:50:31 +02:00
Lubomír Sedlář
bcc440491e gather-dnf: Run latest() later
The initial version of the filtered the latest builds at the start. That
doesn't matter in many cases:

* When there are no lookaside repos, there is generally a single version
  of each package.
* When lookaside repos do not overlap with compose repos, or contain
  only older versions.

It is however a problem when the lookaside repos contain higher version
of a package than what is in a compose repo, and some package explicitly
requires the older version.

Consider this scenario:

* lookaside contains bar-1.1
* compose repo contains bar-1.0 and foo-1.0
* foo-1.0 `Requires: bar < 1.1`

The original code would filter out the bar-1.0 package, and then fail on
unresolved dependencies.

This patch moves the computation of latest packages much later, to part
of code where all options to satisfy a dependency are selected and the
best match is chosen. At that point if there are multiple versions
available, we do want the latest one.

JIRA: SPMM-13483
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2023-06-06 12:48:36 +00:00
Lubomír Sedlář
fa50eedfad iso: Support joliet long names
Without this option the names reported by joliet tree are truncated.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2023-06-05 15:30:18 +02:00
Lubomír Sedlář
b7adbf8a91 Drop pungi-orchestrator code
This was never actually used.

JIRA: RHELCMP-10218
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2023-06-02 06:14:10 +00:00
Lubomír Sedlář
82ae9e86d5 isos: Ensure proper file ownership and permissions
The genisoimage backend uses the -rational-rock option, which sets uid
and gid to 0, and makes file readable by everyone.

With xorriso this must be done explicitly. Setting ownership is a single
command, but the permissions require a per-file command to not make
files executable where not needed.

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=2203888
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2023-06-01 06:29:02 +00:00
Lubomír Sedlář
2ad341a01c gather: Always get latest packages
If lookaside contains an older version of a package, but with a
different arch, the depsolver doesn't notice that and prefers the
lookaside version.

This is not correct. The latest package should be used no matter if
there are different arches available.

The filtering in DNF doesn't ensure this, so we have to build it
ourselves. To limit the performance impact, only run this filtering when
there actually are some lookaside repos configured.

JIRA: RHELCMP-11728

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2023-05-31 12:32:35 +00:00
Lubomír Sedlář
e888e76992 Add back compatibility with jsonschema <3.0.0
Resolves: https://pagure.io/pungi/issue/1667
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2023-05-31 07:29:51 +00:00
Lubomír Sedlář
6e72de7efe Remove useless debug message
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2023-05-30 15:46:17 +02:00
Lubomír Sedlář
c8263fcd39 Remove fedmsg from requirements
The code for sending messages in Fedora actually relies on
fedora-messaging library now. However, we do not have any tests for
that, so there's little reason to pull the library in via
requirements.txt

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2023-05-17 09:29:55 +02:00
Lubomír Sedlář
82ca4f4e65 gather: Support dotarch in DNF backend
The documentation claims that dotarch syntax is supported for additional
packages. For yum backend this seems to be handled automatically, but
the dnf backend could not interpret this.

This patch checks if a package is specified in the syntax and contains a
valid architecture. If so, the query will honor the arch.

JIRA: RHELCMP-11728
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2023-05-15 08:42:21 +02:00
Aurélien Bompard
b8b6b46ce7
Set the priority in the fedora-messaging notifier
According to [infra ticket #10899](https://pagure.io/fedora-infrastructure/issue/10899),
ostree messages should have prioriy 3.

Signed-off-by: Aurélien Bompard <aurelien@bompard.org>
2023-05-03 14:20:57 +02:00
Lubomír Sedlář
e9d836c115 Fix compatibility with createrepo_c 0.21.1
The length of the file entry tuple has changed, it can not be unpacked
reliably.

Relates: https://github.com/rpm-software-management/createrepo_c/issues/360
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2023-04-25 10:25:19 +00:00
Lubomír Sedlář
d3f0701e01 comps: Apply arch filtering to environment/optionlist
Let's filter this list too, not just the grouplist tag.

JIRA: RHELCMP-7926
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2023-04-24 08:29:15 +02:00
Haibo Lin
8f6f0f463f Add config file for cleaning up cache files
systemd-tmpfiles is required to enable the auto clean up.

JIRA: RHELCMP-6327
Signed-off-by: Haibo Lin <hlin@redhat.com>
2023-04-12 09:56:43 +08:00
Haibo Lin
467c7a7f6a 4.3.8 release
JIRA: RHELCMP-11448
Signed-off-by: Haibo Lin <hlin@redhat.com>
2023-03-28 18:05:15 +08:00
Lubomír Sedlář
e1d7544c2b createiso: Update possibly changed file on DVD
There's no good way of detecting if buildinstall phase tweaked boot
configuration (and efiboot.img). We should update those files in the DVD
just to be sure.

The .discinfo file is always different and needs to be updated.

Relates: https://pagure.io/pungi/issue/1647
JIRA: RHELCMP-10811
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2023-03-27 12:40:39 +00:00
Lubomír Sedlář
a71c8e23be pkgset: Stop reuse if configuration changed
When options controlling excluding arches change, it should break reuse.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2023-03-22 12:56:02 +00:00
Lubomír Sedlář
ab508c1511 Allow disabling inheriting ExcludeArch to noarch packages
Copying ExcludeArch/ExclusiveArch from source rpm to noarch is an easy
option to block shipping that particular noarch package from a certain
architecture. However, there is no way to bypass it, and it is rather
confusing and not discoverable.

An alternative way to remove an unwanted package is to use the good old
`filter_packages`, which has enough granularity to remove pretty much
anything from anywhere. The only downside is that it requires a change
in configuration, so it can't be done by a packager directly from a spec
file.

When we decide to break backwards compatibility, this option should be
removed and the entire ExcludeArch/ExclusiveArch inheritance removed
completely.

JIRA: ENGCMP-2606
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2023-03-22 12:56:02 +00:00
Lubomír Sedlář
f960b4d155 pkgset: Support extra builds with no tags
This is a rather fringe use case. If the configuration contains
pkgset_koji_builds or pkgset_koji_scratch_tasks but no pkgset_koji_tag,
the compose will be empty.

The expectation though is that the packages should be pulled.

The extra RPMs are added to all non-modular tags because they are
supposed to mask builds from the same packages (e.g. user may want to
explicitly pull in older version than tagged).

This patch adds support for composes containing only explicitly listed
builds by creating a dummy package set that is not actually using any
tag.

JIRA: RHELCMP-11385
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2023-03-17 15:10:35 +01:00
Lubomír Sedlář
602b698080 buildinstall: Avoid pointlessly tweaking the boot images
Only modify boot images if there actually is some change.

The tweak function updates config files with volume id and kickstart
file. Even if we don't have a kickstart and there is no change in the
config files, the image will be regenerated. This leads to a change in
checksum for no good reason.

This patch keeps track of modified config files. If there are none, it
avoids touching anything else.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2023-03-16 07:46:56 +00:00
Haibo Lin
b30f7e0d83 Prevent to reuse if unsigned packages are allowed
JIRA: RHELCMP-8415
Signed-off-by: Haibo Lin <hlin@redhat.com>
2023-03-16 15:32:09 +08:00
Lubomír Sedlář
0c3b6e22f9 Pass parent id/respin id to CTS
When the --target-dir option is used, the compose can be created in CTS,
but the parent and respin information is not passed through. That leads
to data missing later on.

JIRA: RHELCMP-11411
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2023-03-14 10:51:34 +01:00
Haibo Lin
3175ede38a Exclude existing files in boot.iso
JIRA: RHELCMP-10811
Fixes: https://pagure.io/pungi/issue/1647
Signed-off-by: Haibo Lin <hlin@redhat.com>
2023-03-09 15:33:25 +08:00
Lubomír Sedlář
8920eef339 image-build/osbuild: Pull ISOs into the compose
OSBuild tasks can produce ISO files. If they do, we should include them
in the compose, and we should pull them into the iso/ subdirectory
together with other ISOs.

Fixes: https://pagure.io/pungi/issue/1657
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2023-03-06 09:35:47 +01:00
Lubomír Sedlář
58036eab84 Retry 401 error from CTS
This could be a transient error caused by kerberos server instability.

JIRA: RHELCMP-11251
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2023-02-28 10:14:02 +01:00
Lubomír Sedlář
a4476f2570 gather: Better detection of debuginfo in lookaside
If the depsolver wants to include a package that is present in both the
source repo and a lookaside repo, it reliably detects binary packages
present in lookaside, but for debuginfo it's not so reliable.

There is a separate package object for each package in each repo.
Depending on which one is used, debuginfo could be included in the
result or not. This patch fixes that by actually looking if the same
package is present in any lookaside repo.

JIRA: RHELCMP-9373
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2023-02-27 15:33:19 +01:00
Haibo Lin
8c06b7a3f1 Log versions of all installed packages
JIRA: RHELCMP-9493
Signed-off-by: Haibo Lin <hlin@redhat.com>
2023-02-06 18:24:20 +08:00
Lubomír Sedlář
64ae81b416 Use authentication for all CTS calls
The update of compose URL relied on environment being set from the
initial import. This got broken when a unique credentials cache started
to be used, and was cleaned up after the import.

JIRA: RHELCMP-11072
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2023-02-02 13:52:11 +00:00
Lubomír Sedlář
826169af7c Fix black complaints
These are newly detected by black 23.1.0.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2023-02-02 12:53:32 +01:00
Lubomír Sedlář
d97b8bdd33 Add vhd.gz extension to compressed VHD images
JIRA: RHELCMP-11027
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2023-01-31 11:16:58 +01:00
Lubomír Sedlář
8768b23cbe Add vhd-compressed image type
JIRA: RHELCMP-11027
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2023-01-30 09:27:22 +00:00
Lubomír Sedlář
51628a974d Update to work with latest mock
The `called_once` attribute now raises an exception. Switch to
`assert_called_once` method. Also replace `assertTrue(x.called)` with
`x.assert_called()`.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2023-01-26 13:05:48 +01:00
Ondrej Nosek
88327d5784 Default bztar format for sdist command
Usage of the 'bztar' format is unchanged, just changing the way
of configuration. The previous method was deprecated.

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2022-12-12 12:10:54 +01:00
Ondrej Nosek
6e0a9385f2 4.3.7 release
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2022-12-09 13:50:53 +01:00
Lubomír Sedlář
8be0d84f8a
osbuild: test passing of rich repos from configuration
Test that "rich" repositories defined as dicts in the configuration
stay as dicts in the arguments passed to the osbuild phase.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2022-11-28 14:47:11 +01:00
Tomáš Hozza
8f0906be53
osbuild: support specifying package_sets for repos
The `koji-osbuild` plugin supports additional formats for the `repo`
property since v4 [1]. Specifically, a repo can be specified as a
dictionary with `baseurl` key and `package_sets` list containing
specific package set names, that the repository should be used for.

Extend the configuration schema to reflect the plugin change.
Extend the documentation to cover the new repository format.
Extend an existing unit test to specify additional repository using the
added format.

[1] https://github.com/osbuild/koji-osbuild/pull/82

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2022-11-28 14:47:11 +01:00
Tomáš Hozza
e3072c3d5f
osbuild: don't use util.get_repo_urls()
Don't use `util.get_repo_urls()` to resolve provided repositories, but
implement osbuild-specific variant of the function named
`_get_repo_urls(). The reason is that the function from `utils`
transforms repositories defined as dicts to strings, which is
undesired for osbuild. The requirement for osbuild is to preserve the
dict as is, just to resolve the string in `baseurl` to the actual
repository URL.

Add a unit test covering the newly added function. It is inspired by a
similar test from `test_util.py`.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2022-11-28 14:47:11 +01:00
Tomáš Hozza
ef6d40dce4
osbuild: update schema and config documentation
The `koji-osbuild` Hub schema has been relaxed a bit in the latest
release (v11). Adjust the schema in Pungi to reflect changes in
`koji-osbuild`.

For more information on the changes in `koji-osbuild`, see:
https://github.com/osbuild/koji-osbuild/pull/108

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2022-11-28 14:17:42 +01:00
Lubomír Sedlář
df6664098d Speed up tests by 30 seconds
The retry test for CTS doesn't actually need to wait. Let's mock the
sleep function.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-11-23 11:48:12 +01:00
Lubomír Sedlář
147df93f75 Stop sending compose paths to CTS
The tracking service will reject it as it's not an HTTP URL. Let's not
even try.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-11-23 11:48:12 +01:00
Lubomír Sedlář
dd8c1002d4 Report errors from CTS
If the service returns a status code indicating a user error, report
that and do not retry.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-11-23 11:48:12 +01:00
Lubomír Sedlář
12e3a46390 createiso: Create Joliet tree with xorriso
This structure is important for isoinfo -J, which is in turn called by
virt-install.

This can be tested by using a bootable ISO by modifying it with a dummy
additional file and preserving boot records:

    $ xorriso -indev netinst.iso -outdev test.iso -boot_image any replay -map setup.py setup.py -end
    ...
    $ isoinfo -J -i test.iso
    isoinfo: Unable to find Joliet SVD
    $ rm test.iso
    $ xorriso -indev netinst.iso -outdev test.iso -joliet on -boot_image any replay -map setup.py setup.py -end
    ...
    $ isoinfo -J -i test.iso
    $

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=2144105
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-11-22 12:58:46 +01:00
Lubomír Sedlář
479849042f init: Filter comps for modular variants with tags
Modular variants can either be specified by a list of modules, or by a
list of Koji tags. In terms of comps preprocessing there should not be
any difference between the two.

Resolves: https://pagure.io/pungi/issue/1640
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-11-03 11:11:01 +01:00
Haibo Lin
8cd19605bd Retry failed cts requests
JIRA: RHELCMP-10033
Signed-off-by: Haibo Lin <hlin@redhat.com>
2022-10-31 12:29:44 +08:00
Lubomír Sedlář
fa967f79b5 Ignore existing kerberos ticket for CTS auth
When there is an existing kerberos ticket, it gets precedence over the
environment variable with path to a keytab. That is not expected and the
user ticket can possibly lack permissions in CTS to be able to run the
compose successfully.

This patch fixes that by setting KRB5CCNAME to a fresh path. That way
there will not be any valid ticket, since the credentials cache does not
exist yet.

JIRA: RHELCMP-9742
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-09-13 12:38:20 +02:00
Tomas Hozza
57739c238f
osbuild: support specifying upload_options
Since version 9, the `koji-osbuild` plugin supports specifying upload
options as part of a Koji build. This enables one to upload the built
image directly to the cloud environment as part of the image build in
Koji.

Extend the configuration schema with `upload_options`.
Extend the documentation and describe valid `upload_options` values.
Add a unit test testing a scenario when `upload_options` are specified.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-09-06 11:20:01 +02:00
Tomas Hozza
805a1083a2
osbuild: accept only a single image type in the configuration
Modify the osbuild configuration schema to accept only an array with a
single value as the `image_types`, in addition to a single string. The
single string was supported by the schema also before, but this fact was
not mentioned in the documentation, nor it was supported by the
`koji-osbuild` plugin of version lower than `9`.

Update the documentation accordingly.

Add unit test for invalid configuration containing more than one image
type.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-09-06 10:55:25 +02:00
Haibo Lin
57ea640916 Add Jenkinsfile for CI
JIRA: RHELCMP-9800
Signed-off-by: Haibo Lin <hlin@redhat.com>
2022-08-29 17:38:04 +08:00
Lubomír Sedlář
c7121f9378 profiler: Flush stdout before printing
Apparently redirecting stderr to the same pipe as stdout does not
guarantee that the data will not be mangled together.

Flushing stdout before the profiler data is printed should ensure that
it does not end up in the middle of some RPM path.

Fixes: https://pagure.io/pungi/issue/1627
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-08-29 05:57:14 +00:00