Pungi would by default only ever add files to the cache. That would
eventually result in essentially a mirror of the Koji volume.
This patch adds a helper cleanup script. When called, it goes through
files in the cache and deletes anything that is not hardlinked from
elsewhere and with mtime not updated recently.
Cleaning up files that hardlinked from some compose would not save any
space anyway. The mtime check should account for cases like subpackage
being downloaded but not included in any compose. This would avoid it
from being downloaded over and over again.
When a compose fails or is aborted, there can be a stale lock file left
behind in the cache. This script cleans that up too.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit e6d9f31ef4 (centos_master))
This patch extends the ability to download files from Koji to image
building phases too.
There is no integrity checking for the downloaded images.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit bf3e9bc53a)
With this patch, Pungi can be configured with a local directory to be
used as a cache for RPMs, and it will download packages from Koji over
HTTP instead of reading them from filesystem directly.
The files from the cache can then be hardlink as usual.
There is locking in place to avoid different composes running at the
same time to step on each other.
This is now supported for RPMs only, be it real builds or scratch
builds.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 631bb01d8f)
It's not a great name anymore though, because it will fail the compose
if any input package is missing, no matter whether it's from comps,
prepopulate or additional_packages.
JIRA: RHELCMP-12484
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 1c4275bbfa)
The exceptions from libmodulemd are not particularly helpful as they do
not contain information about what file caused it.
modulemd-yaml-error-quark: Failed to open file: Permission denied (0)
This patch should add the path to the problematic file into the message.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 14e025a5a1)
This patch adds an additional field `options` to scm_dict, which can be
used to provide additional information to the backends.
It implements a single new option for GitWrapper. This option allows
setting a custom git credentials wrapper. This can be useful if Pungi
needs to get files from a git repository that requires authentication.
The helper can be as simple as this (assuming the username is already
provided in the url):
#!/bin/sh
echo password=i-am-secret
The helper would need to be referenced by an absolute path from the
pungi configuration, or prefixed with ! to have git interpret it as a
shell script and look it up in PATH.
See https://git-scm.com/docs/gitcredentials for more details.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
JIRA: RHELCMP-11808
(cherry picked from commit ada8f4e346)
The initial version of the filtered the latest builds at the start. That
doesn't matter in many cases:
* When there are no lookaside repos, there is generally a single version
of each package.
* When lookaside repos do not overlap with compose repos, or contain
only older versions.
It is however a problem when the lookaside repos contain higher version
of a package than what is in a compose repo, and some package explicitly
requires the older version.
Consider this scenario:
* lookaside contains bar-1.1
* compose repo contains bar-1.0 and foo-1.0
* foo-1.0 `Requires: bar < 1.1`
The original code would filter out the bar-1.0 package, and then fail on
unresolved dependencies.
This patch moves the computation of latest packages much later, to part
of code where all options to satisfy a dependency are selected and the
best match is chosen. At that point if there are multiple versions
available, we do want the latest one.
JIRA: SPMM-13483
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit bcc440491e)
Without this option the names reported by joliet tree are truncated.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit fa50eedfad)
This was never actually used.
JIRA: RHELCMP-10218
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit b7adbf8a91 (centos_master))
The genisoimage backend uses the -rational-rock option, which sets uid
and gid to 0, and makes file readable by everyone.
With xorriso this must be done explicitly. Setting ownership is a single
command, but the permissions require a per-file command to not make
files executable where not needed.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=2203888
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 82ae9e86d5 (centos_master))
If lookaside contains an older version of a package, but with a
different arch, the depsolver doesn't notice that and prefers the
lookaside version.
This is not correct. The latest package should be used no matter if
there are different arches available.
The filtering in DNF doesn't ensure this, so we have to build it
ourselves. To limit the performance impact, only run this filtering when
there actually are some lookaside repos configured.
JIRA: RHELCMP-11728
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 2ad341a01c)
The code for sending messages in Fedora actually relies on
fedora-messaging library now. However, we do not have any tests for
that, so there's little reason to pull the library in via
requirements.txt
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit c8263fcd39 (centos_master))
The documentation claims that dotarch syntax is supported for additional
packages. For yum backend this seems to be handled automatically, but
the dnf backend could not interpret this.
This patch checks if a package is specified in the syntax and contains a
valid architecture. If so, the query will honor the arch.
JIRA: RHELCMP-11728
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 82ca4f4e65)
Let's filter this list too, not just the grouplist tag.
JIRA: RHELCMP-7926
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit d3f0701e01)
systemd-tmpfiles is required to enable the auto clean up.
JIRA: RHELCMP-6327
Signed-off-by: Haibo Lin <hlin@redhat.com>
(cherry picked from commit 8f6f0f463f)
There's no good way of detecting if buildinstall phase tweaked boot
configuration (and efiboot.img). We should update those files in the DVD
just to be sure.
The .discinfo file is always different and needs to be updated.
Relates: https://pagure.io/pungi/issue/1647
JIRA: RHELCMP-10811
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit e1d7544c2b)
When options controlling excluding arches change, it should break reuse.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit a71c8e23be)
Copying ExcludeArch/ExclusiveArch from source rpm to noarch is an easy
option to block shipping that particular noarch package from a certain
architecture. However, there is no way to bypass it, and it is rather
confusing and not discoverable.
An alternative way to remove an unwanted package is to use the good old
`filter_packages`, which has enough granularity to remove pretty much
anything from anywhere. The only downside is that it requires a change
in configuration, so it can't be done by a packager directly from a spec
file.
When we decide to break backwards compatibility, this option should be
removed and the entire ExcludeArch/ExclusiveArch inheritance removed
completely.
JIRA: ENGCMP-2606
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit ab508c1511)
This is a rather fringe use case. If the configuration contains
pkgset_koji_builds or pkgset_koji_scratch_tasks but no pkgset_koji_tag,
the compose will be empty.
The expectation though is that the packages should be pulled.
The extra RPMs are added to all non-modular tags because they are
supposed to mask builds from the same packages (e.g. user may want to
explicitly pull in older version than tagged).
This patch adds support for composes containing only explicitly listed
builds by creating a dummy package set that is not actually using any
tag.
JIRA: RHELCMP-11385
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit f960b4d155)
Only modify boot images if there actually is some change.
The tweak function updates config files with volume id and kickstart
file. Even if we don't have a kickstart and there is no change in the
config files, the image will be regenerated. This leads to a change in
checksum for no good reason.
This patch keeps track of modified config files. If there are none, it
avoids touching anything else.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 602b698080)
When the --target-dir option is used, the compose can be created in CTS,
but the parent and respin information is not passed through. That leads
to data missing later on.
JIRA: RHELCMP-11411
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 0c3b6e22f9 (centos_master))
OSBuild tasks can produce ISO files. If they do, we should include them
in the compose, and we should pull them into the iso/ subdirectory
together with other ISOs.
Fixes: https://pagure.io/pungi/issue/1657
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 8920eef339)
This could be a transient error caused by kerberos server instability.
JIRA: RHELCMP-11251
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 58036eab84)
If the depsolver wants to include a package that is present in both the
source repo and a lookaside repo, it reliably detects binary packages
present in lookaside, but for debuginfo it's not so reliable.
There is a separate package object for each package in each repo.
Depending on which one is used, debuginfo could be included in the
result or not. This patch fixes that by actually looking if the same
package is present in any lookaside repo.
JIRA: RHELCMP-9373
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit a4476f2570)
The update of compose URL relied on environment being set from the
initial import. This got broken when a unique credentials cache started
to be used, and was cleaned up after the import.
JIRA: RHELCMP-11072
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 64ae81b416)
The `called_once` attribute now raises an exception. Switch to
`assert_called_once` method. Also replace `assertTrue(x.called)` with
`x.assert_called()`.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 51628a974d)
Usage of the 'bztar' format is unchanged, just changing the way
of configuration. The previous method was deprecated.
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
(cherry picked from commit 88327d5784)
