We can have a compose with unsigned packages.
By the time the next compose is generated, the packages could have been
signed. However, the new compose would still reuse the ISO with unsigned
copies.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit d546a49299)
Running xorriso to modify an ISO image can update content of included
images such as images/eltorito.img, unless we explicitly update the
image, which is undesirable (https://pagure.io/pungi/issue/1647).
However, when the file is changed, the checksum changes and .treeinfo no
longer matches.
This patch implements a workaround: once the DVD is written, it looks
for incorrect checksums, recalculates them and updates the .treeinfo on
the DVD. Since only the checksum is changing and the size of the file
remains the same, this seems to help fix the issue.
An additional step for implanting MD5 is needed again, as that gets
erased by the workaround.
JIRA: RHELCMP-13664
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 3b2c6ae72a)
This phase was used to create live images with livecd-creator
and 32-bit ARM images with appliance-creator. We also remove
get_create_image_cmd from the Koji wrapper as it was only used
for this phase, remove associated tests, and remove related
configuration settings and documentation.
Fixes: https://pagure.io/pungi/issue/1753
Merges: https://pagure.io/pungi/pull-request/1774
Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit 531f0ef389)
When variant X depends on variant A, Pungi creates a temporary local
lookaside with packages from A. If there's an external lookaside
configured, the list of package for variant A can contain URLs to the
external repo.
Newer versions of createrepo fail when pkglist specifies an unreachable
package, and it doesn't do downloading.
JIRA: RHELCMP-13648
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 4a5106375e)
If a module is not built for specific arches, pungi will skip adding it
to these arches in pkgset phase.
JIRA: RHELCMP-13625
Signed-off-by: Haibo Lin <hlin@redhat.com>
(cherry picked from commit 627b72597e)
Pungi can use either genisoimage or xorriso to create ISOs.
It also needed isoinfo utility for querying volume ID from the ISO
image. However, the utility is part of the genisoimage suite of tools.
On systems that no longer provide genisoimage, the image would be
successfully generate with xorriso, but then pungi would fail to extract
the volume id leading to metadata with missing values.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit bc0334cc09)
The ostree and ostree_container phases both log messages in the
exact same form, which is rather confusing. This will make it
much clearer which message comes from which phase.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit 5c9e79f535)
composetracker expects the failure message to be in a specific
form, but some phases weren't using it. They were phrasing it
slightly differently, which throws off composetracker's parsing.
We could extend composetracker to handle both forms, but it seems
simpler to just make all the phases use a consistent form.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit 9f8377abab)
As discussed in
https://pagure.io/releng/failed-composes/issue/6047#comment-899622
the list of 'acceptable' types and formats (in productmd terms)
is locked down in productmd, we cannot just 'declare' new formats
in pungi as we kinda wound up doing by adding these Kiwi
extensions to the EXTENSIONS dict in image_build phase. So
instead, let's return the image_build phase to the way it was,
and add an additional layer of handling in kiwibuild phase for
these awkward cases, which 'translates' the file suffix to a
format productmd knows about already. This is actually how we
would rather behave anyway, because a Kiwi-produced
`vagrant.libvirt.box` file really is the same kind of thing as an
ImageFactory-produced `vagrant-libvirt.box` file; we want them to
have compatible metadata, we don't want them to look like
different things.
Merges: https://pagure.io/pungi/pull-request/1740
Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit 8fb694f000)
The task needs just URLs, the dics don't bring anything here.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit c80ebb029b)
Neal wanted this to work - he tried using global_description_scm
and global_description_path in the initial PR - but it wasn't
wired up to work. This should make it possible to set
`kiwibuild_description_scm` and `kiwibuild_description_path`.
It also technically lets you set `global_` for both, since the
`get_config` implementation is very generic, but it doesn't add
it to the checks, so you'd still get an "unrecognized config
option" warning, I think. It seems appropriate to encourage
setting this as a phase-level option rather than a global one
since it seems quite specific to the kiwibuild phase.
Merges: https://pagure.io/pungi/pull-request/1737
Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit e90ffdfd93)
The mechanisms here are a bit subtle and the kiwibuild phase
didn't quite get them right. The arg passed to `util.failable`
is supposed to be a boolean, but kiwibuild was passing it the
list of failable arches (which will always evaluate True).
How this is meant to work is that we only make *the Koji task
as a whole* failable (by passing `True` to `util.failable`) if
*all* the arches in it are failable. If *any* arch in the task
is not failable, the task should not be failable.
We allow a subset of arches to fail by passing the Koji task a
list of `optional_arches`, later. If an arch is 'optional', that
arch failing won't cause the Koji task itself to be considered
failed.
This commit fixes the logic (I hope), renames all the variables
and adds a couple of comments to make it clearer what's going on,
and does a bit of making the code simpler.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit 0d306d4964)
Kiwi builds for Azure fixed VHD images are suffixed with "vhdfixed"
instead of plain "vhd". Add that to the list of suffixes.
Signed-off-by: Jeremy Cline <jeremycline@microsoft.com>
(cherry picked from commit 1494f203ce)
According to Neal, Vagrant images produced by Kiwi end in e.g.
`vagrant.libvirt.box` and `vagrant.virtualbox.box` - with a
period between `vagrant` and the image type, not a dash as with
oz. We should accept this slightly different format so we can
correctly derive the productmd `type` and `format` for these.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit 93b4b4ae0f)
The osbuild Koji plugin supports passing customizations for an image
build. This is also supported in the Koji CLI plugin. Some teams want to
pass image customizations for images built as part of Pungi composes.
Extend the osbuild phase to support passing customizations in the Pungi
configuration.
Merges: https://pagure.io/pungi/pull-request/1733
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
(cherry picked from commit e738f65458)
Getting the images from task is less hacky then matching on filenames.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit b032425f30)
The keys may not exist anymore. If there's nothing to delete, it's fine.
JIRA: RHELCMP-13464
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 52c2cea0ef)
The plugin supports two modes of operation:
1. Mount a shared storage volume into the runroot and have the output
written there.
2. Have the plugin create a tar.gz with the outputs and upload them to
the hub, from where they can be downloaded.
This patch switches from option 1 to option 2.
This requires all input repositories to be passes in as URLs and not
paths. Once the task finishes, Pungi will download the output archives
and unpack them into the expected locations.
JIRA: RHELCMP-13284
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit f25489d060)
The config repository is cloned into a path that conflicts with the
regular ostree phase. Let's use a unique name to avoid that problem.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 7e779aa90f)
This patch connects the phase into the main script, and adds other
modifications:
* The archive is now stored in the images/ subdirectory in the compose.
* Documentation is updated to correctly mention that variant repos are
not available.
* Configuration for path and name of the final archive is dropped. There
are reasonable defaults for this and there's no point in having users
configure it.
* The extra message for the archive is no longer sent.
* The pungi-make-ostree utility is no longer required in the buildroot.
The pungi-make-ostree utility doesn't do any significant work. It
modifies configuration files (which can happen on the compose host), and
it starts other processes.
This patch changes the ostree-container phase to no longer need the
script in the buildroot. Instead, the utility is called on the compose
host to do the config manipulation and output the needed commands. Those
are then passed into the runroot task.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 081c31238b)
I don't love inferring the type from the filename like this -
it's kinda backwards - but it's an improvement on the current
logic (I don't think 'dvd' is ever currently the correct value
here, I don't think osbuild *can* currently build the type of
image that 'dvd' is meant to indicate). I can't immediately see
any better source of data here (we could use the 'name' or
'package_name' from 'build_info', but those are pretty much
just inputs to the filenames anyway).
Types that are possible in productmd but not covered here are
'cd' (never likely to be used again in Fedora at least, not sure
about RHEL), 'dvd-debuginfo' (again not used in Fedora, may be
used in RHEL), 'ec2', 'kvm' (not sure about those), 'netinst'
(this is a synonym for 'boot', we use 'boot' in practice in
Fedora metadata), 'p2v' and 'rescue' (not sure).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit aa7fcc1c20)
Generally we want all packages to come from particular event.
There are two exceptions: packages configured via `pkgset_koji_builds`
are pulled in by exact NVR and skip event; and modules in
`pkgset_koji_modules` are pulled in by NSVC and also ignore events.
However, the modular content tag did honor event, and could lead to a
crashed compose if the content tag did not exist at the configured
event.
This patch is a slightly too big hammer. It ignores events for all
modules, not just ones configured by explicit NSVC. It's not a huge deal
as the content tags are created before the corresponding module build is
created, and once all rpm builds are tagged into the content tag, MBS
will never change it again.
JIRA: RHELCMP-12765
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit b32c8f3e5e)
If the module tag contains the same module build multiple times (because
it's in multiple tags in the inheritance), Pungi will not process that
correctly and try to include the same NSVC in the compose multiple
times. That leads to a crash.
This patch adds another step to the inheritance filter to ensure the
result contains each module only once.
JIRA: RHELCMP-12768
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 935da7c246)
```
pungi/checks.py:575:17: F601 [*] Dictionary key literal `"type"` repeated
pungi/phases/pkgset/pkgsets.py:617:12: E721 Do not compare types, use `isinstance()`
tests/test_pkgset_source_koji.py:241:16: E721 Do not compare types, use `isinstance()`
tests/test_pkgset_source_koji.py:244:16: E721 Do not compare types, use `isinstance()`
tests/test_pkgset_source_koji.py:370:16: E721 Do not compare types, use `isinstance()`
tests/test_pkgset_source_koji.py:374:20: E721 Do not compare types, use `isinstance()`
```
Signed-off-by: Timothée Ravier <tim@siosm.fr>
(cherry picked from commit 2534ddee99)
Allow the manifest type used to be specified in the pungi configuration
instead of always selecting the manifest type based on the koji output.
Signed-off-by: Simon de Vlieger <cmdr@supakeen.com>
(cherry picked from commit f30a8b4d15)
Rather than trying to use local access when it's accessible, let user
make the decision:
* if koji_cache is configured use it and download stuff
* if not, fall back to local access
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 0d3cd150bd)
When a real build is downloaded, Koji can provide a checksum via API.
This commit adds verification of that checksum.
A mismatch will abort the compose. If Koji doesn't provide a checksum
for the particular sigkey, no checking will happen.
Nothing is still checked for scratch builds and images.
This patch requires Koji 1.32. When talking to an older version, there
is no checking done.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 77f8fa25ad)
This patch extends the ability to download files from Koji to image
building phases too.
There is no integrity checking for the downloaded images.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit bf3e9bc53a)
With this patch, Pungi can be configured with a local directory to be
used as a cache for RPMs, and it will download packages from Koji over
HTTP instead of reading them from filesystem directly.
The files from the cache can then be hardlink as usual.
There is locking in place to avoid different composes running at the
same time to step on each other.
This is now supported for RPMs only, be it real builds or scratch
builds.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 631bb01d8f)
The exceptions from libmodulemd are not particularly helpful as they do
not contain information about what file caused it.
modulemd-yaml-error-quark: Failed to open file: Permission denied (0)
This patch should add the path to the problematic file into the message.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 14e025a5a1)
There's no good way of detecting if buildinstall phase tweaked boot
configuration (and efiboot.img). We should update those files in the DVD
just to be sure.
The .discinfo file is always different and needs to be updated.
Relates: https://pagure.io/pungi/issue/1647
JIRA: RHELCMP-10811
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit e1d7544c2b)
When options controlling excluding arches change, it should break reuse.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit a71c8e23be)
Copying ExcludeArch/ExclusiveArch from source rpm to noarch is an easy
option to block shipping that particular noarch package from a certain
architecture. However, there is no way to bypass it, and it is rather
confusing and not discoverable.
An alternative way to remove an unwanted package is to use the good old
`filter_packages`, which has enough granularity to remove pretty much
anything from anywhere. The only downside is that it requires a change
in configuration, so it can't be done by a packager directly from a spec
file.
When we decide to break backwards compatibility, this option should be
removed and the entire ExcludeArch/ExclusiveArch inheritance removed
completely.
JIRA: ENGCMP-2606
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit ab508c1511)
