Don't use `util.get_repo_urls()` to resolve provided repositories, but
implement osbuild-specific variant of the function named
`_get_repo_urls(). The reason is that the function from `utils`
transforms repositories defined as dicts to strings, which is
undesired for osbuild. The requirement for osbuild is to preserve the
dict as is, just to resolve the string in `baseurl` to the actual
repository URL.
Add a unit test covering the newly added function. It is inspired by a
similar test from `test_util.py`.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Modular variants can either be specified by a list of modules, or by a
list of Koji tags. In terms of comps preprocessing there should not be
any difference between the two.
Resolves: https://pagure.io/pungi/issue/1640
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Since version 9, the `koji-osbuild` plugin supports specifying upload
options as part of a Koji build. This enables one to upload the built
image directly to the cloud environment as part of the image build in
Koji.
Extend the configuration schema with `upload_options`.
Extend the documentation and describe valid `upload_options` values.
Add a unit test testing a scenario when `upload_options` are specified.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Pungi expects each module to be built for all architectures by default.
Unless the module is filtered out, missing metadata for a particular
arch would cause it to crash with a incomprehensible error message. This
should make it a little better.
Relates: https://pagure.io/releng/failed-composes/issue/3889
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
In order to start building Fedora IoT images with osbuild, we need to be able
to pass ostree options from pungi to the koji's osbuildImage task.
This commit adds support for it via new configuration options: ostree_url,
ostree_url and ostree_parent.
A test was added to cover these new options and they are were also added
into the documentation.
JIRA: COMPOSER-1702
Merges: https://pagure.io/pungi/pull-request/1624
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Use a different approach for building DVDs when xorriso is enabled.
The default of using genisoimage is not changed at all. When the config
option is set to use xorriso, the actual execution is different between
bootable and non-bootable images.
The non-bootable images are still created by running xorrisofs (which is
a compatibility tool with same UI as genisoimage). Since the image is
not bootable, there should be no problems with boot options.
For bootable images, Pungi will instead take the boot.iso generated by
Lorax, and use xorriso to inject all the extra files into the image.
The shell script that used to invoke all the commands to build the ISO
now runs the `xorriso` command in interactive mode and feeds another
file into it. The new file contains the xorriso commands to add the
required files to the image.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The pickle files are used for reusing results from old compose and the
failure should not block the compose process.
JIRA: RHELCMP-9494
Signed-off-by: Haibo Lin <hlin@redhat.com>
Checking start of the line is not sufficient for extra_isos that have
the variants in separate directories.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
- Remove validation for modules obsoletes
We can have multiple obsoletes for one module
- Add unit tests to cover basic scenarios for
modules defaults && obsoletes
- Add additional check for invalid yaml file
in Defaults. Previously, empty list of default would
be returned when invalid yaml is present in Defaults
directory.
- Using MergeIndex for Obsoletes only (for now).
https://pagure.io/pungi/issue/1592
Signed-off-by: Marek Kulik <mkulik@redhat.com>
Prior to this change, if a container image used Cachito with OSBS, then
OSBS would store additional "remote-sources" files in the Koji archives
for the build. Pungi cannot parse the metadata for these archive
entries, so it would crash in add_metadata():
File "pungi/phases/osbs.py", line 81, in process
self.worker(compose, variant, config)
File "pungi/phases/osbs.py", line 141, in worker
nvr, archive_ids = add_metadata(variant, task_id, compose, scratch)
File "pungi/phases/osbs.py", line 447, in add_metadata
arch = archive["extra"]["image"]["arch"]
KeyError: 'image'
Tell Koji to only return container image archives, and ignore these
remote-source archives.
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
This should cover case where we there's a build like this:
foo-1-1.src.rpm
foo-sub-1-1.noarch.rpm
foo-debugsource-1-1.x86_64.rpm
The compose contains the noarch package, and should also have the
debugsource package. The original code only checked for
foo-sub-debugsource though.
JIRA: RHELCMP-7628
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
With this param, get_dir_from_scm will try to copy the tmp git dir to
compose target dir when error occurs.
This does not fix the issue but it would be helpful for debugging when
it occurs again.
JIRA: RHELCMP-7244
Signed-off-by: Haibo Lin <hlin@redhat.com>
The JSON source file should not require a mapping for all
variants/architectures. When something is specified, it should be
included.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It was checked in a condition together with the configuration value, and
only ever explicitly used with the same value.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When a compose doesn't build any images, it won't produce any metadata
file for them, and thus it makes no sense to validate it.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Fixes: https://pagure.io/pungi/issue/1565
When nothing in configuration or the image itself changed, let's just
copy the older one.
JIRA: RHELCMP-5969
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch allows Pungi to reuse ISO image created in previous compose
if a list of assumptions proves to hold:
* If image is bootable, buildinstall phase must have been reused too.
* Compose configuration must have not changed (except for a few
whitelisted options).
* Volume ID of the ISO much not have changed.
* No RPM on the ISO must have changed.
The ISO also contains other files. Changes in extra files and product ID
certificates should be visible in configuration (the SHA will differ).
Similarly any repodata configuration would be reflected in
configuration.
JIRA: RHELCMP-5969
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The image type value passed to the task doesn't match the type as it
will be recorded by Koji.
JIRA: RHELCMP-5727
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
- Use ModuleIndex's update_from_file/update_from_string instead of ModuleStream's
read_file/read_string which is deprecated.
- Extend tests to work with real module streams instead of mocks.
Signed-off-by: Filip Valder <fvalder@redhat.com>
Multithreading was added to parallelize the
computation of image checksums. Resulting memory structures
are protected via synchronization primitives. Max number of
threads is uncapped- experiments were done to determine
whether a maximum number of threads would yield greater
efficiency and there were no gains from this.
Likewise, experiments were done to determine whether pools of
threads computed in separate processes could likewise decrease
compute-time. Evidence did not suggest that this was the
case. This indicate that the checksum operation is bounded
by I/O read/write times.
Merges: https://pagure.io/pungi/pull-request/1520
Jira: RHELCMP-5967
Signed-off-by: James Kunstle jkunstle@redhat.com
The original code could cause a source RPM to be present in two variants
that have a dependency relation.
There is always only one source repo for a variant in the final compose.
When gathering packages for a variant that depends on another variant,
we need to build a temporary lookaside repo that has similar content to
the parent variant. This lookaside only contained source RPMs for
packages present the the architecture.
This could result in duplicated SRPMs in the compose.
Example situation:
* Variant B depends on variant A.
* A contains foo.x86_64.rpm (only on x86_64)
* B pulls in subpackage foo-bar.s390x.rpm (on s390x)
Source repo for A will correctly contain foo.src.rpm. With original code
the srpm would also end up in B.src. By adding all sources to the
temporary lookaside Pungi will know that source repo for B doesn't need
to duplicate the package.
The refactoring to use a set to store the packages is meant to avoid
listing the same SRPM multiple times in the repo in the most common
situation when SRPM is listed in multiple architectures.
JIRA: RHELCMP-6002
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It is possible to try to re-run a compose with old event. When trying to
reuse pkgset data, we must use set the bounds not based on
current/reused event, but actually check which was first.
JIRA: CWFHEALTH-495
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
- Do not reuse when there is any external lookaside repo
- Do not reuse when lookaside variant is not reused
JIRA: RHELCMP-4596
Signed-off-by: Haibo Lin <hlin@redhat.com>
Tomáš Hozza
Lubomír Sedlář
Ondřej Budai
Timothée Ravier
Haibo Lin
Marek Kulik
Lingyan Zhuang
Christopher O'Brien
Ken Dreyer
Filip Valder
Ozan Unsal
fdiprete
Jan Kaluza
Dominik Rumian
JamesKunstle
Lev Veyde