Commit Graph

870 Commits

Author SHA1 Message Date
Haibo Lin
57ea640916 Add Jenkinsfile for CI
JIRA: RHELCMP-9800
Signed-off-by: Haibo Lin <hlin@redhat.com>
2022-08-29 17:38:04 +08:00
Ondřej Budai
779793386c osbuild: add support for building ostree artifacts
In order to start building Fedora IoT images with osbuild, we need to be able
to pass ostree options from pungi to the koji's osbuildImage task.

This commit adds support for it via new configuration options: ostree_url,
ostree_url and ostree_parent.

A test was added to cover these new options and they are were also added
into the documentation.

JIRA: COMPOSER-1702
Merges: https://pagure.io/pungi/pull-request/1624
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-08-25 15:35:17 +02:00
Timothée Ravier
603c61a033 ostree: Add unified core mode for compose in rpm-ostree
rpm-ostree is moving to unified core composes and this is now working
for Silverblue & Kinoite.

This is untested for IoT but they should move to os-build with Fedora
37.

See: https://github.com/coreos/rpm-ostree/issues/729
Merges: https://pagure.io/pungi/pull-request/1626
Signed-off-by: Timothée Ravier <tim@siosm.fr>
2022-08-23 10:52:45 +02:00
Lubomír Sedlář
11fa342507 createiso: Make ISO level more granular
Make it possible to set the level separately for each variant and
architecture.

JIRA: RHELCMP-9341

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-08-16 14:38:18 +02:00
Lubomír Sedlář
13ea8e5834 Create DVDs with xorriso
Use a different approach for building DVDs when xorriso is enabled.

The default of using genisoimage is not changed at all. When the config
option is set to use xorriso, the actual execution is different between
bootable and non-bootable images.

The non-bootable images are still created by running xorrisofs (which is
a compatibility tool with same UI as genisoimage). Since the image is
not bootable, there should be no problems with boot options.

For bootable images, Pungi will instead take the boot.iso generated by
Lorax, and use xorriso to inject all the extra files into the image.

The shell script that used to invoke all the commands to build the ISO
now runs the `xorriso` command in interactive mode and feeds another
file into it. The new file contains the xorriso commands to add the
required files to the image.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-08-16 14:30:41 +02:00
Lubomír Sedlář
778dcfa587 Fix black complaint
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-08-10 09:04:41 +02:00
Marek Kulik
ca185aaea8 Fix module defaults and obsoletes validation
- Remove validation for modules obsoletes
  We can have multiple obsoletes for one module
- Add unit tests to cover basic scenarios for
  modules defaults && obsoletes
- Add additional check for invalid yaml file
  in Defaults. Previously, empty list of default would
  be returned when invalid yaml is present in Defaults
  directory.
- Using MergeIndex for Obsoletes only (for now).

https://pagure.io/pungi/issue/1592

Signed-off-by: Marek Kulik <mkulik@redhat.com>
2022-06-10 11:35:26 +00:00
Lubomír Sedlář
80957f5205 kojiwrapper: Ignore warnings before task id
When looking for task ID in output of koji runroot command, do not check
just the first line. Instead look for first line that contains just a
number.

Most of the time, this should really be the first line. But if koji
client decides to print any warnings, this patch should skip that.

JIRA: RHELCMP-8944
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-04-26 08:15:13 +02:00
Haibo Lin
c5cdd498ac Revert "Do not clone the same repository multiple times, re-use already cloned repository"
This reverts commit 330ba9b9c4.

As of RHELCMP-8874, revert this patch as a quick fix.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2022-04-13 16:21:55 +08:00
Christopher O'Brien
d55770898c nomacboot option for livemedia koji tasks
Merges: https://pagure.io/pungi/pull-request/1591
Signed-off-by: Christopher O'Brien <cobrien@redhat.com>
2022-03-23 09:36:51 +01:00
Ken Dreyer
b805ce3d12 osbs: only handle archives of type "image"
Prior to this change, if a container image used Cachito with OSBS, then
OSBS would store additional "remote-sources" files in the Koji archives
for the build. Pungi cannot parse the metadata for these archive
entries, so it would crash in add_metadata():

  File "pungi/phases/osbs.py", line 81, in process
    self.worker(compose, variant, config)
  File "pungi/phases/osbs.py", line 141, in worker
    nvr, archive_ids = add_metadata(variant, task_id, compose, scratch)
  File "pungi/phases/osbs.py", line 447, in add_metadata
    arch = archive["extra"]["image"]["arch"]
  KeyError: 'image'

Tell Koji to only return container image archives, and ignore these
remote-source archives.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2022-03-17 11:04:02 -04:00
Ozan Unsal
ecb1646042 Fix the wrong working directory for the progress_notification script
Jira: RHELCMP-7901

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
2022-03-01 11:25:06 +00:00
Haibo Lin
6c280f2c46 Filter out environment groups unmatch given arch
JIRA: RHELCMP-7926
Signed-off-by: Haibo Lin <hlin@redhat.com>
2022-02-22 11:33:25 +08:00
Ozan Unsal
330ba9b9c4 Do not clone the same repository multiple times, re-use already cloned repository
Clone the directory to the compose tmp directory
Update the test_scm in order to create real Compose object. Mock objects are not allowed
to create/delete files for preventing multiple clones

JIRA: RHELCMP-5250

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
2022-01-26 15:57:13 +01:00
Lubomír Sedlář
32221e8f36 hybrid: Explicitly pull in debugsource packages
This should cover case where we there's a build like this:

foo-1-1.src.rpm
  foo-sub-1-1.noarch.rpm
  foo-debugsource-1-1.x86_64.rpm

The compose contains the noarch package, and should also have the
debugsource package. The original code only checked for
foo-sub-debugsource though.

JIRA: RHELCMP-7628
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-01-12 13:59:09 +00:00
Filip Valder
fe986d68b9 Add module obsoletes feature
JIRA: MODULAR-113
Merges: https://pagure.io/pungi/pull-request/1578
Signed-off-by: Filip Valder <fvalder@redhat.com>
2022-01-12 12:51:37 +01:00
Haibo Lin
20c2e59218 Pass compose parameter for debugging git issue
With this param, get_dir_from_scm will try to copy the tmp git dir to
compose target dir when error occurs.

This does not fix the issue but it would be helpful for debugging when
it occurs again.

JIRA: RHELCMP-7244
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-11-26 15:40:35 +08:00
Haibo Lin
f681956cf1 Fix tests for python 2.6
It failed to build RHEL 6 package as logging.NullHandler does not exist
in python 2.6

JIRA: RHELCMP-7188
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-11-12 17:04:17 +08:00
Lubomír Sedlář
1d654522be Remove with_jigdo argument
It was checked in a condition together with the configuration value, and
only ever explicitly used with the same value.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-11-04 13:37:51 +00:00
Lubomír Sedlář
80bd254347 Check dependencies after config validation
This way the checks can rely on default values from the config.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-11-04 13:37:51 +00:00
Ken Dreyer
94ffa1c5c6 default "with_jigdo" to False
Fedora has not composed with jigdo in a long time. Disable it by
default.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
Merges: https://pagure.io/pungi/pull-request/1561
Fixes: https://pagure.io/pungi/issue/1560
2021-11-04 13:37:51 +00:00
Lubomír Sedlář
9d02f87c99 Stop trying to validate non-existent metadata
When a compose doesn't build any images, it won't produce any metadata
file for them, and thus it makes no sense to validate it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Fixes: https://pagure.io/pungi/issue/1565
2021-11-04 09:57:20 +01:00
Lubomír Sedlář
e2b3002726 repoclosure: Use --forcearch for dnf repoclosure
DNF repoclosure requires this option when checking a repository that is
not compatible with host architecture. It seems that when it is
compatible, it works as well.

Based on how the list of architectures is generated, we know that the
main one will always be first.

Fixes: https://pagure.io/pungi/issue/1562
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-11-02 08:51:40 +01:00
Lubomír Sedlář
e8305f3978 extra_isos: Allow reusing old images
When nothing in configuration or the image itself changed, let's just
copy the older one.

JIRA: RHELCMP-5969
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-11-02 07:08:00 +00:00
Lubomír Sedlář
ac66c3d7f3 createiso: Allow reusing old images
This patch allows Pungi to reuse ISO image created in previous compose
if a list of assumptions proves to hold:

 * If image is bootable, buildinstall phase must have been reused too.
 * Compose configuration must have not changed (except for a few
   whitelisted options).
 * Volume ID of the ISO much not have changed.
 * No RPM on the ISO must have changed.

The ISO also contains other files. Changes in extra files and product ID
certificates should be visible in configuration (the SHA will differ).
Similarly any repodata configuration would be reflected in
configuration.

JIRA: RHELCMP-5969
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-11-02 07:08:00 +00:00
Lubomír Sedlář
eb61c97cdb Remove default runroot channel
When the value is not specified in the configuration file, let Koji pick
the default channel.

JIRA: RHELBLD-8088
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-10-25 15:56:24 +02:00
Lubomír Sedlář
204d88a351 Add missing mock to osbs tests
We don't want the test to try to a dummy URL.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-10-21 13:34:37 +02:00
Haibo Lin
8133676270 osbs: Reuse images from old compose
JIRA: RHELCMP-5972
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-10-20 14:22:19 +08:00
Haibo Lin
e42e65783d image_build: Allow reusing old image_build results
JIRA: RHELCMP-5970
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-10-20 14:07:02 +08:00
Jan Kaluza
0530cf2712 When cts_url is configured, use CTS /repo API for buildContainer yum_repourls.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2021-09-24 10:29:28 +02:00
Jan Kaluza
195bfbefa4 Allow specifying $COMPOSE_ID in the repo value for osbs phase.
There should be an option for `yum_repourls` to point to static
URL, for example when CTS is used. The idea is that instead of
setting `repo` to `AppStream`, we could use link similar to this one:

`https://cts.localhost/api/1/composes/$COMPOSE_ID/repo/?variant=AppStream`

This would be translated to real static link during the OSBS phase:

`https://cts.localhost/api/1/composes/CentOS-Stream-9-20210803.0/repo/?variant=AppStream`

That way this statis link would appear in the yum_repourls.

Merges: https://pagure.io/pungi/pull-request/1543
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2021-09-09 10:46:33 +02:00
Lubomír Sedlář
20dc4beb6b Make getting old compose config reusable
The file will only be loaded once, it gets cached afterwards.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-09-09 10:38:34 +02:00
Dominik Rumian
7c3e8d4276 Fix tests for createrepo
Tests for createrepo failed when pungi is installed in system.

JIRA: RHELCMP-6209

Signed-off-by: Dominik Rumian <drumian@redhat.com>
2021-08-30 13:48:29 +02:00
Dominik Rumian
9cd42a2b5e Formatted files according to flake8 and black feedback
Signed-off-by: Dominik Rumian <drumian@redhat.com>
2021-08-30 09:55:17 +02:00
Haibo Lin
1bb038ca72 Install missing deps in ci image
tests requiring libmodulemd are skipped due to missing deps and
this patch could fix the issue.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-08-17 12:01:51 +08:00
Filip Valder
a7c111643d Supersede ModuleStream loading with ModuleIndex
- Use ModuleIndex's update_from_file/update_from_string instead of ModuleStream's
read_file/read_string which is deprecated.
- Extend tests to work with real module streams instead of mocks.

Signed-off-by: Filip Valder <fvalder@redhat.com>
2021-08-12 16:32:38 +02:00
Dominik Rumian
5831d4ae1e Better error message than 'KeyError' in pungi
Jira: RHELCMP-6107

Signed-off-by: Dominik Rumian <drumian@redhat.com>
2021-08-12 10:52:36 +00:00
Dominik Rumian
8a2d0162d9 Better error message than 'KeyError' in pungi
JIRA: RHELCMP-6107

Signed-off-by: Dominik Rumian <drumian@redhat.com>
2021-08-10 07:56:47 +00:00
fdiprete
446334fb95 show and log command when using the run_blocking_cmd() method [RHELCMP-2243]
Signed-off-by: fdiprete <fdipretre@redhat.com>
2021-08-03 10:58:18 +00:00
Haibo Lin
56a55db966 Use cachedir when createrepo
Then createrepo can reuse checksum values from cache to make it faster.

JIRA: RHELCMP-5984
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-07-27 09:23:47 +08:00
Lubomír Sedlář
7fe32ae758 util: Strip file:// from local urls
Make sure that the function returns a path even for local files
specified by file:// urls.

JIRA: RHELCMP-5340
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-05-06 12:37:55 +02:00
Lev Veyde
00a9861367 Updated the deprecated ks argument name (to the current inst.ks)
Signed-off-by: Lev Veyde <lveyde@redhat.com>
2021-04-21 21:04:01 +03:00
Haibo Lin
e866d22c04 gather: Adjust reusing with lookaside
- Do not reuse when there is any external lookaside repo
- Do not reuse when lookaside variant is not reused

JIRA: RHELCMP-4596
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-04-16 10:37:47 +08:00
Lubomír Sedlář
ab1b5b48ec hybrid: Optimize getting lookaside packages
The original code ended up downloading all repodata from the lookaside
repo. This could cause a lot of memory to be used.

The new code only downloads the repomd.xml and then primary record,
which is sufficient to obtain all needed information. A lot less memory
is used and the code is also significantly faster.

Here are some alternative ways of getting a list of packages from the
lookaside repo and reasons why they did not work:

 * dnf repoquery - this doesn't include modular packages unless the
   stream is default
 * dnf reposync - requires `--urls` option to only print the names,
   which is not available on RHEL 7

JIRA: RHELCMP-4761
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-04-15 11:02:04 +02:00
Haibo Lin
c8091899b2 gather: Copy old logs when reusing gather result
This would be helpful for debugging.

JIRA: RHELCMP-4594
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-04-01 14:42:20 +08:00
Haibo Lin
035b37c566 Cancel koji tasks when pungi terminated
JIRA: RHELCMP-4148
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-03-23 14:47:48 +08:00
Haibo Lin
edb4517e80 Add Dockerfile for building testing image
There are two images because it's hard to install both Python 2
and Python 3 packages (e.g. libcomps) in latest fedora release.

JIRA: RHELCMP-4580
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-03-15 18:13:25 +08:00
Lubomír Sedlář
535034ef91 image_container: Fix incorrect arch processing
OSBS will reject no scratch builds with arch_override.

When the option is not specified in Pungi, it would do `"".split(" ")`
to get list of arches, which returns a list with empty string instead of
an empty list.

With this fixed, it might be possible to have multiple images match the
spec (unless arch is used in the filter). To fix that, we can replace
arch with $basearch variable.

JIRA: RHELCMP-3824
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-03-08 11:57:48 +01:00
Haibo Lin
2769232b72 runroot: Adjust permissions always
Previously commands to adjust permissions do not run when main
command failed and then files can't be cleaned up due to
Permission Denied problem.

JIRA: RHELCMP-4253
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-03-04 16:08:06 +08:00
Haibo Lin
b217470464 Format code
Code didn't get well formatted when jenkins unusable.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-03-02 18:31:45 +08:00