Commit Graph

2805 Commits

Author SHA1 Message Date
Haibo Lin
894cce6a5a Ignore osbs/osbuild config when reusing iso images
JIRA: RHELCMP-7562
Signed-off-by: Haibo Lin <hlin@redhat.com>
2022-01-04 16:35:29 +08:00
soksanichenko
0484426e0c ALBS-97: Build AlmaLinux PPC64le repos and ISOs with pungi
- Changelog
- Version is bumped

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: I933925b7a27a5e1b642020e060f59212fdc6ebf4
2021-12-30 12:42:34 +02:00
soksanichenko
b9d86b90e1 ALBS-97: Build AlmaLinux PPC64le repos and ISOs with pungi
- Scripts `create_packages_json` & `gather_modules` can process lzma compressed yaml files
- Script `create_package_json` can use repodata there are packages with different
  arch by compare with passed to the script

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: Ia9a3bacfa4344f0cf33b9f416649fd4a5f8d3c37
2021-12-28 16:08:04 +02:00
soksanichenko
58a16e5688 - The version is bumped
- The changelog is updated
- The test `create_packages_json` is fixed

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: I173013da990eb296e58ca8f3555a05913ca1c852
2021-12-20 14:11:17 +02:00
Lubomír Sedlář
260b3fce8d compose: Make sure temporary dirs are world readable
When the temporary directory is created with 0700, other programs
(potentially on another host) will have problems reading it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
JIRA: RHELCMP-7635
2021-12-17 08:10:48 +01:00
Haibo Lin
20c2e59218 Pass compose parameter for debugging git issue
With this param, get_dir_from_scm will try to copy the tmp git dir to
compose target dir when error occurs.

This does not fix the issue but it would be helpful for debugging when
it occurs again.

JIRA: RHELCMP-7244
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-11-26 15:40:35 +08:00
Haibo Lin
5e6248e3e0 Generate images.json for extra_isos phase
JIRA: RHELCMP-7241
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-11-15 14:38:53 +08:00
Haibo Lin
f681956cf1 Fix tests for python 2.6
It failed to build RHEL 6 package as logging.NullHandler does not exist
in python 2.6

JIRA: RHELCMP-7188
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-11-12 17:04:17 +08:00
Haibo Lin
cfb9882269 4.3.2 release
JIRA: RHELCMP-7182
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-11-11 16:08:54 +08:00
soksanichenko
f2ed64d952 ALBS-66: Prepare Jenkins jobs for building distribution of AlmaLinux 8.5
- Script `create_packages_json` can duplicate the packages with
  same version in different variants

Change-Id: I3c79ad06c4c22442423c12d5fa06baf82d663a3f
2021-11-10 15:29:59 +02:00
Lubomír Sedlář
b652119d54 gather: Load JSON mapping relative to config dir
JIRA: RHELCMP-7195
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-11-10 11:01:53 +01:00
Lubomír Sedlář
33d7290d78 gather: Stop requiring all variants/arches in JSON
The JSON source file should not require a mapping for all
variants/architectures. When something is specified, it should be
included.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-11-10 11:01:53 +01:00
Ken Dreyer
9bae86a51e doc: make dnf "backend" settings easier to discover
Mention the corresponding "gather" or "repoclosure" backend settings in
the documentation for each setting.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2021-11-04 10:55:54 -04:00
Lubomír Sedlář
1d654522be Remove with_jigdo argument
It was checked in a condition together with the configuration value, and
only ever explicitly used with the same value.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-11-04 13:37:51 +00:00
Lubomír Sedlář
80bd254347 Check dependencies after config validation
This way the checks can rely on default values from the config.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-11-04 13:37:51 +00:00
Ken Dreyer
94ffa1c5c6 default "with_jigdo" to False
Fedora has not composed with jigdo in a long time. Disable it by
default.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
Merges: https://pagure.io/pungi/pull-request/1561
Fixes: https://pagure.io/pungi/issue/1560
2021-11-04 13:37:51 +00:00
Lubomír Sedlář
9d02f87c99 Stop trying to validate non-existent metadata
When a compose doesn't build any images, it won't produce any metadata
file for them, and thus it makes no sense to validate it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Fixes: https://pagure.io/pungi/issue/1565
2021-11-04 09:57:20 +01:00
fdiprete
7b9e08ab28 test images for metadata deserialization error
Merges: https://pagure.io/pungi/pull-request/1559
Jira: https://issues.redhat.com/browse/RHELCMP-6685
Signed-off-by: fdiprete <fdipretre@redhat.com>
2021-11-03 08:50:07 +01:00
Lubomír Sedlář
e2b3002726 repoclosure: Use --forcearch for dnf repoclosure
DNF repoclosure requires this option when checking a repository that is
not compatible with host architecture. It seems that when it is
compatible, it works as well.

Based on how the list of architectures is generated, we know that the
main one will always be first.

Fixes: https://pagure.io/pungi/issue/1562
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-11-02 08:51:40 +01:00
Lubomír Sedlář
e8305f3978 extra_isos: Allow reusing old images
When nothing in configuration or the image itself changed, let's just
copy the older one.

JIRA: RHELCMP-5969
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-11-02 07:08:00 +00:00
Lubomír Sedlář
ac66c3d7f3 createiso: Allow reusing old images
This patch allows Pungi to reuse ISO image created in previous compose
if a list of assumptions proves to hold:

 * If image is bootable, buildinstall phase must have been reused too.
 * Compose configuration must have not changed (except for a few
   whitelisted options).
 * Volume ID of the ISO much not have changed.
 * No RPM on the ISO must have changed.

The ISO also contains other files. Changes in extra files and product ID
certificates should be visible in configuration (the SHA will differ).
Similarly any repodata configuration would be reflected in
configuration.

JIRA: RHELCMP-5969
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-11-02 07:08:00 +00:00
Lubomír Sedlář
eb61c97cdb Remove default runroot channel
When the value is not specified in the configuration file, let Koji pick
the default channel.

JIRA: RHELBLD-8088
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-10-25 15:56:24 +02:00
Ozan Unsal
b03490bf18 4.3.1 release
JIRA: RHELCMP-7116
Signed-off-by: Ozan Unsal <ounsal@redhat.com>
2021-10-25 15:18:26 +02:00
Dan Čermák
ab19043773 Correct irc network name & add matrix room
Signed-off-by: Dan Čermák <dan.cermak@cgc-instruments.com>
2021-10-25 07:26:18 +00:00
Lubomír Sedlář
204d88a351 Add missing mock to osbs tests
We don't want the test to try to a dummy URL.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-10-21 13:34:37 +02:00
Haibo Lin
8133676270 osbs: Reuse images from old compose
JIRA: RHELCMP-5972
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-10-20 14:22:19 +08:00
Haibo Lin
e42e65783d image_build: Allow reusing old image_build results
JIRA: RHELCMP-5970
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-10-20 14:07:02 +08:00
Ozan Unsal
7475d2a3a9 Allow ISO-Level configuration within the config file
In order to enable this feature set "iso_level=<Value from 1 to 4>"
in config file

Jira: RHELCMP-6880

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
2021-10-18 16:32:04 +02:00
Lubomír Sedlář
ac061b2ea8 Work around ODCS creating COMPOSE_ID later
When ODCS starts a compose, it will provide base composeinfo file, but
it doesn't create COMPOSE_ID. This leads to a crash when updating CTS,
since the compose id can't be read from the file. We can instead use the
value we already have in memory.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-10-04 11:32:59 +00:00
Jan Kaluza
0530cf2712 When cts_url is configured, use CTS /repo API for buildContainer yum_repourls.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2021-09-24 10:29:28 +02:00
Ozan Unsal
9612241396 Add COMPOSE_ID into the pungi log file
Jira: RHELCMP-6739

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
2021-09-21 10:55:03 +02:00
Lubomír Sedlář
ba6f7429ee buildinstall: Add easy way to check if previous result was reused
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-09-14 10:09:44 +02:00
Lubomír Sedlář
72bcee01be 4.3.0 release
JIRA: RHELCMP-6614
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-09-10 11:38:11 +02:00
Lubomír Sedlář
a1ebd234a4 Only build CTS url when configured
JIRA: RHELCMP-6611
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-09-10 11:29:53 +02:00
Lubomír Sedlář
5c26aa9127 Require requests_kerberos only when needed
If CTS integration is not used, let's not import a module that is not
needed.

JIRA: RHELCMP-6611
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-09-10 10:00:54 +02:00
Jan Kaluza
195bfbefa4 Allow specifying $COMPOSE_ID in the repo value for osbs phase.
There should be an option for `yum_repourls` to point to static
URL, for example when CTS is used. The idea is that instead of
setting `repo` to `AppStream`, we could use link similar to this one:

`https://cts.localhost/api/1/composes/$COMPOSE_ID/repo/?variant=AppStream`

This would be translated to real static link during the OSBS phase:

`https://cts.localhost/api/1/composes/CentOS-Stream-9-20210803.0/repo/?variant=AppStream`

That way this statis link would appear in the yum_repourls.

Merges: https://pagure.io/pungi/pull-request/1543
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2021-09-09 10:46:33 +02:00
Lubomír Sedlář
20dc4beb6b Make getting old compose config reusable
The file will only be loaded once, it gets cached afterwards.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-09-09 10:38:34 +02:00
Lubomír Sedlář
d8d1cc520b paths: Allow customizing log file extension
If the file contents is JSON, it would be nice to have matching
extension.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-09-09 10:38:34 +02:00
Ozan Unsal
904a1c3271 Add authentication for updating the compose URL in CTS.
Put authentication steps in a function in order to prevent code duplication.

Jira: RHELCMP-6318

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
2021-09-09 07:27:55 +00:00
Lubomír Sedlář
e8ddacd10e Fix type detection for osbuild images
The image type value passed to the task doesn't match the type as it
will be recorded by Koji.

JIRA: RHELCMP-5727
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-09-08 09:07:00 +00:00
Ozan Unsal
b7666ba4a4 Enable pungi to send compose_url patches to CTS
If cts_keytab is also enabled then the HTTP requests are handled with
Kerberos Authentication otherwise no authentication is used.

If cts_url is defined in the configuration, translate_paths is required.
This is needed in order to get the host and the path of the composes.

Jira: RHELCMP-6318

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
2021-09-07 16:41:35 +02:00
Ozan Unsal
3d9335e90e Use xorriso instead of isoinfo when createiso_use_xorrisofs is enabled
Update get_mkisofs_cmd in createiso.py file in order to prevent using
default value. With this change it is possible to enable xorriso format

Jira: RHELCMP-6325

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
2021-08-31 08:09:09 +00:00
Dominik Rumian
7c3e8d4276 Fix tests for createrepo
Tests for createrepo failed when pungi is installed in system.

JIRA: RHELCMP-6209

Signed-off-by: Dominik Rumian <drumian@redhat.com>
2021-08-30 13:48:29 +02:00
Dominik Rumian
9cd42a2b5e Formatted files according to flake8 and black feedback
Signed-off-by: Dominik Rumian <drumian@redhat.com>
2021-08-30 09:55:17 +02:00
Ozan Unsal
980c7ba8fb Handle the pungi failures to ensure creation of log files
If the given directory is not a valid git directory, it raises RuntimeError.
This can be catched and raised as GitUrlResolveError, so compose can continue
to log the failure.

Jira: RHELCMP-6077

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
2021-08-25 11:15:23 +00:00
Haibo Lin
66dacb21e0 Add createrepo_enable_cache to configuration doc
JIRA: RHELCMP-5984
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-08-18 11:14:55 +08:00
Haibo Lin
795bbe31e3 Fix formatting
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-08-17 14:23:43 +08:00
Haibo Lin
1bb038ca72 Install missing deps in ci image
tests requiring libmodulemd are skipped due to missing deps and
this patch could fix the issue.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-08-17 12:01:51 +08:00
Filip Valder
efff2c9504 Use pytest directly incl. support for posargs, e.g.:
tox -- -s -vvv tests/path/to/a/single/test_something.py

Signed-off-by: Filip Valder <fvalder@redhat.com>
2021-08-12 16:35:47 +02:00
Filip Valder
a7c111643d Supersede ModuleStream loading with ModuleIndex
- Use ModuleIndex's update_from_file/update_from_string instead of ModuleStream's
read_file/read_string which is deprecated.
- Extend tests to work with real module streams instead of mocks.

Signed-off-by: Filip Valder <fvalder@redhat.com>
2021-08-12 16:32:38 +02:00