Commit Graph

1438 Commits

Author SHA1 Message Date
soksanichenko 0128b0918a - The unittests are fixed 2022-10-19 14:11:07 +03:00
soksanichenko f98f1b20ab Do not lose a module from koji if we have more than one arch (e.g. x86_64 + i686) 2022-10-19 04:38:22 +03:00
soksanichenko 941a02722c Replace list of cr.packages by cr.PackageIterator 2022-10-18 22:59:04 +03:00
soksanichenko 885b3376d6 ALBS-334: Make the ability of Pungi to give module_defaults from remote sources 2022-05-06 16:43:47 +03:00
soksanichenko 75dcabbfae ALBS-334: Make the ability of Pungi to give module_defaults from remote sources 2022-05-06 16:43:47 +03:00
soksanichenko e79120278d ALBS-334: Make the ability of Pungi to give module_defaults from remote sources 2022-05-06 16:43:46 +03:00
soksanichenko a3c320715a ALBS-334: Make the ability of Pungi to give module_defaults from remote sources 2022-05-06 16:43:21 +03:00
soksanichenko f61bfa6df7 ALBS-334: Make the ability of Pungi to give module_defaults from remote sources 2022-05-06 16:43:20 +03:00
soksanichenko 6471a74f3b ALBS-334: Make the ability of Pungi to give module_defaults from remote sources 2022-05-06 16:43:03 +03:00
soksanichenko 31c521196a ALBS-334: Make the ability of Pungi to give module_defaults from remote sources 2022-05-06 16:43:02 +03:00
soksanichenko 39026c4d00 ALBS-334: Make the ability of Pungi to give module_defaults from remote sources 2022-05-04 20:08:49 +03:00
soksanichenko 5b540e60d0 ALBS-334: Make the ability of Pungi to give module_defaults from remote sources 2022-05-04 20:08:49 +03:00
soksanichenko 67712933af ALBS-334: Make the ability of Pungi to give module_defaults from remote sources 2022-05-04 20:08:49 +03:00
soksanichenko 1144cc5e53 ALBS-334: Make the ability of Pungi to give module_defaults from remote sources 2022-05-04 20:08:49 +03:00
soksanichenko acfdfcef15 ALBS-226: Patch pungi/lorax for building AL9
- Unit tests are fixed
2022-03-19 03:26:10 +02:00
soksanichenko 700ae3cbac ALBS-226: Patch pungi/lorax for building AL9
- Defaults modules can be empty, but pungi detects
  empty folder while copying and raises the exception in this case
2022-03-18 23:43:33 +02:00
soksanichenko b9d86b90e1 ALBS-97: Build AlmaLinux PPC64le repos and ISOs with pungi
- Scripts `create_packages_json` & `gather_modules` can process lzma compressed yaml files
- Script `create_package_json` can use repodata there are packages with different
  arch by compare with passed to the script

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: Ia9a3bacfa4344f0cf33b9f416649fd4a5f8d3c37
2021-12-28 16:08:04 +02:00
soksanichenko f2ed64d952 ALBS-66: Prepare Jenkins jobs for building distribution of AlmaLinux 8.5
- Script `create_packages_json` can duplicate the packages with
  same version in different variants

Change-Id: I3c79ad06c4c22442423c12d5fa06baf82d663a3f
2021-11-10 15:29:59 +02:00
stepan_oksanichenko 14dd6a195f LNX-326: Add the ability to include any package by mask in packages.json to the generator
- The reference packages should be replaced only by the newer reference packages
- The non-reference packages should be replaced by both of types packages

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: I881bd4e58527ae219ef6e1adbc6332b3b05933c1
2021-06-18 14:23:42 +03:00
stepan_oksanichenko 084321dd97 LNX-326: Add the ability to include any package by mask in packages.json to the generator
- The ability is added
- Also the generator includes only the latest by version packages to packages.json
- The generator has key `--is-reference` for an each repo. This key marks a repo as reference.
  An reference repo is used as main source of packages. A not reference repo is used as source
  of packages which don't exist in the reference repos.
- All cases are covered by the unittest

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: I2f80ba4fbfce27fb9a30500ae46c0b8a2f2aabcd
2021-06-15 17:42:12 +03:00
stepan_oksanichenko 941d6b064a LNX-318: Modify build scripts for building CloudLinux OS 8.4
- [Fixed] The script `create_packages_json` selects a first
          comer package from variant, but it should select the
          higher by version of package

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: I36268f2a493897fc11e787c040066d2d501a1c81
2021-06-04 12:36:03 +03:00
stepan_oksanichenko cc4d99441c LNX-108: Add multiarch support to pungi
@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: Ibfd540454941922d790ae4e56cc0992c0c85635d
2021-05-24 18:07:11 +03:00
stepan_oksanichenko b9f554bf39 LNX-311: Add ability to productmd set a main variant while dumping TreeInfo
@BS-NOBUILD
@BS-TARGET-CL8
@BS-LINKED-608ab56299ce8ac801a396c5  # python3-productmd

Change-Id: Id86d627ae8ae0b9a73b5ce6531c20538f3d040b1
2021-04-29 17:01:49 +03:00
stepan_oksanichenko ebf028ca3b LNX-286: Prepare pungi configuration and setup Jenkins job for AlmaLinux 8.4 beta
- The modules from a parsend output of FUS should be have a stream
  with replaced dash by underscore

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: If36d3d0a1ef8010bf85a4a0218b9838e0888453c
2021-04-27 13:39:09 +03:00
stepan_oksanichenko 305103a38e LNX-286: Prepare pungi configuration and setup Jenkins job for AlmaLinux 8.4 beta
- Some modules can be absent in koji env but be present in variants.xml.
  And Pungi will fail in this case. So we must filter out those modules
  from expected modules list by list from pungi build config

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: I22c15c42868412e34fd554030130bd7c3e25b8ef
2021-04-23 13:03:05 +03:00
stepan_oksanichenko 01bce26275 LNX-286: Prepare pungi configuration and setup Jenkins job for AlmaLinux 8.4 beta
- The script `gather_modules` should replace `-` by `_`
  in stream of modules as pungi does it in self

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: Iea05b70afbf80f3ccd20ad4943c9d86c7ed7aa90
2021-04-22 13:40:48 +03:00
Danylo Kuropiatnyk 41381df6a5 LU-2202: Start unittests during installation or build of pungi
* added section with tests and pytest module to requires
IMPORTANT - build.sh script is commented
* added pyfakefs dependency
* fixed little mock_open issue for runroot test
* bumped version

@BS-TARGET-CL8

Change-Id: I036db225646875eb610736cd26f473850a78447c
2021-02-23 07:55:36 -05:00
soksanichenko 02686d7bdf LU-2186 .treeinfo file in AlmaLinux public kickstart repo should contain AppStream variant
- We are modifying existing repo's .treeinfo:
-- Take info about included variants from iso's .treeinfo and put it to repo's .treeinfo

Change-Id: I29bf655d90994e8a1bda40ad04568dd7364f5dca
2021-02-23 06:48:15 -05:00
soksanichenko 2e48c9a56f LU-2195 Change path to sources and iso when generating repositories
- We should add the images to the compose if they will be used only as netinstall image.
  E.g. *-boot.iso.
- And we shouldn't add if the images will be modified in phase `extra_isos`.
  E.g. *-minimal.iso

Change-Id: I9095cfd87414ecca46b1213553589731c82dd2e2
2021-02-22 13:23:48 +02:00
soksanichenko 5434d24027 LU-2133: Prepare CI for iso builds of CLOSS 8
@BS-TARGET-CL8
@BS-NOBUILD

- It's added the script which can collect packages/modules
  from the remote repos (including BS repos) and merge them
  to an one local repo with the right repodata (including
  modules.yaml.gz)
- The script `create_packages_json` can use regexps for list of excluded packages

Change-Id: I1365b712460959db6bb451d1199d640bff6ffe5e
2021-02-09 10:47:46 +02:00
soksanichenko 3b5501b4bf LNX-133: Create a server for building nightly builds of AlmaLinux
- It's added key argument '--json-output-path' to script `pungi-generate-package-json`

Change-Id: Ic18fa2708cc4913002023828b3be018d4907de25
2021-01-28 14:03:40 +02:00
Stepan Oksanichenko 103c3dc608 LNX-133: Create a server for building nightly builds of AlmaLinux
- Script `pungi-gather-modules` can find valid *modules.yaml.gz in the repo dirs by itself

@BS-LINKED-5ffda6156f44affc6c5ea239  # pungi & dependencies
@BS-TARGET-CL8

Change-Id: I3cddc0cf41ea1087183e23de39126a52c69bc9ac
2021-01-25 16:17:35 +02:00
Stepan Oksanichenko 94ad7603b8 LNX-104: Create gather_prepopulate file generator for Pungi
- It's added the tool which can generate json like as `centos-packages.json` using repodata from completed repos.

@BS-LINKED-5ffda6156f44affc6c5ea239  # pungi & dependencies
@BS-TARGET-CL8

Change-Id: Ib0466a1d8e06feb855e81fb7160fe170e2e82e04
2021-01-25 16:17:34 +02:00
oshyshatskyi 903db91c0f LNX-102: Patch pungi tool to use local koji mock
Instead of koji.mbox use local koji-like wrapper.

@BS-LINKED-5ff8b8cb6f44affc6c5e9a7a
@BS-TARGET-CL8

Change-Id: I82a2bc8bc71ae06240656898f3df71bb28bcb9e9
2021-01-25 16:17:33 +02:00
oshyshatskyi 552343fffe LNX-102: Add tool that gathers directory for all rpms
Tool that finds all available rpm files in directory
and creates special tree for pungi:
 # ls /mnt/koji/
   i686/  noarch/  x86_64/

Change-Id: Ibcf2d23c46411ad89477058f4d56e07ca117f0d1
2021-01-25 16:17:33 +02:00
oshyshatskyi 5806217041 LNX-102: Add tool that collects information about modules
Add special tool that gathers given modules.tar.gz files
and collects information about modules into two dirs:
 - module_defaults
 - modules

 First one is used by pungi during repocreate phase and
 the second one is used by koji mock to get list of
 available modules and their versions.

Change-Id: I50a095a5f3bafa7e7a1effc2c0d4a2fc52ba603b
2021-01-25 16:17:33 +02:00
Lubomír Sedlář 3735aaa443 comps: Preserve default arg on groupid
When the wrapper processes comps file, it wasn't emitting "default"
argument for groupid element. The default is false and most entries are
actually using the default, so let's only emit it if set to true.

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1882358
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 9ea1098eae)
2021-01-25 14:06:33 +02:00
Haibo Lin 2c1603c414 Stop copying .git directory with module defaults
JIRA: RHELCMP-3016
Fixes: https://pagure.io/pungi/issue/1464

Signed-off-by: Haibo Lin <hlin@redhat.com>
(cherry picked from commit f518c1bb7c)
2021-01-25 14:06:33 +02:00
Haibo Lin f2fd10b0ab React to SIGINT signal
ODCS sends SIGINT signal.

JIRA: RHELCMP-3687
Signed-off-by: Haibo Lin <hlin@redhat.com>
(cherry picked from commit f470599f6c)
2021-01-25 14:06:33 +02:00
oshyshatskyi 757a6ed653 Revert unneeded commit to match upstream sources
This reverts commit b2e439e5

Change-Id: Ia6706415039681a6fe7b5ec6a735c3bda66d6bb1
2020-12-30 13:58:06 +02:00
Oleksandr Shyshatskyi b2e439e561 current 2020-12-29 10:44:49 +02:00
Lubomír Sedlář cda67776d9 scm: Only copy debugging data if we have a compose
If we don't have a compose, this copy will fail. Let's prevent that.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-12-08 08:57:06 +01:00
Lubomír Sedlář 98ddc74c16 osbuild: Fix not failing on failable tasks
The task can only fail as a whole, thus it only makes sense to set all
architectures (`*`) as failable. The correct value needs to be checked
though.

JIRA: RHELCMP-3412
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-12-02 14:30:20 +01:00
Lubomír Sedlář 4a048d4a85 kojiwrapper: Use gssapi_login
The krb_login method is deprecated and will be removed in 1.22

JIRA: RHELCMP-3383
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-12-02 14:21:31 +01:00
Christian Kellner 6998ffe694 osbuild: use task result to get build info
Instead of parsing the log file to get the NVR and then in turn
use that to get to the build info use the structured return value
from the koji task. The return value of the osbuild plugin is:
    result = {
        "composer": {
            "server": <COMPOSER_URL>,
            "id": <COMPOSE_ID>
        },
        "koji": {
            "build": <BUILD_ID>
        }
    }
This means we have direct access to the koji build id, which was
returned by composer to the plugin via its status API. Using that
removes the need to parse the log file.

Adapt the test accordingly.

Merges: https://pagure.io/pungi/pull-request/1475
Signed-off-by: Christian Kellner <christian@kellner.me>
2020-11-30 09:57:48 +01:00
Lubomír Sedlář e7af6d2ac2 osbuild: Only send release when not empty
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-11-25 15:37:49 +01:00
Lubomír Sedlář 27bab19a5e Fix config validation for osbuild
Tests are now added for it as well.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-11-25 14:14:18 +01:00
Haibo Lin 4c88e7dc0e Use xorrisofs for creating ISOs when needed
A new configuration *createiso_use_xorrisofs* is added to determine
which tool to use for creating ISOs.

By default, createiso_use_xorrisofs = False and genisoimage will be used.
When set to True, xorrisofs will be used.

JIRA: RHELCMP-2875
Fixes: https://pagure.io/pungi/issue/1130

Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-11-20 09:47:49 +08:00
Jan Kaluza c27e21ccf8 Add --respin-of argument.
It is used to defined the relation between original compose
and respun compose with a hotfix in CTS.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-11-19 12:26:38 +00:00
Jan Kaluza 4562fba459 Optimize link_files by creating temporary dict mapping path to pkg_obj.
Previously, the pkg_object has been found by iterating over pkg_sets
for each package. This was quite slow given the number of RPMs in the
compose.

In this commit, the temporary dict is created which stores mapping
between path and pkg_obj and is used instead.

In this commit, the get_package_path is called just once instead of
twice in the loop which might also save some time.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-11-11 12:20:01 +00:00