Commit Graph

2507 Commits

Author SHA1 Message Date
stepan_oksanichenko
084321dd97 LNX-326: Add the ability to include any package by mask in packages.json to the generator
- The ability is added
- Also the generator includes only the latest by version packages to packages.json
- The generator has key `--is-reference` for an each repo. This key marks a repo as reference.
  An reference repo is used as main source of packages. A not reference repo is used as source
  of packages which don't exist in the reference repos.
- All cases are covered by the unittest

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: I2f80ba4fbfce27fb9a30500ae46c0b8a2f2aabcd
2021-06-15 17:42:12 +03:00
stepan_oksanichenko
941d6b064a LNX-318: Modify build scripts for building CloudLinux OS 8.4
- [Fixed] The script `create_packages_json` selects a first
          comer package from variant, but it should select the
          higher by version of package

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: I36268f2a493897fc11e787c040066d2d501a1c81
2021-06-04 12:36:03 +03:00
stepan_oksanichenko
aaeee7132d - It's bumped version
- It's added changelog

@BS-TARGET-CL8

Change-Id: I51eef1eb45ba54d034e6bed46d99b0470f4e9221
2021-05-25 21:28:47 +03:00
stepan_oksanichenko
cc4d99441c LNX-108: Add multiarch support to pungi
@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: Ibfd540454941922d790ae4e56cc0992c0c85635d
2021-05-24 18:07:11 +03:00
stepan_oksanichenko
a435eeed06 - it's added changelog
@BS-NOBUILD

Change-Id: I3a0a0377f9c1cefabf52c33fbc0d19ab0e4fe4f1
2021-04-29 17:15:17 +03:00
stepan_oksanichenko
b9f554bf39 LNX-311: Add ability to productmd set a main variant while dumping TreeInfo
@BS-NOBUILD
@BS-TARGET-CL8
@BS-LINKED-608ab56299ce8ac801a396c5  # python3-productmd

Change-Id: Id86d627ae8ae0b9a73b5ce6531c20538f3d040b1
2021-04-29 17:01:49 +03:00
stepan_oksanichenko
ebf028ca3b LNX-286: Prepare pungi configuration and setup Jenkins job for AlmaLinux 8.4 beta
- The modules from a parsend output of FUS should be have a stream
  with replaced dash by underscore

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: If36d3d0a1ef8010bf85a4a0218b9838e0888453c
2021-04-27 13:39:09 +03:00
stepan_oksanichenko
305103a38e LNX-286: Prepare pungi configuration and setup Jenkins job for AlmaLinux 8.4 beta
- Some modules can be absent in koji env but be present in variants.xml.
  And Pungi will fail in this case. So we must filter out those modules
  from expected modules list by list from pungi build config

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: I22c15c42868412e34fd554030130bd7c3e25b8ef
2021-04-23 13:03:05 +03:00
stepan_oksanichenko
01bce26275 LNX-286: Prepare pungi configuration and setup Jenkins job for AlmaLinux 8.4 beta
- The script `gather_modules` should replace `-` by `_`
  in stream of modules as pungi does it in self

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: Iea05b70afbf80f3ccd20ad4943c9d86c7ed7aa90
2021-04-22 13:40:48 +03:00
soksanichenko
4d763514c1 - Version is bumped
- Changelog is added

Change-Id: I440b44f12c4a1aa41619acd3ba5ca354dc71b419
2021-02-24 17:42:22 +02:00
Danylo Kuropiatnyk
41381df6a5 LU-2202: Start unittests during installation or build of pungi
* added section with tests and pytest module to requires
IMPORTANT - build.sh script is commented
* added pyfakefs dependency
* fixed little mock_open issue for runroot test
* bumped version

@BS-TARGET-CL8

Change-Id: I036db225646875eb610736cd26f473850a78447c
2021-02-23 07:55:36 -05:00
soksanichenko
02686d7bdf LU-2186 .treeinfo file in AlmaLinux public kickstart repo should contain AppStream variant
- We are modifying existing repo's .treeinfo:
-- Take info about included variants from iso's .treeinfo and put it to repo's .treeinfo

Change-Id: I29bf655d90994e8a1bda40ad04568dd7364f5dca
2021-02-23 06:48:15 -05:00
soksanichenko
2e48c9a56f LU-2195 Change path to sources and iso when generating repositories
- We should add the images to the compose if they will be used only as netinstall image.
  E.g. *-boot.iso.
- And we shouldn't add if the images will be modified in phase `extra_isos`.
  E.g. *-minimal.iso

Change-Id: I9095cfd87414ecca46b1213553589731c82dd2e2
2021-02-22 13:23:48 +02:00
soksanichenko
b3a8c3f28a - Version is bumped
- Changelog is added

Change-Id: Ib1366f1fe2639037db99b8e939537bb63801058e
2021-02-11 14:50:12 +02:00
soksanichenko
5434d24027 LU-2133: Prepare CI for iso builds of CLOSS 8
@BS-TARGET-CL8
@BS-NOBUILD

- It's added the script which can collect packages/modules
  from the remote repos (including BS repos) and merge them
  to an one local repo with the right repodata (including
  modules.yaml.gz)
- The script `create_packages_json` can use regexps for list of excluded packages

Change-Id: I1365b712460959db6bb451d1199d640bff6ffe5e
2021-02-09 10:47:46 +02:00
soksanichenko
3b5501b4bf LNX-133: Create a server for building nightly builds of AlmaLinux
- It's added key argument '--json-output-path' to script `pungi-generate-package-json`

Change-Id: Ic18fa2708cc4913002023828b3be018d4907de25
2021-01-28 14:03:40 +02:00
soksanichenko
cea8d92906 Bump version for setup.py
Change-Id: I980e9ebb728c3a88597c987d585e1b5937499e81
2021-01-28 00:06:40 +02:00
soksanichenko
1a29de435e - It's added changelog
- Its bumped version

Change-Id: I4c7b8d9c64da3379a24d93837657cec2686a8511
2021-01-27 23:47:39 +02:00
soksanichenko
69ed7699e8 LNX-133: Create a server for building nightly builds of AlmaLinux
- It's added dependency `python3-dataclasses` to spec

Change-Id: Id6b6f33ca6621ddc1408d9ab51e278801e4dd0a2
2021-01-27 07:47:07 -05:00
Stepan Oksanichenko
103c3dc608 LNX-133: Create a server for building nightly builds of AlmaLinux
- Script `pungi-gather-modules` can find valid *modules.yaml.gz in the repo dirs by itself

@BS-LINKED-5ffda6156f44affc6c5ea239  # pungi & dependencies
@BS-TARGET-CL8

Change-Id: I3cddc0cf41ea1087183e23de39126a52c69bc9ac
2021-01-25 16:17:35 +02:00
Stepan Oksanichenko
94ad7603b8 LNX-104: Create gather_prepopulate file generator for Pungi
- It's added the tool which can generate json like as `centos-packages.json` using repodata from completed repos.

@BS-LINKED-5ffda6156f44affc6c5ea239  # pungi & dependencies
@BS-TARGET-CL8

Change-Id: Ib0466a1d8e06feb855e81fb7160fe170e2e82e04
2021-01-25 16:17:34 +02:00
oshyshatskyi
903db91c0f LNX-102: Patch pungi tool to use local koji mock
Instead of koji.mbox use local koji-like wrapper.

@BS-LINKED-5ff8b8cb6f44affc6c5e9a7a
@BS-TARGET-CL8

Change-Id: I82a2bc8bc71ae06240656898f3df71bb28bcb9e9
2021-01-25 16:17:33 +02:00
oshyshatskyi
552343fffe LNX-102: Add tool that gathers directory for all rpms
Tool that finds all available rpm files in directory
and creates special tree for pungi:
 # ls /mnt/koji/
   i686/  noarch/  x86_64/

Change-Id: Ibcf2d23c46411ad89477058f4d56e07ca117f0d1
2021-01-25 16:17:33 +02:00
oshyshatskyi
5806217041 LNX-102: Add tool that collects information about modules
Add special tool that gathers given modules.tar.gz files
and collects information about modules into two dirs:
 - module_defaults
 - modules

 First one is used by pungi during repocreate phase and
 the second one is used by koji mock to get list of
 available modules and their versions.

Change-Id: I50a095a5f3bafa7e7a1effc2c0d4a2fc52ba603b
2021-01-25 16:17:33 +02:00
67eacf8483 LNX-103 Update .spec file for AlmaLinux
New binaries added to pungi rpm:
pungi-gather-rpms
pungi-gather-modules

Change-Id: Idb25dffb10d50fa9f566c99d714d32df962b6f52
2021-01-25 16:17:32 +02:00
Ken Dreyer
38789d07ee doc: remove default createrepo_checksum value from example
createrepo_checksum already defaults to sha256. Remove this setting from
the documented Minimal Example configuration to make it easier to read.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
(cherry picked from commit 39b847094a)
2021-01-25 14:06:34 +02:00
Lubomír Sedlář
3735aaa443 comps: Preserve default arg on groupid
When the wrapper processes comps file, it wasn't emitting "default"
argument for groupid element. The default is false and most entries are
actually using the default, so let's only emit it if set to true.

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1882358
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 9ea1098eae)
2021-01-25 14:06:33 +02:00
Haibo Lin
2c1603c414 Stop copying .git directory with module defaults
JIRA: RHELCMP-3016
Fixes: https://pagure.io/pungi/issue/1464

Signed-off-by: Haibo Lin <hlin@redhat.com>
(cherry picked from commit f518c1bb7c)
2021-01-25 14:06:33 +02:00
Haibo Lin
f2fd10b0ab React to SIGINT signal
ODCS sends SIGINT signal.

JIRA: RHELCMP-3687
Signed-off-by: Haibo Lin <hlin@redhat.com>
(cherry picked from commit f470599f6c)
2021-01-25 14:06:33 +02:00
Sergey Fokin
ac601ab8ea change Source0 in spec file 2021-01-08 13:30:17 +03:00
oshyshatskyi
757a6ed653 Revert unneeded commit to match upstream sources
This reverts commit b2e439e5

Change-Id: Ia6706415039681a6fe7b5ec6a735c3bda66d6bb1
2020-12-30 13:58:06 +02:00
Oleksandr Shyshatskyi
b2e439e561 current 2020-12-29 10:44:49 +02:00
Lubomír Sedlář
cda67776d9 scm: Only copy debugging data if we have a compose
If we don't have a compose, this copy will fail. Let's prevent that.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-12-08 08:57:06 +01:00
Lubomír Sedlář
62a97c0e1b 4.2.7 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-12-03 14:33:22 +01:00
Lubomír Sedlář
98ddc74c16 osbuild: Fix not failing on failable tasks
The task can only fail as a whole, thus it only makes sense to set all
architectures (`*`) as failable. The correct value needs to be checked
though.

JIRA: RHELCMP-3412
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-12-02 14:30:20 +01:00
Lubomír Sedlář
4a048d4a85 kojiwrapper: Use gssapi_login
The krb_login method is deprecated and will be removed in 1.22

JIRA: RHELCMP-3383
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-12-02 14:21:31 +01:00
Christian Kellner
6998ffe694 osbuild: use task result to get build info
Instead of parsing the log file to get the NVR and then in turn
use that to get to the build info use the structured return value
from the koji task. The return value of the osbuild plugin is:
    result = {
        "composer": {
            "server": <COMPOSER_URL>,
            "id": <COMPOSE_ID>
        },
        "koji": {
            "build": <BUILD_ID>
        }
    }
This means we have direct access to the koji build id, which was
returned by composer to the plugin via its status API. Using that
removes the need to parse the log file.

Adapt the test accordingly.

Merges: https://pagure.io/pungi/pull-request/1475
Signed-off-by: Christian Kellner <christian@kellner.me>
2020-11-30 09:57:48 +01:00
Lubomír Sedlář
9accf5ecf4 docs: Add osbuild phase to overview diagram
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-11-27 09:56:10 +01:00
Lubomír Sedlář
e7af6d2ac2 osbuild: Only send release when not empty
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-11-25 15:37:49 +01:00
Lubomír Sedlář
27bab19a5e Fix config validation for osbuild
Tests are now added for it as well.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-11-25 14:14:18 +01:00
Haibo Lin
4c88e7dc0e Use xorrisofs for creating ISOs when needed
A new configuration *createiso_use_xorrisofs* is added to determine
which tool to use for creating ISOs.

By default, createiso_use_xorrisofs = False and genisoimage will be used.
When set to True, xorrisofs will be used.

JIRA: RHELCMP-2875
Fixes: https://pagure.io/pungi/issue/1130

Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-11-20 09:47:49 +08:00
Jan Kaluza
c27e21ccf8 Add --respin-of argument.
It is used to defined the relation between original compose
and respun compose with a hotfix in CTS.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-11-19 12:26:38 +00:00
Lubomír Sedlář
e3a500ca50 Fix typo in configuration
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-11-16 10:50:01 +01:00
Jan Kaluza
4562fba459 Optimize link_files by creating temporary dict mapping path to pkg_obj.
Previously, the pkg_object has been found by iterating over pkg_sets
for each package. This was quite slow given the number of RPMs in the
compose.

In this commit, the temporary dict is created which stores mapping
between path and pkg_obj and is used instead.

In this commit, the get_package_path is called just once instead of
twice in the loop which might also save some time.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-11-11 12:20:01 +00:00
Jan Kaluza
bb8cd030ec Try reuse old gather_phase even if pkgset_koji_builds changed.
The pkgset_koji_builds influences the gather phase, but the
change itself is not a reason to not reuse old gather phase. if
new pkgset_koji_builds value leads to significant change in input
package set, we will find that later in this function when comparing
old and new package set.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-11-10 12:52:02 +01:00
Jan Kaluza
94bc5e286d Do not use shlex_quote in get_pungi_buildinstall_cmd and get_pungi_ostree_cmd.
These methods return command as list which is never serialized to str and
never executed using bash. It is instead passed directly to
`kobo.shortcuts.run`.

The `shlex_quote` usage here actually breaks the code, because it adds
quotes there which are needed only if this command would be serialized
to string and passed to bash. But this never happens. As a result,
the arguments passed to `kobo.shortcuts.run` contain those extra
quotes.

In this commit the shlex_quote is removed completely from this
part of code.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-11-10 07:45:05 +00:00
Lubomír Sedlář
295a60a704 gather: Fix test for module presence
When testing if a variant has some modules,
it's no longer enough to look at `variant.modules`. That attribute
contains only names of modules configured in variants.xml.

With modules being specified in `pkgset_koji_module_builds` or
`pkgset_scratch_modules` options it is possible the code wouldn't
trigger even if there was a module.

JIRA: RHELCMP-3054
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-11-09 09:33:03 +00:00
Lubomír Sedlář
99c1e2eb5e Kill all subprocess in signal handler
When Pungi receives a signal to terminate, it can sometimes get stuck
if there are threads running. It has to wait for all worker threads to
finish. They generally do finish, unless they get stuck waiting on a
subprocess.

This patch should reduce the likelihood of this happening by stopping
all subprocesses.

JIRA: RHELCMP-3056
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-11-09 08:45:57 +01:00
Bohdan Khomutskyi
4c4c816e70 Pungi 4.2.6 release
Signed-off-by: Bohdan Khomutskyi <bkhomuts@redhat.com>
2020-11-04 12:33:06 +00:00
Lubomír Sedlář
a45f4969f3 Add phase for building images with osbuild
This is similar to image-build in terms of what it does, and somewhat
similar to OSBS phase in how it's implemented.

The phase reads configuration, submits the build via XMLRPC call and
waits for the task to finish. Then it downloads the built image and
includes it in the compose metadata.

JIRA: RHELCMP-315
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-11-04 13:04:43 +01:00