Commit Graph

2819 Commits

Author SHA1 Message Date
Haibo Lin
2769232b72 runroot: Adjust permissions always
Previously commands to adjust permissions do not run when main
command failed and then files can't be cleaned up due to
Permission Denied problem.

JIRA: RHELCMP-4253
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-03-04 16:08:06 +08:00
Haibo Lin
b217470464 Format code
Code didn't get well formatted when jenkins unusable.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-03-02 18:31:45 +08:00
soksanichenko
4d763514c1 - Version is bumped
- Changelog is added

Change-Id: I440b44f12c4a1aa41619acd3ba5ca354dc71b419
2021-02-24 17:42:22 +02:00
Lubomír Sedlář
735bfaa0d6 pkgset: Fix meaning of retries
The name brings a different expectation than how it actually worked.
This patch makes the code work similarly to the expectation.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-23 16:01:41 +01:00
Danylo Kuropiatnyk
41381df6a5 LU-2202: Start unittests during installation or build of pungi
* added section with tests and pytest module to requires
IMPORTANT - build.sh script is commented
* added pyfakefs dependency
* fixed little mock_open issue for runroot test
* bumped version

@BS-TARGET-CL8

Change-Id: I036db225646875eb610736cd26f473850a78447c
2021-02-23 07:55:36 -05:00
soksanichenko
02686d7bdf LU-2186 .treeinfo file in AlmaLinux public kickstart repo should contain AppStream variant
- We are modifying existing repo's .treeinfo:
-- Take info about included variants from iso's .treeinfo and put it to repo's .treeinfo

Change-Id: I29bf655d90994e8a1bda40ad04568dd7364f5dca
2021-02-23 06:48:15 -05:00
soksanichenko
2e48c9a56f LU-2195 Change path to sources and iso when generating repositories
- We should add the images to the compose if they will be used only as netinstall image.
  E.g. *-boot.iso.
- And we shouldn't add if the images will be modified in phase `extra_isos`.
  E.g. *-minimal.iso

Change-Id: I9095cfd87414ecca46b1213553589731c82dd2e2
2021-02-22 13:23:48 +02:00
Lubomír Sedlář
5b5069175d pkgset: Store module tag only if module is used
When a module is skipped from the compose, we should not add it to a
mapping of module tags. If it's there, we then spend time building a
repo for the module, and it get's passed to buildinstall, despite the
packages not being supposed to be included in the compose.

If the packages are not included in any variant, they shouldn't be
available to buildinstall either.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-22 10:21:56 +01:00
Lubomír Sedlář
477dcf37d9 Store extended traceback for gather errors
When a gathering thread raises an exception, it gets forwarded to the
main thread and re-raised there. However, during this transition it
loses details about exact location of the problem.

This patch creates an extended traceback in the worker, which should
make it easier to track the problem down later.

JIRA: RHELCMP-4259
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-17 11:11:26 +01:00
Ondrej Nosek
98359654cf 4.2.8 release
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2021-02-12 15:20:40 +01:00
Lubomír Sedlář
64897d7d48 pkgset: Add ability to wait for signed packages
If packages are appearing quickly in Koji, and signing them is triggered
by automation, there may be a delay between the package being signed and
compose running. In such case it may be preferable to wait for the
signed copy rather than fail the compose.

JIRA: RHELCMP-3932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-11 15:31:14 +01:00
soksanichenko
b3a8c3f28a - Version is bumped
- Changelog is added

Change-Id: Ib1366f1fe2639037db99b8e939537bb63801058e
2021-02-11 14:50:12 +02:00
Lubomír Sedlář
40133074b3 Add image-container phase
This phase runs after image-build and osbuild and can embed an image
into a container.

JIRA: RHELCMP-3820
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-09 14:47:57 +01:00
soksanichenko
5434d24027 LU-2133: Prepare CI for iso builds of CLOSS 8
@BS-TARGET-CL8
@BS-NOBUILD

- It's added the script which can collect packages/modules
  from the remote repos (including BS repos) and merge them
  to an one local repo with the right repodata (including
  modules.yaml.gz)
- The script `create_packages_json` can use regexps for list of excluded packages

Change-Id: I1365b712460959db6bb451d1199d640bff6ffe5e
2021-02-09 10:47:46 +02:00
Lubomír Sedlář
61e90fd7e0 osbs: Move metadata processing to standalone function
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-08 13:23:28 +01:00
Lubomír Sedlář
36373479db Move container metadata into compose object
Rather than tracking this directly in OSBS phase, move this into Compose
object, which will allow access to this from multiple phases.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-08 13:23:26 +01:00
Lubomír Sedlář
44f7eff1b7 Move UnsignedPackagesError to a separate file
This file can contain all Pungi specific exceptions.

It should also fix an issue encountered on Python 2.7:

    AttributeError: 'module' object has no attribute 'pkgsets'

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-08 10:16:32 +00:00
Haibo Lin
daa0ca6106 pkgset: Include just one version of module
When adding extra modules via option *pkgset_koji_module_builds*, all
other versions of the same stream potentially available in a Brew tag
should be skipped.

JIRA: RHELCMP-3689
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-02-02 18:16:52 +08:00
Haibo Lin
d4ee42ec23 pkgset: Check tag inheritance change before reuse
JIRA: RHELCMP-2453
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-02-01 09:22:42 +08:00
Lubomír Sedlář
49a5661521 pkgset: Remove reuse file when packages are not signed
In such case we never want to reuse the pkgset, as it risks leaking
unsigned packages. Safest option is to remove the file completely.

Fixes: https://pagure.io/pungi/issue/1480
JIRA: RHELCMP-3720
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-01-29 13:06:29 +01:00
soksanichenko
3b5501b4bf LNX-133: Create a server for building nightly builds of AlmaLinux
- It's added key argument '--json-output-path' to script `pungi-generate-package-json`

Change-Id: Ic18fa2708cc4913002023828b3be018d4907de25
2021-01-28 14:03:40 +02:00
soksanichenko
cea8d92906 Bump version for setup.py
Change-Id: I980e9ebb728c3a88597c987d585e1b5937499e81
2021-01-28 00:06:40 +02:00
soksanichenko
1a29de435e - It's added changelog
- Its bumped version

Change-Id: I4c7b8d9c64da3379a24d93837657cec2686a8511
2021-01-27 23:47:39 +02:00
Lubomír Sedlář
c87fce30ac pkgset: Drop kobo.plugin usage from PkgsetSource
Relates: https://pagure.io/pungi/issue/1488
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-01-27 15:55:21 +01:00
Lubomír Sedlář
0f4b0577f7 gather: Drop kobo.plugins usage from GatherMethod
Relates: https://pagure.io/pungi/issue/1488
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-01-27 15:55:21 +01:00
Lubomír Sedlář
83458f26c2 pkgset: Drop kobo.plugins usage from GatherSources
Relates: https://pagure.io/pungi/issue/1488
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-01-27 15:55:21 +01:00
soksanichenko
69ed7699e8 LNX-133: Create a server for building nightly builds of AlmaLinux
- It's added dependency `python3-dataclasses` to spec

Change-Id: Id6b6f33ca6621ddc1408d9ab51e278801e4dd0a2
2021-01-27 07:47:07 -05:00
Stepan Oksanichenko
103c3dc608 LNX-133: Create a server for building nightly builds of AlmaLinux
- Script `pungi-gather-modules` can find valid *modules.yaml.gz in the repo dirs by itself

@BS-LINKED-5ffda6156f44affc6c5ea239  # pungi & dependencies
@BS-TARGET-CL8

Change-Id: I3cddc0cf41ea1087183e23de39126a52c69bc9ac
2021-01-25 16:17:35 +02:00
Stepan Oksanichenko
94ad7603b8 LNX-104: Create gather_prepopulate file generator for Pungi
- It's added the tool which can generate json like as `centos-packages.json` using repodata from completed repos.

@BS-LINKED-5ffda6156f44affc6c5ea239  # pungi & dependencies
@BS-TARGET-CL8

Change-Id: Ib0466a1d8e06feb855e81fb7160fe170e2e82e04
2021-01-25 16:17:34 +02:00
oshyshatskyi
903db91c0f LNX-102: Patch pungi tool to use local koji mock
Instead of koji.mbox use local koji-like wrapper.

@BS-LINKED-5ff8b8cb6f44affc6c5e9a7a
@BS-TARGET-CL8

Change-Id: I82a2bc8bc71ae06240656898f3df71bb28bcb9e9
2021-01-25 16:17:33 +02:00
oshyshatskyi
552343fffe LNX-102: Add tool that gathers directory for all rpms
Tool that finds all available rpm files in directory
and creates special tree for pungi:
 # ls /mnt/koji/
   i686/  noarch/  x86_64/

Change-Id: Ibcf2d23c46411ad89477058f4d56e07ca117f0d1
2021-01-25 16:17:33 +02:00
oshyshatskyi
5806217041 LNX-102: Add tool that collects information about modules
Add special tool that gathers given modules.tar.gz files
and collects information about modules into two dirs:
 - module_defaults
 - modules

 First one is used by pungi during repocreate phase and
 the second one is used by koji mock to get list of
 available modules and their versions.

Change-Id: I50a095a5f3bafa7e7a1effc2c0d4a2fc52ba603b
2021-01-25 16:17:33 +02:00
67eacf8483 LNX-103 Update .spec file for AlmaLinux
New binaries added to pungi rpm:
pungi-gather-rpms
pungi-gather-modules

Change-Id: Idb25dffb10d50fa9f566c99d714d32df962b6f52
2021-01-25 16:17:32 +02:00
Ken Dreyer
38789d07ee doc: remove default createrepo_checksum value from example
createrepo_checksum already defaults to sha256. Remove this setting from
the documented Minimal Example configuration to make it easier to read.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
(cherry picked from commit 39b847094a)
2021-01-25 14:06:34 +02:00
Lubomír Sedlář
3735aaa443 comps: Preserve default arg on groupid
When the wrapper processes comps file, it wasn't emitting "default"
argument for groupid element. The default is false and most entries are
actually using the default, so let's only emit it if set to true.

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1882358
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 9ea1098eae)
2021-01-25 14:06:33 +02:00
Haibo Lin
2c1603c414 Stop copying .git directory with module defaults
JIRA: RHELCMP-3016
Fixes: https://pagure.io/pungi/issue/1464

Signed-off-by: Haibo Lin <hlin@redhat.com>
(cherry picked from commit f518c1bb7c)
2021-01-25 14:06:33 +02:00
Haibo Lin
f2fd10b0ab React to SIGINT signal
ODCS sends SIGINT signal.

JIRA: RHELCMP-3687
Signed-off-by: Haibo Lin <hlin@redhat.com>
(cherry picked from commit f470599f6c)
2021-01-25 14:06:33 +02:00
Ken Dreyer
39b847094a doc: remove default createrepo_checksum value from example
createrepo_checksum already defaults to sha256. Remove this setting from
the documented Minimal Example configuration to make it easier to read.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2021-01-25 08:22:17 +00:00
Lubomír Sedlář
9ea1098eae comps: Preserve default arg on groupid
When the wrapper processes comps file, it wasn't emitting "default"
argument for groupid element. The default is false and most entries are
actually using the default, so let's only emit it if set to true.

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1882358
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-01-22 08:33:09 +01:00
Haibo Lin
f518c1bb7c Stop copying .git directory with module defaults
JIRA: RHELCMP-3016
Fixes: https://pagure.io/pungi/issue/1464

Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-01-14 10:36:38 +08:00
Haibo Lin
f470599f6c React to SIGINT signal
ODCS sends SIGINT signal.

JIRA: RHELCMP-3687
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-01-11 10:02:10 +08:00
Sergey Fokin
ac601ab8ea change Source0 in spec file 2021-01-08 13:30:17 +03:00
oshyshatskyi
757a6ed653 Revert unneeded commit to match upstream sources
This reverts commit b2e439e5

Change-Id: Ia6706415039681a6fe7b5ec6a735c3bda66d6bb1
2020-12-30 13:58:06 +02:00
Oleksandr Shyshatskyi
b2e439e561 current 2020-12-29 10:44:49 +02:00
Lubomír Sedlář
cda67776d9 scm: Only copy debugging data if we have a compose
If we don't have a compose, this copy will fail. Let's prevent that.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-12-08 08:57:06 +01:00
Lubomír Sedlář
62a97c0e1b 4.2.7 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-12-03 14:33:22 +01:00
Lubomír Sedlář
98ddc74c16 osbuild: Fix not failing on failable tasks
The task can only fail as a whole, thus it only makes sense to set all
architectures (`*`) as failable. The correct value needs to be checked
though.

JIRA: RHELCMP-3412
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-12-02 14:30:20 +01:00
Lubomír Sedlář
4a048d4a85 kojiwrapper: Use gssapi_login
The krb_login method is deprecated and will be removed in 1.22

JIRA: RHELCMP-3383
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-12-02 14:21:31 +01:00
Christian Kellner
6998ffe694 osbuild: use task result to get build info
Instead of parsing the log file to get the NVR and then in turn
use that to get to the build info use the structured return value
from the koji task. The return value of the osbuild plugin is:
    result = {
        "composer": {
            "server": <COMPOSER_URL>,
            "id": <COMPOSE_ID>
        },
        "koji": {
            "build": <BUILD_ID>
        }
    }
This means we have direct access to the koji build id, which was
returned by composer to the plugin via its status API. Using that
removes the need to parse the log file.

Adapt the test accordingly.

Merges: https://pagure.io/pungi/pull-request/1475
Signed-off-by: Christian Kellner <christian@kellner.me>
2020-11-30 09:57:48 +01:00
Lubomír Sedlář
9accf5ecf4 docs: Add osbuild phase to overview diagram
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-11-27 09:56:10 +01:00