Commit Graph

2732 Commits

Author SHA1 Message Date
Lubomír Sedlář
c87fce30ac pkgset: Drop kobo.plugin usage from PkgsetSource
Relates: https://pagure.io/pungi/issue/1488
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-01-27 15:55:21 +01:00
Lubomír Sedlář
0f4b0577f7 gather: Drop kobo.plugins usage from GatherMethod
Relates: https://pagure.io/pungi/issue/1488
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-01-27 15:55:21 +01:00
Lubomír Sedlář
83458f26c2 pkgset: Drop kobo.plugins usage from GatherSources
Relates: https://pagure.io/pungi/issue/1488
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-01-27 15:55:21 +01:00
Ken Dreyer
39b847094a doc: remove default createrepo_checksum value from example
createrepo_checksum already defaults to sha256. Remove this setting from
the documented Minimal Example configuration to make it easier to read.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2021-01-25 08:22:17 +00:00
Lubomír Sedlář
9ea1098eae comps: Preserve default arg on groupid
When the wrapper processes comps file, it wasn't emitting "default"
argument for groupid element. The default is false and most entries are
actually using the default, so let's only emit it if set to true.

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1882358
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-01-22 08:33:09 +01:00
Haibo Lin
f518c1bb7c Stop copying .git directory with module defaults
JIRA: RHELCMP-3016
Fixes: https://pagure.io/pungi/issue/1464

Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-01-14 10:36:38 +08:00
Haibo Lin
f470599f6c React to SIGINT signal
ODCS sends SIGINT signal.

JIRA: RHELCMP-3687
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-01-11 10:02:10 +08:00
Lubomír Sedlář
cda67776d9 scm: Only copy debugging data if we have a compose
If we don't have a compose, this copy will fail. Let's prevent that.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-12-08 08:57:06 +01:00
Lubomír Sedlář
62a97c0e1b 4.2.7 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-12-03 14:33:22 +01:00
Lubomír Sedlář
98ddc74c16 osbuild: Fix not failing on failable tasks
The task can only fail as a whole, thus it only makes sense to set all
architectures (`*`) as failable. The correct value needs to be checked
though.

JIRA: RHELCMP-3412
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-12-02 14:30:20 +01:00
Lubomír Sedlář
4a048d4a85 kojiwrapper: Use gssapi_login
The krb_login method is deprecated and will be removed in 1.22

JIRA: RHELCMP-3383
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-12-02 14:21:31 +01:00
Christian Kellner
6998ffe694 osbuild: use task result to get build info
Instead of parsing the log file to get the NVR and then in turn
use that to get to the build info use the structured return value
from the koji task. The return value of the osbuild plugin is:
    result = {
        "composer": {
            "server": <COMPOSER_URL>,
            "id": <COMPOSE_ID>
        },
        "koji": {
            "build": <BUILD_ID>
        }
    }
This means we have direct access to the koji build id, which was
returned by composer to the plugin via its status API. Using that
removes the need to parse the log file.

Adapt the test accordingly.

Merges: https://pagure.io/pungi/pull-request/1475
Signed-off-by: Christian Kellner <christian@kellner.me>
2020-11-30 09:57:48 +01:00
Lubomír Sedlář
9accf5ecf4 docs: Add osbuild phase to overview diagram
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-11-27 09:56:10 +01:00
Lubomír Sedlář
e7af6d2ac2 osbuild: Only send release when not empty
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-11-25 15:37:49 +01:00
Lubomír Sedlář
27bab19a5e Fix config validation for osbuild
Tests are now added for it as well.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-11-25 14:14:18 +01:00
Haibo Lin
4c88e7dc0e Use xorrisofs for creating ISOs when needed
A new configuration *createiso_use_xorrisofs* is added to determine
which tool to use for creating ISOs.

By default, createiso_use_xorrisofs = False and genisoimage will be used.
When set to True, xorrisofs will be used.

JIRA: RHELCMP-2875
Fixes: https://pagure.io/pungi/issue/1130

Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-11-20 09:47:49 +08:00
Jan Kaluza
c27e21ccf8 Add --respin-of argument.
It is used to defined the relation between original compose
and respun compose with a hotfix in CTS.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-11-19 12:26:38 +00:00
Lubomír Sedlář
e3a500ca50 Fix typo in configuration
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-11-16 10:50:01 +01:00
Jan Kaluza
4562fba459 Optimize link_files by creating temporary dict mapping path to pkg_obj.
Previously, the pkg_object has been found by iterating over pkg_sets
for each package. This was quite slow given the number of RPMs in the
compose.

In this commit, the temporary dict is created which stores mapping
between path and pkg_obj and is used instead.

In this commit, the get_package_path is called just once instead of
twice in the loop which might also save some time.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-11-11 12:20:01 +00:00
Jan Kaluza
bb8cd030ec Try reuse old gather_phase even if pkgset_koji_builds changed.
The pkgset_koji_builds influences the gather phase, but the
change itself is not a reason to not reuse old gather phase. if
new pkgset_koji_builds value leads to significant change in input
package set, we will find that later in this function when comparing
old and new package set.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-11-10 12:52:02 +01:00
Jan Kaluza
94bc5e286d Do not use shlex_quote in get_pungi_buildinstall_cmd and get_pungi_ostree_cmd.
These methods return command as list which is never serialized to str and
never executed using bash. It is instead passed directly to
`kobo.shortcuts.run`.

The `shlex_quote` usage here actually breaks the code, because it adds
quotes there which are needed only if this command would be serialized
to string and passed to bash. But this never happens. As a result,
the arguments passed to `kobo.shortcuts.run` contain those extra
quotes.

In this commit the shlex_quote is removed completely from this
part of code.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-11-10 07:45:05 +00:00
Lubomír Sedlář
295a60a704 gather: Fix test for module presence
When testing if a variant has some modules,
it's no longer enough to look at `variant.modules`. That attribute
contains only names of modules configured in variants.xml.

With modules being specified in `pkgset_koji_module_builds` or
`pkgset_scratch_modules` options it is possible the code wouldn't
trigger even if there was a module.

JIRA: RHELCMP-3054
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-11-09 09:33:03 +00:00
Lubomír Sedlář
99c1e2eb5e Kill all subprocess in signal handler
When Pungi receives a signal to terminate, it can sometimes get stuck
if there are threads running. It has to wait for all worker threads to
finish. They generally do finish, unless they get stuck waiting on a
subprocess.

This patch should reduce the likelihood of this happening by stopping
all subprocesses.

JIRA: RHELCMP-3056
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-11-09 08:45:57 +01:00
Bohdan Khomutskyi
4c4c816e70 Pungi 4.2.6 release
Signed-off-by: Bohdan Khomutskyi <bkhomuts@redhat.com>
2020-11-04 12:33:06 +00:00
Lubomír Sedlář
a45f4969f3 Add phase for building images with osbuild
This is similar to image-build in terms of what it does, and somewhat
similar to OSBS phase in how it's implemented.

The phase reads configuration, submits the build via XMLRPC call and
waits for the task to finish. Then it downloads the built image and
includes it in the compose metadata.

JIRA: RHELCMP-315
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-11-04 13:04:43 +01:00
Jan Kaluza
609a555597 Allow setting int arguments for pungi-buildinstall plugin.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-11-02 12:14:14 +00:00
Jan Kaluza
d1eac95cda Use shlex_quote for complete --foo=bar argument.
Without that, the resulting string is `--foo="'bar'"`
which results in `'bar'` being passed to Koji task.

With this commit, the resulting string is `"'--foo=bar'"`
which results in `bar` being passed to Koji which is expected.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-10-29 16:17:21 +01:00
Haibo Lin
4c297beb65 Include images info in composeinfo.json
This change requires https://github.com/release-engineering/productmd/pull/150

JIRA: RHELCMP-2296
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-10-14 17:57:30 +08:00
Lubomír Sedlář
f27b120cfc tests: Fix mock usage for Python 2
The `args` attribute doesn't work, let's instead access the arguments
via index.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-09-25 12:41:05 +02:00
Lubomír Sedlář
9df3f42a44 Fix typo in config validation
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-09-25 09:29:25 +00:00
Lubomír Sedlář
327019264f 4.2.5 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-09-25 09:52:31 +02:00
Jan Kaluza
1595e188a9 Allow setting --development compose_type.
This is related to https://github.com/release-engineering/productmd/pull/149.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-09-25 07:31:20 +02:00
Haibo Lin
866b881072 Make sure old pkgset arch repo exists when reuing
JIRA: RHELCMP-2482
Fixes: https://pagure.io/pungi/issue/1424

Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-09-24 10:43:35 +08:00
Jan Kaluza
4623536b24 Fix wrong condition when reusing old gather phase results.
When list is used in gather_lookaside_repos, the Pungi currently
fails with an exception. This is caused by inverted condition
in the code which tries to filter-out the lookaside repos
generated during the Pungin execution pointing to different
compose variants.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-09-23 07:02:11 +00:00
Haibo Lin
27a825de48 pkgset: Allow to include extra module builds
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-09-22 13:22:56 +08:00
Jiri Konecny
fdb2449c0e
Adapt tests to new patch-iso --work-dir parameter
Signed-off-by: Jiri Konecny <jkonecny@redhat.com>
2020-09-15 16:07:18 +02:00
Jiri Konecny
59727f84b1
Support change of the patch-iso temp dir
This is useful when running pungi-patch-iso on VM with low amount of
memory but higher disk space. Without this option the operation will
fail because /tmp is tmpfs filesystem.

Signed-off-by: Jiri Konecny <jkonecny@redhat.com>
2020-09-15 15:47:16 +02:00
Lubomír Sedlář
22efe15379 pkgset: Handle exceptions in pkgset threads
There are two thread pools for making package sets. If Pungi is being
terminated by external event and the exception is handled in the first
thread, the second one never gets to the `stop` method and the process
keeps hanging.

This patch should make sure that `stop()` is called on both pools.

JIRA: RHELCMP-2459
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-09-15 09:55:32 +02:00
Haibo Lin
b3a55fd863 gitwrapper: Re-run git init before do full clone
Sometimes full clone failed with error fatal: Not a git repository
and we found that .git/refs/ dir is missing after shallow clone failed.

JIRA: RHELCMP-724
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-09-10 10:09:46 +08:00
Lubomír Sedlář
c6312b34d0 doc: Improve description of module defaults dir
JIRA: RHELCMP-812
Resolves: https://pagure.io/pungi/issue/1394
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-09-08 07:31:38 +00:00
Lubomír Sedlář
e12331db78 util: Refactor retry function
When running a Bodhi update, somehow the raise condition was triggered
before any exception was raised, which caused the compose to fail.

This shouldn't really happen often, but it's possible if the machine
adjusts time for any reason (e.g. DST).

This commit moves the check for timeout after the function is called.
This can cause an extra invocation of the function, which shouldn't be a
huge deal really.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-09-01 12:41:06 +00:00
Haibo Lin
160fc4f7df createrepo: Ignore error when cleaning up tmp dir
JIRA: RHELCMP-998
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-09-01 18:27:00 +08:00
Lubomír Sedlář
a6a96e40db Preserve environment when running koji commands
JIRA: RHELBLD-2479
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-08-25 09:07:28 +02:00
Lubomír Sedlář
e628bb91ec Fix formatting
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-08-20 15:15:13 +02:00
Jiri Konecny
40bbb4325a Adapt tests to a new get_graft_points() argument
Signed-off-by: Jiri Konecny <jkonecny@redhat.com>
2020-08-20 15:15:13 +02:00
Jiri Konecny
8d4fea7890 Fix pungi-patch-iso to work with a new compose top dir solution
get_graft_points has changed but without adapting pungi-patch-iso utility so the
exception was raised when executed.

Fixes: https://pagure.io/pungi/issue/1438
Signed-off-by: Jiri Konecny <jkonecny@redhat.com>
2020-08-20 15:15:02 +02:00
Jiri Konecny
981b69c699 Don't use compose in get_graft_points
Not all usage of the get_graft_points have access to the compose object.

Signed-off-by: Jiri Konecny <jkonecny@redhat.com>
2020-08-20 15:06:49 +02:00
Lubomír Sedlář
b557bf160f Fall back to rpm2cpio
JIRA: RHELCMP-2030
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-08-20 10:19:17 +02:00
Haibo Lin
1e1c8533ac tests: Fix tests for old version of git
GitSCMTestCaseReal failed with error 'Unknown option: -C' when building
rpm for el7.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-08-18 12:38:20 +08:00
Haibo Lin
29761d1656 4.2.4 release
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-08-17 09:19:10 +08:00