Commit Graph

2895 Commits

Author SHA1 Message Date
Lubomír Sedlář
22efe15379 pkgset: Handle exceptions in pkgset threads
There are two thread pools for making package sets. If Pungi is being
terminated by external event and the exception is handled in the first
thread, the second one never gets to the `stop` method and the process
keeps hanging.

This patch should make sure that `stop()` is called on both pools.

JIRA: RHELCMP-2459
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-09-15 09:55:32 +02:00
Haibo Lin
b3a55fd863 gitwrapper: Re-run git init before do full clone
Sometimes full clone failed with error fatal: Not a git repository
and we found that .git/refs/ dir is missing after shallow clone failed.

JIRA: RHELCMP-724
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-09-10 10:09:46 +08:00
Lubomír Sedlář
c6312b34d0 doc: Improve description of module defaults dir
JIRA: RHELCMP-812
Resolves: https://pagure.io/pungi/issue/1394
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-09-08 07:31:38 +00:00
Lubomír Sedlář
e12331db78 util: Refactor retry function
When running a Bodhi update, somehow the raise condition was triggered
before any exception was raised, which caused the compose to fail.

This shouldn't really happen often, but it's possible if the machine
adjusts time for any reason (e.g. DST).

This commit moves the check for timeout after the function is called.
This can cause an extra invocation of the function, which shouldn't be a
huge deal really.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-09-01 12:41:06 +00:00
Haibo Lin
160fc4f7df createrepo: Ignore error when cleaning up tmp dir
JIRA: RHELCMP-998
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-09-01 18:27:00 +08:00
Lubomír Sedlář
a6a96e40db Preserve environment when running koji commands
JIRA: RHELBLD-2479
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-08-25 09:07:28 +02:00
Lubomír Sedlář
e628bb91ec Fix formatting
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-08-20 15:15:13 +02:00
Jiri Konecny
40bbb4325a Adapt tests to a new get_graft_points() argument
Signed-off-by: Jiri Konecny <jkonecny@redhat.com>
2020-08-20 15:15:13 +02:00
Jiri Konecny
8d4fea7890 Fix pungi-patch-iso to work with a new compose top dir solution
get_graft_points has changed but without adapting pungi-patch-iso utility so the
exception was raised when executed.

Fixes: https://pagure.io/pungi/issue/1438
Signed-off-by: Jiri Konecny <jkonecny@redhat.com>
2020-08-20 15:15:02 +02:00
Jiri Konecny
981b69c699 Don't use compose in get_graft_points
Not all usage of the get_graft_points have access to the compose object.

Signed-off-by: Jiri Konecny <jkonecny@redhat.com>
2020-08-20 15:06:49 +02:00
Lubomír Sedlář
b557bf160f Fall back to rpm2cpio
JIRA: RHELCMP-2030
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-08-20 10:19:17 +02:00
Haibo Lin
1e1c8533ac tests: Fix tests for old version of git
GitSCMTestCaseReal failed with error 'Unknown option: -C' when building
rpm for el7.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-08-18 12:38:20 +08:00
Haibo Lin
29761d1656 4.2.4 release
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-08-17 09:19:10 +08:00
Jan Kaluza
2657a12c96 Allow setting CTS parent_compose_ids using --parent-compose-id option.
This is needed to track dependencies between composes in the Compose
Tracking Service.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-08-13 13:22:43 +02:00
Bohdan Khomutskyi
7a6d8303dc Replace -c parameter with --config
Also, reformat the invocation of Lorax, since
Lorax does not accept -c=%s, only -c %s.

Jira: RHELCMP-713
Signed-off-by: Bohdan Khomutskyi <bkhomuts@redhat.com>
2020-08-13 11:16:43 +02:00
Haibo Lin
c273350fe5 doc: Update doc/contributing.rst
JIRA: RHELCMP-1773
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-08-11 20:36:22 +08:00
Haibo Lin
4dcb6dee0d Use requirements.txt
JIRA: RHELCMP-1773
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-08-11 20:36:22 +08:00
Bohdan Khomutskyi
b899126b7e Allow squashfs-only and configuration_file in lorax_options
The change allows for setting the parameters as described below to Lorax.
Lorax, a program called during the buildInstall phase, creates the SquashFS
during the buildInstall phase.
The Squash filesystem is present both on the DVD and the BOOT.ISO.

squashfs_only --- (str) passes --squashfs-only option.
configuration_file --- (str or scm_dict) passes -c option to Lorax.

The final goal of this change is to allow for optimization of
the installation medium size.

This pull request is related to the Fedora change proposal, which is available
at this location:
https://fedoraproject.org/wiki/Category:Changes/OptimizeSquashFS
See the change proposal for more information about the benefits of higher
compression ratio.

Jira: RHELCMP-693
Signed-off-by: Bohdan Khomutskyi <bkhomuts@redhat.com>
2020-08-11 09:29:50 +00:00
Haibo Lin
9920aa7a74 Hardlink or copy scratch builds always
JIRA: RHELCMP-1566
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-08-07 18:00:19 +08:00
Lubomír Sedlář
a294a05726 Retry ostree installer task on losetup error
This uses a previous change that added ability to retry and simply
extends the conditions when it fires.

JIRA: RHELCMP-1863
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-08-05 10:58:04 +02:00
Haibo Lin
3c72755814 Use pytest instead of nosetests
Nose has been in maintenance mode for the past several years
and pytest is a good replacement.

Other changes:
- Replace deprecated assertRegexpMatches with assertRegex
- Replace deprecated assertRaisesRegexp with assertRaisesRegex
- Replace deprecated SafeConfigParser with ConfigParser
- Force reinstall pytest and mock in tox virtualenv. This is because
  the globally installed packages may not work as expected(occured in
  jenkins job).

JIRA: RHELCMP-1619
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-07-29 14:57:16 +08:00
Haibo Lin
495a4c48b2 Run tests via tox
This could simplify jenkins job configuration and make the tasks
trackable in git.

JIRA: RHELCMP-1618
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-07-27 13:28:32 +08:00
Haibo Lin
05a5e2b1f0 Make sure old_repo_dir for reusing exists
Fixes: https://pagure.io/pungi/issue/1424
JIRA: RHELCMP-1519
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-07-22 15:11:25 +08:00
Haibo Lin
7e6bed9713 Retry buildinstall tasks on losetup error
JIRA: RHELCMP-1394
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-07-21 17:15:25 +08:00
Haibo Lin
f7167fa3b6 Allow including scratch module builds
JIRA: RHELCMP-439
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-07-17 09:08:36 +08:00
Lubomír Sedlář
f5e33950c1 Unpack RPMs using rpm2archive
This should support both older files compressed with cpio as well as
newer zstd-compressed files.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-07-09 07:25:17 +00:00
Lubomír Sedlář
7d00942d13 Allow test calls in any order
There are threads involved and the order is undefined. We need to accept
any order.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-07-08 08:26:59 +02:00
Haibo Lin
0ab6f48de3 Create arch repo when reusing failed
Reusing old arch repo may fail for reasons such as arch not
available in old compose or unexpected error when copying data
from old compose.

JIRA: RHELCMP-994
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-07-07 17:59:12 +08:00
Haibo Lin
b193fa0ab7 createiso: Ignore errors when deleting staging dir
JIRA: RHELCMP-975
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-07-07 16:32:10 +08:00
Ondrej Nosek
d9f111edae Remove buffering when running koji commands
If the compose is aborted while koji tasks are running, we can be
left with empty log files. That complicates debugging.

JIRA: RHELCMP-1218

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2020-07-02 09:45:46 +02:00
Lubomír Sedlář
54882a0fc4 Fix typos
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-06-24 14:16:42 +02:00
Jan Kaluza
b6573fab92 Check if composeinfo-base.json exists before creating it.
Previously, we checked only for `compose_dir`, but it makes
more sense to really check for `composeinfo-base.json` instead.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-06-23 15:27:53 +02:00
Pierre-Yves Chibon
658a5f805f Port scripts/wait_for_signed_ostree_handler.py to fedora-messaging
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
2020-06-17 23:28:27 +02:00
Pierre-Yves Chibon
ad1a3360bc Port scripts/fedmsg_notification.py to fedora-messaging
This commit also adds a --config argument allowing to
override/specify a specific fedora-messaging configuration
file to use.

Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
2020-06-17 10:11:39 +02:00
Haibo Lin
b6605827b3 buildinstall: Improve error reporting when lorax fails
Logs of DepsolveError will be printed in the main log.

Fixes: https://pagure.io/pungi/issue/1399
JIRA: RHELCMP-955
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-06-17 13:17:22 +08:00
Lubomír Sedlář
634d30fac5 4.2.3 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-06-11 14:17:49 +02:00
Jan Kaluza
e35c250700 Move test for unsigned packages with pkgset_koji_scratch_tasks to PkgsetPhase class.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-06-11 12:05:47 +00:00
Jan Kaluza
4a15d1351a Allow building compose with scratch builds defined by pkgset_koji_scratch_tasks.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-06-11 12:05:47 +00:00
Lubomír Sedlář
b59bdcea92 createrepo: Allow making productid glob stricter
This patch updates the documentation to match the actual behavior, and
adds a configuration option to remove the leading prefix.

The extra wildcard is causing problems when there are two variants in
the compose and one UID is a suffix of the other (e.g. DevTools and
Tools), since multiple files will match the shorter name and an error
will be reported.

JIRA: RHELCMP-1086
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-06-11 10:36:37 +02:00
Lubomír Sedlář
649ff095c0 docs: Remove outdated note
Fixes: https://pagure.io/pungi/issue/1396
JIRA: RHELCMP-850
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-06-10 14:01:42 +02:00
Lubomír Sedlář
3bb1e3df11 createrepo: Add extra modulemd files to the repo
This is a workaround for modularity design issues and DNF bugs. If there
were gaps in contexts, DNF has trouble handling the upgrades. Thus we
may need to add module metadata for older versions of previously
released module streams and add the missing contexts.

JIRA: RHELCMP-982
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-06-10 09:50:44 +02:00
Haibo Lin
6ac12af343 pkgset: handle exception when using dogpile cache
JIRA: RHELCMP-562
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-06-10 13:35:12 +08:00
Haibo Lin
f5bfd509ab notification: Add compose_path into the messages
It's needed by the notification script added in RHELCMP-401.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-06-09 06:40:26 +00:00
Frédéric Pierret (fepitre)
b973657197 gather: handle mirrorlist in kickstart
Merges: https://pagure.io/pungi/pull-request/1406
Signed-off-by: Frédéric Pierret (fepitre) <frederic.pierret@qubes-os.org>
2020-06-09 08:34:58 +02:00
Haibo Lin
0196d7fd00 Allow only creating unified ISO for specified arch
Fixes: https://pagure.io/pungi/issue/1393
JIRA: RHELCMP-807
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-06-09 09:47:12 +08:00
Haibo Lin
ffb65e8770 docs: update doc for gather_lookaside_repos option
JIRA: RHELCMP-566
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-06-02 14:41:56 +08:00
Haibo Lin
7c2743fb50 Include the output of getisoimage in the error message
JIRA: RHELCMP-720
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-06-01 13:29:55 +08:00
Lubomír Sedlář
d6caf0785b tests: Patch time in CTS related tests
The value must be fixed to match the expected JSON.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-05-29 10:24:17 +02:00
Jan Kaluza
b8c3ca1abe Allow using Pungi Koji plugin for ostree phases.
This commits changes `ostree` and `ostree_installer` phases
so they can run with Koji Pungi plugin instead of the plain runroot.

It is similar to `buildinstall` phase running with Koji plugin.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-05-26 13:23:58 +00:00
Jan Kaluza
f1eea0b5a6 Allow getting the compose id from CTS (Compose Tracking Service).
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-05-26 13:18:38 +00:00