Commit Graph

2416 Commits

Author SHA1 Message Date
Ondrej Nosek
d9f111edae Remove buffering when running koji commands
If the compose is aborted while koji tasks are running, we can be
left with empty log files. That complicates debugging.

JIRA: RHELCMP-1218

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2020-07-02 09:45:46 +02:00
Lubomír Sedlář
54882a0fc4 Fix typos
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-06-24 14:16:42 +02:00
Jan Kaluza
b6573fab92 Check if composeinfo-base.json exists before creating it.
Previously, we checked only for `compose_dir`, but it makes
more sense to really check for `composeinfo-base.json` instead.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-06-23 15:27:53 +02:00
Pierre-Yves Chibon
658a5f805f Port scripts/wait_for_signed_ostree_handler.py to fedora-messaging
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
2020-06-17 23:28:27 +02:00
Pierre-Yves Chibon
ad1a3360bc Port scripts/fedmsg_notification.py to fedora-messaging
This commit also adds a --config argument allowing to
override/specify a specific fedora-messaging configuration
file to use.

Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
2020-06-17 10:11:39 +02:00
Haibo Lin
b6605827b3 buildinstall: Improve error reporting when lorax fails
Logs of DepsolveError will be printed in the main log.

Fixes: https://pagure.io/pungi/issue/1399
JIRA: RHELCMP-955
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-06-17 13:17:22 +08:00
Lubomír Sedlář
634d30fac5 4.2.3 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-06-11 14:17:49 +02:00
Jan Kaluza
e35c250700 Move test for unsigned packages with pkgset_koji_scratch_tasks to PkgsetPhase class.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-06-11 12:05:47 +00:00
Jan Kaluza
4a15d1351a Allow building compose with scratch builds defined by pkgset_koji_scratch_tasks.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-06-11 12:05:47 +00:00
Lubomír Sedlář
b59bdcea92 createrepo: Allow making productid glob stricter
This patch updates the documentation to match the actual behavior, and
adds a configuration option to remove the leading prefix.

The extra wildcard is causing problems when there are two variants in
the compose and one UID is a suffix of the other (e.g. DevTools and
Tools), since multiple files will match the shorter name and an error
will be reported.

JIRA: RHELCMP-1086
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-06-11 10:36:37 +02:00
Lubomír Sedlář
649ff095c0 docs: Remove outdated note
Fixes: https://pagure.io/pungi/issue/1396
JIRA: RHELCMP-850
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-06-10 14:01:42 +02:00
Lubomír Sedlář
3bb1e3df11 createrepo: Add extra modulemd files to the repo
This is a workaround for modularity design issues and DNF bugs. If there
were gaps in contexts, DNF has trouble handling the upgrades. Thus we
may need to add module metadata for older versions of previously
released module streams and add the missing contexts.

JIRA: RHELCMP-982
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-06-10 09:50:44 +02:00
Haibo Lin
6ac12af343 pkgset: handle exception when using dogpile cache
JIRA: RHELCMP-562
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-06-10 13:35:12 +08:00
Haibo Lin
f5bfd509ab notification: Add compose_path into the messages
It's needed by the notification script added in RHELCMP-401.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-06-09 06:40:26 +00:00
Frédéric Pierret (fepitre)
b973657197 gather: handle mirrorlist in kickstart
Merges: https://pagure.io/pungi/pull-request/1406
Signed-off-by: Frédéric Pierret (fepitre) <frederic.pierret@qubes-os.org>
2020-06-09 08:34:58 +02:00
Haibo Lin
0196d7fd00 Allow only creating unified ISO for specified arch
Fixes: https://pagure.io/pungi/issue/1393
JIRA: RHELCMP-807
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-06-09 09:47:12 +08:00
Haibo Lin
ffb65e8770 docs: update doc for gather_lookaside_repos option
JIRA: RHELCMP-566
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-06-02 14:41:56 +08:00
Haibo Lin
7c2743fb50 Include the output of getisoimage in the error message
JIRA: RHELCMP-720
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-06-01 13:29:55 +08:00
Lubomír Sedlář
d6caf0785b tests: Patch time in CTS related tests
The value must be fixed to match the expected JSON.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-05-29 10:24:17 +02:00
Jan Kaluza
b8c3ca1abe Allow using Pungi Koji plugin for ostree phases.
This commits changes `ostree` and `ostree_installer` phases
so they can run with Koji Pungi plugin instead of the plain runroot.

It is similar to `buildinstall` phase running with Koji plugin.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-05-26 13:23:58 +00:00
Jan Kaluza
f1eea0b5a6 Allow getting the compose id from CTS (Compose Tracking Service).
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-05-26 13:18:38 +00:00
Lubomír Sedlář
59e2aa9607 Fix flake8 issues
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-05-26 11:49:50 +02:00
Bohdan Khomutskyi
694b7f3d28 Optimize the _link_file function to not call os.stat redundantly.
This will eliminate 2 calls to os.stat per one invocation of the _link_file function.
Assuming during the compose build 50000 files are linked, this change will eliminate 100000 redundant calls to os.stat.

Jira: RHELCMP-797
Signed-off-by: Bohdan Khomutskyi <bkhomuts@redhat.com>
2020-05-21 10:13:28 +02:00
Ken Dreyer
4ba65710c2 doc: explain sigkey behavior
Explain how Pungi operates on the sigkeys list, and what happens when
there is only one item in the list.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2020-05-18 15:42:37 -06:00
Ken Dreyer
5ed5646bca doc: explain tradeoffs with pkgset_allow_reuse
Explain why someone would want to enable or disable the
pkgset_allow_reuse option.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2020-05-15 12:56:46 -06:00
Ken Dreyer
d3acb0fa9e doc: fix grammar for pkgset_allow_reuse setting
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2020-05-15 12:56:37 -06:00
Haibo Lin
d4efe17328 Support --skip-branding option in lorax
JIRA: RHELCMP-572
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-05-14 11:49:40 +08:00
Haibo Lin
fe4b2dd302 docs: Embed phases.svg directly
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-05-08 17:44:27 +08:00
Haibo Lin
20ba1a7639 Execute image_checksum phase right after the dependent phases
JIRA: RHELCMP-468
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-05-08 16:12:30 +08:00
Haibo Lin
5d9dcf61fb 4.2.2 release
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-04-29 19:25:59 +08:00
Lubomír Sedlář
9fced77140 scm: Workaround incorrect permissions on created directory
We have seen the directory created with wrong permissions. Since we
haven't been able to find out why it happens this is a workaround.

JIRA: RHELCMP-142
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-04-29 09:51:53 +02:00
Lubomír Sedlář
9a1b9dd154 Fix warning about productimg in skip_phases option
If phase is skipped in a config option (and not on command line), we
should just print a warning or do nothing in quiet mode.

JIRA: RHELCMP-399
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-04-28 07:51:40 +00:00
Haibo Lin
0525768519 Add pkgset_allow_reuse option
JIRA: RHELCMP-492
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-04-28 09:54:36 +08:00
Haibo Lin
153eb628e8 Delete outdated comments
JIRA: RHELCMP-395
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-04-22 17:14:51 +08:00
Lubomír Sedlář
f7944a406e gather: Fix nodeps method to not prefix match
If the input packages contain e.g. `ansible`, and there's
`ansible-runner-service` in the package set, we don't want to pull it
in.

JIRA: RHELCMP-446
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-04-21 09:58:00 +02:00
Lubomír Sedlář
1a5cd9e0bf linker: Remove check after copying file
After a file is copied, a check was done if size and mtime of the files
match.

This has very few benefits in what problems it can detect, and can cause
problems on NFS where it can take a short time before the updated
attributes are visible on the new location.

JIRA: RHELCMP-378
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-04-20 14:52:00 +02:00
Haibo Lin
30f4771db1 Stop creating iso_stage_dir before deleting
It's unreasonable to create the dir and then delete it immediately.

JIRA: RHELCMP-151
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-04-20 18:10:58 +08:00
Jan Kaluza
5395af416c Allow reusing old buildinstall phase results.
New `buildinstall.metadata` file is created once the buildinstall
phase is done. This file contains:

- list of lorax command line arguments.
- list of RPMs installed in the buildinstall buildroot.
- list of RPMs installed in the resulting boot.iso.

This file is checked in the next compose run to find out if
the result of buildinstall phase from the previous compose
can be reused. Following is checked:

- lorax commandline arguments are the same (except of expected
  differences).
- The NVRAs of RPMs in the runroot_tag are the same as the ones
  installed in the old buildinstall buildroot.
- The NVRAs of RPMs installed in the boot.iso are the same as
  the ones in package sets in the current compose.

By its implementation, this reuse strategy is used only if
pungi_buildinstall Koji plugin is used.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>

Add tests for buildinstall reuse and buildinstall_allow_reuse option.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-04-17 11:37:13 +00:00
Dusty Mabe
3509d7a36c ostree: set umask to be more permissive for ostree operations
We need to set the umask to be more permissive so directories get group
write permissions. See https://pagure.io/releng/issue/8811#comment-629051.

Signed-off-by: Dusty Mabe <dusty@dustymabe.com>
Merges: https://pagure.io/pungi/pull-request/1373
2020-04-17 09:20:24 +02:00
Haibo Lin
477b43d4e9 Split repoclosure into separate phase
Move repoclosure out from test phase into its own phase and
run parallel with image building phases(osbs, imagebuild, ...)
to speed things up.

JIRA: RHELCMP-8
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-04-14 09:25:57 +08:00
Lubomír Sedlář
e187b5ea79 Break too long line
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-04-06 14:20:47 +02:00
Buvanesh Kumar
4cf11906e8 RCM-79601: Increase time delay in race condition
Signed-off-by: Buvanesh Kumar <bsivasub@redhat.com>
2020-04-06 08:15:57 -04:00
Jan Kaluza
fdfaae8b71 Allow gather phase reuse on product_id change.
I've analysed multiple nigthly composes and often the only difference
in the configuration between two nightly composes is different
`product_id` commit hash.

The `product_id` is used in later `createrepo` phase and does not
influence the gather phase at all. I therefore think it can be
whitelisted in gather phase reuse code.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-04-06 08:23:19 +00:00
Haibo Lin
c5e59fa732 Gather more debug data for GitWrapper clone
GitWrapper._clone crashes occasionally but unfortunately
the root cause is not found yet. We need more info for debugging.

JIRA: COMPOSE-4219
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-04-03 17:08:51 +08:00
Haibo Lin
65251d983a Reuse arch pkgset repos
JIRA: COMPOSE-4217
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-03-31 13:32:12 +08:00
Haibo Lin
63ec1adc22 Get non-rpm build to pungi's extra_files with inheritance
JIRA: COMPOSE-4214
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-03-20 13:19:57 +08:00
Lubomír Sedlář
4a78514162 util: Fix regex for detecting debuginfo packages
The `re.match` function already anchors the pattern at the start of the
string, but allows for other characters to continue after match.

This is causing problems with packages like `elfutils-debuginfod-client`
which are not debuginfo.

Let's be safe and explicitly anchor both start and end.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-03-16 11:13:02 +01:00
Lubomír Sedlář
4efdacd0a0 4.2.1 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-03-13 12:55:44 +01:00
Jan Kaluza
a209bda73c Create MaterializedPackageSets in threads to make pkgset faster.
When modules are used, there are lot of small package sets. These
package sets have usually less than 500 packages. The createrepo
part of `MaterializedPackageSet.create` executed for such small
set of packages takes around 1 second. Most of this time
the createrepo_c runs in single thread. It does the initialization,
it writes the XML files, ...

The parts of createrepo which can be run in parallel and therefore
would use all the CPUs are quite small for very small package sets.

This commit therefore executes multiple threads with
`MaterializedPackageSet.create` for these very small package sets.

This saves around 40 seconds from pkgset phase for RHEL compose.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-03-13 12:51:02 +01:00
Lon Hohberger
9391ce3065 pungi-koji: Allow user to specify version component count in latest symlink
This change adds an option which allows users to specify
the number of version components (i.e. values between dots)

The default behavior is preserved, and is equivalent to using
'-1' as the value.

- Negative values remove items from the end.
- Positive values specify the count of of components
- Zero will remove the version field entirely from the symlink
- When nonzero, at least one version component will appear

Merges: https://pagure.io/pungi/pull-request/1361
Signed-off-by: Lon Hohberger <lhh@redhat.com>
2020-03-13 10:50:08 +01:00