Commit Graph

301 Commits

Author SHA1 Message Date
Haibo Lin
0525768519 Add pkgset_allow_reuse option
JIRA: RHELCMP-492
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-04-28 09:54:36 +08:00
Jan Kaluza
5395af416c Allow reusing old buildinstall phase results.
New `buildinstall.metadata` file is created once the buildinstall
phase is done. This file contains:

- list of lorax command line arguments.
- list of RPMs installed in the buildinstall buildroot.
- list of RPMs installed in the resulting boot.iso.

This file is checked in the next compose run to find out if
the result of buildinstall phase from the previous compose
can be reused. Following is checked:

- lorax commandline arguments are the same (except of expected
  differences).
- The NVRAs of RPMs in the runroot_tag are the same as the ones
  installed in the old buildinstall buildroot.
- The NVRAs of RPMs installed in the boot.iso are the same as
  the ones in package sets in the current compose.

By its implementation, this reuse strategy is used only if
pungi_buildinstall Koji plugin is used.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>

Add tests for buildinstall reuse and buildinstall_allow_reuse option.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-04-17 11:37:13 +00:00
Haibo Lin
477b43d4e9 Split repoclosure into separate phase
Move repoclosure out from test phase into its own phase and
run parallel with image building phases(osbs, imagebuild, ...)
to speed things up.

JIRA: RHELCMP-8
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-04-14 09:25:57 +08:00
Lubomír Sedlář
4efdacd0a0 4.2.1 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-03-13 12:55:44 +01:00
Jan Kaluza
169fa5b453 Allow reusing gather phase results.
- Get also requires/provides of RPMs in package set.
- Store the results of gather phase as pickle file.
- Reuse old gather phase results in case Pungi configuration
  did not change, the "names" of RPMs in global package set
  did not change and their requires/provides did not change.
- Add `gather_allow_reuse` option to enable this feature.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>

Add gather_allow_reuse, add more tests and better handling of gather_lookaside_repos.
2020-03-13 10:17:59 +01:00
Jan Kaluza
e70ad8aaa5 Run extra_files phase in parallel to other phases.
It seems the other phases executed in Weaver phase does not need
extra_files output, so it can be run in parallel to them.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-03-12 07:58:57 +01:00
Haibo Lin
ff269a8675 gather: Make additional_packages accept debuginfo packages
Now debuginfo packages defined in additional_packages option
could be handled correctly.

JIRA: COMPOSE-4085
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-12 16:38:08 +08:00
Haibo Lin
3a31d47c83 doc: add code convention requirements
JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 17:00:39 +08:00
Lubomír Sedlář
6f23c7b8ba 4.2.0 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-02-03 08:26:02 +01:00
Jan Kaluza
3cde5c3a87 Add support for new Pungi Buildinstall Koji plugin.
We would like to start generating the buildinstall phase using the safer
Koji Pungi Buildinstall plugin and stop the direct use of Runroot plugin.

The plugin so far exists only as PR for Koji:
https://pagure.io/koji/pull-request/1939

This commit adds support for this plugin when `lorax_use_koji_plugin`
is set to `True`.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-01-23 16:09:15 +01:00
Haibo Lin
93ed40ad2d Remove invalid parameters from osbs config
JIRA: COMPOSE-4049
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-13 09:05:55 +00:00
Lubomír Sedlář
aa6a213c8d Add a new option for customizing version in .treeinfo
There was an attempt to achieve this by customizing --version and
--release arguments for Lorax, but Pungi does not take the [general] and
[release] sections from its .treeinfo. Instead it was always using
release version.

The value from this new option will be put into .treeinfo and used for
--version and --release arguments in Lorax (unless explicitly defined in
lorax_options to another value).

JIRA: COMPOSE-4029
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-07 10:55:48 +01:00
Jan Kaluza
3750d6795f Update runroot_method documentation.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-01-06 07:47:38 +01:00
Haibo Lin
794d151bef Remove productimg phase
JIRA: COMPOSE-4004
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-03 11:42:38 +00:00
Haibo Lin
3cf16eb42d cleanup: refactor scripts via entry points
Fixes: https://pagure.io/pungi/issue/1045
JIRA: COMPOSE-2946
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-12-10 10:53:31 +08:00
Lubomír Sedlář
ea2cd448a0 4.1.41 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-12-02 14:51:35 +01:00
Lubomír Sedlář
20c3614fb3 scm: Add backend for downloading archives from Koji
Tests and documentation included.

JIRA: COMPOSE-3816
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-05 08:51:26 +01:00
Lubomír Sedlář
8829fc32ab 4.1.40 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-23 09:02:41 +02:00
Juliana Rodrigueiro
f8dcda9dcb doc: Make it clear that the field 'repo' has to be empty for 'file' backend
Leaving the field 'repo' out for the 'file' backend makes the compose fail.

    [ERROR   ] Compose run failed: 'repo'

Signed-off-by: Juliana Rodrigueiro <juliana.rodrigueiro@intra2net.com>
2019-10-18 10:30:25 +02:00
Lubomír Sedlář
014560f8bd doc: Add full examples of compose configuration
JIRA: COMPOSE-3797
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-14 11:04:39 +02:00
Owen W. Taylor
72bf795bd4 ostree-install: allow configuring additional depenencies for runroot
A lorax template used for the ostree-installer might need an additional
package dependency (e.g., flatpak to embed a flatpak repository) - add
a config key 'extra_runroot_pkgs' to the ostree installer configuration
to allow supplementing the set of packages installed into the runroot.

Signed-off-by: Owen W. Taylor <otaylor@fishsoup.net>
2019-10-10 15:13:38 -04:00
Lubomír Sedlář
c346492df4 buildinstall: Allow customizing dracut arguments
JIRA: COMPOSE-3853
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-09 15:42:10 +02:00
Lubomír Sedlář
21d45eb243 pkgset: Allow filtering modules from Koji tags
Add a configuration option to enable skipping some modules found in the
configured tag.

Fixes: https://pagure.io/pungi/issue/1260
JIRA: COMPOSE-3794
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-24 10:13:00 +02:00
Lubomír Sedlář
e814e1445a 4.1.39 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-09 09:30:45 +02:00
Haibo Lin
3811c0a176 config: Deprecate bootable option
JIRA: COMPOSE-2635
Fixes: https://pagure.io/pungi/issue/976
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-07-22 16:51:38 +08:00
Lubomír Sedlář
3097019338 test: Allow turning strictness off as well
When the value is explicitly set to `False`, the check should not be
strict.

JIRA: COMPOSE-3658
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-16 08:24:18 +02:00
Lubomír Sedlář
62c6d4ddcf test: Option to make size check strict
Sometimes it's practical not just warn when ISO is larger than expected,
but to also abort the compose.

JIRA: COMPOSE-3658
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-15 13:57:50 +02:00
Lubomír Sedlář
a340663f00 doc: Clarify relationship to Koji
Fixes: https://pagure.io/pungi/issue/1143
JIRA: COMPOSE-3372
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-03 09:06:49 +02:00
Lubomír Sedlář
46333b01ad 4.1.38 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-02 10:32:01 +02:00
Haibo Lin
c4ed2bf3b2 config: Deprecate runroot option
Fixes: https://pagure.io/pungi/issue/975
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-06-27 18:00:07 +08:00
Haibo Lin
ce066707c1 config: Deprecate release_is_layered option
Fixes: https://pagure.io/pungi/issue/977
Merges: https://pagure.io/pungi/pull-request/1204
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-06-06 13:45:49 +02:00
Lubomír Sedlář
163d69713d 4.1.37 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-27 10:40:38 +02:00
Jan Kaluza
cfb7b71fca Fix issues in OpenSSH Runroot method found by real tests.
- Pass the runroot_tag to init command in OpenSSH Runroot method.
  This is needed for the init command as a source for initial packages
  for the buildroot.
- Rename the "runroot_ssh_init_command" to "runroot_ssh_init_template"
  to make it consistent with the rest of "runroot_ssh_*" options.
- Add missing "runroot_ssh_*" options to checks.py.
- Use chmod/chown to `output_dir` in OpenSSH Runroot method the same way
  as it is used in Koji runroot method to make the runroot output readable
  for Pungi user.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2019-05-15 07:40:31 +02:00
Lubomír Sedlář
479d17c033 4.1.36 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-12 12:10:39 +02:00
Jan Kaluza
49d137b444 Extend "openssh" runroot_method to be able to execute "mock"
This adds few new config options which are well described in the
configuration documentation. Please refer to it for more information.

Merges: https://pagure.io/pungi/pull-request/1170
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2019-04-12 11:10:40 +02:00
Lubomír Sedlář
d521711957 osbs: Rework configuration for image pushes
Embedding the registry configuration into OSBS config itself is
simple, but makes it impossible to reuse the same configuration for
multiple different composes.

A nice example is a nightly pushing images to a testing registry, and
production compose building the same images but pushing to staging
location. The original design requires duplication of all the
configuration just because registries are different.

With this option, the push information is stored in a separate option as
a mapping from NVR patterns to arbitrary data. The patterns are used to
match finished builds to registry.

The old configuration is marked as deprecated in code and will
eventually be removed. The deprecation handling in config validation
does not allow emitting warnings for nested values.

JIRA: COMPOSE-3394
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-05 10:28:34 +02:00
Jan Kaluza
959d6979d4 Add "openssh" runroot method
Merges: https://pagure.io/pungi/pull-request/1166
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2019-04-04 14:44:05 +02:00
Lubomír Sedlář
5f8b519941 4.1.35 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-26 10:03:26 +01:00
Lubomír Sedlář
358fdd50ce buildinstall: Allow overwriting version for lorax
Sometimes the release version can be more specific than what should be
exposed to users of the boot iso.

JIRA: COMPOSE-3295
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-22 12:47:16 +01:00
Jan Kaluza
75bb48a882 Create new Runroot class and use it everywhere where runroot task is executed.
This adds new `Runroot` class and new `runroot_method` option which makes
it possible to choose between two currently available runroot methods:

- Local
- Koji

The main goal of this commit is to make it possible to add new runroot
methods in the future and this is the first step in that direction.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2019-03-21 14:15:58 +01:00
Lubomír Sedlář
45cdbb2faf orchestrator: Send messages about the main compose
Only start/finish messages will be sent if a handler is configured.

JIRA: COMPOSE-3288
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-21 09:29:54 +01:00
Lubomír Sedlář
088ea7fe37 orchestrator: Support generic pre- and post- scripts
Run arbitrary commands before and after the compose.

The example config is updated to generate latest symlink with a
post-compose script. The pre compose script runs always, post compose
runs only if the compose is not doomed.

JIRA: COMPOSE-3288
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-21 09:29:54 +01:00
Lubomír Sedlář
86fb93d603 orchestrator: Support getting kerberos ticket
If the configuration sets keytab path and principal, run kinit with
custom cache file, and delete the file at the end of the run.

JIRA: COMPOSE-3288
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-21 09:29:54 +01:00
Ken Dreyer
b5bef68c60 doc: explain koji_profile
Add more details to the documentation for the koji_profile setting.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2019-03-14 15:53:15 -06:00
Lubomír Sedlář
cc456d3e75 4.1.34 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-05 12:37:07 +01:00
Lubomír Sedlář
3efcfede6a image-build: Support repo/install_tree as path
If a repo or install tree is specified as an absolute path on the local
filesystem, we should either translate it using the configured mappings,
or if no mapping matches, it should be return unchanged. A variant name
can not start with a slash, so attempting that translation does not make
much sense.

JIRA: COMPOSE-3290
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-25 08:18:47 +01:00
Lubomír Sedlář
f33973ee65 Make the Apple/HFS compatibility configurable
The default is the original behaviour. On F30+ a new option should be
added to config to make it work.

Over time as users move to this option (which requires a new enough
version of lorax), the default should be switched and then the option
removed.

Resolves: https://pagure.io/pungi/issue/1126
Merges: https://pagure.io/pungi/pull-request/1128
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-21 16:11:01 +01:00
Lubomír Sedlář
e21c49d9e2 4.1.33 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-13 10:37:48 +01:00
Lubomír Sedlář
6c6d4759f5 isos: Check maximum expected size
This patch allows the configuration to express maximum expected size for
ISOs created in createiso and extra_isos phases. If the image is larger
than this limit, a warning is emitted in test phase. The compose itself
is not affected in any way.

JIRA: COMPOSE-2824
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-12 15:15:30 +01:00
Lubomír Sedlář
7693e562b1 osbs: Process data about pushing images to registries
This patch does not do any actual pushing. It will only extract data
about push targets from the main configuration and store it together
with exact Koji NVR in a well-defined location, and also send the data
to message bus for another service to handle.

JIRA: COMPOSE-3228
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-12 15:12:50 +01:00