Commit Graph

78 Commits

Author SHA1 Message Date
soksanichenko
323d31df2b Merge branch 'master' into a8_updated
# Conflicts:
#	pungi.spec
#	pungi/wrappers/kojiwrapper.py
#	setup.py
#	tests/test_extra_isos_phase.py
#	tests/test_pkgset_pkgsets.py
2022-11-07 23:38:38 +02:00
Lev Veyde
00a9861367 Updated the deprecated ks argument name (to the current inst.ks)
Signed-off-by: Lev Veyde <lveyde@redhat.com>
2021-04-21 21:04:01 +03:00
Haibo Lin
b217470464 Format code
Code didn't get well formatted when jenkins unusable.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-03-02 18:31:45 +08:00
soksanichenko
2e48c9a56f LU-2195 Change path to sources and iso when generating repositories
- We should add the images to the compose if they will be used only as netinstall image.
  E.g. *-boot.iso.
- And we shouldn't add if the images will be modified in phase `extra_isos`.
  E.g. *-minimal.iso

Change-Id: I9095cfd87414ecca46b1213553589731c82dd2e2
2021-02-22 13:23:48 +02:00
Bohdan Khomutskyi
b899126b7e Allow squashfs-only and configuration_file in lorax_options
The change allows for setting the parameters as described below to Lorax.
Lorax, a program called during the buildInstall phase, creates the SquashFS
during the buildInstall phase.
The Squash filesystem is present both on the DVD and the BOOT.ISO.

squashfs_only --- (str) passes --squashfs-only option.
configuration_file --- (str or scm_dict) passes -c option to Lorax.

The final goal of this change is to allow for optimization of
the installation medium size.

This pull request is related to the Fedora change proposal, which is available
at this location:
https://fedoraproject.org/wiki/Category:Changes/OptimizeSquashFS
See the change proposal for more information about the benefits of higher
compression ratio.

Jira: RHELCMP-693
Signed-off-by: Bohdan Khomutskyi <bkhomuts@redhat.com>
2020-08-11 09:29:50 +00:00
Haibo Lin
b6605827b3 buildinstall: Improve error reporting when lorax fails
Logs of DepsolveError will be printed in the main log.

Fixes: https://pagure.io/pungi/issue/1399
JIRA: RHELCMP-955
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-06-17 13:17:22 +08:00
Haibo Lin
d4efe17328 Support --skip-branding option in lorax
JIRA: RHELCMP-572
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-05-14 11:49:40 +08:00
Jan Kaluza
5395af416c Allow reusing old buildinstall phase results.
New `buildinstall.metadata` file is created once the buildinstall
phase is done. This file contains:

- list of lorax command line arguments.
- list of RPMs installed in the buildinstall buildroot.
- list of RPMs installed in the resulting boot.iso.

This file is checked in the next compose run to find out if
the result of buildinstall phase from the previous compose
can be reused. Following is checked:

- lorax commandline arguments are the same (except of expected
  differences).
- The NVRAs of RPMs in the runroot_tag are the same as the ones
  installed in the old buildinstall buildroot.
- The NVRAs of RPMs installed in the boot.iso are the same as
  the ones in package sets in the current compose.

By its implementation, this reuse strategy is used only if
pungi_buildinstall Koji plugin is used.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>

Add tests for buildinstall reuse and buildinstall_allow_reuse option.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-04-17 11:37:13 +00:00
Jan Kaluza
145c3adbef Move buildinstall results to final directories if using pungi-buildinstall Koji plugin.
If Koji pungi-buildinstall is used, then the buildinstall results are
stored in the `output_dir` dir, but in "results" and "logs" subdirectories.
We need to move them to final_output_dir.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-03-03 10:35:25 +01:00
Haibo Lin
c0193c9fca Fix flake8 complaints - E501
E501 line too long (92 > 88 characters)
E501 line too long (103 > 88 characters)
...

JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 14:36:46 +08:00
Haibo Lin
3eddcfccd8 Fix flake8 complaints - F401
F401 'dnf' imported but unused
F401 'imp' imported but unused
F401 'os' imported but unused
F401 'subprocess' imported but unused
F401 'sys' imported but unused
F401 'yum' imported but unused

JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 11:48:31 +08:00
Haibo Lin
38142d30ba Format tests with black
JIRA: COMPOSE-4086
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-05 17:29:15 +08:00
Jan Kaluza
3cde5c3a87 Add support for new Pungi Buildinstall Koji plugin.
We would like to start generating the buildinstall phase using the safer
Koji Pungi Buildinstall plugin and stop the direct use of Runroot plugin.

The plugin so far exists only as PR for Koji:
https://pagure.io/koji/pull-request/1939

This commit adds support for this plugin when `lorax_use_koji_plugin`
is set to `True`.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-01-23 16:09:15 +01:00
Haibo Lin
6eb6511aa6 Make --task-id mandatory in get_runroot_cmd
JIRA: COMPOSE-4027
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-20 12:56:01 +08:00
Haibo Lin
0ed70fc8b6 tests: drop sys.path modification
It's unnecessary to add the path manually.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-12-10 10:53:38 +08:00
Haibo Lin
6afbe6d20a buildinstall: Log message for boot config change
JIRA: COMPOSE-3945
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-21 12:16:05 +08:00
Lubomír Sedlář
c346492df4 buildinstall: Allow customizing dracut arguments
JIRA: COMPOSE-3853
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-09 15:42:10 +02:00
Lubomír Sedlář
908a6a759d Use custom assertion for checking content of files
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 15:25:44 +02:00
Lubomír Sedlář
2f54745715 Remove usage of unittest2 on Python 3
It was needed to provide assertItemsEqual method. Starting with Python
3.2, there's assertCountEqual that does the same thing. Six provides a
helper that will dispatch to the existing method. With this change,
unittest 2 is only needed on Python 2.6 to backport the method.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 15:25:18 +02:00
Lubomír Sedlář
43fb06e00f Remove shebangs from test files
They should be executed with a runner like nose, not directly.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 15:24:17 +02:00
Lubomír Sedlář
26ddd46acb ostree: Remove arch_repo path usage
Pass all pkgset repos as input to the task.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
ab9122be2a buildinstall: Remove arch_repo usage
Simply use all existing package set repos as input for the runroot
task. The command line gets a bit long, but the actual behaviour should
remain the same.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Haibo Lin
3811c0a176 config: Deprecate bootable option
JIRA: COMPOSE-2635
Fixes: https://pagure.io/pungi/issue/976
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-07-22 16:51:38 +08:00
Lubomír Sedlář
bf1b3e8421 Remove remaining mentions of runroot option
This also cleans up the runroot method detection code to not rely on the
now removed option.

JIRA: COMPOSE-2634
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-02 09:43:13 +02:00
Haibo Lin
ce066707c1 config: Deprecate release_is_layered option
Fixes: https://pagure.io/pungi/issue/977
Merges: https://pagure.io/pungi/pull-request/1204
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-06-06 13:45:49 +02:00
Lubomír Sedlář
551f52922f runroot: Remove useless argument output_path
The same logic is handled by `chown_paths`, which does the same thing
but supports multiple paths.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-22 15:10:32 +02:00
Lubomír Sedlář
1951b0a521 buildinstall: Change owner of lorax logs
It's created in runroot, the owner and permissions should be updated so
that anyone can read it and user running compose can delete the files.

JIRA: COMPOSE-3545
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-22 15:10:32 +02:00
Lubomír Sedlář
f2bbf35429 kojiwrapper: Allow changing mode of multiple files
The directory with logs should have updated owner and permission as
well as the actual output. This patch lays foundation for that by
allowing multiple paths to be specified.

JIRA: COMPOSE-3545
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-22 15:10:32 +02:00
Lubomír Sedlář
2dd30008ae buildinstall: Copy files in thread
Instead of running the copy from the main script explicitly, make it
part of the thread.

This should make things very slightly faster, and the code is much
simpler.

Fixes: https://pagure.io/pungi/issue/959
JIRA: COMPOSE-2604
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-13 11:04:37 +02:00
Lubomír Sedlář
358fdd50ce buildinstall: Allow overwriting version for lorax
Sometimes the release version can be more specific than what should be
exposed to users of the boot iso.

JIRA: COMPOSE-3295
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-22 12:47:16 +01:00
Jan Kaluza
75bb48a882 Create new Runroot class and use it everywhere where runroot task is executed.
This adds new `Runroot` class and new `runroot_method` option which makes
it possible to choose between two currently available runroot methods:

- Local
- Koji

The main goal of this commit is to make it possible to add new runroot
methods in the future and this is the first step in that direction.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2019-03-21 14:15:58 +01:00
Lubomír Sedlář
8065239e04 buildinstall: Pick correct config with rootfs_size
If there are multiple matching configuration blocks, we should take the
value that is defined in the last one, but only if it actually is there.
That's how it works for other options.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-01 14:24:20 +01:00
Lubomír Sedlář
ba260c24e8 buildinstall: Expose lorax's --rootfs-size argument
We already make this possible for the ostree installer, but it was
missing from the traditional one. The default behaviour is to let lorax
decide, but if user knows better, they can overwrite in configuration.

JIRA: COMPOSE-3188
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-01-07 11:00:43 +01:00
Lubomír Sedlář
5926858b58 util: Remove escaping spaces from volume ID
This was not configurable for users, and the default was always used,
which meant no escaping. Might as well just remove the dead code.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-03 07:57:09 +02:00
Lubomír Sedlář
23c454ff67 buildinstall: Make output world readable
This makes it possible to run a compose as non-root user, plus removes
the need for workarounds to publish the results directly.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-17 10:12:52 +02:00
Lubomír Sedlář
e3de4dcccf Report failed failable deliverables as errors
It does not abort the compose, but it's still an error. It should be
marked as such in the log. This will allow easier searching for the
failures.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-17 09:31:50 +02:00
Lubomír Sedlář
ea0964eeef buildinstall: Use metadata if skipped
If the phase is skipped, it could mean that we are doing a debug run and
we don't want to mess up the .treefile by missing arch specific images.
The other alternative is that the phase was really skipped, in which
case there will be no files generated and we already handle that fine.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-23 10:42:33 +02:00
Lubomír Sedlář
15ccd309fa Remove comps from arch repo
The comps could potentially be different in different variants, so
instead we can create the comps repo for every variant separately and
use two repos instead of one (packages in one repository, comps in
another one).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-12 13:52:36 +02:00
Lubomír Sedlář
d9e2101b08 buildinstall: Add extra repos
A new configuration option is added that allows users to point lorax to
extra repositories. This can be handy if some tools to create the
bootable image are not part of the product itself.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
JIRA: COMPOSE-2253
2018-04-05 07:56:20 +02:00
Lubomír Sedlář
95bb147015 buildinstall: Add option to disable it
Fixes: https://pagure.io/pungi/issue/854
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-13 15:54:52 +01:00
Ondrej Nosek
59c162d46f Remove strace from buildinstall runroot
The package is not really needed.

Relates: https://pagure.io/pungi/issue/799

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2017-12-11 17:36:28 +01:00
Jan Kaluza
953fb4c54c buildinstall: Allow using external dire for runroot task
A new `buildinstall_topdir` option allows using buildinstall even when
the compose is created on a different volume that Koji is using.

The files are created in this external directory and then copies into
the usual location.

Merges: https://pagure.io/pungi/pull-request/807
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2017-11-30 10:55:52 +01:00
Lubomír Sedlář
ed22e07ef9 Port to Python 3
This should make all tests pass on both Python 2 and Python 3.

Unittest2 is required on Py 2.6 and Py 3.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-24 15:15:31 +02:00
Lubomír Sedlář
be39dc3caf buildinstall: Expose template arguments for lorax
This would be useful for modularity. The templates can be added now and
variables set via the existing `lorax_option`.

It's not possible to use custom templates not shipped with lorax, as
passing the path to a random directory is a little bit more tricky.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-23 16:08:09 +02:00
Lubomír Sedlář
fcbc3ed4ae buildinstall: No copy if task fails
Even a failed lorax task can leave behind some in-progress images. Pungi
needs to copy the files into compose/ subdirectory only if the task
finished successfully.

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1485021
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-28 13:43:32 +02:00
Lubomír Sedlář
05a666fb3b buildinstall: Move tweaking configs into a function
This will allow us to more easily test the code and also reuse it in
other places.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-21 07:55:19 +01:00
Lubomír Sedlář
4ea1916a87 Enable customizing runroot task weight
For different cases where runroot is used it's now possible to set
custom weight. The usecase for this is to avoid one builder taking too
many tasks. Especially buildinstall is quite resource intensive, so one
builder taking multiple tasks at the same time leads to very slow
compose time.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-23 12:44:27 +01:00
Qixiang Wan
6fbf1e8f59 Create temporary dirs under compose's workdir
Add compose.mkdtemp which creates and returns a unique temporary
directory under <compose_topdir>/work/{global,<arch>}/tmp[-<variant>]/

Change tempfile.mkdtemp in code to compose.mkdtemp, so temporary
dirs are created under the compose's workdir, this makes it more
easier to maintain the temporary dirs/files especially when compose
fails with amount of temporary files left there.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-01-11 15:25:47 +08:00
Qixiang Wan
80fa723b1d Include phase name in log for some phases
Phases createiso, liveimages, image_build, ostree_installer and osbs are
done in parallel, logs from these phases are mixed and and it's not
obvious which log message belongs to which phase. This change adds phase
name in log message for these phases.

The new mixin 'PhaseLoggerMixin' is added to extend a Pungi phase with a
logging logger which copy handlers from compose's logger but with
formatter changed.

Fixes: #58
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-11-23 16:47:22 +08:00
Dennis Gilmore
d4a78a1553 Merge #459 Fix building images and live media for addons 2016-11-15 21:00:50 +00:00