Commit Graph

874 Commits

Author SHA1 Message Date
Haibo Lin
edb4517e80 Add Dockerfile for building testing image
There are two images because it's hard to install both Python 2
and Python 3 packages (e.g. libcomps) in latest fedora release.

JIRA: RHELCMP-4580
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-03-15 18:13:25 +08:00
Lubomír Sedlář
535034ef91 image_container: Fix incorrect arch processing
OSBS will reject no scratch builds with arch_override.

When the option is not specified in Pungi, it would do `"".split(" ")`
to get list of arches, which returns a list with empty string instead of
an empty list.

With this fixed, it might be possible to have multiple images match the
spec (unless arch is used in the filter). To fix that, we can replace
arch with $basearch variable.

JIRA: RHELCMP-3824
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-03-08 11:57:48 +01:00
Haibo Lin
2769232b72 runroot: Adjust permissions always
Previously commands to adjust permissions do not run when main
command failed and then files can't be cleaned up due to
Permission Denied problem.

JIRA: RHELCMP-4253
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-03-04 16:08:06 +08:00
Haibo Lin
b217470464 Format code
Code didn't get well formatted when jenkins unusable.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-03-02 18:31:45 +08:00
Lubomír Sedlář
735bfaa0d6 pkgset: Fix meaning of retries
The name brings a different expectation than how it actually worked.
This patch makes the code work similarly to the expectation.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-23 16:01:41 +01:00
Lubomír Sedlář
477dcf37d9 Store extended traceback for gather errors
When a gathering thread raises an exception, it gets forwarded to the
main thread and re-raised there. However, during this transition it
loses details about exact location of the problem.

This patch creates an extended traceback in the worker, which should
make it easier to track the problem down later.

JIRA: RHELCMP-4259
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-17 11:11:26 +01:00
Lubomír Sedlář
64897d7d48 pkgset: Add ability to wait for signed packages
If packages are appearing quickly in Koji, and signing them is triggered
by automation, there may be a delay between the package being signed and
compose running. In such case it may be preferable to wait for the
signed copy rather than fail the compose.

JIRA: RHELCMP-3932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-11 15:31:14 +01:00
Lubomír Sedlář
40133074b3 Add image-container phase
This phase runs after image-build and osbuild and can embed an image
into a container.

JIRA: RHELCMP-3820
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-09 14:47:57 +01:00
Lubomír Sedlář
36373479db Move container metadata into compose object
Rather than tracking this directly in OSBS phase, move this into Compose
object, which will allow access to this from multiple phases.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-08 13:23:26 +01:00
Haibo Lin
d4ee42ec23 pkgset: Check tag inheritance change before reuse
JIRA: RHELCMP-2453
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-02-01 09:22:42 +08:00
Haibo Lin
f518c1bb7c Stop copying .git directory with module defaults
JIRA: RHELCMP-3016
Fixes: https://pagure.io/pungi/issue/1464

Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-01-14 10:36:38 +08:00
Lubomír Sedlář
98ddc74c16 osbuild: Fix not failing on failable tasks
The task can only fail as a whole, thus it only makes sense to set all
architectures (`*`) as failable. The correct value needs to be checked
though.

JIRA: RHELCMP-3412
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-12-02 14:30:20 +01:00
Lubomír Sedlář
4a048d4a85 kojiwrapper: Use gssapi_login
The krb_login method is deprecated and will be removed in 1.22

JIRA: RHELCMP-3383
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-12-02 14:21:31 +01:00
Christian Kellner
6998ffe694 osbuild: use task result to get build info
Instead of parsing the log file to get the NVR and then in turn
use that to get to the build info use the structured return value
from the koji task. The return value of the osbuild plugin is:
    result = {
        "composer": {
            "server": <COMPOSER_URL>,
            "id": <COMPOSE_ID>
        },
        "koji": {
            "build": <BUILD_ID>
        }
    }
This means we have direct access to the koji build id, which was
returned by composer to the plugin via its status API. Using that
removes the need to parse the log file.

Adapt the test accordingly.

Merges: https://pagure.io/pungi/pull-request/1475
Signed-off-by: Christian Kellner <christian@kellner.me>
2020-11-30 09:57:48 +01:00
Lubomír Sedlář
e7af6d2ac2 osbuild: Only send release when not empty
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-11-25 15:37:49 +01:00
Lubomír Sedlář
27bab19a5e Fix config validation for osbuild
Tests are now added for it as well.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-11-25 14:14:18 +01:00
Haibo Lin
4c88e7dc0e Use xorrisofs for creating ISOs when needed
A new configuration *createiso_use_xorrisofs* is added to determine
which tool to use for creating ISOs.

By default, createiso_use_xorrisofs = False and genisoimage will be used.
When set to True, xorrisofs will be used.

JIRA: RHELCMP-2875
Fixes: https://pagure.io/pungi/issue/1130

Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-11-20 09:47:49 +08:00
Jan Kaluza
c27e21ccf8 Add --respin-of argument.
It is used to defined the relation between original compose
and respun compose with a hotfix in CTS.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-11-19 12:26:38 +00:00
Jan Kaluza
bb8cd030ec Try reuse old gather_phase even if pkgset_koji_builds changed.
The pkgset_koji_builds influences the gather phase, but the
change itself is not a reason to not reuse old gather phase. if
new pkgset_koji_builds value leads to significant change in input
package set, we will find that later in this function when comparing
old and new package set.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-11-10 12:52:02 +01:00
Jan Kaluza
94bc5e286d Do not use shlex_quote in get_pungi_buildinstall_cmd and get_pungi_ostree_cmd.
These methods return command as list which is never serialized to str and
never executed using bash. It is instead passed directly to
`kobo.shortcuts.run`.

The `shlex_quote` usage here actually breaks the code, because it adds
quotes there which are needed only if this command would be serialized
to string and passed to bash. But this never happens. As a result,
the arguments passed to `kobo.shortcuts.run` contain those extra
quotes.

In this commit the shlex_quote is removed completely from this
part of code.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-11-10 07:45:05 +00:00
Lubomír Sedlář
a45f4969f3 Add phase for building images with osbuild
This is similar to image-build in terms of what it does, and somewhat
similar to OSBS phase in how it's implemented.

The phase reads configuration, submits the build via XMLRPC call and
waits for the task to finish. Then it downloads the built image and
includes it in the compose metadata.

JIRA: RHELCMP-315
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-11-04 13:04:43 +01:00
Lubomír Sedlář
f27b120cfc tests: Fix mock usage for Python 2
The `args` attribute doesn't work, let's instead access the arguments
via index.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-09-25 12:41:05 +02:00
Jan Kaluza
4623536b24 Fix wrong condition when reusing old gather phase results.
When list is used in gather_lookaside_repos, the Pungi currently
fails with an exception. This is caused by inverted condition
in the code which tries to filter-out the lookaside repos
generated during the Pungin execution pointing to different
compose variants.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-09-23 07:02:11 +00:00
Jiri Konecny
fdb2449c0e
Adapt tests to new patch-iso --work-dir parameter
Signed-off-by: Jiri Konecny <jkonecny@redhat.com>
2020-09-15 16:07:18 +02:00
Haibo Lin
b3a55fd863 gitwrapper: Re-run git init before do full clone
Sometimes full clone failed with error fatal: Not a git repository
and we found that .git/refs/ dir is missing after shallow clone failed.

JIRA: RHELCMP-724
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-09-10 10:09:46 +08:00
Lubomír Sedlář
a6a96e40db Preserve environment when running koji commands
JIRA: RHELBLD-2479
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-08-25 09:07:28 +02:00
Lubomír Sedlář
e628bb91ec Fix formatting
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-08-20 15:15:13 +02:00
Jiri Konecny
40bbb4325a Adapt tests to a new get_graft_points() argument
Signed-off-by: Jiri Konecny <jkonecny@redhat.com>
2020-08-20 15:15:13 +02:00
Haibo Lin
1e1c8533ac tests: Fix tests for old version of git
GitSCMTestCaseReal failed with error 'Unknown option: -C' when building
rpm for el7.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-08-18 12:38:20 +08:00
Jan Kaluza
2657a12c96 Allow setting CTS parent_compose_ids using --parent-compose-id option.
This is needed to track dependencies between composes in the Compose
Tracking Service.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-08-13 13:22:43 +02:00
Bohdan Khomutskyi
7a6d8303dc Replace -c parameter with --config
Also, reformat the invocation of Lorax, since
Lorax does not accept -c=%s, only -c %s.

Jira: RHELCMP-713
Signed-off-by: Bohdan Khomutskyi <bkhomuts@redhat.com>
2020-08-13 11:16:43 +02:00
Bohdan Khomutskyi
b899126b7e Allow squashfs-only and configuration_file in lorax_options
The change allows for setting the parameters as described below to Lorax.
Lorax, a program called during the buildInstall phase, creates the SquashFS
during the buildInstall phase.
The Squash filesystem is present both on the DVD and the BOOT.ISO.

squashfs_only --- (str) passes --squashfs-only option.
configuration_file --- (str or scm_dict) passes -c option to Lorax.

The final goal of this change is to allow for optimization of
the installation medium size.

This pull request is related to the Fedora change proposal, which is available
at this location:
https://fedoraproject.org/wiki/Category:Changes/OptimizeSquashFS
See the change proposal for more information about the benefits of higher
compression ratio.

Jira: RHELCMP-693
Signed-off-by: Bohdan Khomutskyi <bkhomuts@redhat.com>
2020-08-11 09:29:50 +00:00
Haibo Lin
3c72755814 Use pytest instead of nosetests
Nose has been in maintenance mode for the past several years
and pytest is a good replacement.

Other changes:
- Replace deprecated assertRegexpMatches with assertRegex
- Replace deprecated assertRaisesRegexp with assertRaisesRegex
- Replace deprecated SafeConfigParser with ConfigParser
- Force reinstall pytest and mock in tox virtualenv. This is because
  the globally installed packages may not work as expected(occured in
  jenkins job).

JIRA: RHELCMP-1619
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-07-29 14:57:16 +08:00
Haibo Lin
7e6bed9713 Retry buildinstall tasks on losetup error
JIRA: RHELCMP-1394
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-07-21 17:15:25 +08:00
Haibo Lin
f7167fa3b6 Allow including scratch module builds
JIRA: RHELCMP-439
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-07-17 09:08:36 +08:00
Lubomír Sedlář
7d00942d13 Allow test calls in any order
There are threads involved and the order is undefined. We need to accept
any order.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-07-08 08:26:59 +02:00
Haibo Lin
0ab6f48de3 Create arch repo when reusing failed
Reusing old arch repo may fail for reasons such as arch not
available in old compose or unexpected error when copying data
from old compose.

JIRA: RHELCMP-994
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-07-07 17:59:12 +08:00
Ondrej Nosek
d9f111edae Remove buffering when running koji commands
If the compose is aborted while koji tasks are running, we can be
left with empty log files. That complicates debugging.

JIRA: RHELCMP-1218

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2020-07-02 09:45:46 +02:00
Haibo Lin
b6605827b3 buildinstall: Improve error reporting when lorax fails
Logs of DepsolveError will be printed in the main log.

Fixes: https://pagure.io/pungi/issue/1399
JIRA: RHELCMP-955
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-06-17 13:17:22 +08:00
Jan Kaluza
e35c250700 Move test for unsigned packages with pkgset_koji_scratch_tasks to PkgsetPhase class.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-06-11 12:05:47 +00:00
Jan Kaluza
4a15d1351a Allow building compose with scratch builds defined by pkgset_koji_scratch_tasks.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-06-11 12:05:47 +00:00
Lubomír Sedlář
3bb1e3df11 createrepo: Add extra modulemd files to the repo
This is a workaround for modularity design issues and DNF bugs. If there
were gaps in contexts, DNF has trouble handling the upgrades. Thus we
may need to add module metadata for older versions of previously
released module streams and add the missing contexts.

JIRA: RHELCMP-982
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-06-10 09:50:44 +02:00
Haibo Lin
f5bfd509ab notification: Add compose_path into the messages
It's needed by the notification script added in RHELCMP-401.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-06-09 06:40:26 +00:00
Haibo Lin
7c2743fb50 Include the output of getisoimage in the error message
JIRA: RHELCMP-720
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-06-01 13:29:55 +08:00
Lubomír Sedlář
d6caf0785b tests: Patch time in CTS related tests
The value must be fixed to match the expected JSON.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-05-29 10:24:17 +02:00
Jan Kaluza
b8c3ca1abe Allow using Pungi Koji plugin for ostree phases.
This commits changes `ostree` and `ostree_installer` phases
so they can run with Koji Pungi plugin instead of the plain runroot.

It is similar to `buildinstall` phase running with Koji plugin.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-05-26 13:23:58 +00:00
Jan Kaluza
f1eea0b5a6 Allow getting the compose id from CTS (Compose Tracking Service).
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-05-26 13:18:38 +00:00
Lubomír Sedlář
59e2aa9607 Fix flake8 issues
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-05-26 11:49:50 +02:00
Haibo Lin
d4efe17328 Support --skip-branding option in lorax
JIRA: RHELCMP-572
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-05-14 11:49:40 +08:00
Jan Kaluza
5395af416c Allow reusing old buildinstall phase results.
New `buildinstall.metadata` file is created once the buildinstall
phase is done. This file contains:

- list of lorax command line arguments.
- list of RPMs installed in the buildinstall buildroot.
- list of RPMs installed in the resulting boot.iso.

This file is checked in the next compose run to find out if
the result of buildinstall phase from the previous compose
can be reused. Following is checked:

- lorax commandline arguments are the same (except of expected
  differences).
- The NVRAs of RPMs in the runroot_tag are the same as the ones
  installed in the old buildinstall buildroot.
- The NVRAs of RPMs installed in the boot.iso are the same as
  the ones in package sets in the current compose.

By its implementation, this reuse strategy is used only if
pungi_buildinstall Koji plugin is used.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>

Add tests for buildinstall reuse and buildinstall_allow_reuse option.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-04-17 11:37:13 +00:00
Haibo Lin
477b43d4e9 Split repoclosure into separate phase
Move repoclosure out from test phase into its own phase and
run parallel with image building phases(osbs, imagebuild, ...)
to speed things up.

JIRA: RHELCMP-8
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-04-14 09:25:57 +08:00
Jan Kaluza
fdfaae8b71 Allow gather phase reuse on product_id change.
I've analysed multiple nigthly composes and often the only difference
in the configuration between two nightly composes is different
`product_id` commit hash.

The `product_id` is used in later `createrepo` phase and does not
influence the gather phase at all. I therefore think it can be
whitelisted in gather phase reuse code.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-04-06 08:23:19 +00:00
Haibo Lin
65251d983a Reuse arch pkgset repos
JIRA: COMPOSE-4217
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-03-31 13:32:12 +08:00
Haibo Lin
63ec1adc22 Get non-rpm build to pungi's extra_files with inheritance
JIRA: COMPOSE-4214
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-03-20 13:19:57 +08:00
Haibo Lin
3543f8fb3e pkgset: Reuse pkgset repos
JIRA: COMPOSE-4158
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-03-13 17:29:53 +08:00
Jan Kaluza
169fa5b453 Allow reusing gather phase results.
- Get also requires/provides of RPMs in package set.
- Store the results of gather phase as pickle file.
- Reuse old gather phase results in case Pungi configuration
  did not change, the "names" of RPMs in global package set
  did not change and their requires/provides did not change.
- Add `gather_allow_reuse` option to enable this feature.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>

Add gather_allow_reuse, add more tests and better handling of gather_lookaside_repos.
2020-03-13 10:17:59 +01:00
Jan Kaluza
afcb3e969b Add Compose.old_compose_path and use it when searching for files in old compose.
The current code calls `find_old_compose` followed by multiple `os.path.*`
calls to find out if particular file exists in the old compose. This
duplicates code a lot and makes it harder to read.

In this commit, new `Compose.old_compose_path` is introduced and
used instead of direct calls of `find_old_compose`.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-03-09 12:47:23 +01:00
Jan Kaluza
fc3b5063ca Do not skip symlinks when searching for old compose.
ODCS creates symlinks to real directories containing the composes.
The directory structure is similar to following one:

- `./nightly/Fedora-Rawhide-20200304.n.0` -> `../odcs-3`
- `./nightly/Fedora-Rawhide-20200305.n.0` -> `../odcs-4`
- `./nightly/latest-Fedora-Rawhide` -> `../odcs-5`

The current Pungi code to search for old composes skips symlinks
and therefore old ODCS composes are not found.

This commit removes this check and therefore symlinks are allowed
when searching for old compose.

I think this check existed to prevent using `latest-*` symlink as
source for the compose. But this is not possible, because the
code checks that the old compose directory name has certain pattern
constructed from release_short, release_version, ... The `latest-*`
symlink definitely does not match this pattern.

I also executed test compose and it worked as expected.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-03-06 09:03:10 +00:00
Jan Kaluza
145c3adbef Move buildinstall results to final directories if using pungi-buildinstall Koji plugin.
If Koji pungi-buildinstall is used, then the buildinstall results are
stored in the `output_dir` dir, but in "results" and "logs" subdirectories.
We need to move them to final_output_dir.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-03-03 10:35:25 +01:00
Haibo Lin
ff269a8675 gather: Make additional_packages accept debuginfo packages
Now debuginfo packages defined in additional_packages option
could be handled correctly.

JIRA: COMPOSE-4085
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-12 16:38:08 +08:00
Haibo Lin
9b12be7300 Fix formatting issues
Code should be formatted via black.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 16:16:20 +08:00
Haibo Lin
65aa8fde2f Fix other flake8 complaints
E231 missing whitespace after ','
E265 block comment should start with '# '
E266 too many leading '#' for block comment
E302 expected 2 blank lines, found 1
E501 line too long (115 > 88 characters)
E713 test for membership should be 'not in'
E722 do not use bare 'except'
F812 list comprehension redefines 'g' from line 1499
F821 undefined name 'cmp'
F841 local variable 'ex' is assigned to but never used

JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 16:14:09 +08:00
Haibo Lin
c0193c9fca Fix flake8 complaints - E501
E501 line too long (92 > 88 characters)
E501 line too long (103 > 88 characters)
...

JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 14:36:46 +08:00
Haibo Lin
3eddcfccd8 Fix flake8 complaints - F401
F401 'dnf' imported but unused
F401 'imp' imported but unused
F401 'os' imported but unused
F401 'subprocess' imported but unused
F401 'sys' imported but unused
F401 'yum' imported but unused

JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 11:48:31 +08:00
Haibo Lin
38142d30ba Format tests with black
JIRA: COMPOSE-4086
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-05 17:29:15 +08:00
Jan Kaluza
ef33d00f5b Add --schema-override to pungi-config-validate script
Some composes might need extra validation to ensure they are following
certain strict rules - for example containing only signed packages or
packages only from particular Koji tag.

There is currently no way how to check that Pungi configuration fulfills
these extra requirements.

This commit adds new `--schema-override` option to
`pungi-config-validate` script which allows caller to specify path to
JSON schema overriding the default JSON schema and therefore limitting
it further.

For exmaple, to limit the `pkgset_source` to `koji`, one can use
following JSON schema override:

```
{
    "properties": {
        "pkgset_source": {
            "enum": ["koji"]
        }
    }
}
```

It is possible to use `--schema-override` multiple times to apply
multiple schema overrides.

Merges: https://pagure.io/pungi/pull-request/1341
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-02-05 10:03:30 +01:00
Lubomír Sedlář
46ea0743a9 iso: Clean up cache for guestmount
When partial cleanup messes up the guestfs cache, the call to guestmount
will fail. To fix that, let's check if there is a problem first and
clean up everything if needed.

Relates: https://bugzilla.redhat.com/show_bug.cgi?id=1771976
JIRA: COMPOSE-3932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-30 10:35:42 +01:00
Lubomír Sedlář
4c6396f491 Remove deprecated warn() call
Instead warning() should be called.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-29 15:34:00 +01:00
Jan Kaluza
3cde5c3a87 Add support for new Pungi Buildinstall Koji plugin.
We would like to start generating the buildinstall phase using the safer
Koji Pungi Buildinstall plugin and stop the direct use of Runroot plugin.

The plugin so far exists only as PR for Koji:
https://pagure.io/koji/pull-request/1939

This commit adds support for this plugin when `lorax_use_koji_plugin`
is set to `True`.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-01-23 16:09:15 +01:00
Haibo Lin
6eb6511aa6 Make --task-id mandatory in get_runroot_cmd
JIRA: COMPOSE-4027
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-20 12:56:01 +08:00
Haibo Lin
bce57c2e66 Wait on runroot tasks with wait-task command
JIRA: COMPOSE-4027
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-19 17:56:31 +08:00
Haibo Lin
52f82ccc6e pkgset: Ignore deleted module builds
JIRA: COMPOSE-4058
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-14 16:23:55 +08:00
Lubomír Sedlář
e7a58ccd07 Add tests for new exclude options
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-13 10:54:11 +01:00
Jan Kaluza
fcf1442f71 Support generating ISOs when using link_type="symlink".
When `link_type = "symlink"` is used, the packages are in fact symlinks
to /mnt/koji. When graft points file is generated, the paths in this graft
points file point to symlinks and therefore symlinks are copied into the
generated ISO file instead of real files.

In this commit, the code to generate the graft points file is changed
so it resolves the symlink to real file stored on /mnt/koji. To make
this code safer, it does such resolving only in case the symlink points
outside of `compose.paths.compose.topdir()`. Therefore you can still
generate ISO file with symlink pointing to file stored within the ISO
file itself, although this is not done currently afaik.

The main reason for this is to be able to generate ISO files even
without hardlinks (which would need read-write access on /mnt/koji)
and without copying all the packages from /mnt/koji to local storage.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-01-06 12:30:13 +00:00
Haibo Lin
3f111b559f Check dependency of --just-phase
JIRA: COMPOSE-4020
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-06 08:52:47 +00:00
Haibo Lin
794d151bef Remove productimg phase
JIRA: COMPOSE-4004
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-03 11:42:38 +00:00
Jan Kaluza
817acdbbac Allow setting runroot_method per Pungi phase.
The `runroot_method` now accepts `dict` value with phase name as a key
and runroot method as a value. For backward compatibility, the `str`
value is still supported.

The new `global_runroot_method` option has been added which defines
the runroot method in case it is not set in `dict` in the `runroot_method`.

This commit allows running `createiso` phase locally while keeping the other
phases in Koji.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-01-02 14:10:20 +00:00
Haibo Lin
b415e31f9d ostree: Fix arg passed to scm.get_dir_from_scm
A compose object should be passed instead of logger.

JIRA: COMPOSE-4050
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-02 13:13:14 +00:00
Haibo Lin
12828849d6 gather: Improve logging for gathering
Adding arch and variant to log message to make it clearer.

JIRA: COMPOSE-4009
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-12-16 08:56:22 +00:00
Haibo Lin
0ed70fc8b6 tests: drop sys.path modification
It's unnecessary to add the path manually.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-12-10 10:53:38 +08:00
Haibo Lin
3cf16eb42d cleanup: refactor scripts via entry points
Fixes: https://pagure.io/pungi/issue/1045
JIRA: COMPOSE-2946
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-12-10 10:53:31 +08:00
Lubomír Sedlář
aefe9b186d repoclosure: Parse all fus logs
Originally the list of solvables for fus was growing with each iteration
and nothing was ever removed. That later changed so that fus iterations
are only done on newly added stuff. It's great for performance, but
means that the last log is not a superset of all others.

To get all dependency problems we need to look into all log files, not
just the last one.

JIRA: COMPOSE-3964
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-12-02 11:08:55 +01:00
Lubomír Sedlář
51b1144b70 runroot: Log different commands to different files
JIRA: COMPOSE-3980
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-28 15:12:05 +01:00
Haibo Lin
6afbe6d20a buildinstall: Log message for boot config change
JIRA: COMPOSE-3945
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-21 12:16:05 +08:00
Haibo Lin
0c040e0a69 livemedia: Remove title option
Fixes: https://pagure.io/pungi/issue/1293
JIRA: COMPOSE-3947
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-20 15:33:28 +08:00
Haibo Lin
242100eb72 Make sure repoclosure cache cleaned when running as root
When running repoclosure as root user, it will use other dir instead of
the one returned by getCacheDir().

For yum, with --tempcache option could let the cache dir returned by
getCacheDir() always be used.

For dnf, there's no such an option and we have to handle it specially.

JIRA: COMPOSE-3922
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-20 14:13:19 +08:00
Haibo Lin
e0dd20dffe Prefix repoclosure cache dir with compose id
JIRA: COMPOSE-3922
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-20 13:51:13 +08:00
Lubomír Sedlář
c87d299a20 hybrid: Download remote files when getting platform
When probing lookasides for platform definition, we need to make sure it
works for repos specified as HTTP urls. Createrepo doesn't seem to
automatically download the repodata, so we have to help it.

JIRA: COMPOSE-3958
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-19 08:51:18 +01:00
Lubomír Sedlář
4473b05f10 gather: Use fresh cache for each sequence of fus runs
Each depsolved tree will be using its own cache for fus. This should
still allow for faster loading of metadata after first iteration, but
should prevent errors from using cached files meant for another variant
or architecture. The cache is deleted after the last iteration.

JIRA: COMPOSE-3959
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-18 14:39:47 +01:00
Haibo Lin
722411dfcd tests: Add test for compose logger setup
PR#1267 brings in some uncovered code which introduced
an issue fixed by PR#1292.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-12 18:03:13 +08:00
Haibo Lin
2cbd75803a gather: Do not write metadata if gather phase skipped
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-07 16:19:19 +08:00
Lubomír Sedlář
cba3f1c88f Write global metadata about extra files
This is basically collecting all individual extra_files.json and putting
their content into a single location in
compose/metadata/extra_files.json. The file format is part of productmd
1.23.

JIRA: COMPOSE-3831
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-05 09:11:01 +01:00
Lubomír Sedlář
20c3614fb3 scm: Add backend for downloading archives from Koji
Tests and documentation included.

JIRA: COMPOSE-3816
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-05 08:51:26 +01:00
Lubomír Sedlář
39e8f6f710 Pass compose to SCM wrappers
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-05 08:51:26 +01:00
Haibo Lin
b05295c202 Remove misleading warning about size of the ISO
JIRA: COMPOSE-3885
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-04 14:22:59 +08:00
Haibo Lin
114df77c6b Retry watching koji tasks on server outage
Fixes: https://pagure.io/pungi/issue/1285
JIRA: COMPOSE-3896
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-01 15:57:48 +08:00
Haibo Lin
3901c227f0 Fix crash on unsigned packages
JIRA: COMPOSE-3869
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-10-21 17:02:59 +08:00
Owen W. Taylor
72bf795bd4 ostree-install: allow configuring additional depenencies for runroot
A lorax template used for the ostree-installer might need an additional
package dependency (e.g., flatpak to embed a flatpak repository) - add
a config key 'extra_runroot_pkgs' to the ostree installer configuration
to allow supplementing the set of packages installed into the runroot.

Signed-off-by: Owen W. Taylor <otaylor@fishsoup.net>
2019-10-10 15:13:38 -04:00
Lubomír Sedlář
c346492df4 buildinstall: Allow customizing dracut arguments
JIRA: COMPOSE-3853
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-09 15:42:10 +02:00
Lubomír Sedlář
908a6a759d Use custom assertion for checking content of files
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 15:25:44 +02:00