Commit Graph

1364 Commits

Author SHA1 Message Date
Haibo Lin 90187298f2 Prefix lookaside cache dir with compose id
Then it could be cleaned up leater.

JIRA: COMPOSE-3922
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-20 13:51:13 +08:00
Haibo Lin e0dd20dffe Prefix repoclosure cache dir with compose id
JIRA: COMPOSE-3922
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-20 13:51:13 +08:00
Lubomír Sedlář c87d299a20 hybrid: Download remote files when getting platform
When probing lookasides for platform definition, we need to make sure it
works for repos specified as HTTP urls. Createrepo doesn't seem to
automatically download the repodata, so we have to help it.

JIRA: COMPOSE-3958
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-19 08:51:18 +01:00
Lubomír Sedlář 4473b05f10 gather: Use fresh cache for each sequence of fus runs
Each depsolved tree will be using its own cache for fus. This should
still allow for faster loading of metadata after first iteration, but
should prevent errors from using cached files meant for another variant
or architecture. The cache is deleted after the last iteration.

JIRA: COMPOSE-3959
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-18 14:39:47 +01:00
Lubomír Sedlář d34b0d7900 pkgset: Fix running without any koji tags
All places in the code assume the option to not be required except for
this one line.

An obsolete comment is removed as well.

Relates: https://pagure.io/releng/failed-composes/issue/477
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-11 09:30:37 +01:00
Lubomír Sedlář 254d0cebff Get message from LogRecord with a method
Relates: https://pagure.io/releng/failed-composes/issue/469
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-11 08:46:01 +01:00
Haibo Lin 2cbd75803a gather: Do not write metadata if gather phase skipped
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-07 16:19:19 +08:00
Lubomír Sedlář cba3f1c88f Write global metadata about extra files
This is basically collecting all individual extra_files.json and putting
their content into a single location in
compose/metadata/extra_files.json. The file format is part of productmd
1.23.

JIRA: COMPOSE-3831
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-05 09:11:01 +01:00
Lubomír Sedlář 20c3614fb3 scm: Add backend for downloading archives from Koji
Tests and documentation included.

JIRA: COMPOSE-3816
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-05 08:51:26 +01:00
Lubomír Sedlář 39e8f6f710 Pass compose to SCM wrappers
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-11-05 08:51:26 +01:00
Haibo Lin b05295c202 Remove misleading warning about size of the ISO
JIRA: COMPOSE-3885
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-04 14:22:59 +08:00
Haibo Lin 114df77c6b Retry watching koji tasks on server outage
Fixes: https://pagure.io/pungi/issue/1285
JIRA: COMPOSE-3896
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-01 15:57:48 +08:00
Lubomír Sedlář 6771a21916 pkgset: Only reuse valid old repo
Instead of just checking that the repo directory exists, make sure the
repodata subdirectory is in there. If it's missing, then createrepo_c
has nothing to use anyway, and it may help avoid issues.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-22 15:32:13 +02:00
Haibo Lin 4f96164ec7 Clean up skipping phases
It makes no sense to only skip phases needed by other phase.

Fixes: https://pagure.io/pungi/issue/1274
JIRA: COMPOSE-3871
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-10-22 16:36:01 +08:00
Haibo Lin 3901c227f0 Fix crash on unsigned packages
JIRA: COMPOSE-3869
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-10-21 17:02:59 +08:00
Haibo Lin 2ff1f2fac3 gather: Make depsolving parallel
JIRA: COMPOSE-3827
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-10-18 14:20:32 +08:00
Owen W. Taylor 72bf795bd4 ostree-install: allow configuring additional depenencies for runroot
A lorax template used for the ostree-installer might need an additional
package dependency (e.g., flatpak to embed a flatpak repository) - add
a config key 'extra_runroot_pkgs' to the ostree installer configuration
to allow supplementing the set of packages installed into the runroot.

Signed-off-by: Owen W. Taylor <otaylor@fishsoup.net>
2019-10-10 15:13:38 -04:00
Lubomír Sedlář c346492df4 buildinstall: Allow customizing dracut arguments
JIRA: COMPOSE-3853
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-09 15:42:10 +02:00
Lubomír Sedlář 26962d94ca gather: Resolve dependencies of debug packages
Until now, the behaviour was that all debuginfo from a build would be
included if at least one package with the same arch was included.

This resulted in many debuginfo packages being included even though
their corresponding package was not present.

With this patch, we will always pull in debugsource, but foo-debuginfo
will only be included if foo is included for the same arch. As a
consequence, it is necessary to resolve dependencies of debuginfo
packages. There are cases where foo-debuginfo needs foo-debuginfo-common
for example.

This change means that DNF and YUM backends are no longer identical in
the output. The tests where this is demonstrated are duplicated and
their results are modified.

JIRA: COMPOSE-3823
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:46:21 +02:00
Lubomír Sedlář 4413b0df24 gather: Fix bare except block
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:46:21 +02:00
Lubomír Sedlář 89fcb79aca gather: Fix formatting
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:46:21 +02:00
Lubomír Sedlář 428f8297e1 gather: Simplify adding packages to result
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:46:21 +02:00
Lubomír Sedlář c7f3f38822 gather: Remove unused import
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:46:21 +02:00
Lubomír Sedlář 8ab7d9f7ba Move import of modulemd to a separate module
This should make it possible to only import the library only when it's
really needed.

DNF does not work with libmodulemd v2. If we import libmodulemd2 and
then dnf, the program will just hang forever. We only need DNF in
pungi-gather, where libmodulemd is not needed, and also where we do need
libmodulemd, we don't have DNF.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:40:31 +02:00
Lubomír Sedlář f822ee324a ostree: Run commands in universal_newlines mode
This will mean the output is returned as unicode (decoded as UTF-8).
Thus Kobo will not have to do any decoding. This should work around
possible errors with breaking multibyte unicode character sequences into
different chunks.

Relates: https://pagure.io/releng/failed-composes/issue/237
Relates: https://github.com/release-engineering/kobo/issues/119
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 09:30:53 +02:00
Lubomír Sedlář 21d45eb243 pkgset: Allow filtering modules from Koji tags
Add a configuration option to enable skipping some modules found in the
configured tag.

Fixes: https://pagure.io/pungi/issue/1260
JIRA: COMPOSE-3794
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-24 10:13:00 +02:00
Lubomír Sedlář 44e551317a pkgset: Initialize path_prefix to empty value
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-23 09:40:36 +02:00
Lubomír Sedlář 2b112d53f7 Allow loading overrides for module defaults
This patch adds a new config option. This is expected to be a name of
subdirectory in the repo with module defaults. If supplied, overrides
from that location are loaded every time defaults are loaded.

This raises the minimal required version of libmodulemd to 2.8.0

JIRA: COMPOSE-3828
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-23 09:23:39 +02:00
Haibo Lin 260df24859 Move pkgset skipping logs to separate file
There will be a new log file logs/global/excluding-arch.global.log

Fixes: https://pagure.io/pungi/issue/1251
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-09-20 16:02:41 +08:00
Lubomír Sedlář 150f5f0cb6 Fix crash when pkgset phase is skipped
JIRA: COMPOSE-3832
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-18 15:47:16 +02:00
Lubomír Sedlář 3e263f33f2 Resolve symlinks to images
Koji started to create relative symlinks to images created in various
tasks. We need to check for that and potentially link the actual file.

JIRA: COMPOSE-3822
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-11 15:36:28 +02:00
Lubomír Sedlář 2d99edc8d8 Fix getting platforms from lookaside
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-06 11:14:16 +02:00
Lubomír Sedlář dfd4ff6016 extra-files: Simplify iterating over variants
Finding variants with particular architecture is not simple. Swapping
the two loops makes it faster, and additionally it fixes to correctly
work with sources.

JIRA: COMPOSE-3787
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-04 15:45:15 +02:00
Lubomír Sedlář b8e41d9b1b createiso: Make media.repo sticky
This means the file will be included on all ISOs when splitting.

JIRA: COMPOSE-3787
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-04 15:45:15 +02:00
Lubomír Sedlář 7aa65f00c2 hybrid: Re-add getting platform from lookaside repos
This is needed when a compose does not include any modules itself, but
is using a lookaside with modules.

JIRA: COMPOSE-3720
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:40:46 +02:00
Lubomír Sedlář 1423105802 pkgset: Remove fast_merge method
This is not called anywhere anymore. Instead of merging these package
sets, they exist as separate entities.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář 465ecf229c Add test for materialized pkgset
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář 63a8b7b6c9 Remove package whitelist
This was a workaround to make some packages from the global repo
invisible for depsolving. This is now handled by packages being in
different repos. We can select which repos are enabled at which point.
This achieves the same result, but much faster.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář cec3efed51 paths: Remove arch_repo path
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář a426a83ed6 gather: Remove use of arch_repo
The repo was used to speed up creating lookaside repo from a variant.
This uses a similar approach as createrepo phase: selecting the last
available package set and using that data.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář 2d0ffb56ca ostree-installer: Remove usage of arch_repo
Pass all pkgset repos as input to the task.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář 26ddd46acb ostree: Remove arch_repo path usage
Pass all pkgset repos as input to the task.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář ab9122be2a buildinstall: Remove arch_repo usage
Simply use all existing package set repos as input for the runroot
task. The command line gets a bit long, but the actual behaviour should
remain the same.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář 3824eab15b createrepo: Remove arch_repo usage
There is no longer a single repo with all packages. This means that the
metadata has to be loaded from another location.

When taking packages from Koji, we can assume that the non-modular
package tag will be processed last. The repo for this tag will be used.
This has better chance of being useful than using a random module.

For repo sources, there is only one package set anyway, so this change
makes no difference.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář 054b91c798 Fix tests
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář 11779f6644 pkgset: Update processing file cache
The cache is now always saved when the repo is created on disk. The
loading procedure in Koji source is updated to look for cache for
correct tag.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář fe2df01e8b Remove pickling from source repos
The pickled pkgset was only ever used for debug mode. We don't use that
anymore. Let's stop writing files that are not used anywhere.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář 6fa478e688 gather: Port hybrid method
Update the code to consume multiple source repos.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář a446593698 pkgset: Update Koji source to create multiple pkgsets
With this patch, there should be a separate package set for each tag
that is consumed.

Generally each module will create a separate package set, with the
exception of -devel modules that will be in the same set as their
non-devel version.

Variants no longer need to keep their own package set objects. Instead
they now include a set of package set names that should be used for the
variant. This can replace the whitelist mechanism of deps gather method.

JIRA: COMPOSE-3620
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář eed2aa2753 pkgset: Add object representing a package set on disk
Once a package set repo is written to disk, let's use this object that
connects the repository path with the mapping of packages.

This change also makes it explicit where the dependency on package set
repos are. In the original codebase, any part of code could generate a
path to the repo, even if that repo has not yet been written.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář 3f665937b2 pkgset: Add name to package set
This name will serve as an identifier for the group of packages.

For Koji package sets, it should the name of the tag from which the
packages come. For package sets based on repos a dummy constant name is
used.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář be61da0192 paths: Add function to generate path to pkgset repo
This will be used to hold a repository for different package sets.

JIRA: COMPOSE-3620
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář 90393c4b49 Update code to pass around a list of pkgsets
This opens up a path to having multiple package sets in the compose. The
pkgset phase now creates a list of them (although at this time there is
always a single item in that list).

Any consumer of the package sets objects is updated to handle a list.
Generally this means an extra loop.

JIRA: COMPOSE-3620
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Pat Riehecky a99bf8c828 Store the parsed variants for possible later re-use
Merges: https://pagure.io/pungi/pull-request/1257
Signed-off-by: Pat Riehecky <riehecky@fnal.gov>
2019-09-02 13:44:24 +02:00
Pat Riehecky b420986aa4 compose: additional logging on variants being processed
Merges: https://pagure.io/pungi/pull-request/1254
Signed-off-by: Pat Riehecky <riehecky@fnal.gov>
2019-09-02 13:41:22 +02:00
Haibo Lin 53a93f016a Refactor code for removing repoclosure cache
For 'yum' backend, only cache dirs following repoclosure-$COMPOSE_ID-$variant.$arch
name convention are created, e.g. repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64

But for 'dnf' backend, the dir name looks like
repoclosure-$COMPOSE_ID-$variant.$arch-$suffix and there are other files
created, e.g.
    repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64-df9fe164317e314e
    repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64-filenames.solvx
    repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64.solv

JIRA: COMPOSE-2565
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-08-23 11:00:17 +08:00
Haibo Lin 82349626c6 Delete cache dirs even though repoclosure command failed
JIRA: COMPOSE-2565
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-08-22 08:37:53 +00:00
Haibo Lin 29224b02ff Delete lookaside cache dir
If --lookaside option passed to repoclosure command, extra cache dir
will be created and it should be deleted too.

JIRA: COMPOSE-2565
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-08-22 14:24:40 +08:00
Haibo Lin 048698d885 Delete repoclosure cache
JIRA: COMPOSE-2565
Merges: https://pagure.io/pungi/pull-request/1253
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-08-15 08:42:16 +02:00
Lubomír Sedlář eeaee1c20f Add function to get all arches in a compose
The same logic is used in two places. This deserves a common function.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-08-06 13:30:08 +02:00
Lubomír Sedlář 51d638d5db Remove not-used hack to workaround required option
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-08-06 09:25:29 +02:00
Lubomír Sedlář e674c2f574 pkgset: Create repos in a single place
Both pkgset sources use the same logic to create per-arch repos. There
is no reason to have that code in both places.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-08-06 09:24:28 +02:00
Lubomír Sedlář 8994aa5d88 pkgset: Added modules to variant in correct format
The data parsed from variants.xml uses a different format that what we
added in `_add_module_to_variant`. This leads to crashes later.

JIRA: COMPOSE-3746
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-08-05 12:06:26 +02:00
Haibo Lin c00162413c Add compose_id to repoclosure/lookaside directory names
JIRA: COMPOSE-3616
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-08-05 15:57:27 +08:00
Lubomír Sedlář e9a363bfde Fix shadowed variable
This could result in incorrect log messages being printed.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-24 13:45:27 +02:00
Lubomír Sedlář 27d015543f Remove now unused msg variables
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-24 13:45:00 +02:00
Lubomír Sedlář 6efaae19fd Remove debug mode
This was already discouraged to not be used, and is a bad idea in
current setup anyway. Removing this can simplify the code.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-24 13:36:23 +02:00
Lubomír Sedlář 0891bfbe59 Remove dead code
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-24 12:37:53 +02:00
Lubomír Sedlář b514e20833 pkgset: Check for empty module index
Instead of testing its return value. Future version of libmodulemd will
raise an exception instead of returning empty data.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-24 07:49:12 +02:00
Lubomír Sedlář c6d6367932 gather: Mark repos as containing module hotfixes
This should prevent DNF from doing any unwanted magic with the contents.
Ideally we want it to completely ignore the modular metadata in there,
but there seems to be no way to do that.

This should at least prevent it from hiding non-modular packages that
are masked by some default stream.

Fixes: https://pagure.io/pungi/issue/1241
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-22 12:35:36 +02:00
Haibo Lin 3811c0a176 config: Deprecate bootable option
JIRA: COMPOSE-2635
Fixes: https://pagure.io/pungi/issue/976
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-07-22 16:51:38 +08:00
Haibo Lin cf52665a8d Use createrepo_checksum option instead of hardcoded sha256
JIRA: COMPOSE-3392
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-07-19 17:53:59 +08:00
Lubomír Sedlář 2dfb1cd4c8 pkgset: Skip adding modulemd if there is none
When libmodulemd is available, but the compose contains no modules, we
don't want to add the modular metadata anywhere. This patch actually
avoids a crash as `ModuleIndex.dump_to_string()` will return `None` if
there is no data.

JIRA: COMPOSE-3662
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-16 09:02:49 +02:00
Lubomír Sedlář 3097019338 test: Allow turning strictness off as well
When the value is explicitly set to `False`, the check should not be
strict.

JIRA: COMPOSE-3658
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-16 08:24:18 +02:00
Lubomír Sedlář 61e3cb0ef1 Port to libmodulemd v2
Fixes: https://pagure.io/pungi/issue/1225
JIRA: COMPOSE-3662
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-15 15:27:05 +02:00
Lubomír Sedlář 62c6d4ddcf test: Option to make size check strict
Sometimes it's practical not just warn when ISO is larger than expected,
but to also abort the compose.

JIRA: COMPOSE-3658
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-15 13:57:50 +02:00
Lubomír Sedlář eeec62756f gather: Remove unused argument
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-11 14:25:42 +02:00
Lubomír Sedlář da78b99fc0 pkgset: Load modulemd only when needed
We can avoid parsing source modulemd information since we can get the
same information from the Koji build info.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-11 14:25:42 +02:00
Lubomír Sedlář 04baa2a4db Stop storing modulemd without arch
Historically each variant had a list of modules. This is no longer
needed and can be dropped. We can also stop logging the modulemd since
we know it was retrieved from Koji and not modified locally.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-11 14:25:42 +02:00
Lubomír Sedlář a31fe998d5 Remove unused import
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-11 14:25:42 +02:00
Lubomír Sedlář 8d00f56117 gather: Correctly sort list with multiple data types
On Python 3 it is not possible to sort str and None or RpmWrapper.

First convert everything to strings and then sort it. The sorting is
really to simplify diffing the files, so exact order does not have to be
preserved.

Fixes: https://pagure.io/pungi/issue/1227
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-02 15:13:23 +02:00
Lubomír Sedlář bf1b3e8421 Remove remaining mentions of runroot option
This also cleans up the runroot method detection code to not rely on the
now removed option.

JIRA: COMPOSE-2634
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-02 09:43:13 +02:00
Lubomír Sedlář 49d0ab797c pkgset: Include module metadata in the repos
If the package set repo contains any modular package, the module
metadata is added there as well.

This is needed to accomodate change in DNF that refuses to work with
repo with modular packages if the metadata is not there. This DNF change
can cause issues in buildinstall phase.

Related: https://bugzilla.redhat.com/show_bug.cgi?id=1623128

The hybrid solver is modified to not create a separate repo with the
module metadata anymore, since it will be available in the repo with
packages. This also allows us to drop code to look into lookaside repos.

We still need to iterate over local modules in order to find out what
platform should be used.

JIRA: COMPOSE-3621
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-01 09:09:17 +02:00
Haibo Lin c4ed2bf3b2 config: Deprecate runroot option
Fixes: https://pagure.io/pungi/issue/975
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-06-27 18:00:07 +08:00
Daris A Nevil 920eceaa5a Respect --nomacboot flag when calling isohybrid
Fixes: https://pagure.io/pungi/issue/1222
Merges: https://pagure.io/pungi/pull-request/1223
Signed-off-by: Daris A Nevil <dnevil@intrusion.com>
2019-06-26 13:21:22 +02:00
Lubomír Sedlář 4e59c7595e config: Improve config validation for anyOf and oneOf
Instead of just saying that there is a problem, give all possible
reasons.

JIRA: COMPOSE-3636
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-25 12:50:07 +02:00
Lubomír Sedlář acd3c19618 config-validate: Allow defining variables
When trying to validate a template that should later be filled in with
`pungi-config-dump`, there will be errors about undefined variables.
These are meant to be set when the template is populated.

This patch adds support for `-e`, `--define` argument to the validation
script that can be used to suppress these errors.

Alternatively a JSON file is read from the directory with config file
that can contain values for the variables.

The `--define` option is changed in both validation and dumping to allow
empty string as an accepted value.

JIRA: COMPOSE-3599
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-21 12:47:50 +02:00
Lubomír Sedlář e6a26571e0 scm: Close stdin of processing command
It is possible the user set a command to run in cloned Git repository,
but that command can ask for additional input. However Pungi will
capture all output, so if there is a prompt, it will never be shown.

In order to prevent confusion ("Did this hang?"), let's send empty
string to stdin of the program. That will cause any possible read to see
EOF immediately, which should cause an error that will then be reported
by Pungi to the user.

It is still possible the program will wait for input if it reads
directly from TTY. However in such case the prompt should hopefully also
be sent to TTY directly, so that possible confusion should be cleared.

JIRA: COMPOSE-3598
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-20 09:07:07 +02:00
Lubomír Sedlář 82580ed5b3 pkgset: Create arch repos in parallel
This patch reuses the existing createrepo_num_threads options to limit
maximum number of parallel createrepo processes.

Fixes: https://pagure.io/pungi/issue/955
JIRA: COMPOSE-2575
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-19 15:42:46 +02:00
Lubomír Sedlář f1263eeacb util: Resolve HEAD in repos that have a remote
JIRA: COMPOSE-3597
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-19 09:04:01 +02:00
Lubomír Sedlář 2c3e6a5a74 pkgset: Use highest pickle protocol
Higher protocols should be more efficient in terms of performance and
storage size. Since we don't really care about interoperability with
different python version, we can safely go to the highest version.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-18 09:33:11 +02:00
Haibo Lin 6ec206f9ae gather: fix crash issue when gather_method = "nodeps"
JIRA: COMPOSE-3089
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-06-14 17:50:19 +08:00
Lubomír Sedlář e68e17d8fe pkgset: Check for unused module patterns across all tags
Since there can be multiple tags, the check must be done once for all of
them at the same time. Otherwise any module found only in some and not
all tags would raise this error.

The code builds a set of all existing patterns and then removes items
from it. If there is something left once all tags are processed, it
means such patterns were not matched by anything.

JIRA: COMPOSE-3609
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-13 10:10:30 +02:00
Lubomír Sedlář f784bbd519 util: Fix offline resolving for scm dict
For scm dict resolving the return value should be git ref (or source
branch for offline mode).

JIRA: COMPOSE-3614
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-12 08:22:38 +02:00
Lubomír Sedlář 2a9490526a pkgset: Make serialization more resilient
If the package set is loaded from a pickle, it will not contain this
key, and thus it can not be again written down into a new pickle. While
this is not a common use case, it's still better not to crash in that
case.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-11 08:45:16 +02:00
Lubomír Sedlář 0207260b9f fus: Support HTTP repos
Recent version of fus gained ability to download repodata if repo is
provided as HTTP url.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-07 14:43:04 +02:00
Haibo Lin ce066707c1 config: Deprecate release_is_layered option
Fixes: https://pagure.io/pungi/issue/977
Merges: https://pagure.io/pungi/pull-request/1204
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-06-06 13:45:49 +02:00
Lubomír Sedlář 4e5f74b78d pkgset: Set correct nsvc for devel modules
We need to update the source modulemd to match what we want. For most
modules the name will be correct already, but in case of devel modules
the original name will still be present there.

Fixes: COMPOSE-3596
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-31 16:02:19 +02:00
Lubomír Sedlář f6162b90dd Remove unused variable
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-31 15:57:55 +02:00
Lubomír Sedlář c55ed742cd Whitespace fixes
This patch fixes some issues with inconsistent use of whitespace. It
only modifies lines that do not contain any code to not break git blame
too much.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-31 09:31:01 +02:00
Lubomír Sedlář 68115f3502 Whitespace cleanup
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-31 09:14:26 +02:00
Lubomír Sedlář fa6197246b Remove unused variables
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-31 09:14:15 +02:00
Lubomír Sedlář 0a90f18b1f Remove unused imports
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-31 09:05:50 +02:00
Lubomír Sedlář 33471c38bb pkgset: Ignore modules without metadata in Koji
This is fairly similar to a package only being built for particular
arches.

Fixes: https://pagure.io/pungi/issue/1198
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-23 13:51:15 +02:00
Lubomír Sedlář 551f52922f runroot: Remove useless argument output_path
The same logic is handled by `chown_paths`, which does the same thing
but supports multiple paths.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-22 15:10:32 +02:00
Lubomír Sedlář 1951b0a521 buildinstall: Change owner of lorax logs
It's created in runroot, the owner and permissions should be updated so
that anyone can read it and user running compose can delete the files.

JIRA: COMPOSE-3545
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-22 15:10:32 +02:00
Lubomír Sedlář f2bbf35429 kojiwrapper: Allow changing mode of multiple files
The directory with logs should have updated owner and permission as
well as the actual output. This patch lays foundation for that by
allowing multiple paths to be specified.

JIRA: COMPOSE-3545
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-22 15:10:32 +02:00
Lubomír Sedlář 8acd2c9689 buildinstall: Create toplevel directory on compose host
When using lorax, this directory will contain subdirectories for
individual variants. Those are created in runroot and their ownership
and permissions are fixed there. However the top level dir was only
created and not updated.

JIRA: COMPOSE-3545
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-22 15:10:32 +02:00
fanjinke a74470c18d arch_utils: add Hygon Dhyana CPU support
Add Hygon CPU vendor ID("HygonGenuine") detection and optimize arch to
"athlon" in x86 and "amd64" in x86_64.

Merges: https://pagure.io/pungi/pull-request/1196
Signed-off-by: fanjinke <fanjinke@hygon.cn>
2019-05-21 12:24:28 +02:00
Lubomír Sedlář 187ce8df79 gather: Introduce module source again
This reverts commit ac15f21135.

It is still needed if nodeps gather method is used. It simply returns
all packages listed in all modules.

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1708661
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-21 08:23:18 +02:00
Lubomír Sedlář fe723a2094 metadata: Include empty directories in metadata
For example when a variant does not have any debuginfo packages, the
metadata will contain path to a repository, but it will be missing the
package path despite the (empty) directory being present on the
filesystem.

We should really only skip missing directories.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-20 13:26:27 +02:00
Lubomír Sedlář 283bae11da gather: Relax validations on variant_as_lookaside
Instead of validating both variants exist, let's just check the
existence of only the variant that is being used as a lookaside.

If the configuration says Foo depends on Bar, the error is reported only
if Foo exists but Bar does not. Any other situation is silently ignored.

JIRA: COMPOSE-3393
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-20 10:46:42 +02:00
Lubomír Sedlář d0e8472ab5 Remove invalid escape sequences
This emits a warning on Python 3.8. Let's just fix it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-15 15:47:44 +02:00
Jan Kaluza cfb7b71fca Fix issues in OpenSSH Runroot method found by real tests.
- Pass the runroot_tag to init command in OpenSSH Runroot method.
  This is needed for the init command as a source for initial packages
  for the buildroot.
- Rename the "runroot_ssh_init_command" to "runroot_ssh_init_template"
  to make it consistent with the rest of "runroot_ssh_*" options.
- Add missing "runroot_ssh_*" options to checks.py.
- Use chmod/chown to `output_dir` in OpenSSH Runroot method the same way
  as it is used in Koji runroot method to make the runroot output readable
  for Pungi user.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2019-05-15 07:40:31 +02:00
Lubomír Sedlář 2dd30008ae buildinstall: Copy files in thread
Instead of running the copy from the main script explicitly, make it
part of the thread.

This should make things very slightly faster, and the code is much
simpler.

Fixes: https://pagure.io/pungi/issue/959
JIRA: COMPOSE-2604
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-13 11:04:37 +02:00
Lubomír Sedlář 9da70ee7a9 init: Create comps repos in parallel
This should be mostly IO intensive operation, so running multiple
commands at the same time should save us a tiny bit of time.

Fixes: https://pagure.io/pungi/issue/981
JIRA: COMPOSE-2646
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-10 12:44:14 +02:00
Lubomír Sedlář b0f0579a9e util: Resolve ref if duplicate branches are present
If the repo contains the same name under multiple directories, make the
resolving work by filtering only to refs/heads and refs/tags.

Fixes: https://pagure.io/pungi/issue/1180
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-07 13:44:57 +02:00
Lubomír Sedlář 9517df44c7 config: Fix getting default branch in SCM dict
If user configures branch as explicit None, we want to default to HEAD
(which is most likely refs/heads/master in git).

The original code was getting branch as None, which lead to wrong
resolver being used and the repo url being used as branch.

Fixes: https://pagure.io/pungi/issue/1181
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-07 10:35:49 +02:00
Lubomír Sedlář a9b9ec97fb pkgset: Fix whitelist for modules
The prefix checking only works if there are no streams that would share
prefixes. Let's instead check the value as a whole. There is extra
complexity from the fact that version and context may not be specified.

The stream as specified in input is processed to replace dashes (`-`)
with underscores (`_`) to match how the builds are imported into Koji.

JIRA: COMPOSE-3547
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-06 09:48:44 +02:00
Lubomír Sedlář 2ae742af04 pkgset: Fix filtering excluded modular packages
For modular tags we only include packages that are included in the
module. Originally the filter was taking package names from
rpms.artifacts section of MMD.

This however does not work correctly, as there can easily be module
which lists foo.src but does not want to include foo.x86_64 or other
arches.

This patch fixes this particular problem by included arch in the set of
packages to be kept.

JIRA: COMPOSE-3543
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-02 13:22:13 +02:00
Lubomír Sedlář f858cea466 pkgset: Do not overwrite version in module
This value is important to get correct path to metadata.

JIRA: COMPOSE-3541
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-30 12:22:13 +02:00
Lubomír Sedlář b73d2d7f11 pkgset: Treat modular version as number for sorting
The scheme for generating versions has changed multiple times. MBS is
careful to only modify them so that they always compare correctly. This
only works though if the versions are treated as numbers.

This should be safe in that non-numbers should never be encountered as
module version. Libmodulemd internally stores the version as int (or
some version of int).

JIRA: COMPOSE-3540
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-26 14:51:24 +02:00
Lubomír Sedlář e550686e06 Use absolute path for hardlink
Hardlink command can be installed in /usr/sbin, where it is not visible
to non-priviledged users. They can still run it, but don't have it in
their PATH.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-26 13:48:54 +02:00
Jon Disnard 72b4969832 createiso: Run hardlink on staged content
Even if we want to break hardlinks from Koji volume, there may be files
that can be hardlinked and save some space on the media.

JIRA: COMPOSE-3482
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-26 13:48:54 +02:00
Lubomír Sedlář 1e7ec68bbd comps-wrapper: Emit attributes sorted
Python 3.8 no longer sorts attributes automatically, which is causing
some of the tests to fail. The easiest fix is to update the code to make
sure sorting is in place.

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1698514
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-18 10:23:51 +02:00
Jan Kaluza 49d137b444 Extend "openssh" runroot_method to be able to execute "mock"
This adds few new config options which are well described in the
configuration documentation. Please refer to it for more information.

Merges: https://pagure.io/pungi/pull-request/1170
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2019-04-12 11:10:40 +02:00
Lubomír Sedlář d521711957 osbs: Rework configuration for image pushes
Embedding the registry configuration into OSBS config itself is
simple, but makes it impossible to reuse the same configuration for
multiple different composes.

A nice example is a nightly pushing images to a testing registry, and
production compose building the same images but pushing to staging
location. The original design requires duplication of all the
configuration just because registries are different.

With this option, the push information is stored in a separate option as
a mapping from NVR patterns to arbitrary data. The patterns are used to
match finished builds to registry.

The old configuration is marked as deprecated in code and will
eventually be removed. The deprecation handling in config validation
does not allow emitting warnings for nested values.

JIRA: COMPOSE-3394
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-05 10:28:34 +02:00
Jan Kaluza 959d6979d4 Add "openssh" runroot method
Merges: https://pagure.io/pungi/pull-request/1166
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2019-04-04 14:44:05 +02:00
Lubomír Sedlář 2f8717ec97 Fix printing version on Python 3
Don't crash when getting version from installed package.

Fixes: https://pagure.io/pungi/issue/1152
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-04 11:42:19 +02:00
Lubomír Sedlář 1f95c33e2a gather: Use wildcard for repo selection
If package source is set to repos, honor wildcard and apply that block
every time.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-01 09:49:53 +02:00
Lubomír Sedlář c6a86c444a gather: Apply repo path substitutions for DNF backend
Use a method to add repos that will apply $arch and $basearch
substitution automatically. Yum backend only applies $basearch, so if
compatibility is needed, that one should be used.

Drop code for handling mirrorlist, since Pungi does not ever use it, and
being used externally is not really supported.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-01 09:49:53 +02:00
Lubomír Sedlář 66a127c313 pkgset: Refactor hiding unused modular packages
A module build can create packages that are tagged in the content tag,
but should not be included in the module. Originally Pungi didn't know
what exactly the module contains and so it needed to apply filters to
exclude stuff that was definitely out.

With getting the final MMD from Koji, we can actually make this a bit
more strict by only keeping packages that we know we need.

When processing each content tag, we can put into package set only
packages that are included in some module using that tag. This should
work with -devel modules as well. Both the regular and -devel modules
will contribute to the set and thus all packages will go to the package
set.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-25 08:48:16 +01:00
Lubomír Sedlář 9f9b784e64 Remove configuration for devel modules
They should be loaded in the same way as regular modules straight from
Koji.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-25 08:48:16 +01:00
Lubomír Sedlář ac15f21135 gather: Remove module source
This source does not really return anything useful. It was necessary to
process the source modulemd to fill in list of RPMs. Since we now get
the final files from Koji, this is not needed anymore and the source can
be dropped.

This change requires a lot of tweaks for test.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-25 08:48:16 +01:00
Lubomír Sedlář 9939d09643 createrepo: Stop processing modulemd
The file we get from Koji has all important bits already filled in.
There is no need to add anything.

This patch also stops filtering the artifacts to only contain packages
that are actually in the repo. Thus debuginfo and source packages will
appear there.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-25 08:48:16 +01:00
Lubomír Sedlář 9229699078 pkgset: Load final modulemd files from Koji
Instead of loading the "source" modulemd, always get the final file for
each architecture from Koji.

Logging the downloaded files locally is no longer necessary, when
debugging a problem we can find the files in the respective builds in
Koji.

Tests are updated to work with new code, and an obsolete test is
removed.

JIRA: COMPOSE-3147
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-25 08:48:16 +01:00
Lubomír Sedlář 358fdd50ce buildinstall: Allow overwriting version for lorax
Sometimes the release version can be more specific than what should be
exposed to users of the boot iso.

JIRA: COMPOSE-3295
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-22 12:47:16 +01:00
Jan Kaluza 75bb48a882 Create new Runroot class and use it everywhere where runroot task is executed.
This adds new `Runroot` class and new `runroot_method` option which makes
it possible to choose between two currently available runroot methods:

- Local
- Koji

The main goal of this commit is to make it possible to add new runroot
methods in the future and this is the first step in that direction.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2019-03-21 14:15:58 +01:00
Lubomír Sedlář 45cdbb2faf orchestrator: Send messages about the main compose
Only start/finish messages will be sent if a handler is configured.

JIRA: COMPOSE-3288
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-21 09:29:54 +01:00
Lubomír Sedlář b80efbfd97 hybrid: Refactor handling debuginfo packages
Behavior before this change: debuginfo was added based on source RPMs.
Pungi would get all debuginfo for all architectures that include at
least one binary package. This had a consequence that if foo-x.x86_64
and foo-x.i686 from the same built were included, foo-y-debuginfo.i686
would be included despite foo-y only being present for x86_64.

The patch changes this to work on binary package level: for each
included package `x`, check if there is `x-debuginfo` or
`x-debugsource`, and include them. Packages added in this way have to go
through one iteration of the solver, to account for cases such as
`x-libs-debuginfo` depending on `x-debuginfo` (with only `x` starting
this chain).

To make it slightly faster, the invocation of fus is changed to only
process newly added package in each iteration. It should have no effect
on the results, and subsequent iterations are much faster if they don't
process the same stuff again and again.

JIRA: COMPOSE-3247
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-18 13:56:48 +01:00
Lubomír Sedlář 039b8d44b3 pkgset: Stop loading list of module RPMs
The list is not needed for anything anymore, and it only takes time and
space in memory.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-14 09:08:44 -04:00
Lubomír Sedlář 0fc797a06f pkgset: Only load cache once
The file in old compose does not change, there is no need to load it
again for each tag.

One consequence of this change is that the same cache will be used by
all package sets. They add stuff to the cache as they use it. However
that should not be a problem for the same reason it's okay to use the
cache in the first place. If two packages have the same path, they are
actually the same file and it's okay to reuse the data.

JIRA: COMPOSE-3374
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-12 08:57:35 -04:00
Jan Kaluza 2d39490909 Do not add pkgset_koji_builds to modules, but only to pkgset_koji_tag tags.
Also remove the useless listTaggedRPMs call which has been used *only*
to catch the very rare error which happened from time to time when
we used PDC in Pungi few years ago. This rare error is not relevant
anymore now.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2019-03-11 17:56:27 +01:00
Lubomír Sedlář 9de036b401 scm: Don't retry git fetch
If it fails, we can't really tell if it's a transient error or just too
old git client. Fall back to full clone immediately and retry there.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-11 13:12:06 +00:00
Lubomír Sedlář 9541c75b7e checks: Use GitResolver for scm dicts
Otherwise the offline flag is not honored correctly.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-06 09:46:12 +00:00
Lubomír Sedlář 4431ea8a0c hybrid: Fix opening gzipped files on Python 2.6
GzipFile works in `with` statement starting with 2.7

JIRA: RCM-51070
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-06 08:04:19 +01:00
Lubomír Sedlář 357f556d69 image-build: Accept formats in lists
If the config is loaded from JSON, we will get list instead of tuple.

The validation rule does not really care whether it's a list or tuple,
it only enforces there are two strings. The definition is renamed to be
a bit more descriptive.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-04 08:32:19 +01:00
Lubomír Sedlář 6bd1e9da2a image-build: Resolve git ref in ksurl
The ksurl property was not listed in the config schema, and thus it was
not processed.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-01 09:40:30 +01:00
Lubomír Sedlář 2d694272c0 Resolve git branches in scm_dict
If the config uses SCM dicts that include branch or tag names, they will
be resolved to specific commit ids.

It goes through the caching resolver. The main motivation for that is to
correctly support the --offline flag. It's highly unlikely there will be
two scm_dicts in the config with the same repo.

JIRA: COMPOSE-3279
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-01 09:40:30 +01:00
Lubomír Sedlář 5a7ced5b7d util: Refactor resolving git url
Split out a smaller function that receives a git repo URL and ref and
returns the commit id. The caching resolver class is modified to use
this second function if branch is given to it.

The new function checks if the ref received already looks like a commit
ID, and if so it does not attempt to do anything with it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-01 09:40:30 +01:00