Commit Graph

714 Commits

Author SHA1 Message Date
Lubomír Sedlář
465ecf229c Add test for materialized pkgset
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
63a8b7b6c9 Remove package whitelist
This was a workaround to make some packages from the global repo
invisible for depsolving. This is now handled by packages being in
different repos. We can select which repos are enabled at which point.
This achieves the same result, but much faster.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
a426a83ed6 gather: Remove use of arch_repo
The repo was used to speed up creating lookaside repo from a variant.
This uses a similar approach as createrepo phase: selecting the last
available package set and using that data.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
2d0ffb56ca ostree-installer: Remove usage of arch_repo
Pass all pkgset repos as input to the task.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
26ddd46acb ostree: Remove arch_repo path usage
Pass all pkgset repos as input to the task.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
ab9122be2a buildinstall: Remove arch_repo usage
Simply use all existing package set repos as input for the runroot
task. The command line gets a bit long, but the actual behaviour should
remain the same.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
3824eab15b createrepo: Remove arch_repo usage
There is no longer a single repo with all packages. This means that the
metadata has to be loaded from another location.

When taking packages from Koji, we can assume that the non-modular
package tag will be processed last. The repo for this tag will be used.
This has better chance of being useful than using a random module.

For repo sources, there is only one package set anyway, so this change
makes no difference.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
054b91c798 Fix tests
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
a446593698 pkgset: Update Koji source to create multiple pkgsets
With this patch, there should be a separate package set for each tag
that is consumed.

Generally each module will create a separate package set, with the
exception of -devel modules that will be in the same set as their
non-devel version.

Variants no longer need to keep their own package set objects. Instead
they now include a set of package set names that should be used for the
variant. This can replace the whitelist mechanism of deps gather method.

JIRA: COMPOSE-3620
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
eed2aa2753 pkgset: Add object representing a package set on disk
Once a package set repo is written to disk, let's use this object that
connects the repository path with the mapping of packages.

This change also makes it explicit where the dependency on package set
repos are. In the original codebase, any part of code could generate a
path to the repo, even if that repo has not yet been written.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
3f665937b2 pkgset: Add name to package set
This name will serve as an identifier for the group of packages.

For Koji package sets, it should the name of the tag from which the
packages come. For package sets based on repos a dummy constant name is
used.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
90393c4b49 Update code to pass around a list of pkgsets
This opens up a path to having multiple package sets in the compose. The
pkgset phase now creates a list of them (although at this time there is
always a single item in that list).

Any consumer of the package sets objects is updated to handle a list.
Generally this means an extra loop.

JIRA: COMPOSE-3620
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Haibo Lin
53a93f016a Refactor code for removing repoclosure cache
For 'yum' backend, only cache dirs following repoclosure-$COMPOSE_ID-$variant.$arch
name convention are created, e.g. repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64

But for 'dnf' backend, the dir name looks like
repoclosure-$COMPOSE_ID-$variant.$arch-$suffix and there are other files
created, e.g.
    repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64-df9fe164317e314e
    repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64-filenames.solvx
    repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64.solv

JIRA: COMPOSE-2565
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-08-23 11:00:17 +08:00
Haibo Lin
048698d885 Delete repoclosure cache
JIRA: COMPOSE-2565
Merges: https://pagure.io/pungi/pull-request/1253
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-08-15 08:42:16 +02:00
Lubomír Sedlář
e674c2f574 pkgset: Create repos in a single place
Both pkgset sources use the same logic to create per-arch repos. There
is no reason to have that code in both places.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-08-06 09:24:28 +02:00
Lubomír Sedlář
8994aa5d88 pkgset: Added modules to variant in correct format
The data parsed from variants.xml uses a different format that what we
added in `_add_module_to_variant`. This leads to crashes later.

JIRA: COMPOSE-3746
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-08-05 12:06:26 +02:00
Haibo Lin
c00162413c Add compose_id to repoclosure/lookaside directory names
JIRA: COMPOSE-3616
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-08-05 15:57:27 +08:00
Lubomír Sedlář
6efaae19fd Remove debug mode
This was already discouraged to not be used, and is a bad idea in
current setup anyway. Removing this can simplify the code.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-24 13:36:23 +02:00
Haibo Lin
3811c0a176 config: Deprecate bootable option
JIRA: COMPOSE-2635
Fixes: https://pagure.io/pungi/issue/976
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-07-22 16:51:38 +08:00
Lubomír Sedlář
3097019338 test: Allow turning strictness off as well
When the value is explicitly set to `False`, the check should not be
strict.

JIRA: COMPOSE-3658
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-16 08:24:18 +02:00
Lubomír Sedlář
61e3cb0ef1 Port to libmodulemd v2
Fixes: https://pagure.io/pungi/issue/1225
JIRA: COMPOSE-3662
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-15 15:27:05 +02:00
Lubomír Sedlář
62c6d4ddcf test: Option to make size check strict
Sometimes it's practical not just warn when ISO is larger than expected,
but to also abort the compose.

JIRA: COMPOSE-3658
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-15 13:57:50 +02:00
Lubomír Sedlář
eeec62756f gather: Remove unused argument
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-11 14:25:42 +02:00
Lubomír Sedlář
da78b99fc0 pkgset: Load modulemd only when needed
We can avoid parsing source modulemd information since we can get the
same information from the Koji build info.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-11 14:25:42 +02:00
Lubomír Sedlář
04baa2a4db Stop storing modulemd without arch
Historically each variant had a list of modules. This is no longer
needed and can be dropped. We can also stop logging the modulemd since
we know it was retrieved from Koji and not modified locally.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-11 14:25:42 +02:00
Lubomír Sedlář
8d00f56117 gather: Correctly sort list with multiple data types
On Python 3 it is not possible to sort str and None or RpmWrapper.

First convert everything to strings and then sort it. The sorting is
really to simplify diffing the files, so exact order does not have to be
preserved.

Fixes: https://pagure.io/pungi/issue/1227
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-02 15:13:23 +02:00
Lubomír Sedlář
bf1b3e8421 Remove remaining mentions of runroot option
This also cleans up the runroot method detection code to not rely on the
now removed option.

JIRA: COMPOSE-2634
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-02 09:43:13 +02:00
Lubomír Sedlář
49d0ab797c pkgset: Include module metadata in the repos
If the package set repo contains any modular package, the module
metadata is added there as well.

This is needed to accomodate change in DNF that refuses to work with
repo with modular packages if the metadata is not there. This DNF change
can cause issues in buildinstall phase.

Related: https://bugzilla.redhat.com/show_bug.cgi?id=1623128

The hybrid solver is modified to not create a separate repo with the
module metadata anymore, since it will be available in the repo with
packages. This also allows us to drop code to look into lookaside repos.

We still need to iterate over local modules in order to find out what
platform should be used.

JIRA: COMPOSE-3621
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-01 09:09:17 +02:00
Haibo Lin
c4ed2bf3b2 config: Deprecate runroot option
Fixes: https://pagure.io/pungi/issue/975
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-06-27 18:00:07 +08:00
Lubomír Sedlář
32a6415e58 config: Keep known options defined on CLI
If the validation or dumping script is given some options, they should
only be removed if they are not valid. We have to remove the invalid
ones, otherwise that would cause a warning about unknown options.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-25 15:13:16 +02:00
Lubomír Sedlář
4e59c7595e config: Improve config validation for anyOf and oneOf
Instead of just saying that there is a problem, give all possible
reasons.

JIRA: COMPOSE-3636
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-25 12:50:07 +02:00
Lubomír Sedlář
acd3c19618 config-validate: Allow defining variables
When trying to validate a template that should later be filled in with
`pungi-config-dump`, there will be errors about undefined variables.
These are meant to be set when the template is populated.

This patch adds support for `-e`, `--define` argument to the validation
script that can be used to suppress these errors.

Alternatively a JSON file is read from the directory with config file
that can contain values for the variables.

The `--define` option is changed in both validation and dumping to allow
empty string as an accepted value.

JIRA: COMPOSE-3599
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-21 12:47:50 +02:00
Lubomír Sedlář
82580ed5b3 pkgset: Create arch repos in parallel
This patch reuses the existing createrepo_num_threads options to limit
maximum number of parallel createrepo processes.

Fixes: https://pagure.io/pungi/issue/955
JIRA: COMPOSE-2575
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-19 15:42:46 +02:00
Lubomír Sedlář
f1263eeacb util: Resolve HEAD in repos that have a remote
JIRA: COMPOSE-3597
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-19 09:04:01 +02:00
Lubomír Sedlář
d01084337e tests: Avoid using threads in tests
Starting tests just to run mock functions slows the tests down for no
good reason. Let's instead mock the runner and run the dummy tasks
serially.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-18 13:17:17 +02:00
Lubomír Sedlář
2c3e6a5a74 pkgset: Use highest pickle protocol
Higher protocols should be more efficient in terms of performance and
storage size. Since we don't really care about interoperability with
different python version, we can safely go to the highest version.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-18 09:33:11 +02:00
Lubomír Sedlář
e68e17d8fe pkgset: Check for unused module patterns across all tags
Since there can be multiple tags, the check must be done once for all of
them at the same time. Otherwise any module found only in some and not
all tags would raise this error.

The code builds a set of all existing patterns and then removes items
from it. If there is something left once all tags are processed, it
means such patterns were not matched by anything.

JIRA: COMPOSE-3609
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-13 10:10:30 +02:00
Lubomír Sedlář
f784bbd519 util: Fix offline resolving for scm dict
For scm dict resolving the return value should be git ref (or source
branch for offline mode).

JIRA: COMPOSE-3614
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-12 08:22:38 +02:00
Lubomír Sedlář
0207260b9f fus: Support HTTP repos
Recent version of fus gained ability to download repodata if repo is
provided as HTTP url.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-07 14:43:04 +02:00
Haibo Lin
ce066707c1 config: Deprecate release_is_layered option
Fixes: https://pagure.io/pungi/issue/977
Merges: https://pagure.io/pungi/pull-request/1204
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-06-06 13:45:49 +02:00
Lubomír Sedlář
4e5f74b78d pkgset: Set correct nsvc for devel modules
We need to update the source modulemd to match what we want. For most
modules the name will be correct already, but in case of devel modules
the original name will still be present there.

Fixes: COMPOSE-3596
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-31 16:02:19 +02:00
Lubomír Sedlář
f6162b90dd Remove unused variable
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-31 15:57:55 +02:00
Lubomír Sedlář
c55ed742cd Whitespace fixes
This patch fixes some issues with inconsistent use of whitespace. It
only modifies lines that do not contain any code to not break git blame
too much.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-31 09:31:01 +02:00
Lubomír Sedlář
551f52922f runroot: Remove useless argument output_path
The same logic is handled by `chown_paths`, which does the same thing
but supports multiple paths.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-22 15:10:32 +02:00
Lubomír Sedlář
1951b0a521 buildinstall: Change owner of lorax logs
It's created in runroot, the owner and permissions should be updated so
that anyone can read it and user running compose can delete the files.

JIRA: COMPOSE-3545
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-22 15:10:32 +02:00
Lubomír Sedlář
f2bbf35429 kojiwrapper: Allow changing mode of multiple files
The directory with logs should have updated owner and permission as
well as the actual output. This patch lays foundation for that by
allowing multiple paths to be specified.

JIRA: COMPOSE-3545
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-22 15:10:32 +02:00
Lubomír Sedlář
187ce8df79 gather: Introduce module source again
This reverts commit ac15f21135.

It is still needed if nodeps gather method is used. It simply returns
all packages listed in all modules.

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1708661
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-21 08:23:18 +02:00
Lubomír Sedlář
283bae11da gather: Relax validations on variant_as_lookaside
Instead of validating both variants exist, let's just check the
existence of only the variant that is being used as a lookaside.

If the configuration says Foo depends on Bar, the error is reported only
if Foo exists but Bar does not. Any other situation is silently ignored.

JIRA: COMPOSE-3393
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-20 10:46:42 +02:00
Lubomír Sedlář
cbe8457377 tests: Use correct Python interpreter
Instead of guessing based on a good enough name, we can use
`sys.executable` and get path to current python.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-16 09:07:18 +02:00
Lubomír Sedlář
a33cb0bf91 tests: Ignore warnings when running validation script
The script is run as a standalone process. Anything printed to stderr
breaks the test.

Since Python 3.8, we are getting warnings about invalid escape sequences
in some modules that are imported but not owned by us.

This patch should silence the warnings.

Relates: https://bugzilla.redhat.com/show_bug.cgi?id=1698514
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-15 15:49:20 +02:00