Commit Graph

2535 Commits

Author SHA1 Message Date
Pat Riehecky
b420986aa4 compose: additional logging on variants being processed
Merges: https://pagure.io/pungi/pull-request/1254
Signed-off-by: Pat Riehecky <riehecky@fnal.gov>
2019-09-02 13:41:22 +02:00
Haibo Lin
53a93f016a Refactor code for removing repoclosure cache
For 'yum' backend, only cache dirs following repoclosure-$COMPOSE_ID-$variant.$arch
name convention are created, e.g. repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64

But for 'dnf' backend, the dir name looks like
repoclosure-$COMPOSE_ID-$variant.$arch-$suffix and there are other files
created, e.g.
    repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64-df9fe164317e314e
    repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64-filenames.solvx
    repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64.solv

JIRA: COMPOSE-2565
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-08-23 11:00:17 +08:00
Haibo Lin
82349626c6 Delete cache dirs even though repoclosure command failed
JIRA: COMPOSE-2565
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-08-22 08:37:53 +00:00
Haibo Lin
29224b02ff Delete lookaside cache dir
If --lookaside option passed to repoclosure command, extra cache dir
will be created and it should be deleted too.

JIRA: COMPOSE-2565
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-08-22 14:24:40 +08:00
Haibo Lin
048698d885 Delete repoclosure cache
JIRA: COMPOSE-2565
Merges: https://pagure.io/pungi/pull-request/1253
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-08-15 08:42:16 +02:00
Lubomír Sedlář
eeaee1c20f Add function to get all arches in a compose
The same logic is used in two places. This deserves a common function.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-08-06 13:30:08 +02:00
Lubomír Sedlář
51d638d5db Remove not-used hack to workaround required option
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-08-06 09:25:29 +02:00
Lubomír Sedlář
e674c2f574 pkgset: Create repos in a single place
Both pkgset sources use the same logic to create per-arch repos. There
is no reason to have that code in both places.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-08-06 09:24:28 +02:00
Lubomír Sedlář
8994aa5d88 pkgset: Added modules to variant in correct format
The data parsed from variants.xml uses a different format that what we
added in `_add_module_to_variant`. This leads to crashes later.

JIRA: COMPOSE-3746
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-08-05 12:06:26 +02:00
Haibo Lin
c00162413c Add compose_id to repoclosure/lookaside directory names
JIRA: COMPOSE-3616
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-08-05 15:57:27 +08:00
Lubomír Sedlář
e9a363bfde Fix shadowed variable
This could result in incorrect log messages being printed.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-24 13:45:27 +02:00
Lubomír Sedlář
27d015543f Remove now unused msg variables
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-24 13:45:00 +02:00
Lubomír Sedlář
6efaae19fd Remove debug mode
This was already discouraged to not be used, and is a bad idea in
current setup anyway. Removing this can simplify the code.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-24 13:36:23 +02:00
Lubomír Sedlář
0891bfbe59 Remove dead code
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-24 12:37:53 +02:00
Lubomír Sedlář
b514e20833 pkgset: Check for empty module index
Instead of testing its return value. Future version of libmodulemd will
raise an exception instead of returning empty data.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-24 07:49:12 +02:00
Lubomír Sedlář
c6d6367932 gather: Mark repos as containing module hotfixes
This should prevent DNF from doing any unwanted magic with the contents.
Ideally we want it to completely ignore the modular metadata in there,
but there seems to be no way to do that.

This should at least prevent it from hiding non-modular packages that
are masked by some default stream.

Fixes: https://pagure.io/pungi/issue/1241
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-22 12:35:36 +02:00
Haibo Lin
3811c0a176 config: Deprecate bootable option
JIRA: COMPOSE-2635
Fixes: https://pagure.io/pungi/issue/976
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-07-22 16:51:38 +08:00
Haibo Lin
cf52665a8d Use createrepo_checksum option instead of hardcoded sha256
JIRA: COMPOSE-3392
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-07-19 17:53:59 +08:00
Lubomír Sedlář
2dfb1cd4c8 pkgset: Skip adding modulemd if there is none
When libmodulemd is available, but the compose contains no modules, we
don't want to add the modular metadata anywhere. This patch actually
avoids a crash as `ModuleIndex.dump_to_string()` will return `None` if
there is no data.

JIRA: COMPOSE-3662
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-16 09:02:49 +02:00
Lubomír Sedlář
3097019338 test: Allow turning strictness off as well
When the value is explicitly set to `False`, the check should not be
strict.

JIRA: COMPOSE-3658
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-16 08:24:18 +02:00
Lubomír Sedlář
61e3cb0ef1 Port to libmodulemd v2
Fixes: https://pagure.io/pungi/issue/1225
JIRA: COMPOSE-3662
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-15 15:27:05 +02:00
Lubomír Sedlář
62c6d4ddcf test: Option to make size check strict
Sometimes it's practical not just warn when ISO is larger than expected,
but to also abort the compose.

JIRA: COMPOSE-3658
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-15 13:57:50 +02:00
Lubomír Sedlář
eeec62756f gather: Remove unused argument
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-11 14:25:42 +02:00
Lubomír Sedlář
da78b99fc0 pkgset: Load modulemd only when needed
We can avoid parsing source modulemd information since we can get the
same information from the Koji build info.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-11 14:25:42 +02:00
Lubomír Sedlář
04baa2a4db Stop storing modulemd without arch
Historically each variant had a list of modules. This is no longer
needed and can be dropped. We can also stop logging the modulemd since
we know it was retrieved from Koji and not modified locally.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-11 14:25:42 +02:00
Lubomír Sedlář
a31fe998d5 Remove unused import
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-11 14:25:42 +02:00
Lubomír Sedlář
a340663f00 doc: Clarify relationship to Koji
Fixes: https://pagure.io/pungi/issue/1143
JIRA: COMPOSE-3372
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-03 09:06:49 +02:00
Lubomír Sedlář
8d00f56117 gather: Correctly sort list with multiple data types
On Python 3 it is not possible to sort str and None or RpmWrapper.

First convert everything to strings and then sort it. The sorting is
really to simplify diffing the files, so exact order does not have to be
preserved.

Fixes: https://pagure.io/pungi/issue/1227
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-02 15:13:23 +02:00
Lubomír Sedlář
fc362c5347 git-changelog: Fix running on Python 3
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-02 15:05:25 +02:00
Lubomír Sedlář
46333b01ad 4.1.38 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-02 10:32:01 +02:00
Lubomír Sedlář
bf1b3e8421 Remove remaining mentions of runroot option
This also cleans up the runroot method detection code to not rely on the
now removed option.

JIRA: COMPOSE-2634
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-02 09:43:13 +02:00
Lubomír Sedlář
49d0ab797c pkgset: Include module metadata in the repos
If the package set repo contains any modular package, the module
metadata is added there as well.

This is needed to accomodate change in DNF that refuses to work with
repo with modular packages if the metadata is not there. This DNF change
can cause issues in buildinstall phase.

Related: https://bugzilla.redhat.com/show_bug.cgi?id=1623128

The hybrid solver is modified to not create a separate repo with the
module metadata anymore, since it will be available in the repo with
packages. This also allows us to drop code to look into lookaside repos.

We still need to iterate over local modules in order to find out what
platform should be used.

JIRA: COMPOSE-3621
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-01 09:09:17 +02:00
Haibo Lin
c4ed2bf3b2 config: Deprecate runroot option
Fixes: https://pagure.io/pungi/issue/975
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-06-27 18:00:07 +08:00
Daris A Nevil
920eceaa5a Respect --nomacboot flag when calling isohybrid
Fixes: https://pagure.io/pungi/issue/1222
Merges: https://pagure.io/pungi/pull-request/1223
Signed-off-by: Daris A Nevil <dnevil@intrusion.com>
2019-06-26 13:21:22 +02:00
Lubomír Sedlář
32a6415e58 config: Keep known options defined on CLI
If the validation or dumping script is given some options, they should
only be removed if they are not valid. We have to remove the invalid
ones, otherwise that would cause a warning about unknown options.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-25 15:13:16 +02:00
Lubomír Sedlář
d063217d6f config-dump: Report better error for non-existing sources
JIRA: COMPOSE-3624
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-25 14:42:50 +02:00
Lubomír Sedlář
4e59c7595e config: Improve config validation for anyOf and oneOf
Instead of just saying that there is a problem, give all possible
reasons.

JIRA: COMPOSE-3636
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-25 12:50:07 +02:00
Lubomír Sedlář
acd3c19618 config-validate: Allow defining variables
When trying to validate a template that should later be filled in with
`pungi-config-dump`, there will be errors about undefined variables.
These are meant to be set when the template is populated.

This patch adds support for `-e`, `--define` argument to the validation
script that can be used to suppress these errors.

Alternatively a JSON file is read from the directory with config file
that can contain values for the variables.

The `--define` option is changed in both validation and dumping to allow
empty string as an accepted value.

JIRA: COMPOSE-3599
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-21 12:47:50 +02:00
Lubomír Sedlář
32624c59b1 config: Report validation warning if variants fail to load
In a real compose this would be a blocker issue and the compose would be
aborted, but for validation it may make sense to continue. Instead of
crashing, let's report a clear warning.

JIRA: COMPOSE-3606
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-21 10:02:25 +02:00
Lubomír Sedlář
2f05a71c61 Allow customizing nosetests command
In some environments there may not be a version of the command without a
version. In such case it's quite convenient to be able to specify what
is the actual name of the nosetests command.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-20 13:26:53 +02:00
Lubomír Sedlář
e6a26571e0 scm: Close stdin of processing command
It is possible the user set a command to run in cloned Git repository,
but that command can ask for additional input. However Pungi will
capture all output, so if there is a prompt, it will never be shown.

In order to prevent confusion ("Did this hang?"), let's send empty
string to stdin of the program. That will cause any possible read to see
EOF immediately, which should cause an error that will then be reported
by Pungi to the user.

It is still possible the program will wait for input if it reads
directly from TTY. However in such case the prompt should hopefully also
be sent to TTY directly, so that possible confusion should be cleared.

JIRA: COMPOSE-3598
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-20 09:07:07 +02:00
Lubomír Sedlář
82580ed5b3 pkgset: Create arch repos in parallel
This patch reuses the existing createrepo_num_threads options to limit
maximum number of parallel createrepo processes.

Fixes: https://pagure.io/pungi/issue/955
JIRA: COMPOSE-2575
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-19 15:42:46 +02:00
Lubomír Sedlář
f1263eeacb util: Resolve HEAD in repos that have a remote
JIRA: COMPOSE-3597
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-19 09:04:01 +02:00
Lubomír Sedlář
d01084337e tests: Avoid using threads in tests
Starting tests just to run mock functions slows the tests down for no
good reason. Let's instead mock the runner and run the dummy tasks
serially.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-18 13:17:17 +02:00
Lubomír Sedlář
2c3e6a5a74 pkgset: Use highest pickle protocol
Higher protocols should be more efficient in terms of performance and
storage size. Since we don't really care about interoperability with
different python version, we can safely go to the highest version.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-18 09:33:11 +02:00
Haibo Lin
6ec206f9ae gather: fix crash issue when gather_method = "nodeps"
JIRA: COMPOSE-3089
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-06-14 17:50:19 +08:00
Lubomír Sedlář
e68e17d8fe pkgset: Check for unused module patterns across all tags
Since there can be multiple tags, the check must be done once for all of
them at the same time. Otherwise any module found only in some and not
all tags would raise this error.

The code builds a set of all existing patterns and then removes items
from it. If there is something left once all tags are processed, it
means such patterns were not matched by anything.

JIRA: COMPOSE-3609
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-13 10:10:30 +02:00
Lubomír Sedlář
f784bbd519 util: Fix offline resolving for scm dict
For scm dict resolving the return value should be git ref (or source
branch for offline mode).

JIRA: COMPOSE-3614
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-12 08:22:38 +02:00
Lubomír Sedlář
2a9490526a pkgset: Make serialization more resilient
If the package set is loaded from a pickle, it will not contain this
key, and thus it can not be again written down into a new pickle. While
this is not a common use case, it's still better not to crash in that
case.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-11 08:45:16 +02:00
Lubomír Sedlář
0207260b9f fus: Support HTTP repos
Recent version of fus gained ability to download repodata if repo is
provided as HTTP url.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-07 14:43:04 +02:00