Commit Graph

1904 Commits

Author SHA1 Message Date
Patrick Uiterwijk
8f3c06bd14 Make wait-for-signed-ostree repeat the fedmsg in case the signer crashed
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2018-05-09 23:17:59 +02:00
Lubomír Sedlář
905064775a arch: Remove mocks in tests
This was added back when rpmutils module was used. Since we migrated to
Python 3 we have included the code directly in Pungi (because it was
coming from yum which is not going to Py3 any time soon). The mocks can
be dropped.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-09 13:49:29 +02:00
Lubomír Sedlář
7c237c2c63 ostree-installer: Allow overwriting buildinstall
Relates: https://pagure.io/pungi/issue/909
Relates: https://pagure.io/pungi/issue/695
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-07 15:22:01 +02:00
Lubomír Sedlář
543154d597 ostree-installer: Work with skipped buildinstall
Fixes: https://pagure.io/pungi/issue/909
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-07 12:29:20 +02:00
Lubomír Sedlář
076be762ec createrepo: Use less verbose logs
There is really no need to write out megabytes of logs that are not
really interesting. This should also help the parallelization. With the
verbose log createrepo fills the output buffer and needs to wait for the
busy python program to read it first.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-07 08:38:23 +02:00
Lubomír Sedlář
bd9a0ceda2 pkgset: Create global repo in parallel to merging pkgsets
The work in creating the repo is done in a separate process. This can
easily use two threads.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-07 08:38:23 +02:00
Lubomír Sedlář
dc557b809a ostree-installer: Copy files without owner
The files created in koji runroot will be owned by root. If the compose
is done under different user, there could be a problem with copying the
files preserving the owner. Let's just copy them without that.

Fixes: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-04 14:18:24 +02:00
Lubomír Sedlář
99b6e44a30 createiso: Skip if buildinstall fails
If the ISO is meant to be bootable but lorax fails, there's no point in
creating the ISO as it will not behave as expected.

Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1574585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-04 08:45:45 +02:00
Lubomír Sedlář
490f079c44 Update tests for libmodulemd 1.4.0
It got a little picky about allowed inputs. We need full RPM NEVRA in
all tests.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-02 15:36:52 +02:00
Lubomír Sedlář
182a5a399b 4.1.24 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-02 14:16:44 +02:00
Lubomír Sedlář
ed8fffb6d1 koji-wrapper: Log failed subtasks
If the parent task is successful, there can still be failed child tasks
for failable arches. We need to log those and potentially mark the
compose as incomplete.

Fixes: https://pagure.io/pungi/issue/874
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-02 08:27:44 +02:00
Lubomír Sedlář
f298121bbc Update compose status when config validation fails
If the config is syntactically correct, but it contains some issues
discovered in `validate` method of any phase, we need to raise an
exception, not exit immediately.

JIRA: COMPOSE-2431
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-27 07:41:45 +02:00
Lubomír Sedlář
e419100d5f pkgset: Allow different inheritance for modules
With one compose combining traditional and modular content there might
be different requirements for tag inheritance. This patch adds a new
option that controls whether builds in modular tags should be inherited.
It defaults to False, which is the right option for current MBS
behavior.

JIRA: COMPOSE-2148
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-26 14:42:30 +02:00
Lubomír Sedlář
d3938d7c04 ostree: Recognize force_new_commit option in old config
Fixes: https://pagure.io/pungi/issue/925
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-26 13:22:21 +02:00
Lubomír Sedlář
68f80751cf modules: Correctly report error for unexpected modules
The code checked with `assert` that there is only one module matching
given NSV. In actual package that would not do anything and we would
silently pick the first value.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-26 09:26:37 +02:00
Lubomír Sedlář
b8555b7869 modules: Allow context in variants XML
JIRA: COMPOSE-2508
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-26 09:26:37 +02:00
Lubomír Sedlář
8c22236ad4 gather: Print profiling information to stderr
On stdout it gets mixed with listing of gathered packages. This has no
effect in a real pungi-koji run, where both streams are merged into a
single file which is then parsed, but in manual debugging runs it's a
little obnoxious.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-25 14:21:09 +02:00
Jan Kaluza
4d53a5c9ca pkgset: Stop creating database for repodata
This speeds up the compose quite a bit and there is no need for the
database anyway.

Merges: https://pagure.io/pungi/pull-request/922
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-04-25 14:19:27 +02:00
Lubomír Sedlář
eaf58f7d40 gather: Use another variant as lookaside
Create a temporary repository and add it as another lookaside in the
compose.

JIRA: COMPOSE-2426
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-23 10:51:56 +02:00
Lubomír Sedlář
ea0964eeef buildinstall: Use metadata if skipped
If the phase is skipped, it could mean that we are doing a debug run and
we don't want to mess up the .treefile by missing arch specific images.
The other alternative is that the phase was really skipped, in which
case there will be no files generated and we already handle that fine.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-23 10:42:33 +02:00
Jan Kaluza
9915c7d644 Allow reusing pkgset FileCache from old composes.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-04-23 09:57:03 +02:00
Lubomír Sedlář
1e972885f5 validation: Populate dict of all variants
Until now only list of top level variants was enough, but for variant
as lookaside we need a dict including even child variants.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-20 10:03:01 +02:00
Lubomír Sedlář
c07a4d64a1 gather: Stop pulling debuginfo and source for lookaside packages
If a package is in lookaside, we don't really need a debuginfo nor
source for it. Definitely we should not pull them from the package set
even if there is some suitable package. Therefore we should include them
in the output but marked as lookaside.

Fixes: https://pagure.io/pungi/issue/915
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-19 08:26:42 +02:00
Lubomír Sedlář
58afece033 Only use comps repo if we really have comps
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-17 15:32:02 +02:00
Lubomír Sedlář
267ff86f04 pkgset: Use modules PDC API
Instead of the deprecated and confusing unreleasedvariants.

JIRA: COMPOSE-2363
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-17 12:16:18 +02:00
Patrick Uiterwijk
527394707d Access ci_base date via compose
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2018-04-15 02:16:23 +02:00
Lubomír Sedlář
a217eea24e Allow filtering comps for different variants
We already filter for arches, so this just extends the filter to work
with variants as well.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-13 14:56:06 +02:00
Lubomír Sedlář
d7021c5688 comps: Make filtering by attribute more generic
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-13 14:56:06 +02:00
Lubomír Sedlář
44c1e2dc6f pkgset: Dump downloaded modulemd to logs
This will allow easier inspection.

JIRA: COMPOSE-2316
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-13 14:44:51 +02:00
Lubomír Sedlář
4596020ecd Fix PEP8 warning about if not x in y
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-13 14:44:51 +02:00
Ondrej Nosek
cb3d36be5d Variant as a lookaside - configuration
Relates: COMPOSE-2425

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-04-13 14:14:43 +02:00
Lubomír Sedlář
15ccd309fa Remove comps from arch repo
The comps could potentially be different in different variants, so
instead we can create the comps repo for every variant separately and
use two repos instead of one (packages in one repository, comps in
another one).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-12 13:52:36 +02:00
Lubomír Sedlář
9daaf1e038 init: Stop creating module defaults dir twice
JIRA: COMPOSE-2447
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-12 13:38:36 +02:00
Lubomír Sedlář
9f583eeb6d gather: Reduce logs from DNF gathering
The log file used to contain a dump of the configuration, which could
potentially be very large. This patch removes data that is directly
visible in the input kickstart file. Instead it just displays the number
of items.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-12 07:49:44 +02:00
Lubomír Sedlář
d1d074ce28 Clone module defaults into work/ directory
We can not rely on config_dir being writable, and should not modify
anything in there anyway.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-11 16:29:36 +02:00
Petr Šabata
8b24c7cbf3 Update the configuration JSON schema for module_defaults_dir
Signed-off-by: Petr Šabata <contyk@redhat.com>
2018-04-11 16:29:18 +02:00
Petr Šabata
8238011bdd Update configuration docs with module_defaults_dir
Signed-off-by: Petr Šabata <contyk@redhat.com>
2018-04-11 16:29:18 +02:00
Petr Šabata
b9523ff5b0 Handle relative paths in module_defaults_dir
This now handles strings as well as sct_dicts with type=file that
include relative paths.

Signed-off-by: Petr Šabata <contyk@redhat.com>
2018-04-11 16:29:18 +02:00
Petr Šabata
0e7f770fb7 Include module defaults in the repodata
If the compose configuration includes the module_defaults_dir (an
scm_dict), clone the directory, read the module defaults contained
therein and include relevant defaults in the combined modulemd file.

Only defaults for modules present in the variant are included.

This requires libmodulemd 1.2.0+.

Merges: https://pagure.io/pungi/pull-request/891
Signed-off-by: Petr Šabata <contyk@redhat.com>
2018-04-11 16:29:17 +02:00
Lubomír Sedlář
506ac99f62 Add *.in fixtures to tarball
The recent comps related commit added a fixture file with new extension
which is not getting included in the source tarball and therefore causes
tests to fail.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-10 08:33:41 +02:00
Lubomír Sedlář
8f1beeb54b init: Always filter comps file
Even for Everything we want to filter the comps file to make sure we
remove the stuff that is not compatible with current arch. All groups
are still preserved in that case.

This allows us to do the filtering once in init phase than just use the
prepared file in comps source.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-10 08:21:56 +02:00
Lubomír Sedlář
6daee968ae docs: Describe comps processing
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-10 08:21:56 +02:00
Lubomír Sedlář
aab3b04b08 gather: Use comps for given variant
The file for variant is filtered to only contain relevant parts.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-10 08:21:56 +02:00
Lubomír Sedlář
064f922117 docs: Fix typo
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-10 08:21:56 +02:00
Lubomír Sedlář
c705488505 Add all packages to whitelist for hybrid variant
If a variant contains both modules and comps groups, we need to include
builds from the compose tag in the package whitelist. However only
packages that are not already provided by any module should be added.

JIRA: COMPOSE-2435
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-09 13:10:36 +02:00
Lubomír Sedlář
de231064b7 comps: Add tests for CompsFilter
All use cases that are actually used by pungi-koji are tested. There is
missing coverage for

 * keeping only items with matching arch
 * not reindenting the file

These aren't currently used and should be removed in the future, but
there may be other tools depending on the comps_filter executable.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-09 07:48:28 +02:00
Lubomír Sedlář
7ea4c33d87 comps: Move filtering into wrapper module
The code should not live directly in the executable, that makes it very
hard to test.

Other than the move there is no functional change.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-09 07:48:28 +02:00
Ondrej Nosek
7798174b30 Tests fail if unittest2 library is missing
Merges: https://pagure.io/pungi/pull-request/894
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-04-09 07:47:17 +02:00
Robert Marshall
6454f30714 Add unittest2 and rpmdevools to contributing doc
- Add rpmdevtools and python-unittest2 to the list of packages
  required to set up a testing environment

Signed-off-by: Robert Marshall <rmarshall@redhat.com>
2018-04-06 19:27:22 -04:00
Lubomír Sedlář
f38770c67d pkgset: Construct UID for PDC modules
We can't rely on the UID to be correctly joined with colons. There may
be historical data that still uses dashes.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-06 10:47:22 +02:00