Commit Graph

2335 Commits

Author SHA1 Message Date
Lubomír Sedlář
cd493bc4c1 init: Add validation for module defaults
After cloning the repository with defaults, open each file and try to
check if there is more than one definition for the same module. It's not
a problem for the compose process, but consumers of the compose would
get confused and possibly explode. Better alert people early.

Conceptually this should be part of the test phase, but that would mean
waiting for the compose to finish before reporting the error. The
earlier the better.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-07 10:25:35 +02:00
Lubomír Sedlář
c10a4ca337 ostree-installer: Skip comps repo if there are no comps
When the compose is not using comps, we can't pass the comps to lorax,
since it doesn't exist and it would cause a crash.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-07 09:27:14 +02:00
Lubomír Sedlář
663a07068e kojiwrapper: Call chmod recursively
Related: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-04 13:11:37 +02:00
Lubomír Sedlář
71f2c61020 Add test for getting licenses from a repo
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-04 08:26:26 +02:00
Stephen Gallagher
23ced26588 Add content_licenses to module metadata
During the createrepo phase for Modular variants, this will now
interrogate the repodata from the "work" repositories for the set of
licenses in use by each of the RPMs in a module and add those to the
metadata to be written out into the final repodata location.

Merges: https://pagure.io/pungi/pull-request/968
Signed-off-by: Stephen Gallagher <sgallagh@redhat.com>
2018-06-04 08:26:17 +02:00
Lubomír Sedlář
8b9508b027 Update virtualenv instructions
* Koji can now be installed from PyPI and it will also pull in rpm. This
  needs rpm-devel install on the system. This means we can get kobo from
  PyPI as well.
* Using print function should make it work for python 3 virtualenvs.
* Newer we need kobo in 0.6.0 at least, but newer is fine too.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-01 12:11:55 +02:00
Lubomír Sedlář
102fec83b3 kojiwrapper: Don't mark runroot as successful by chmod
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-31 14:50:03 +02:00
Lubomír Sedlář
cbcebe90e1 Allow extracting koji event from another compose
When multiple composes are chained, they should reuse the same event.
However it is tricky as the value would have to be passed by hand. This
patch makes it possible to read the value from another compose (the
first one in the chain).

JIRA: COMPOSE-2571
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-31 11:11:21 +02:00
Stephen Gallagher
ec96757707 Copy modules instead of reparsing them
The Modulemd.copy() method has been available since libmodulemd 1.1
and is much faster than dumping to a string and parsing it again.

Signed-off-by: Stephen Gallagher <sgallagh@redhat.com>
Merges: https://pagure.io/pungi/pull-request/965
2018-05-31 09:04:28 +02:00
Lubomír Sedlář
49e8aa0c7e Silence config warnings in quiet mode
It's a bit unfortunate we can't use the logger for printing these, but
there is cyclic dependency: to set up logger, we need Compose object (to
generate path to log file), but to create Compose object we need a valid
config.

JIRA: COMPOSE-2577
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-30 15:03:07 +02:00
Lubomír Sedlář
92b5ad2e05 kojiwrapper: Make result of runroot world readable
The commands in runroot run as root every time. If they create files
that are not readable to other users, the reset of compose could have
problems with it if it does not run as root too. Particularly updates
composes in Bodhi run under apache user.

Relates: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-29 10:31:42 +02:00
Lubomír Sedlář
e53da69db3 osbs: Add nvr to metadata
All the components are there already separately, but having the full NVR
should simplify searching the metadata with grep.

JIRA: COMPOSE-2519
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-28 13:16:01 +02:00
Patrick Uiterwijk
f1cd1ae562 Always get old compose with release type suffix
Signed-off-by: Patrick Uiterwijk <patrick@puiterwijk.org>
Merges: https://pagure.io/pungi/pull-request/792
2018-05-25 09:49:14 +02:00
Patrick Uiterwijk
288d9ecc90 Make ostree_installer check if buildinstall is skipped correctly
The _skipped attribute is only set after the buildinstall phase is started.

Fixes: #909
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2018-05-24 13:25:34 +02:00
Ondrej Nosek
420b744193 4.1.25 release
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-05-22 14:12:31 +02:00
Lubomír Sedlář
fdea7878f1 comps-wrapper: Make tests pass on EL6
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-22 12:34:39 +02:00
Lubomír Sedlář
a4bbf475f1 pkgset: Add option to ignore noarch in ExclusiveArch
The `add_noarch` option of `get_valid_arches` is broken and doesn't
really do anything (noarch is always present in the result).

This causes packages that have ExclusiveArch including noarch to
actually not be excluded. They should be.

Changing this globally could have a very big impact. Therefore we can
hide it behind a configuration option so that it's opt-in.

JIRA: COMPOSE-2457
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-21 12:36:16 +02:00
Ondrej Nosek
c85d80f3c2 Handling multiple modules with the same NSV - PDC
Signed-off-by: Ondrej Nosek <onosek@redhat.com>

JIRA: COMPOSE-2510
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-05-21 11:24:46 +02:00
Lubomír Sedlář
b4e746aa71 createrepo: Allow disabling SQLite database
This is an optimization for Yum. DNF does not care at all.

The behaviour is configurable, but the default depends on gather
backend, as that is what users should be using to consume the packages
from the repo.

Fixes: https://pagure.io/pungi/issue/951
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-21 08:13:29 +02:00
Lubomír Sedlář
1afb709404 init: Drop database from comps repo
It should not be needed there, since the repo is empty anyway. A test is
added for the variant specific comps repo.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-21 08:13:29 +02:00
Lubomír Sedlář
89d798006b createrepo: Add module arch to metadata
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-18 08:56:22 +02:00
Lubomír Sedlář
f1b71d1eeb arch: Drop mapping ppc64 -> ppc64p7
The ppc64p7 flavour is not used anymore.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-16 16:25:58 +02:00
Lubomír Sedlář
5324c6441f arch: Make i386 map to i686 instead of athlon
This is used for mapping basearch to binary arch.

One use for this is when running depsolving to tell yum/dnf what arch to
work with. With this change it will get i686 instead of athlon when
working on i386 basearch.

The other use case is finding arches compatible with given basearch. The
change will remove athlon from the list for i386.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-16 15:22:31 +02:00
Lubomír Sedlář
2862ae0b28 ostree-installer: Use Python function to copy
This should give us better error reporting. The `copy_all` function
should preserve permissions on all files.

Relates: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-16 12:35:32 +02:00
Lubomír Sedlář
b2f995d516 Add a phase for creating extra ISOs
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-16 12:20:01 +02:00
Lubomír Sedlář
4544b454f8 Stop using .message attribute on exceptions
It does not exist on Python 3. Converting the exception to string works
identically.

The validate methods on many phases are simplified by not calling the
parent (which does not do anything).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-15 14:17:46 +02:00
Ondrej Nosek
b2190c1c3f Validation of parameter skip_phases
It checks config file value 'skip_phases' for valid phases names.
Also checks command-line attribute 'skip-phase' of 'bin/pungi-koji'.

JIRA: COMPOSE-2493

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-05-15 13:30:00 +02:00
Patrick Uiterwijk
ab2faa85b3 Capture sigterm and mark the compose as DOOMED
Merges: https://pagure.io/pungi/pull-request/946
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2018-05-15 12:36:20 +02:00
Lubomír Sedlář
057751381a createiso: Remove useless method
We call the parent, call a method without side effects and return...

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-11 12:42:21 +02:00
Lubomír Sedlář
48d0f2f643 createiso: Refactor code into smaller functions
This way some parts of the code will be reusable. This should have no
effects on the outcome, the tests still pass without any changes needed.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-11 12:42:21 +02:00
Lubomír Sedlář
fc78a3cbb3 init: Stop filtering comps environments all the time
For variants that contain all packages (Fedora's Everything) we don't
want to lose any environments.

Fixes: https://pagure.io/pungi/issue/940
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-10 10:50:21 +02:00
Patrick Uiterwijk
8f3c06bd14 Make wait-for-signed-ostree repeat the fedmsg in case the signer crashed
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2018-05-09 23:17:59 +02:00
Lubomír Sedlář
905064775a arch: Remove mocks in tests
This was added back when rpmutils module was used. Since we migrated to
Python 3 we have included the code directly in Pungi (because it was
coming from yum which is not going to Py3 any time soon). The mocks can
be dropped.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-09 13:49:29 +02:00
Lubomír Sedlář
7c237c2c63 ostree-installer: Allow overwriting buildinstall
Relates: https://pagure.io/pungi/issue/909
Relates: https://pagure.io/pungi/issue/695
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-07 15:22:01 +02:00
Lubomír Sedlář
543154d597 ostree-installer: Work with skipped buildinstall
Fixes: https://pagure.io/pungi/issue/909
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-07 12:29:20 +02:00
Lubomír Sedlář
076be762ec createrepo: Use less verbose logs
There is really no need to write out megabytes of logs that are not
really interesting. This should also help the parallelization. With the
verbose log createrepo fills the output buffer and needs to wait for the
busy python program to read it first.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-07 08:38:23 +02:00
Lubomír Sedlář
bd9a0ceda2 pkgset: Create global repo in parallel to merging pkgsets
The work in creating the repo is done in a separate process. This can
easily use two threads.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-07 08:38:23 +02:00
Lubomír Sedlář
dc557b809a ostree-installer: Copy files without owner
The files created in koji runroot will be owned by root. If the compose
is done under different user, there could be a problem with copying the
files preserving the owner. Let's just copy them without that.

Fixes: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-04 14:18:24 +02:00
Lubomír Sedlář
99b6e44a30 createiso: Skip if buildinstall fails
If the ISO is meant to be bootable but lorax fails, there's no point in
creating the ISO as it will not behave as expected.

Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1574585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-04 08:45:45 +02:00
Lubomír Sedlář
490f079c44 Update tests for libmodulemd 1.4.0
It got a little picky about allowed inputs. We need full RPM NEVRA in
all tests.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-02 15:36:52 +02:00
Lubomír Sedlář
182a5a399b 4.1.24 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-02 14:16:44 +02:00
Lubomír Sedlář
ed8fffb6d1 koji-wrapper: Log failed subtasks
If the parent task is successful, there can still be failed child tasks
for failable arches. We need to log those and potentially mark the
compose as incomplete.

Fixes: https://pagure.io/pungi/issue/874
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-02 08:27:44 +02:00
Lubomír Sedlář
f298121bbc Update compose status when config validation fails
If the config is syntactically correct, but it contains some issues
discovered in `validate` method of any phase, we need to raise an
exception, not exit immediately.

JIRA: COMPOSE-2431
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-27 07:41:45 +02:00
Lubomír Sedlář
e419100d5f pkgset: Allow different inheritance for modules
With one compose combining traditional and modular content there might
be different requirements for tag inheritance. This patch adds a new
option that controls whether builds in modular tags should be inherited.
It defaults to False, which is the right option for current MBS
behavior.

JIRA: COMPOSE-2148
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-26 14:42:30 +02:00
Lubomír Sedlář
d3938d7c04 ostree: Recognize force_new_commit option in old config
Fixes: https://pagure.io/pungi/issue/925
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-26 13:22:21 +02:00
Lubomír Sedlář
68f80751cf modules: Correctly report error for unexpected modules
The code checked with `assert` that there is only one module matching
given NSV. In actual package that would not do anything and we would
silently pick the first value.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-26 09:26:37 +02:00
Lubomír Sedlář
b8555b7869 modules: Allow context in variants XML
JIRA: COMPOSE-2508
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-26 09:26:37 +02:00
Lubomír Sedlář
8c22236ad4 gather: Print profiling information to stderr
On stdout it gets mixed with listing of gathered packages. This has no
effect in a real pungi-koji run, where both streams are merged into a
single file which is then parsed, but in manual debugging runs it's a
little obnoxious.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-25 14:21:09 +02:00
Jan Kaluza
4d53a5c9ca pkgset: Stop creating database for repodata
This speeds up the compose quite a bit and there is no need for the
database anyway.

Merges: https://pagure.io/pungi/pull-request/922
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-04-25 14:19:27 +02:00
Lubomír Sedlář
eaf58f7d40 gather: Use another variant as lookaside
Create a temporary repository and add it as another lookaside in the
compose.

JIRA: COMPOSE-2426
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-23 10:51:56 +02:00