Commit Graph

1090 Commits

Author SHA1 Message Date
Marek Marczykowski-Górecki a12c2b9ea0 Use xorriso instead of genisoimage
xorriso make the image reproducible (given the same input files),
including support for SOURCE_DATE_EPOCH in various metadata.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
2018-10-08 13:33:29 +02:00
Marek Marczykowski-Górecki 41e144e47d Use $SOURCE_DATE_EPOCH (if set) in discinfo file
This helps the output image to be reproducible.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
2018-10-08 13:33:29 +02:00
Lubomír Sedlář aab2fc4519 extra_iso: Add list of variants to metadata
The main variant is already available, this patch adds information about
additional variants that are included in the image.

JIRA: COMPOSE-2917
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-05 12:01:43 +02:00
Lubomír Sedlář 1aff7fc3ac linker: Simplify creating pool
The pool is created in multiple places, and the process is always to
create an instance and add workers to it. Let's abstract the loop in a
method.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-03 10:27:56 +02:00
Lubomír Sedlář e2e8df3f09 gather: Hide pid of fus process
There's an environment variable that can suppress printing of PID for
every debug message. It provides us with no real information and only
makes the log larger.

JIRA: COMPOSE-2973
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-03 10:23:39 +02:00
Lubomír Sedlář 4537001ff6 fus: Strip protocol from repo path
Only local paths are supported currently. As such, `file://` can be
stripped, and for anything else we should raise an exception.

JIRA: COMPOSE-2996
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-03 09:47:00 +02:00
Jan Kaluza 37c89dfde6 Add 'pkgset_koji_builds' option to include extra builds in a compose
This PR adds new pkgset_koji_builds configuration option.

This option allows setting list of extra Koji build NVRs which will be
included in a compose. This is useful in two cases:

a) It allows generating standard composes with few packages update to
certain version to test how the compose behaves when the package is
updated for real.

b) It allows generating compose consisting only from particular builds
when pkgset_koji_tag = '' or None. This is useful when one want to
regenerate the compose with packages which are not tagged in single Koji
tag. This is very useful for ODCS when reproducing old composes.

Merges: https://pagure.io/pungi/pull-request/1049
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-09-21 14:06:49 +02:00
Lubomír Sedlář 36c347eb79 ostree: Use --touch-if-changed
There are three different cases:

 * we expect commitid and it's there
 * we expect commitid and it's missing
 * we don't expect commitid

This patch helps differentiate between the second two. In former one we
should report an error and mark the phase as failed. The latter is
perfectly fine and no error should be reported

Fixes: https://pagure.io/pungi/issue/1046
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-19 13:07:03 +02:00
Lubomír Sedlář be81aeaa7d gather: Filter arches similarly to pkgset
There should not be multilib arches when checking if noarch package is
compatible with current arch. Otherwise we might be excluding a package
from x86_64 just because it does not work on i686.

JIRA: COMPOSE-2885
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-14 13:27:39 +02:00
Lubomír Sedlář 5f6ee61c70 gather: Fix multilib query for hybrid solver
The multilib library expects us to ask about the candidate package for
another arch (should I include this as multilib?) and not the current
arch (should I include other arches for this package?).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-05 16:04:40 +02:00
Lubomír Sedlář a53dc6f1bb gather: Expand multilib lists for hybrid method
If there are globs in the blacklist or whitelist, we need to expand it
to full package name.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-05 15:08:31 +02:00
Lubomír Sedlář 1350684c31 Index arch modulemd by full NSVC
There can be multiple modules with the same name and stream. They should
all have the same version, but will have different contexts. Fus takes
only N:S as input, but should pull in all matching modules. We just need
to give it correct data in the repo.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-05 13:10:53 +02:00
Lubomír Sedlář fbb739ef17 pkgset: Apply whitelist to modules in the tag
This patch changes the behaviour when both module tag and NSV?C? is
specified. The NSVC are used as a whitelist and only matching modules
will be included in the compose.

Additionally this patch adds filtering based on inheritance: when
finding the latest module for each N:S combination, only the top tag in
which the module is tagged is used. Even if a newer build is available
somewhere deeper in the inheritance, it's not going to be used.

Example inheritance and tagged modules

    f29-compose (foo:1:2018:cafe)
    └─ f29-candidate (foo:1:2019:cafe)

The compose will use 2018 version, because it's in the topmost tag.

JIRA: COMPOSE-2685
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-03 12:19:50 +02:00
Lubomír Sedlář 3419762830 extra_iso: Set unified flag in metadata
This will avoid conflict in productmd that two images share the same
attributes.

JIRA: COMPOSE-2908
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-31 08:46:21 +02:00
Lubomír Sedlář 828557b4d6 pkgset: Respect koji event when searching for modules
In the search result, we should ignore any module build that finished
after the event that we are working with.

Fixes: https://pagure.io/pungi/issue/999
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-30 14:58:45 +02:00
Jan Kaluza d2f392fac8 Use dogpile.cache to cache the listTaggedRPMS calls if possible
If the same tag is queried with the same event, Pungi can cache the
response and call the API again. Particularly for small composes this
can save up significant amount of time.

Merges: https://pagure.io/pungi/pull-request/1022
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-08-30 13:21:01 +02:00
Lubomír Sedlář 92968fe52d gather: Keep original rpms.json in debug mode
If we're running in debug mode and the file is already present, it
should not be modified. This means that in order to rerun the actual
gather phase the file needs to be manually deleted first. However the
much more common use is to skip gather phase (because only images should
be re-run). In that case the manifest will be preserved correctly.

JIRA: COMPOSE-2756
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-29 15:52:45 +02:00
Lubomír Sedlář 5346e000f0 Create non-bootable ISO for variant without buildinstall
If the configuration specifically excludes a variant from buildinstall,
but does not also disable ISO creation, we should just create a
non-bootable ISO instead of reporting a warning.

JIRA: COMPOSE-2887
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-28 08:56:03 +02:00
Lubomír Sedlář 03ee632cc8 gather: Honor module whitelist
If the modulemd contains a whitelist of packages (under buildopts), it
means the packages are possibly renamed, and we need to check that list
instead of components.rpms.

Multilib does not really work now, anything with non-native arch is
skipped.

JIRA: RCM-38019
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-27 09:08:16 +02:00
Lubomír Sedlář 6f527ae5b9 Clarify error about non-existing module
JIRA: COMPOSE-2825
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-24 13:03:18 +02:00
Lubomír Sedlář a5fa4457c3 gather: Print full unresolved dependency
We should not lose the flag and version. The name is not sufficient to
fully determine the problem.

The log information printed in `pungi/gather.py` is parsed by regex in
`pungi/wrappers/pungi.py`. It handles the dependency with spaces and
versions fine.

JIRA: COMPOSE-2880
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-24 08:38:22 +02:00
Lubomír Sedlář 9278008ba0 Fix tests on Python 2.6
It does not like the flags kwargs for re.sub(), which should not really
be needed, since \W is case insensitive anyway.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-17 10:44:32 +02:00
Lubomír Sedlář ac0d5e4ede createiso: Use correct python version
On some systems we need to use python3, in other places it can be
another version. Instead of guessing, let's look at shebang line in
lorax executable and use the same.

JIRA: COMPOSE-2852
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-15 08:37:12 +02:00
Lubomír Sedlář b85cd7ff9f ostree: Update tests for working with YAML file
Merges: https://pagure.io/pungi/pull-request/1019
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-14 16:13:23 +02:00
Colin Walters b5d5e8da4a pungi/ostree: Convert rpm-ostree YAML to JSON
We'd like to use YAML for future rpm-ostree work; among
other things it supports comments.

See https://pagure.io/fedora-atomic/pull-request/125
and the original https://github.com/projectatomic/rpm-ostree/pull/1377

Signed-off-by: Colin Walters <walters@verbum.org>
2018-08-14 16:13:23 +02:00
Lubomír Sedlář 2fa1f09827 createrepo: Allow passing arbitrary arguments
This could be used to enable zchunk generation, which can require up to
4 different options. Instead of hardcoding every single one, let's just
allow more direct access to the executed command.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-08 16:15:51 +02:00
Lubomír Sedlář b12deab153 gather: Get modular packages from fus
Fus returns also RPMs in modules, but until latest version it only
worked if the package was in the same repo as the metadata. This changed
in latest version and now Pungi does not need to expand the list
anymore.

JIRA: COMPOSE-2779
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-03 08:00:37 +02:00
Lubomír Sedlář 5926858b58 util: Remove escaping spaces from volume ID
This was not configurable for users, and the default was always used,
which meant no escaping. Might as well just remove the dead code.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-03 07:57:09 +02:00
Lubomír Sedlář add9835b56 Allow removing non-alnum chars from volid
This mimics similar change in lorax.

JIRA: RCM-36970
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-03 07:57:09 +02:00
Lubomír Sedlář 10bdb370ba extra-isos: Include treeinfo pointing to all variants
This will be used by Anaconda to consume multiple repos for
installation.

JIRA: RCM-36970
JIRA: COMPOSE-2753
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>

Fixup
2018-08-03 07:57:09 +02:00
Lubomír Sedlář 90291d7c73 createiso: Use unique paths for breaking hardlinks
If the data needs to be split into multiple ISOs, we need to make sure
the paths are unique for each image. Otherwise all files will be copied
into the same directory, and once the first image is finished, the whole
staging dir is deleting. That obviously breaks the tasks that are still
in progress.

JIRA: COMPOSE-2610
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-02 14:53:18 +02:00
Lubomír Sedlář 4d4c6555e2 gather: Detect hybrid variant with additional packages
The check if a variant is hybrid (or modular only) currently only looks
at comps groups. However it's possible there will be no comps groups,
but packages will be listed explicitly in config as additional_packages.

Relates: RCM-37979
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-26 17:11:13 -04:00
Mohan Boddu 80c3fd1170 Include exact version of pungi in the logs
Fixes #990

Signed-off-by: Mohan Boddu <mboddu@bhujji.com>
2018-07-26 17:03:58 -04:00
Lubomír Sedlář 4ecf75295a gather: Allow empty result for gather
This only affects Yum backend, and allows to mention packages in comps
groups before they become available. This means a compose can be set up
ahead of all packages being available, and it will simply produce empty
repos.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-23 09:38:39 -04:00
Lubomír Sedlář b772d4a773 gather: Add langpacks in hybrid solver
Comps file specifies a pattern for some packages. If that package is
installed, all packages matching the pattern are added as well. This can
be added to fus as another pass.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-23 09:08:33 -04:00
Lubomír Sedlář 16ac225013 comps: Add get_langpacks function
This also simplifies the test data by removing a lot of the langpacks.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-23 09:08:33 -04:00
Lubomír Sedlář 9b576bf976 pungi-legacy: Add --joliet-long option
Using `-J` each path component can only be 64 characters long. This is
not sufficient for some packages. Adding `--joliet-long` increases the
limit to 103.

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1605103
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-20 10:17:38 +02:00
Lubomír Sedlář 7d099012aa Fix tests for DNF 3
The configuration for modules has changed.

Lookaside handling has changed, and there are now test failures. This
is a not a bug in DNF, so we need to fix it on our side.

Relates: https://bugzilla.redhat.com/show_bug.cgi?id=1603123
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-19 14:22:10 +02:00
Lubomír Sedlář 9ca454007a gather: Early exit for non-comps sources
When getting list of initial packages, only run the source and do
nothing else. Additional package, system-release etc. will be added only
to comps.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-19 09:11:35 +02:00
Jan Kaluza 6080b45178 Fix Koji search for modules with dash in stream
MBS replaces the dash by underscore. We have to match that.

Merges: https://pagure.io/pungi/pull-request/1005
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-07-18 10:31:19 +02:00
Lubomír Sedlář 23c454ff67 buildinstall: Make output world readable
This makes it possible to run a compose as non-root user, plus removes
the need for workarounds to publish the results directly.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-17 10:12:52 +02:00
Lubomír Sedlář 60917fdc77 buildinstall: Copy file without preserving owner
The files are generally owned by root. If the compose is running as
root, this will still create files owned by root. If it's running as
non-priviledged user, it will crash. With this patch it will work.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-17 10:12:52 +02:00
Lubomír Sedlář e3de4dcccf Report failed failable deliverables as errors
It does not abort the compose, but it's still an error. It should be
marked as such in the log. This will allow easier searching for the
failures.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-17 09:31:50 +02:00
Lubomír Sedlář 878eaeaaf6 Stop importing PDCClient
We no longer use it anywhere, so drop the import.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-16 15:52:42 +02:00
Lubomír Sedlář ba0193ca28 gather: Add a hybrid depsolver backend
This patch adds a new gather method called `hybrid`, which uses a `fus`
binary, which must exist somewhere on the `$PATH`. It will call it
multiple times to add multilib packages.

The solver can handle packages, modules and comps groups as input.
However comps groups are never passed in. Pungi will expand it to a list
of packages to avoid issues with comps handling in fus. It ignores
optional packages, and if the group mentions a package that does not
exist, nothing else from the group is included.

Multilib is also handled outside of fus. Pungi will run it, parse the
packages from output, determines multilib packages and adds them as
input. Then it runs the solver again. This is done until nothing new is
added. Usually two passes should be enough.

Source packages and debuginfo are added as a final step. All debuginfo
packages from any included source are added. If the source or debuginfo
package is included in any lookaside repo, it will be skipped.

The tool expects to get a platform stream that should be provided for
modules to depend on. Pungi looks into the modules and gets the platform
from there. If there are more requests, an error is raised.

There is some missing functionality and options that are ignored.
Particularly these are:

 * gather_fulltree
 * gather_selfhosting
 * greedy_method

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-16 10:21:41 +02:00
Lubomír Sedlář 2c6b784f70 Always use lookasides for repoclosure
There's no point in checking for a layered release first. If there are
no repos, the loop will simply not execute even once. If there are
lookasides configured, we want to use them no matter if the release is
layered or not.

Also the log is updated to include the actual command for easier
debugging next time.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-16 08:50:34 +02:00
Lubomír Sedlář d2849d3826 osbs: Generate unique repo names
And include variant in repo file name. The whole path is unique already,
but not the filename itself.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-09 13:21:58 +02:00
Sinny Kumari 1759c1ba80 Expand version field during image_build using version_generator
We can specify !VERSION_FROM_VERSION in version field during
image_build to expand it to correct release number without any label
information.

Also implemented !RELEASE_FROM_DATE_RESPIN to provide correct
release number. This helps to keep Atomic Host media files name
produced by image_build during bodhi updates compose run
consistent with nightly run.

Fixes: https://pagure.io/pungi/issue/987
Merges: https://pagure.io/pungi/pull-request/995

Signed-off-by: Sinny Kumari <sinny@redhat.com>
2018-07-09 12:56:00 +02:00
Lubomír Sedlář d29e56c7d8 createrepo: Stop including modulemd in debug repos
Debug packages are usually installed by exact NVR, and having two repos
providing the same module (but with different packages) is confusing for
the tooling.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-27 13:35:58 +02:00
Lubomír Sedlář 6c14236562 Simplify iterating over module defaults
There are now two places where we need to do this, so we can simplify
the logic of finding and filtering them.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-27 12:19:12 +02:00
Lubomír Sedlář 98e7106f3e pkgset: Apply module filters on pkgset level
If a module says to filter a package out, we can do it immediately when
getting the build information from Koji.

This avoids a possible problem of something pulling the module package
in as a dependency, but it should also make the package set slightly
smaller.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-25 15:15:42 +02:00
Lubomír Sedlář 470c0ab3be init: Validate whitespace in comps groups
If a package name contains leading or trailing whitespace, it will
eventually lead to issues: pungi will try to include that group, but
since it does not exist, the packages will not make it in.

The root cause is hard to find. Better report an error immediately.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-25 11:58:52 +02:00
Lubomír Sedlář 096075848b createrepo: Include empty modules
A module without any RPMs is still valid and should be included. It
should also be in the metadata.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-22 12:54:39 +02:00
Lubomír Sedlář 38f1a8509e createiso: Break hardlinks by copying files
If a file has multiple hard links, genisoimage will put the wrong number
on the ISO. This patch can work around it by copying hard-linked files
into a temporary staging directory.

JIRA: COMPOSE-2610
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-20 12:40:43 +02:00
Martin Curlej d8c03f6239 pkgset: Query Koji instead of PDC
PDC is deprecated in upstream. The usecase for getting list of modules
by NS, NSV or NSVC can however be satisfied by querying modules imported
into Koji.

This makes it possible to deprecate PDC configuration.

Merges: https://pagure.io/pungi/pull-request/985
Signed-off-by: Martin Curlej <mcurlej@redhat.com>
2018-06-18 15:17:04 +02:00
Lubomír Sedlář 4a61de1e8a variants: Reject values with whitespace
If there is leading or trailing whitespace in a comps group name, it
will not be included in the compose and there will even be no error
message. Whitespace on module name results in a failure.

To avoid these errors, validating the variants file will now also check
that there is no whitespace in significant places, and abort the compose
if there a problem.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-12 12:48:14 +02:00
Lubomír Sedlář fa92e54c22 osbs: Fresh koji session for getting metadata
The task can take a while to finish, and it's possible for the
authenticated session to expire in the mean time. Watching the task will
work, because that happens by spawning `koji watch-task` as subprocess.

We can work around this by creating a fresh unauthenticated session for
getting the task results.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-11 10:27:39 +02:00
Lubomír Sedlář 004ef31917 gather: Ignore comps in lookaside repo
This fixes a case where extra packages are pulled in.

The scenario is:

 * there's a lookaside repo which contains group G which has package P
 * we want to pull group G into the compose, but our definition of G
   does not contain P
 * lookaside repo does not contain package P
 * current package set has P

DNF depsolver will then merge the two definitions and try to get all the
packages. For most cases this is not a problem, since the package is in
the lookaside repo and will not be pulled into the compose. But in the
example above since P is not in lookaside, Pungi will put it into
current compose.

This is also ugly in the depsolving log says it includes package P
because it was in input. But checking comps file does not show it.

The result of this change is that some packages might disappear from
current composes. This can only happen if there is a lookaside which
defines groups with similar IDs, which should be very rare. In cases
where this would be a problem the fix is to explicitly add wanted
packages either to comps or to additional packages.

Fixes: https://pagure.io/pungi/issue/978
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-08 13:32:24 +02:00
Lubomír Sedlář cd493bc4c1 init: Add validation for module defaults
After cloning the repository with defaults, open each file and try to
check if there is more than one definition for the same module. It's not
a problem for the compose process, but consumers of the compose would
get confused and possibly explode. Better alert people early.

Conceptually this should be part of the test phase, but that would mean
waiting for the compose to finish before reporting the error. The
earlier the better.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-07 10:25:35 +02:00
Lubomír Sedlář c10a4ca337 ostree-installer: Skip comps repo if there are no comps
When the compose is not using comps, we can't pass the comps to lorax,
since it doesn't exist and it would cause a crash.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-07 09:27:14 +02:00
Lubomír Sedlář 663a07068e kojiwrapper: Call chmod recursively
Related: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-04 13:11:37 +02:00
Stephen Gallagher 23ced26588 Add content_licenses to module metadata
During the createrepo phase for Modular variants, this will now
interrogate the repodata from the "work" repositories for the set of
licenses in use by each of the RPMs in a module and add those to the
metadata to be written out into the final repodata location.

Merges: https://pagure.io/pungi/pull-request/968
Signed-off-by: Stephen Gallagher <sgallagh@redhat.com>
2018-06-04 08:26:17 +02:00
Lubomír Sedlář 102fec83b3 kojiwrapper: Don't mark runroot as successful by chmod
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-31 14:50:03 +02:00
Lubomír Sedlář cbcebe90e1 Allow extracting koji event from another compose
When multiple composes are chained, they should reuse the same event.
However it is tricky as the value would have to be passed by hand. This
patch makes it possible to read the value from another compose (the
first one in the chain).

JIRA: COMPOSE-2571
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-31 11:11:21 +02:00
Stephen Gallagher ec96757707 Copy modules instead of reparsing them
The Modulemd.copy() method has been available since libmodulemd 1.1
and is much faster than dumping to a string and parsing it again.

Signed-off-by: Stephen Gallagher <sgallagh@redhat.com>
Merges: https://pagure.io/pungi/pull-request/965
2018-05-31 09:04:28 +02:00
Lubomír Sedlář 92b5ad2e05 kojiwrapper: Make result of runroot world readable
The commands in runroot run as root every time. If they create files
that are not readable to other users, the reset of compose could have
problems with it if it does not run as root too. Particularly updates
composes in Bodhi run under apache user.

Relates: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-29 10:31:42 +02:00
Lubomír Sedlář e53da69db3 osbs: Add nvr to metadata
All the components are there already separately, but having the full NVR
should simplify searching the metadata with grep.

JIRA: COMPOSE-2519
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-28 13:16:01 +02:00
Patrick Uiterwijk f1cd1ae562 Always get old compose with release type suffix
Signed-off-by: Patrick Uiterwijk <patrick@puiterwijk.org>
Merges: https://pagure.io/pungi/pull-request/792
2018-05-25 09:49:14 +02:00
Patrick Uiterwijk 288d9ecc90 Make ostree_installer check if buildinstall is skipped correctly
The _skipped attribute is only set after the buildinstall phase is started.

Fixes: #909
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2018-05-24 13:25:34 +02:00
Lubomír Sedlář a4bbf475f1 pkgset: Add option to ignore noarch in ExclusiveArch
The `add_noarch` option of `get_valid_arches` is broken and doesn't
really do anything (noarch is always present in the result).

This causes packages that have ExclusiveArch including noarch to
actually not be excluded. They should be.

Changing this globally could have a very big impact. Therefore we can
hide it behind a configuration option so that it's opt-in.

JIRA: COMPOSE-2457
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-21 12:36:16 +02:00
Ondrej Nosek c85d80f3c2 Handling multiple modules with the same NSV - PDC
Signed-off-by: Ondrej Nosek <onosek@redhat.com>

JIRA: COMPOSE-2510
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-05-21 11:24:46 +02:00
Lubomír Sedlář b4e746aa71 createrepo: Allow disabling SQLite database
This is an optimization for Yum. DNF does not care at all.

The behaviour is configurable, but the default depends on gather
backend, as that is what users should be using to consume the packages
from the repo.

Fixes: https://pagure.io/pungi/issue/951
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-21 08:13:29 +02:00
Lubomír Sedlář 1afb709404 init: Drop database from comps repo
It should not be needed there, since the repo is empty anyway. A test is
added for the variant specific comps repo.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-21 08:13:29 +02:00
Lubomír Sedlář 89d798006b createrepo: Add module arch to metadata
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-18 08:56:22 +02:00
Lubomír Sedlář f1b71d1eeb arch: Drop mapping ppc64 -> ppc64p7
The ppc64p7 flavour is not used anymore.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-16 16:25:58 +02:00
Lubomír Sedlář 5324c6441f arch: Make i386 map to i686 instead of athlon
This is used for mapping basearch to binary arch.

One use for this is when running depsolving to tell yum/dnf what arch to
work with. With this change it will get i686 instead of athlon when
working on i386 basearch.

The other use case is finding arches compatible with given basearch. The
change will remove athlon from the list for i386.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-16 15:22:31 +02:00
Lubomír Sedlář 2862ae0b28 ostree-installer: Use Python function to copy
This should give us better error reporting. The `copy_all` function
should preserve permissions on all files.

Relates: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-16 12:35:32 +02:00
Lubomír Sedlář b2f995d516 Add a phase for creating extra ISOs
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-16 12:20:01 +02:00
Lubomír Sedlář 4544b454f8 Stop using .message attribute on exceptions
It does not exist on Python 3. Converting the exception to string works
identically.

The validate methods on many phases are simplified by not calling the
parent (which does not do anything).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-15 14:17:46 +02:00
Ondrej Nosek b2190c1c3f Validation of parameter skip_phases
It checks config file value 'skip_phases' for valid phases names.
Also checks command-line attribute 'skip-phase' of 'bin/pungi-koji'.

JIRA: COMPOSE-2493

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-05-15 13:30:00 +02:00
Patrick Uiterwijk ab2faa85b3 Capture sigterm and mark the compose as DOOMED
Merges: https://pagure.io/pungi/pull-request/946
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2018-05-15 12:36:20 +02:00
Lubomír Sedlář 057751381a createiso: Remove useless method
We call the parent, call a method without side effects and return...

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-11 12:42:21 +02:00
Lubomír Sedlář 48d0f2f643 createiso: Refactor code into smaller functions
This way some parts of the code will be reusable. This should have no
effects on the outcome, the tests still pass without any changes needed.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-11 12:42:21 +02:00
Lubomír Sedlář fc78a3cbb3 init: Stop filtering comps environments all the time
For variants that contain all packages (Fedora's Everything) we don't
want to lose any environments.

Fixes: https://pagure.io/pungi/issue/940
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-10 10:50:21 +02:00
Lubomír Sedlář 7c237c2c63 ostree-installer: Allow overwriting buildinstall
Relates: https://pagure.io/pungi/issue/909
Relates: https://pagure.io/pungi/issue/695
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-07 15:22:01 +02:00
Lubomír Sedlář 543154d597 ostree-installer: Work with skipped buildinstall
Fixes: https://pagure.io/pungi/issue/909
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-07 12:29:20 +02:00
Lubomír Sedlář 076be762ec createrepo: Use less verbose logs
There is really no need to write out megabytes of logs that are not
really interesting. This should also help the parallelization. With the
verbose log createrepo fills the output buffer and needs to wait for the
busy python program to read it first.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-07 08:38:23 +02:00
Lubomír Sedlář bd9a0ceda2 pkgset: Create global repo in parallel to merging pkgsets
The work in creating the repo is done in a separate process. This can
easily use two threads.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-07 08:38:23 +02:00
Lubomír Sedlář dc557b809a ostree-installer: Copy files without owner
The files created in koji runroot will be owned by root. If the compose
is done under different user, there could be a problem with copying the
files preserving the owner. Let's just copy them without that.

Fixes: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-04 14:18:24 +02:00
Lubomír Sedlář 99b6e44a30 createiso: Skip if buildinstall fails
If the ISO is meant to be bootable but lorax fails, there's no point in
creating the ISO as it will not behave as expected.

Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1574585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-04 08:45:45 +02:00
Lubomír Sedlář ed8fffb6d1 koji-wrapper: Log failed subtasks
If the parent task is successful, there can still be failed child tasks
for failable arches. We need to log those and potentially mark the
compose as incomplete.

Fixes: https://pagure.io/pungi/issue/874
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-02 08:27:44 +02:00
Lubomír Sedlář e419100d5f pkgset: Allow different inheritance for modules
With one compose combining traditional and modular content there might
be different requirements for tag inheritance. This patch adds a new
option that controls whether builds in modular tags should be inherited.
It defaults to False, which is the right option for current MBS
behavior.

JIRA: COMPOSE-2148
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-26 14:42:30 +02:00
Lubomír Sedlář d3938d7c04 ostree: Recognize force_new_commit option in old config
Fixes: https://pagure.io/pungi/issue/925
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-26 13:22:21 +02:00
Lubomír Sedlář 68f80751cf modules: Correctly report error for unexpected modules
The code checked with `assert` that there is only one module matching
given NSV. In actual package that would not do anything and we would
silently pick the first value.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-26 09:26:37 +02:00
Lubomír Sedlář b8555b7869 modules: Allow context in variants XML
JIRA: COMPOSE-2508
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-26 09:26:37 +02:00
Lubomír Sedlář 8c22236ad4 gather: Print profiling information to stderr
On stdout it gets mixed with listing of gathered packages. This has no
effect in a real pungi-koji run, where both streams are merged into a
single file which is then parsed, but in manual debugging runs it's a
little obnoxious.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-25 14:21:09 +02:00
Jan Kaluza 4d53a5c9ca pkgset: Stop creating database for repodata
This speeds up the compose quite a bit and there is no need for the
database anyway.

Merges: https://pagure.io/pungi/pull-request/922
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-04-25 14:19:27 +02:00
Lubomír Sedlář eaf58f7d40 gather: Use another variant as lookaside
Create a temporary repository and add it as another lookaside in the
compose.

JIRA: COMPOSE-2426
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-23 10:51:56 +02:00
Lubomír Sedlář ea0964eeef buildinstall: Use metadata if skipped
If the phase is skipped, it could mean that we are doing a debug run and
we don't want to mess up the .treefile by missing arch specific images.
The other alternative is that the phase was really skipped, in which
case there will be no files generated and we already handle that fine.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-23 10:42:33 +02:00
Jan Kaluza 9915c7d644 Allow reusing pkgset FileCache from old composes.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-04-23 09:57:03 +02:00