Commit Graph

2015 Commits

Author SHA1 Message Date
Marek Marczykowski-Górecki
d6e72c8e61 Set repodata mtime to SOURCE_DATE_EPOCH
repodata/repomd.xml include timestamps of all the other repodata files.
Even when those files are created reproducibly, they have current
modification time. In general case this is a good thing (ease checking
if repodata cache is up to date). But in case of composing installation
image, it breaks reproducibility.
Avoid this by instructing createrepo to set mtime and revision to
$SOURCE_DATE_EPOCH.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
2018-10-08 13:33:29 +02:00
Marek Marczykowski-Górecki
2aeb8de459 Make sure .treeinfo file is sorted
OrderedDict used by default by ConfigParser isn't enough because order
of entries being added may not be deterministic (depends on directory
list order). To solve this problem, use SortedDict as a base.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
2018-10-08 13:33:29 +02:00
Marek Marczykowski-Górecki
d469cbfef5 Use constant MBR ID for isohybrid
If not set explicitly, isohybrid choose it randomly, which harm
reproducibility.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
2018-10-08 13:33:29 +02:00
Marek Marczykowski-Górecki
a12c2b9ea0 Use xorriso instead of genisoimage
xorriso make the image reproducible (given the same input files),
including support for SOURCE_DATE_EPOCH in various metadata.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
2018-10-08 13:33:29 +02:00
Marek Marczykowski-Górecki
41e144e47d Use $SOURCE_DATE_EPOCH (if set) in discinfo file
This helps the output image to be reproducible.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
2018-10-08 13:33:29 +02:00
Lubomír Sedlář
cb33b0278d unified_isos: Add extra variants to metadata
Each unified ISO contains all toplevel variants, and they are listed in
the metadata multiple times (once for each variant). In each case the
metadata should include all other variants that are included on the
image.

JIRA: COMPOSE-2918
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-05 12:01:43 +02:00
Lubomír Sedlář
aab2fc4519 extra_iso: Add list of variants to metadata
The main variant is already available, this patch adds information about
additional variants that are included in the image.

JIRA: COMPOSE-2917
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-05 12:01:43 +02:00
Lubomír Sedlář
1aff7fc3ac linker: Simplify creating pool
The pool is created in multiple places, and the process is always to
create an instance and add workers to it. Let's abstract the loop in a
method.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-03 10:27:56 +02:00
Lubomír Sedlář
e2e8df3f09 gather: Hide pid of fus process
There's an environment variable that can suppress printing of PID for
every debug message. It provides us with no real information and only
makes the log larger.

JIRA: COMPOSE-2973
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-03 10:23:39 +02:00
Lubomír Sedlář
4537001ff6 fus: Strip protocol from repo path
Only local paths are supported currently. As such, `file://` can be
stripped, and for anything else we should raise an exception.

JIRA: COMPOSE-2996
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-03 09:47:00 +02:00
Jan Kaluza
37c89dfde6 Add 'pkgset_koji_builds' option to include extra builds in a compose
This PR adds new pkgset_koji_builds configuration option.

This option allows setting list of extra Koji build NVRs which will be
included in a compose. This is useful in two cases:

a) It allows generating standard composes with few packages update to
certain version to test how the compose behaves when the package is
updated for real.

b) It allows generating compose consisting only from particular builds
when pkgset_koji_tag = '' or None. This is useful when one want to
regenerate the compose with packages which are not tagged in single Koji
tag. This is very useful for ODCS when reproducing old composes.

Merges: https://pagure.io/pungi/pull-request/1049
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-09-21 14:06:49 +02:00
Lubomír Sedlář
2a65b8fb7d ostree: Reduce duplication in tests
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-19 13:07:03 +02:00
Lubomír Sedlář
36c347eb79 ostree: Use --touch-if-changed
There are three different cases:

 * we expect commitid and it's there
 * we expect commitid and it's missing
 * we don't expect commitid

This patch helps differentiate between the second two. In former one we
should report an error and mark the phase as failed. The latter is
perfectly fine and no error should be reported

Fixes: https://pagure.io/pungi/issue/1046
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-19 13:07:03 +02:00
Lubomír Sedlář
93bc843682 ostree: Fix handler crash without commit ID
If there is no new commit in the repo, we should not wait for a
signature, as there is nothing to sign.

Fixes: https://pagure.io/pungi/issue/1046
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-19 13:07:03 +02:00
Lubomír Sedlář
be81aeaa7d gather: Filter arches similarly to pkgset
There should not be multilib arches when checking if noarch package is
compatible with current arch. Otherwise we might be excluding a package
from x86_64 just because it does not work on i686.

JIRA: COMPOSE-2885
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-14 13:27:39 +02:00
Peter Robinson
e580ad6db1 Stop shipping and remove RELEASE-NOTES
The RELEASE-NOTES hasn't been updated in years and when it was it was
only done a handful of times, clearly no one has complained, and git
history is likely more useful so just drop it.

Signed-off-by: Peter Robinson <pbrobinson@gmail.com>
2018-09-13 10:30:17 +01:00
Lubomír Sedlář
21b8f3af3c 4.1.28 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-06 13:43:04 +02:00
Lubomír Sedlář
5f6ee61c70 gather: Fix multilib query for hybrid solver
The multilib library expects us to ask about the candidate package for
another arch (should I include this as multilib?) and not the current
arch (should I include other arches for this package?).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-05 16:04:40 +02:00
Lubomír Sedlář
a53dc6f1bb gather: Expand multilib lists for hybrid method
If there are globs in the blacklist or whitelist, we need to expand it
to full package name.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-05 15:08:31 +02:00
Lubomír Sedlář
1350684c31 Index arch modulemd by full NSVC
There can be multiple modules with the same name and stream. They should
all have the same version, but will have different contexts. Fus takes
only N:S as input, but should pull in all matching modules. We just need
to give it correct data in the repo.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-05 13:10:53 +02:00
Lubomír Sedlář
fbb739ef17 pkgset: Apply whitelist to modules in the tag
This patch changes the behaviour when both module tag and NSV?C? is
specified. The NSVC are used as a whitelist and only matching modules
will be included in the compose.

Additionally this patch adds filtering based on inheritance: when
finding the latest module for each N:S combination, only the top tag in
which the module is tagged is used. Even if a newer build is available
somewhere deeper in the inheritance, it's not going to be used.

Example inheritance and tagged modules

    f29-compose (foo:1:2018:cafe)
    └─ f29-candidate (foo:1:2019:cafe)

The compose will use 2018 version, because it's in the topmost tag.

JIRA: COMPOSE-2685
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-03 12:19:50 +02:00
Lubomír Sedlář
833ba64c51 ostree: Wait for updated ref as well as signature
Fixes: https://pagure.io/pungi/issue/1036
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-03 09:54:12 +02:00
Lubomír Sedlář
3419762830 extra_iso: Set unified flag in metadata
This will avoid conflict in productmd that two images share the same
attributes.

JIRA: COMPOSE-2908
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-31 08:46:21 +02:00
Lubomír Sedlář
828557b4d6 pkgset: Respect koji event when searching for modules
In the search result, we should ignore any module build that finished
after the event that we are working with.

Fixes: https://pagure.io/pungi/issue/999
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-30 14:58:45 +02:00
Jan Kaluza
d2f392fac8 Use dogpile.cache to cache the listTaggedRPMS calls if possible
If the same tag is queried with the same event, Pungi can cache the
response and call the API again. Particularly for small composes this
can save up significant amount of time.

Merges: https://pagure.io/pungi/pull-request/1022
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-08-30 13:21:01 +02:00
Lubomír Sedlář
92968fe52d gather: Keep original rpms.json in debug mode
If we're running in debug mode and the file is already present, it
should not be modified. This means that in order to rerun the actual
gather phase the file needs to be manually deleted first. However the
much more common use is to skip gather phase (because only images should
be re-run). In that case the manifest will be preserved correctly.

JIRA: COMPOSE-2756
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-29 15:52:45 +02:00
Lubomír Sedlář
7c5020e82d Reduce duplication in tests
Set the dummy compose into non-debug mode in a single place, and only
change it when needed.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-29 15:52:45 +02:00
Lubomír Sedlář
19a42792db docs: Add better description for package globs
This is used in multiple places, but not defined anywhere.

Fixes: https://pagure.io/pungi/issue/1028
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-29 13:26:56 +02:00
Lubomír Sedlář
5346e000f0 Create non-bootable ISO for variant without buildinstall
If the configuration specifically excludes a variant from buildinstall,
but does not also disable ISO creation, we should just create a
non-bootable ISO instead of reporting a warning.

JIRA: COMPOSE-2887
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-28 08:56:03 +02:00
Lubomír Sedlář
45fa9d3273 Clean up after yum tests
The tests create empty directories in current working directory, but
they never clean them up. Instead we can switch to a temporary location
that will be cleaned up automatically.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-27 09:16:40 +02:00
Lubomír Sedlář
03ee632cc8 gather: Honor module whitelist
If the modulemd contains a whitelist of packages (under buildopts), it
means the packages are possibly renamed, and we need to check that list
instead of components.rpms.

Multilib does not really work now, anything with non-native arch is
skipped.

JIRA: RCM-38019
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-27 09:08:16 +02:00
Lubomír Sedlář
6f527ae5b9 Clarify error about non-existing module
JIRA: COMPOSE-2825
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-24 13:03:18 +02:00
Lubomír Sedlář
a5fa4457c3 gather: Print full unresolved dependency
We should not lose the flag and version. The name is not sufficient to
fully determine the problem.

The log information printed in `pungi/gather.py` is parsed by regex in
`pungi/wrappers/pungi.py`. It handles the dependency with spaces and
versions fine.

JIRA: COMPOSE-2880
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-24 08:38:22 +02:00
Lubomír Sedlář
9278008ba0 Fix tests on Python 2.6
It does not like the flags kwargs for re.sub(), which should not really
be needed, since \W is case insensitive anyway.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-17 10:44:32 +02:00
Lubomír Sedlář
0c38ecdc85 Include all test data in tarball
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-17 09:34:56 +02:00
Lubomír Sedlář
5be2798c6b 4.1.27 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-17 08:37:41 +02:00
Lubomír Sedlář
4b05e7b9fd extra-iso: Rename test data file
The file is not being included in the source tarball, which breaks
internal jenkins and will eventually cause the test to fail at RPM build
time.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-15 14:57:43 +02:00
Lubomír Sedlář
ac0d5e4ede createiso: Use correct python version
On some systems we need to use python3, in other places it can be
another version. Instead of guessing, let's look at shebang line in
lorax executable and use the same.

JIRA: COMPOSE-2852
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-15 08:37:12 +02:00
Lubomír Sedlář
b85cd7ff9f ostree: Update tests for working with YAML file
Merges: https://pagure.io/pungi/pull-request/1019
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-14 16:13:23 +02:00
Colin Walters
b5d5e8da4a pungi/ostree: Convert rpm-ostree YAML to JSON
We'd like to use YAML for future rpm-ostree work; among
other things it supports comments.

See https://pagure.io/fedora-atomic/pull-request/125
and the original https://github.com/projectatomic/rpm-ostree/pull/1377

Signed-off-by: Colin Walters <walters@verbum.org>
2018-08-14 16:13:23 +02:00
Lubomír Sedlář
2fa1f09827 createrepo: Allow passing arbitrary arguments
This could be used to enable zchunk generation, which can require up to
4 different options. Instead of hardcoding every single one, let's just
allow more direct access to the executed command.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-08 16:15:51 +02:00
Lubomír Sedlář
b12deab153 gather: Get modular packages from fus
Fus returns also RPMs in modules, but until latest version it only
worked if the package was in the same repo as the metadata. This changed
in latest version and now Pungi does not need to expand the list
anymore.

JIRA: COMPOSE-2779
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-03 08:00:37 +02:00
Lubomír Sedlář
5926858b58 util: Remove escaping spaces from volume ID
This was not configurable for users, and the default was always used,
which meant no escaping. Might as well just remove the dead code.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-03 07:57:09 +02:00
Lubomír Sedlář
add9835b56 Allow removing non-alnum chars from volid
This mimics similar change in lorax.

JIRA: RCM-36970
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-03 07:57:09 +02:00
Lubomír Sedlář
10bdb370ba extra-isos: Include treeinfo pointing to all variants
This will be used by Anaconda to consume multiple repos for
installation.

JIRA: RCM-36970
JIRA: COMPOSE-2753
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>

Fixup
2018-08-03 07:57:09 +02:00
Lubomír Sedlář
90291d7c73 createiso: Use unique paths for breaking hardlinks
If the data needs to be split into multiple ISOs, we need to make sure
the paths are unique for each image. Otherwise all files will be copied
into the same directory, and once the first image is finished, the whole
staging dir is deleting. That obviously breaks the tasks that are still
in progress.

JIRA: COMPOSE-2610
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-02 14:53:18 +02:00
Lubomír Sedlář
4d4c6555e2 gather: Detect hybrid variant with additional packages
The check if a variant is hybrid (or modular only) currently only looks
at comps groups. However it's possible there will be no comps groups,
but packages will be listed explicitly in config as additional_packages.

Relates: RCM-37979
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-26 17:11:13 -04:00
Mohan Boddu
80c3fd1170 Include exact version of pungi in the logs
Fixes #990

Signed-off-by: Mohan Boddu <mboddu@bhujji.com>
2018-07-26 17:03:58 -04:00
Lubomír Sedlář
4ecf75295a gather: Allow empty result for gather
This only affects Yum backend, and allows to mention packages in comps
groups before they become available. This means a compose can be set up
ahead of all packages being available, and it will simply produce empty
repos.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-23 09:38:39 -04:00
Lubomír Sedlář
b772d4a773 gather: Add langpacks in hybrid solver
Comps file specifies a pattern for some packages. If that package is
installed, all packages matching the pattern are added as well. This can
be added to fus as another pass.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-23 09:08:33 -04:00