Commit Graph

2864 Commits

Author SHA1 Message Date
Lubomír Sedlář
ca7d6256e5 Move resolving git reference to config validation
Instead of multiple places handling the same thing duplicating the
logic, it's better to do it once upfront. This allows easy caching of
the results.

Additional advantage of this approach is that the config dump will
include resolved URLs. The original reference will still be available in
the copy of the original config.

JIRA: COMPOSE-3065
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-21 13:25:35 +01:00
Lubomír Sedlář
444af0396e util: Add a cache for resolved git urls
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-21 10:15:52 +01:00
Mohan Boddu
4864a0f58e Copy config files into logs/global/config-copy/ directory
Fixes: https://pagure.io/pungi/issue/994
Merges: https://pagure.io/pungi/pull-request/1013
Signed-off-by: Mohan Boddu <mboddu@bhujji.com>
2018-11-19 14:42:38 +01:00
Lubomír Sedlář
7e923d3823 Remove timestamp from config dump
This makes it hard to programatically access the dump.

The timestamp was included originally to make sure that restarting a
compose in debug mode will preserve all configs. However we don't really
support that usecase anyway.

JIRA: COMPOSE-3063
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-19 13:53:34 +01:00
Lubomír Sedlář
7c2e701a74 extra_iso: Support extra files in directory
It works, only the config schema does not allow it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-19 13:18:42 +01:00
Lubomír Sedlář
c2a4700446 extra_iso: Include extra_files.json metadata
This should be on top level of the ISO, and list files added
specifically to the ISO. If there's anything inherited from one some
variant, the files will be listed in metadata in the variant directory.

JIRA: COMPOSE-3069
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-19 13:18:42 +01:00
Lubomír Sedlář
605c9ca435 Allow reading configuration from JSON
JIRA: COMPOSE-3064
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-19 12:02:19 +01:00
Lubomír Sedlář
f809cac0b2 Cleanup parsing treefile
...and also add tests for it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-19 11:59:32 +01:00
Mohan Boddu
324b371cff Fix convert rpm_ostree config to YAML
Merges: https://pagure.io/pungi/pull-request/1088
Signed-off-by: Mohan Boddu <mboddu@bhujji.com>
2018-11-19 11:59:32 +01:00
Lubomír Sedlář
07d08627c6 koji_wrapper: Change owner of runroot output
The files created in runroot task are owned by root by default (because
that's who is running the processes to create them). Making the results
world readable allows the compose to work, but it still can be difficult
to clean up old composes if they contain random files owned by root.

Fixes: https://pagure.io/pungi/issue/1039
JIRA: COMPOSE-2906
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-15 12:07:27 +01:00
Lubomír Sedlář
1d020dbedd util: Preserve symlinks when copying
Currently the `copy_all` function resolves all symlinks and crashes on
broken symlinks. Instead it should copy symlinks as symlinks.

Fixes: https://pagure.io/pungi/issue/1084
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-15 09:10:16 +01:00
Patrick Uiterwijk
8e88373a82 Move from yaml.load to yaml.safe_load
yaml.load is equally powerful as python pickles, and we don't
need that level of power for the ostree yaml files.
Better safe than sorry.

Signed-off-by: Patrick Uiterwijk <patrick@puiterwijk.org>
2018-11-13 21:11:02 +01:00
Lubomír Sedlář
dc692bc604 extra_iso: Stop including variant extra files
They are not always wanted, so let's not include them by default.
There's a new option to include the same files that extra files phases
uses, or alternatively they can be configured specifically and put into
the variant subdirectory.

JIRA: COMPOSE-3084
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-09 12:06:25 +01:00
Lubomír Sedlář
7c7f997d74 gather: Expand wildcards in package names for nodeps
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-08 08:44:44 +01:00
Lubomír Sedlář
fa752eb2b5 Configure image name per variant
Up to now it was possible to change the pattern for all images, but
there are use-cases where different variants might want different names.
For example there could be one main variant that should only have
product name in the ISO filename, but addons should still be marked with
variant name.

JIRA: COMPOSE-3041
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-05 08:48:36 +01:00
Lubomír Sedlář
85bf5535bc init: Keep parent groups in addon comps environments
The environment in comps for a variant can refer to groups in parent
variant (either for addons, or because of other configuration). We
should not remove the groups in this case.

This requires changes in two places:
 * teaching `comps_filter` about groups that should not be removed
 * fixing writing comps so that it does not actually change the data as
   well

JIRA: COMPOSE-2612
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-02 12:43:56 +01:00
Lubomír Sedlář
a73099d446 Support more specific config for devel modules
The initial implementation is very coarse. It enables it for all
variants and all modules. That is not always wanted.

With this patch, the config file has to explicitly list the devel
modules for each variant that should have it. The variant must be
configured also to include the non-devel module (but the module may be
in lookaside so it won't be included).

We now include module metadata in the internal lookaside repo, so that
this whole thing works if one variant is built on top of another.

JIRA: COMPOSE-3034
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-30 15:58:35 +01:00
Lubomír Sedlář
32bb9aeabe Load supported milestones from productmd
Since 1.18 productmd will make the list available for consumers. If
possible, we should use it, and fall back to hardcoded list.

JIRA: COMPOSE-3044
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-30 14:31:56 +01:00
Lubomír Sedlář
04715f4906 hybrid: Remove dead code
We don't explicitly add modular packages anymore, fus is supposed to
handle that.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-26 14:56:53 +02:00
Lubomír Sedlář
2153c5fe21 Remove dead code
These variables are not used anymore, instead it is taken from extra
data in the build.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-26 14:56:53 +02:00
Lubomír Sedlář
65a1779e3a 4.1.30 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-26 14:27:53 +02:00
Lubomír Sedlář
e10b893adc gather: Expand wildcards in Pungi
When the configuration lists `*` in `additional_packages`, it has a
special meaning. If it's passed to fus directly, it will use it to match
all modules and RPMs that are not masked by a package available in some
default stream. Neither is good. We don't want it to match modules, and
we want even the masked packages.

The fix is to expand the wildcard to a list of NVRs and give that to
fus. It should include the package even if it is masked.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-26 13:16:12 +02:00
Lubomír Sedlář
fe39056431 repoclosure: Extract logs from hybrid solver
There is no repoclosure that correctly understands modules. The best
thing we can offer is the errors reported by the depsolver.

JIRA: COMPOSE-2321
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-26 08:42:21 +02:00
Lubomír Sedlář
12f949fe84 gather: Track multilib that doesn't exist
If a multilib package fails to be added (broken dependencies, or it
doesn't exist at all), we want to track it and eventually stop the
iterations.

Merges: https://pagure.io/pungi/pull-request/1071
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-24 08:13:56 +02:00
Jan Kaluza
5fc0f915c6 Get the NSVC from Koji module CG build metadata
Stream can contain dash sign and when MBS imports such NSVC to Koji, the
dash is replaced with underscore. The current Pungi code does not
respect that and tries to use the stream from Koji directly, which
results in wrong stream being using in some Pungi internal data.

In this PR, the NSVC is taken from module metadata section of CG Koji
build, which contains real stream including the dashes.

Merges: https://pagure.io/pungi/pull-request/1072
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-10-23 15:26:37 +02:00
Lubomír Sedlář
10fa53a6ac gather: Prepare module metadata before starting depsolving
Without this we don't have artifacts from all modules when the first
modular variant is solved. That makes some modular packages appear as
bare.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-19 10:28:17 +02:00
Lubomír Sedlář
0c3e02eeb0 hybrid: Include modulemd from all variants in temporary repo
If there are two variants with different module sets, the missing
modulemd from variant B causeing depsolving in variant A to consider the
packages as non-modular. That is wrong.

Related: https://bugzilla.redhat.com/show_bug.cgi?id=1640125
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-19 10:28:17 +02:00
Lubomír Sedlář
66d9c10a6f extra_iso: Include media.repo and .discinfo
JIRA: COMPOSE-2994
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-18 16:16:39 +02:00
Lubomír Sedlář
736772f954 hybrid: Don't add debuginfo as langpacks
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-17 10:14:04 +02:00
Lubomír Sedlář
9afb8e6801 fus: Write solvables to file
Instead of having one giant command line, write the long list into a
file and pass that to the solver.

The items on the input list are sorted for easy processing.

JIRA: COMPOSE-3012
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-17 10:14:04 +02:00
Lubomír Sedlář
4f712259d7 hybrid: Honor filter_packages
JIRA: COMPOSE-3011
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-17 10:14:04 +02:00
Lubomír Sedlář
111af836ea Include all test fixtures in source tarball
Listing all possible file extensions is error-prone.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-17 10:09:46 +02:00
Lubomír Sedlář
9d9f0add83 Drop the memory saving code
It actually still does not work if there is inheritance between
variants.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-16 15:35:47 +02:00
Lubomír Sedlář
844f2005a0 Save memory less agressively
We can't drop the whole package set once the first solver iteration
finishes, because it runs for each arch separately and we need the data
for each of them. We can however delete the arch specific portion.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-15 15:34:35 +02:00
Lubomír Sedlář
cbfb556f7c extra-iso: Use correct efiboot.img file
If the file is generated, Pungi will modify it at the end of
buildinstall phase. We need to point ISO creation to the modified file,
otherwise the checksum in .treeinfo will be incorrect.

It's sufficient to just update the graft point to correct file, copying
is not needed.

JIRA: COMPOSE-2976
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-12 13:41:36 +02:00
Lubomír Sedlář
a00ea3d4a6 extra-iso: Fix treeinfo
Remove boot.iso references and add [media] section.

JIRA: COMPOSE-2974
JIRA: COMPOSE-2975
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-11 13:19:07 +02:00
Lubomír Sedlář
bb6e68a853 createiso: Move code for tweaking treeinfo into a function
The function loads existing treeinfo, removes reference to boot.iso and
adds [media] section. This is the basic tweak that should happen for all
ISOs. Additional changes depend on the actual content.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-11 13:19:07 +02:00
Lubomír Sedlář
740df1bc6c extra-iso: Generate jigdo by default
It can be turned off by the same option that is used in createiso phase.

JIRA: COMPOSE-2962
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-10 16:06:46 +02:00
Lubomír Sedlář
0c8702cd6d 4.1.29 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-10 12:51:04 +02:00
Lubomír Sedlář
ce9ac35640 hybrid: Only include modules that are not in lookaside
If we want to include the -devel modules, the original modules should be
present for the solver. However if we put them in both local and
lookaside repos, fus will get confused. Let's not include modules that
are in lookaside in the module repo created for the solving job.

Even if the modular packages are present in the local repo, they are
identical to the ones in lookaside, and so they should not make it into
the result anyway.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-08 14:41:04 +02:00
Lubomír Sedlář
814103d87f Try to be more conservative about memory usage
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-08 14:41:04 +02:00
Lubomír Sedlář
3fea217b9c hybrid: Remove modules not listed by fus
It's possible we ask to include module X, but it's in lookaside and as
such it should not be in the output. Therefore we need to remove it from
the variant.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-08 14:41:04 +02:00
Lubomír Sedlář
1a161982c0 gather: Make devel modules configurable
By default nothing should change. This patch adds a new config option
that enables all this new craziness:

 * it turns of applying module filters at pkgset level
 * it creates new modules and adds them to the compose

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-08 14:41:04 +02:00
Lubomír Sedlář
06b32b5d80 pkgset: Stop prefilling RPM artifacts
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-08 14:41:04 +02:00
Lubomír Sedlář
08d65bdde6 gather: Create devel module for each normal module
The module has same S:V:C, but the name is suffixed with `-devel`. The
module should contain all packages from the module koji tag that were
not included in the actual module.

The devel module has the same dependencies as the regular module, but
also additionally depends on the original module. The API and profiles
are cleared in the new module.

In the metadata it shows the same koji tag.

The test if package goes to the module is refactored to a function to
make work with the negated case a bit easier.

There may be unneeded multilib packages in the -devel module, because
there might be buildtime dependencies between things that we don't see.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>

Clear API and profiles

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-08 14:40:13 +02:00
Lubomír Sedlář
410d0125bc pkgset: Save package set for each module
This can be used in figuring out which packages go into the module and
which not.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-08 14:36:43 +02:00
Lubomír Sedlář
1bff5ccfa2 fus: List lookaside repos first
This works around a bug where fus prioritizes first repo and thus could
include packages even if they were in lookaside.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-08 14:36:43 +02:00
Lubomír Sedlář
604ec40c8e gather: Work with repos without location_base
A repo does not necessarily have the location_base attribute if the
packages are next to the repodata. This can easily happen for lookaside
repos.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-08 14:36:43 +02:00
Lubomír Sedlář
e78f8d1f13 Remove extra dependencies
There is a sorted dict implementation in Productmd used to achieve the
exact same thing as Pungi does here. No need for an extra dependency.
While we're at it, we can also sort the imports.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-08 14:15:00 +02:00
Marek Marczykowski-Górecki
d6e72c8e61 Set repodata mtime to SOURCE_DATE_EPOCH
repodata/repomd.xml include timestamps of all the other repodata files.
Even when those files are created reproducibly, they have current
modification time. In general case this is a good thing (ease checking
if repodata cache is up to date). But in case of composing installation
image, it breaks reproducibility.
Avoid this by instructing createrepo to set mtime and revision to
$SOURCE_DATE_EPOCH.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
2018-10-08 13:33:29 +02:00