Commit Graph

2124 Commits

Author SHA1 Message Date
Lubomír Sedlář
86fb93d603 orchestrator: Support getting kerberos ticket
If the configuration sets keytab path and principal, run kinit with
custom cache file, and delete the file at the end of the run.

JIRA: COMPOSE-3288
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-21 09:29:54 +01:00
Lubomír Sedlář
b80efbfd97 hybrid: Refactor handling debuginfo packages
Behavior before this change: debuginfo was added based on source RPMs.
Pungi would get all debuginfo for all architectures that include at
least one binary package. This had a consequence that if foo-x.x86_64
and foo-x.i686 from the same built were included, foo-y-debuginfo.i686
would be included despite foo-y only being present for x86_64.

The patch changes this to work on binary package level: for each
included package `x`, check if there is `x-debuginfo` or
`x-debugsource`, and include them. Packages added in this way have to go
through one iteration of the solver, to account for cases such as
`x-libs-debuginfo` depending on `x-debuginfo` (with only `x` starting
this chain).

To make it slightly faster, the invocation of fus is changed to only
process newly added package in each iteration. It should have no effect
on the results, and subsequent iterations are much faster if they don't
process the same stuff again and again.

JIRA: COMPOSE-3247
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-18 13:56:48 +01:00
Ken Dreyer
b5bef68c60 doc: explain koji_profile
Add more details to the documentation for the koji_profile setting.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2019-03-14 15:53:15 -06:00
Lubomír Sedlář
039b8d44b3 pkgset: Stop loading list of module RPMs
The list is not needed for anything anymore, and it only takes time and
space in memory.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-14 09:08:44 -04:00
Lubomír Sedlář
0fc797a06f pkgset: Only load cache once
The file in old compose does not change, there is no need to load it
again for each tag.

One consequence of this change is that the same cache will be used by
all package sets. They add stuff to the cache as they use it. However
that should not be a problem for the same reason it's okay to use the
cache in the first place. If two packages have the same path, they are
actually the same file and it's okay to reuse the data.

JIRA: COMPOSE-3374
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-12 08:57:35 -04:00
Jan Kaluza
2d39490909 Do not add pkgset_koji_builds to modules, but only to pkgset_koji_tag tags.
Also remove the useless listTaggedRPMs call which has been used *only*
to catch the very rare error which happened from time to time when
we used PDC in Pungi few years ago. This rare error is not relevant
anymore now.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2019-03-11 17:56:27 +01:00
Lubomír Sedlář
9de036b401 scm: Don't retry git fetch
If it fails, we can't really tell if it's a transient error or just too
old git client. Fall back to full clone immediately and retry there.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-11 13:12:06 +00:00
Marek Marczykowski-Górecki
b3bf6cd9cd
tests: fix metadata tests when SOURCE_DATE_EPOCH is set
SOURCE_DATE_EPOCH is preferred over time.time() call in metadata
generation, which makes time.time() mock ineffective. Set
SOURCE_DATE_EPOCH instead of mocking time.time(). This additionally test
if SOURCE_DATE_EPOCH is really used.
Adjust expected output, as SOURCE_DATE_EPOCH is defined without
fractional component.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
2019-03-09 15:42:57 +01:00
Lubomír Sedlář
9541c75b7e checks: Use GitResolver for scm dicts
Otherwise the offline flag is not honored correctly.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-06 09:46:12 +00:00
Lubomír Sedlář
4431ea8a0c hybrid: Fix opening gzipped files on Python 2.6
GzipFile works in `with` statement starting with 2.7

JIRA: RCM-51070
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-06 08:04:19 +01:00
Lubomír Sedlář
cc456d3e75 4.1.34 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-05 12:37:07 +01:00
Lubomír Sedlář
a206a73db1 config: Allow validating configuration in JSON
JIRA: COMPOSE-3318
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-04 14:36:28 +01:00
Lubomír Sedlář
357f556d69 image-build: Accept formats in lists
If the config is loaded from JSON, we will get list instead of tuple.

The validation rule does not really care whether it's a list or tuple,
it only enforces there are two strings. The definition is renamed to be
a bit more descriptive.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-04 08:32:19 +01:00
Lubomír Sedlář
6bd1e9da2a image-build: Resolve git ref in ksurl
The ksurl property was not listed in the config schema, and thus it was
not processed.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-01 09:40:30 +01:00
Lubomír Sedlář
2d694272c0 Resolve git branches in scm_dict
If the config uses SCM dicts that include branch or tag names, they will
be resolved to specific commit ids.

It goes through the caching resolver. The main motivation for that is to
correctly support the --offline flag. It's highly unlikely there will be
two scm_dicts in the config with the same repo.

JIRA: COMPOSE-3279
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-01 09:40:30 +01:00
Lubomír Sedlář
5a7ced5b7d util: Refactor resolving git url
Split out a smaller function that receives a git repo URL and ref and
returns the commit id. The caching resolver class is modified to use
this second function if branch is given to it.

The new function checks if the ref received already looks like a commit
ID, and if so it does not attempt to do anything with it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-01 09:40:30 +01:00
Lubomír Sedlář
ff4e6d4782 scm-wrapper: Refactor getting files from Git
Before this patch, there were two code paths: either getting the only
the wanted content by calling git-archive, or cloning the repository and
copying the files.

Both these approaches have the downside of not allowing retriving
content from a specific git commit.

The workaround is to create a new empty repo (in the location to which
we cloned previously), fetching the specific commit to there and then
checking it out.

This supports any commit and works identically for any protocol. The
downside is that all files in that commit will be downloaded. It should
be no worse than the git-clone path, but can result in bigger transfers
than git-archive.

Unfortunately this is only supported with git 2.5+. On older version
fetch will fail with no error message (tested with 1.8.3). This can be
used to fall back to full clone. This fallback is clearly suboptimal in
terms of data transfer, but it should work reliably.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-01 09:40:30 +01:00
Lubomír Sedlář
ef6bb20a0e osbs: Fix wrong message in logs
We run a thread for each task, not one per variant. If there are
multiple containers in a single variant, the logs contain confusing
entries about starting the phase multiple times.

2019-02-28 01:58:54 ... [BEGIN] OSBS phase for variant Foo
2019-02-28 01:58:54 ... [BEGIN] OSBS phase for variant Foo

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-28 09:00:47 +01:00
Lubomír Sedlář
afd2d3ae66 orchestrator: Log exception to log file
This data can be useful for debugging.

JIRA: COMPOSE-3284
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-27 09:56:29 +01:00
Lubomír Sedlář
8b84aa384a config-dump: Allow freezing koji event
Accept similar argument to pungi-koji: either direct event ID, or a path
to a compose from which the event will be extracted.

It would be nice if we could reuse the path to compose given as source,
but there may be more that one compose, so ultimately we need a way to
overwrite it anyway.

JIRA: COMPOSE-3278
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-26 10:52:45 +01:00
Lubomír Sedlář
fa47d9edba Read koji event from config file
This can be useful for archiving configuration to freeze the koji
package set to a particular event.

JIRA: COMPOSE-3278
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-26 10:52:45 +01:00
Lubomír Sedlář
42a8965e87 image-build: Fix typo in file extension for vmdk image
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-25 16:06:59 +01:00
Lubomír Sedlář
95596f1c69 osbs: Accept local paths as repo URLs
Translate the using configured patterns, and give OSBS a repo file with
that URL.

JIRA: COMPOSE-3290
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-25 08:18:47 +01:00
Lubomír Sedlář
3efcfede6a image-build: Support repo/install_tree as path
If a repo or install tree is specified as an absolute path on the local
filesystem, we should either translate it using the configured mappings,
or if no mapping matches, it should be return unchanged. A variant name
can not start with a slash, so attempting that translation does not make
much sense.

JIRA: COMPOSE-3290
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-25 08:18:47 +01:00
Lubomír Sedlář
8c82cfc1c7 osbs: Remove format requirement for registry
Pungi does not really care if it's an object, list or string. Anything
should be allowed.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-22 15:10:45 +01:00
Lubomír Sedlář
f33973ee65 Make the Apple/HFS compatibility configurable
The default is the original behaviour. On F30+ a new option should be
added to config to make it work.

Over time as users move to this option (which requires a new enough
version of lorax), the default should be switched and then the option
removed.

Resolves: https://pagure.io/pungi/issue/1126
Merges: https://pagure.io/pungi/pull-request/1128
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-21 16:11:01 +01:00
Dan Horák
071d11a883 update iso creation for ppc64le
Don't produce ISO image with Apple/HFS compatibility stuff on ppc64le.

Signed-off-by: Dan Horák <dan@danny.cz>
2019-02-21 16:11:01 +01:00
Lubomír Sedlář
59c9c150ef orchestrator: Use prefix for config substitutions
This will avoid possible conflicts. The names for parts are chosen by
users, and if we decide to add more options in the future, this can
become handy.

JIRA: COMPOSE-3285
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-21 09:42:14 +01:00
Ken Dreyer
5a251ff24c README: add link to documentation
Make it easier for README users to discover the Pungi docs on Pagure.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2019-02-18 14:36:48 -07:00
Lubomír Sedlář
e21c49d9e2 4.1.33 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-13 10:37:48 +01:00
Lubomír Sedlář
6c6d4759f5 isos: Check maximum expected size
This patch allows the configuration to express maximum expected size for
ISOs created in createiso and extra_isos phases. If the image is larger
than this limit, a warning is emitted in test phase. The compose itself
is not affected in any way.

JIRA: COMPOSE-2824
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-12 15:15:30 +01:00
Lubomír Sedlář
7693e562b1 osbs: Process data about pushing images to registries
This patch does not do any actual pushing. It will only extract data
about push targets from the main configuration and store it together
with exact Koji NVR in a well-defined location, and also send the data
to message bus for another service to handle.

JIRA: COMPOSE-3228
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-12 15:12:50 +01:00
Lubomír Sedlář
23bf01bb45 hybrid: Apply filters to debuginfo and source packages
For binary packages the filters are handled at the depsolver level.
However sources and debuginfo is added later in the process, so the
filters have to be explicitly applied.

JIRA: COMPOSE-3114
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-12 12:20:46 +01:00
Lubomír Sedlář
e573246a2a hybrid: Get platform from lookaside repos
If the hybrid solver is used in a situation where there are modules in
lookaside repo, but not in the compose itself, it will fail to detect
any platform. Since we are already opening the module repodata, we can
retrieve platforms from all modules in there as well.

If there are conflicts (e.g. multiple modules depending on different
platforms), an error will be reported.

JIRA: COMPOSE-3277
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-08 14:41:56 +01:00
Jan Kaluza
c3aa297d8c Return RPMs added to -devel module in GatherSourceModule.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2019-02-08 14:24:16 +01:00
Jan Kaluza
31bafa29c5 Allow setting wildcard as a module name in variants to include all the modules.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2019-02-07 13:59:57 +01:00
Lubomír Sedlář
940a581bd9 gather: Link files in order for dependant variants
If there are variants that depend on another, they should be processed
in order to make sure packages from the base variant are linked first.
That way the srpm cache is populated and any package in layered variant
but with source in base will have access to correct epoch information.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-04 08:24:22 +01:00
Lubomír Sedlář
8065239e04 buildinstall: Pick correct config with rootfs_size
If there are multiple matching configuration blocks, we should take the
value that is defined in the last one, but only if it actually is there.
That's how it works for other options.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-01 14:24:20 +01:00
Lubomír Sedlář
f59034b22d hybrid: Add packages from prepopulate to input
This is really an oversight. The list of packages from prepopulate list
should be treated the same as if the packagees were coming from comps or
additional packages.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-01-29 16:16:44 +01:00
Lubomír Sedlář
da1ea83561 ostree_installer: Pass --buildarch to lorax
This should tell lorax what arch to use and avoids fragile detection
based on contents of source repo.

It uses the same logic buildinstall phase uses to get the buildarch.

Related: https://pagure.io/teamsilverblue/issue/67
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-01-28 13:41:49 +01:00
Lubomír Sedlář
f92c71683f orchestrator: Add missing function arguments
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-01-21 09:13:03 +01:00
Lubomír Sedlář
7320bf4943 orchestrator: Compatibility with Python 2.6
This is why we can't have nice things.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-01-14 10:57:05 +01:00
Frédéric Pierret (fepitre)
86314fdc83 pungi-legacy: expose lorax's --rootfs-size argument
Merges: https://pagure.io/pungi/pull-request/1112
Fixes: https://pagure.io/pungi/issue/1107
Signed-off-by: Frédéric Pierret (fepitre) <frederic.pierret@qubes-os.org>
2019-01-14 10:41:37 +01:00
Miro Hrončok
3fe4beb20c Only require enum34 on Legacy Python
This resolves https://bugzilla.redhat.com/show_bug.cgi?id=1665501

Signed-off-by: Why Do You <enforce@this.qm>
2019-01-14 09:16:14 +01:00
Lubomír Sedlář
6a32b3b741 ostree: Add test for expanding basearch for message
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-01-10 07:53:46 +01:00
Patrick Uiterwijk
c9c5fcac94 Make sure ${basearch} is also replaced with config['ostree_ref']
Signed-off-by: Patrick Uiterwijk <patrick@puiterwijk.org>
2019-01-09 21:39:33 +01:00
Lubomír Sedlář
dbc0d5736c 4.1.32 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-01-08 12:17:29 +01:00
Lubomír Sedlář
90c60f8e64 Add script to orchestrate multiple composes
It may make sense to break a big compose into smaller chunks that can be
done independently. This script allows describing the smaller parts,
runs them with correct dependencies and arranges the result to look like
a single big compose.

All parts use the same koji event, that is either obtained from Koji, or
from command line argument.

JIRA: COMPOSE-2654
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-01-08 09:40:36 +01:00
Lubomír Sedlář
ba260c24e8 buildinstall: Expose lorax's --rootfs-size argument
We already make this possible for the ostree installer, but it was
missing from the traditional one. The default behaviour is to let lorax
decide, but if user knows better, they can overwrite in configuration.

JIRA: COMPOSE-3188
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-01-07 11:00:43 +01:00
Frédéric Pierret (fepitre)
682f959ee0 Support for pungi-legacy with productmd format
Merges: https://pagure.io/pungi/pull-request/1099
Signed-off-by: Frédéric Pierret (fepitre) <frederic.pierret@qubes-os.org>
2019-01-04 14:11:23 +01:00