Commit Graph

2259 Commits

Author SHA1 Message Date
Lubomír Sedlář
b0f0579a9e util: Resolve ref if duplicate branches are present
If the repo contains the same name under multiple directories, make the
resolving work by filtering only to refs/heads and refs/tags.

Fixes: https://pagure.io/pungi/issue/1180
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-07 13:44:57 +02:00
Lubomír Sedlář
9517df44c7 config: Fix getting default branch in SCM dict
If user configures branch as explicit None, we want to default to HEAD
(which is most likely refs/heads/master in git).

The original code was getting branch as None, which lead to wrong
resolver being used and the repo url being used as branch.

Fixes: https://pagure.io/pungi/issue/1181
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-07 10:35:49 +02:00
Lubomír Sedlář
a9b9ec97fb pkgset: Fix whitelist for modules
The prefix checking only works if there are no streams that would share
prefixes. Let's instead check the value as a whole. There is extra
complexity from the fact that version and context may not be specified.

The stream as specified in input is processed to replace dashes (`-`)
with underscores (`_`) to match how the builds are imported into Koji.

JIRA: COMPOSE-3547
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-06 09:48:44 +02:00
Lubomír Sedlář
2ae742af04 pkgset: Fix filtering excluded modular packages
For modular tags we only include packages that are included in the
module. Originally the filter was taking package names from
rpms.artifacts section of MMD.

This however does not work correctly, as there can easily be module
which lists foo.src but does not want to include foo.x86_64 or other
arches.

This patch fixes this particular problem by included arch in the set of
packages to be kept.

JIRA: COMPOSE-3543
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-02 13:22:13 +02:00
Lubomír Sedlář
f858cea466 pkgset: Do not overwrite version in module
This value is important to get correct path to metadata.

JIRA: COMPOSE-3541
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-30 12:22:13 +02:00
Lubomír Sedlář
b73d2d7f11 pkgset: Treat modular version as number for sorting
The scheme for generating versions has changed multiple times. MBS is
careful to only modify them so that they always compare correctly. This
only works though if the versions are treated as numbers.

This should be safe in that non-numbers should never be encountered as
module version. Libmodulemd internally stores the version as int (or
some version of int).

JIRA: COMPOSE-3540
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-26 14:51:24 +02:00
Lubomír Sedlář
e550686e06 Use absolute path for hardlink
Hardlink command can be installed in /usr/sbin, where it is not visible
to non-priviledged users. They can still run it, but don't have it in
their PATH.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-26 13:48:54 +02:00
Jon Disnard
72b4969832 createiso: Run hardlink on staged content
Even if we want to break hardlinks from Koji volume, there may be files
that can be hardlinked and save some space on the media.

JIRA: COMPOSE-3482
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-26 13:48:54 +02:00
Lubomír Sedlář
1e7ec68bbd comps-wrapper: Emit attributes sorted
Python 3.8 no longer sorts attributes automatically, which is causing
some of the tests to fail. The easiest fix is to update the code to make
sure sorting is in place.

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1698514
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-18 10:23:51 +02:00
Jiri Konecny
0cd089802f
patch-iso supports multiple graft directories
Pungi patch iso now supports multiple graft directories. This should
make usage more comfortable.

Signed-off-by: Jiri Konecny <jkonecny@redhat.com>
2019-04-12 15:27:54 +02:00
Lubomír Sedlář
479d17c033 4.1.36 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-12 12:10:39 +02:00
Jan Kaluza
49d137b444 Extend "openssh" runroot_method to be able to execute "mock"
This adds few new config options which are well described in the
configuration documentation. Please refer to it for more information.

Merges: https://pagure.io/pungi/pull-request/1170
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2019-04-12 11:10:40 +02:00
Lubomír Sedlář
d521711957 osbs: Rework configuration for image pushes
Embedding the registry configuration into OSBS config itself is
simple, but makes it impossible to reuse the same configuration for
multiple different composes.

A nice example is a nightly pushing images to a testing registry, and
production compose building the same images but pushing to staging
location. The original design requires duplication of all the
configuration just because registries are different.

With this option, the push information is stored in a separate option as
a mapping from NVR patterns to arbitrary data. The patterns are used to
match finished builds to registry.

The old configuration is marked as deprecated in code and will
eventually be removed. The deprecation handling in config validation
does not allow emitting warnings for nested values.

JIRA: COMPOSE-3394
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-05 10:28:34 +02:00
Jan Kaluza
959d6979d4 Add "openssh" runroot method
Merges: https://pagure.io/pungi/pull-request/1166
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2019-04-04 14:44:05 +02:00
Lubomír Sedlář
2f8717ec97 Fix printing version on Python 3
Don't crash when getting version from installed package.

Fixes: https://pagure.io/pungi/issue/1152
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-04 11:42:19 +02:00
Lubomír Sedlář
b165866f39 config-dump: Fix crash when used without --define
The default value is None, which is not iterable.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-03 13:09:12 +02:00
Lubomír Sedlář
59a9c01304 setup: Fix missing comma in a list
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-03 10:25:46 +02:00
Lubomír Sedlář
724255a984 setup: Install some deps on Py2.7 only
Lockfile and dict.sorted are only used in pungi.gather module, which
does not work with Python 3 due to dependency on yum. There's no point
in always pulling in these two libraries there.

This change fixes issues with python dependencies generator that puts
python3.6dist(lockfile) into RPM requires.

For builds on Python 2.6 these libraries are actually needed, but there
are no generators on RHEL 6. So this change can only break development
on RHEL 6. And if someone needs to do that, they have bigger problems...

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-02 14:46:19 +02:00
Lubomír Sedlář
fd0117f38c config-dump: Allow defining variables on CLI
This can be used to fill particular values to a predefined template
without editing any file disk.

JIRA: COMPOSE-3316
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-02 09:54:25 +02:00
Lubomír Sedlář
e71e91982b Update test data
The parts in multi compose are now named reasonably.

The package set repos are selected only once and not per variant. We
can't use repos with packages for specific arch only as that would
require more transformations during generating test data.

Lookasides for ResilientStorage part are added to point to Server part.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-01 09:49:53 +02:00
Lubomír Sedlář
1f95c33e2a gather: Use wildcard for repo selection
If package source is set to repos, honor wildcard and apply that block
every time.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-01 09:49:53 +02:00
Lubomír Sedlář
c6a86c444a gather: Apply repo path substitutions for DNF backend
Use a method to add repos that will apply $arch and $basearch
substitution automatically. Yum backend only applies $basearch, so if
compatibility is needed, that one should be used.

Drop code for handling mirrorlist, since Pungi does not ever use it, and
being used externally is not really supported.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-01 09:49:53 +02:00
Lubomír Sedlář
c2c36dc3c2 tests: Stop overwriting modulesdir for DNF
Tests are not passing in Fedora builds with this code, since it was
deleted in DNF. I think it's not actually doing anything, so we should
be able to drop it as well.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-26 14:58:09 +01:00
Lubomír Sedlář
5f8b519941 4.1.35 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-26 10:03:26 +01:00
Lubomír Sedlář
31ef7736aa orchestrator: Monitor status of parts
When a phase is started or stopped, add a line to the to output. This
should help users keep track of what is happening in case the part takes
a long time to run.

JIRA: COMPOSE-3287
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-25 14:20:04 +01:00
Lubomír Sedlář
d7ef86293e tests: Skip tests if libmodulemd is not available
These tests need the library, and will crash without it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-25 10:41:09 +01:00
Lubomír Sedlář
66a127c313 pkgset: Refactor hiding unused modular packages
A module build can create packages that are tagged in the content tag,
but should not be included in the module. Originally Pungi didn't know
what exactly the module contains and so it needed to apply filters to
exclude stuff that was definitely out.

With getting the final MMD from Koji, we can actually make this a bit
more strict by only keeping packages that we know we need.

When processing each content tag, we can put into package set only
packages that are included in some module using that tag. This should
work with -devel modules as well. Both the regular and -devel modules
will contribute to the set and thus all packages will go to the package
set.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-25 08:48:16 +01:00
Lubomír Sedlář
9f9b784e64 Remove configuration for devel modules
They should be loaded in the same way as regular modules straight from
Koji.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-25 08:48:16 +01:00
Lubomír Sedlář
ac15f21135 gather: Remove module source
This source does not really return anything useful. It was necessary to
process the source modulemd to fill in list of RPMs. Since we now get
the final files from Koji, this is not needed anymore and the source can
be dropped.

This change requires a lot of tweaks for test.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-25 08:48:16 +01:00
Lubomír Sedlář
9939d09643 createrepo: Stop processing modulemd
The file we get from Koji has all important bits already filled in.
There is no need to add anything.

This patch also stops filtering the artifacts to only contain packages
that are actually in the repo. Thus debuginfo and source packages will
appear there.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-25 08:48:16 +01:00
Lubomír Sedlář
9229699078 pkgset: Load final modulemd files from Koji
Instead of loading the "source" modulemd, always get the final file for
each architecture from Koji.

Logging the downloaded files locally is no longer necessary, when
debugging a problem we can find the files in the respective builds in
Koji.

Tests are updated to work with new code, and an obsolete test is
removed.

JIRA: COMPOSE-3147
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-25 08:48:16 +01:00
Lubomír Sedlář
358fdd50ce buildinstall: Allow overwriting version for lorax
Sometimes the release version can be more specific than what should be
exposed to users of the boot iso.

JIRA: COMPOSE-3295
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-22 12:47:16 +01:00
Jan Kaluza
75bb48a882 Create new Runroot class and use it everywhere where runroot task is executed.
This adds new `Runroot` class and new `runroot_method` option which makes
it possible to choose between two currently available runroot methods:

- Local
- Koji

The main goal of this commit is to make it possible to add new runroot
methods in the future and this is the first step in that direction.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2019-03-21 14:15:58 +01:00
Lubomír Sedlář
45cdbb2faf orchestrator: Send messages about the main compose
Only start/finish messages will be sent if a handler is configured.

JIRA: COMPOSE-3288
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-21 09:29:54 +01:00
Lubomír Sedlář
088ea7fe37 orchestrator: Support generic pre- and post- scripts
Run arbitrary commands before and after the compose.

The example config is updated to generate latest symlink with a
post-compose script. The pre compose script runs always, post compose
runs only if the compose is not doomed.

JIRA: COMPOSE-3288
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-21 09:29:54 +01:00
Lubomír Sedlář
86fb93d603 orchestrator: Support getting kerberos ticket
If the configuration sets keytab path and principal, run kinit with
custom cache file, and delete the file at the end of the run.

JIRA: COMPOSE-3288
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-21 09:29:54 +01:00
Lubomír Sedlář
b80efbfd97 hybrid: Refactor handling debuginfo packages
Behavior before this change: debuginfo was added based on source RPMs.
Pungi would get all debuginfo for all architectures that include at
least one binary package. This had a consequence that if foo-x.x86_64
and foo-x.i686 from the same built were included, foo-y-debuginfo.i686
would be included despite foo-y only being present for x86_64.

The patch changes this to work on binary package level: for each
included package `x`, check if there is `x-debuginfo` or
`x-debugsource`, and include them. Packages added in this way have to go
through one iteration of the solver, to account for cases such as
`x-libs-debuginfo` depending on `x-debuginfo` (with only `x` starting
this chain).

To make it slightly faster, the invocation of fus is changed to only
process newly added package in each iteration. It should have no effect
on the results, and subsequent iterations are much faster if they don't
process the same stuff again and again.

JIRA: COMPOSE-3247
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-18 13:56:48 +01:00
Ken Dreyer
b5bef68c60 doc: explain koji_profile
Add more details to the documentation for the koji_profile setting.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2019-03-14 15:53:15 -06:00
Lubomír Sedlář
039b8d44b3 pkgset: Stop loading list of module RPMs
The list is not needed for anything anymore, and it only takes time and
space in memory.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-14 09:08:44 -04:00
Lubomír Sedlář
0fc797a06f pkgset: Only load cache once
The file in old compose does not change, there is no need to load it
again for each tag.

One consequence of this change is that the same cache will be used by
all package sets. They add stuff to the cache as they use it. However
that should not be a problem for the same reason it's okay to use the
cache in the first place. If two packages have the same path, they are
actually the same file and it's okay to reuse the data.

JIRA: COMPOSE-3374
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-12 08:57:35 -04:00
Jan Kaluza
2d39490909 Do not add pkgset_koji_builds to modules, but only to pkgset_koji_tag tags.
Also remove the useless listTaggedRPMs call which has been used *only*
to catch the very rare error which happened from time to time when
we used PDC in Pungi few years ago. This rare error is not relevant
anymore now.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2019-03-11 17:56:27 +01:00
Lubomír Sedlář
9de036b401 scm: Don't retry git fetch
If it fails, we can't really tell if it's a transient error or just too
old git client. Fall back to full clone immediately and retry there.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-11 13:12:06 +00:00
Marek Marczykowski-Górecki
b3bf6cd9cd
tests: fix metadata tests when SOURCE_DATE_EPOCH is set
SOURCE_DATE_EPOCH is preferred over time.time() call in metadata
generation, which makes time.time() mock ineffective. Set
SOURCE_DATE_EPOCH instead of mocking time.time(). This additionally test
if SOURCE_DATE_EPOCH is really used.
Adjust expected output, as SOURCE_DATE_EPOCH is defined without
fractional component.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
2019-03-09 15:42:57 +01:00
Lubomír Sedlář
9541c75b7e checks: Use GitResolver for scm dicts
Otherwise the offline flag is not honored correctly.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-06 09:46:12 +00:00
Lubomír Sedlář
4431ea8a0c hybrid: Fix opening gzipped files on Python 2.6
GzipFile works in `with` statement starting with 2.7

JIRA: RCM-51070
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-06 08:04:19 +01:00
Lubomír Sedlář
cc456d3e75 4.1.34 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-05 12:37:07 +01:00
Lubomír Sedlář
a206a73db1 config: Allow validating configuration in JSON
JIRA: COMPOSE-3318
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-04 14:36:28 +01:00
Lubomír Sedlář
357f556d69 image-build: Accept formats in lists
If the config is loaded from JSON, we will get list instead of tuple.

The validation rule does not really care whether it's a list or tuple,
it only enforces there are two strings. The definition is renamed to be
a bit more descriptive.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-04 08:32:19 +01:00
Lubomír Sedlář
6bd1e9da2a image-build: Resolve git ref in ksurl
The ksurl property was not listed in the config schema, and thus it was
not processed.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-01 09:40:30 +01:00
Lubomír Sedlář
2d694272c0 Resolve git branches in scm_dict
If the config uses SCM dicts that include branch or tag names, they will
be resolved to specific commit ids.

It goes through the caching resolver. The main motivation for that is to
correctly support the --offline flag. It's highly unlikely there will be
two scm_dicts in the config with the same repo.

JIRA: COMPOSE-3279
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-01 09:40:30 +01:00