Commit Graph

156 Commits

Author SHA1 Message Date
Lubomír Sedlář
a217eea24e Allow filtering comps for different variants
We already filter for arches, so this just extends the filter to work
with variants as well.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-13 14:56:06 +02:00
Lubomír Sedlář
d7021c5688 comps: Make filtering by attribute more generic
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-13 14:56:06 +02:00
Lubomír Sedlář
de231064b7 comps: Add tests for CompsFilter
All use cases that are actually used by pungi-koji are tested. There is
missing coverage for

 * keeping only items with matching arch
 * not reindenting the file

These aren't currently used and should be removed in the future, but
there may be other tools depending on the comps_filter executable.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-09 07:48:28 +02:00
Lubomír Sedlář
7ea4c33d87 comps: Move filtering into wrapper module
The code should not live directly in the executable, that makes it very
hard to test.

Other than the move there is no functional change.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-09 07:48:28 +02:00
Ondrej Nosek
1f0739831c Update tests for Python 2.6
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-03-29 16:34:52 +02:00
Ondrej Nosek
e3aa2f769b Write module metadata
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-03-29 09:47:55 +02:00
Jan Kaluza
9be2d6a920 Allow setting <kojitag/> in <modules/> in variants.xml to get the modules from this Koji tag.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-03-21 14:33:45 +01:00
Lubomír Sedlář
6bff4bd10e Fix modular content in non-modular variant
When allowing empty list of modules, the check for variant tags got
broken, causing Everything to no longer have an associated list of
allowed packages.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Relates: https://pagure.io/pungi/issue/862
2018-03-16 14:47:10 +01:00
Lubomír Sedlář
5c902592ae pkgset: Allow empty list of modules
This should indicate that it's a modular variant, but there is no
modular content yet. We don't want to treat that as Everything.

The end result will be an empty repository.

Fixes: https://pagure.io/pungi/issue/871
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-14 14:49:09 +01:00
Lubomír Sedlář
41d0139b39 Write package whitelist for each variant
If we have a package set for the variant (which happens if there are
modules), include a list of all NEVRAs in the pungi kickstart.

This can be used to make sure only packages from correct tag get into
the compose. If two packages with same name but different version get
into the compose, this can help get even older version into a particular
variant.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-08 14:29:23 +01:00
Lubomír Sedlář
a49704b2b8 scm: Stop decoding output of post-clone command
There is no guarantee that it will print any text. We don't even need
the output, we just print it to error log if there is a problem.

Fixes: https://pagure.io/pungi/issue/847
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-02-20 12:18:48 +01:00
Jan Kaluza
930c2f1a42 Add buildinstall_use_guestmount boolean option
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-01-16 09:18:35 +01:00
Ondrej Nosek
2152e7ea26 Fall back to mount if guestmount is not available
Relates: https://pagure.io/pungi/issue/803

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-01-02 17:13:47 +01:00
Ondrej Nosek
116e7ca3bd El-Torito boot information on s390x
Relates: https://github.com/rhinstaller/lorax/pull/236

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2017-12-14 10:08:30 +01:00
Lubomír Sedlář
aa9c137412 Drop check for repoquery command
There is no place that would possibly call it, so we can drop the check
and function for generating cli.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-12-06 13:26:43 +01:00
Lubomír Sedlář
001ae1d7b4 Use modifyrepo_c if possible
When configured to use createrepo_c, we should also use modifyrepo_c.
That allows us to relax the check for createrepo package.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-12-06 13:26:43 +01:00
Lubomír Sedlář
d2804b5d89 iso-wrapper: Fix calling wrong logger method
The mounting function receives logger as an argument, and its method is
called just error.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-11-13 09:04:34 +01:00
Lubomír Sedlář
388be481ea iso-wrapper: Remove hacks for sorting
We can use a key function instead of relying to the deprecated cmp. This
makes the code work on Python 2.6 and on recent versions it makes it
faster.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-11-08 15:35:29 +01:00
Lubomír Sedlář
026ba10987 kojiwrapper: Deal with multiple values for image-build
When the config for image-build command contains multiple values, they
should be joined with commas into a single value.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-11-01 12:47:36 +01:00
Lubomír Sedlář
f3806f7c77 Stop using deprecated pipes.quote
Instead use the definition from python-six. Once we drop Py 2 support
completely, we'll just swap underscores with dots.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-24 15:15:31 +02:00
Lubomír Sedlář
797b13b34a Convert configparser values to string
On Python 3, configparser will reject non string values, and
theoretically we could have some in the configuration.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-24 15:15:31 +02:00
Lubomír Sedlář
ed9d7f69a6 Use universal_newlines when running other commands
This will automatically convert the output to unicode/str and we will
not have to worry about decoding ourselves.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-24 15:15:31 +02:00
Lubomír Sedlář
ed22e07ef9 Port to Python 3
This should make all tests pass on both Python 2 and Python 3.

Unittest2 is required on Py 2.6 and Py 3.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-24 15:15:31 +02:00
Ralph Bean
5b6e468952 Allow extracting profiling information from pungi-gather.
`pungi-gather` (the tool that underlies both the `pkgset` and `gather`
phases) contains profiling code that will log statistics about how long
different function calls take.  However, pungi-koji did not contain a
way to pass the ``--profiler`` argument to enable this.

This change adds a new configuration option ``gather_profiler`` which,
when set to true, simply passes the argument to `pungi-koji`.  Hopefully
this can help shed some light on what is happening in some of our
longer-running composes.

Merges: https://pagure.io/pungi/pull-request/727
Signed-off-by: Ralph Bean <rbean@redhat.com>
2017-09-04 10:25:11 +02:00
Lubomír Sedlář
5379fb5e28 Check for correct string class
Use python-six and check against correct string class on both Python 2
and Python 3.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-28 14:31:09 +02:00
Lubomír Sedlář
65910f2c33 Open files as binary where needed
In many cases we need to open files as binary to avoid errors on Py3
about writing binary data to file opened in text mode.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-28 13:47:18 +02:00
Owen W. Taylor
49c6abcfea KojiWrapper: include serverca in session_opts
If we have a custom server CA certificate, it needs to be generally
available, and not just used when logging in so that SSL verification
works.

Signed-off-by: Owen W. Taylor <otaylor@fishsoup.net>
2017-08-21 14:16:40 -04:00
Lubomír Sedlář
c3b49f7ffb pkgset: Download packages with dnf
When using repos as gather_source, we should use DNF backend even for
constructing initial package set and to download the packages from
source repos. Without this the repos source would not be usable on
Python 3.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-16 14:32:26 +02:00
Lubomír Sedlář
22fdd59ca4 scm: Accept unicode as local path
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-14 14:36:47 +02:00
Lubomír Sedlář
cae202c17b scm-wrapper: Allow running command after git clone
When a file should be obtained from a git repository, allow running an
arbitrary command (like `make`) after clone but before copying the files
out. This only works for the Git backend.

The downside is that a clone is needed and we can no longer use `git
archive` to speed things up.

Fixes: https://pagure.io/pungi/issue/5
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-10 15:32:24 +02:00
Lubomír Sedlář
c9f34b6684 gather: Only parse pungi log once
No need to read the file three times, we can just get all the data in
one go.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-09 11:04:14 +02:00
Lubomír Sedlář
cef8650c3e gather: Report missing comps packages
When a package mentioned in comps is not available in the package set,
print a warning about this. Additionally there is a config option that
allows to turn this warning into a fatal error.

Fixes: https://pagure.io/pungi/issue/50
Fixes: https://pagure.io/pungi/issue/683
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-09 11:04:14 +02:00
Lubomír Sedlář
1d7617f783 gather: Avoid reading whole log into memory
There's no reason for reading the whole log of depsolving into memory
just to split it into lines and process one line at a time.

We can just as well read it in chunks.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-09 11:04:14 +02:00
Qixiang Wan
0a3e5b27bf remove the dependency of rpmUtils
The rpmUtils module is provided yum-utils package, which is only
available for Python 2. There is no replacement for the functionality in
DNF.

There is a proposal to add this functionality to rpm itself, but it's
not really moving forward very much:
https://bugzilla.redhat.com/show_bug.cgi?id=1072972

As a short term solution let's copy the needed parts of rpmUtils.arch
module directly to pungi code base.

Fixes: https://pagure.io/pungi/issue/533
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-07-26 11:23:16 +02:00
Lubomír Sedlář
8c48dfb93a gather: nodeps should take packages from comps groups
When gather_method is set to nodeps, we should not ignore the comps
group that the method received. Instead it should find out which
packages are in those groups and take them into the compose.

In order for this to be of any reasonable use, the comps file needs to
include all dependencies for the packages.

Fixes: #653
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-11 10:21:13 +02:00
Lubomír Sedlář
ecbb43ab86 variants-wrapper: Remove main() function
Nothing is calling this code, and it's only really useful for manual
testing of parsing.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-27 09:36:48 +02:00
Lubomír Sedlář
81b71b9ed3 koji-wrapper: Handle failed subtasks
If a subtask fails, we can't ask about it's results as that would raise
an exception. We can safely assume that since the parent succeeded, any
failed child is actually allowed to fail.

Fixes: #641
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-12 18:15:29 +02:00
Lubomír Sedlář
1db1abbb82 koji-wrapper: Stop mangling env variables
When koji is authenticated with a keytab, by setting the private
directory we erased rest of existing environment. In non-keytab path,
the environment variables got removed as well.

This patch makes sure that the environment will not be modified more
than necessary (by setting KRB5CCNAME if needed).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-05 12:46:45 +02:00
Lubomír Sedlář
8c237b78c2 comps-wrapper: Report unknown package types
When there is a typo in the comps file, instead of crashing with a
non-descript KeyError we should raise a nice error with details about
the problem.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-01 09:59:41 +02:00
Lubomír Sedlář
63c0bbc5c9 iso-wrapper: Capture debug information for mounting
Occasionally we have seen the mount command fail. The default error
message says to set some environment variables and try again. We can
just always set the environment and only print the output on failure.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-05-29 13:18:07 +02:00
Lubomír Sedlář
77ee882c21 comps-wrapper: Fix crash on conditional packages
A `requires` attribute is taken from a wrong package (because of a wrong
variable used: `pkg` vs. `package`). On RHEL 6 this actually leads to a
crash. Let's use only one name to avoid such problems.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-05-29 13:11:49 +02:00
Lubomír Sedlář
f27f3ce4ba koji-wrapper: Run all blocking commands with fresh ccache
If keytab is used for authentication, other commands than runroot can
possibly fail due to the credentials cache being overwritten.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-05-29 12:53:29 +02:00
Jan Kaluza
68351fa5a8 Add @retry decorator and use it to retry connection on PDC on IOError and in SCM's retry_run.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2017-05-29 11:33:23 +02:00
Jan Kaluza
d037d61521 Store RPM artifacts in resulting repository in modulemd metadata.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2017-05-04 21:12:30 +02:00
Lubomír Sedlář
6ce88630ec variants: Remove redundant check
This situation is handled by schema validation already, so this
condition can never really fire.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-05-03 14:51:39 +02:00
Lubomír Sedlář
55058f1590 scm-wrapper: Remove unused arguments
There are a couple arguments that are never used. We can just remove
them. This also fixes a bug where temporary files would be left
undeleted in tests.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-27 15:59:26 +02:00
Lubomír Sedlář
059449e140 koji-wrapper: Run koji runroot with fresh credentials cache
If the koji profile we are using is configured to use keytab, we should
run koji executable with a fresh credentials cache. Otherwise we risk a
race condition as multiple processes will trample over the same
directory in /tmp/krbcc_0.

This is currently only implemented for calling `koji runroot`. We might
need to do it for other commands as well (currently there is a sleep to
avoid the race condition for other commands).

Fixes: https://pagure.io/releng/issue/6715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-25 15:45:00 +02:00
Lubomír Sedlář
7028399403 util: Move get_buildroot_rpms to koji wrapper
This way the util module does not import the particular wrapper.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-25 15:22:41 +02:00
Lubomír Sedlář
180a5b94a9 comps-wrapper: Reduce duplication in code
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-13 11:45:23 +02:00
Lubomír Sedlář
3861be3e08 comps-wrapper: Port to libcomps
Instead of replacing yum.comps with an something from DNF, we can go
directly to libcomps. DNF does not have the equivalent functionality
(particularly it's impossible to load comps from file directly).

We would have depended on libcomps anyway transitively, so this is not a
big deal.

Fixes: #587
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-13 11:45:23 +02:00