Commit Graph

788 Commits

Author SHA1 Message Date
Lubomír Sedlář ec39514fba Use Python 3 print function
This is updated in all files for consistency, even the modules that will
never be ported to Py 3 completely due to dependency on Yum.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-23 09:41:22 +02:00
Owen W. Taylor 49c6abcfea KojiWrapper: include serverca in session_opts
If we have a custom server CA certificate, it needs to be generally
available, and not just used when logging in so that SSL verification
works.

Signed-off-by: Owen W. Taylor <otaylor@fishsoup.net>
2017-08-21 14:16:40 -04:00
Lubomír Sedlář a63e4746c9 Report warning when config sections are not used
It's possible a variant is excluded via tree_variants option and the
section does not match anything. It can be confusing to users why
nothing is happening. This patch lets Pungi log all unmatched patterns.

Fixes: https://pagure.io/pungi/issue/692
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-21 09:07:10 +02:00
Lubomír Sedlář c3b49f7ffb pkgset: Download packages with dnf
When using repos as gather_source, we should use DNF backend even for
constructing initial package set and to download the packages from
source repos. Without this the repos source would not be usable on
Python 3.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-16 14:32:26 +02:00
Lubomír Sedlář be4501c54b gather: Fix duplicated log line
Due to overwriting an existing variable the logs are getting duplicated
line about missing comps packages instead of announcement of gathering
being finished. Rename the variable to fix the problem.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-16 13:14:21 +02:00
Lubomír Sedlář a8cd78faf9 gather: Add fulltree-exclude flag to DNF backend
This is needed for correct trimming of addons and optional. A package
pulled in as a dependency but that matches something on fulltree exclude
list should have this flag. It will then be moved from addon to base
variant and therefore excluded from optional.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-16 13:06:55 +02:00
Lubomír Sedlář 23ca2fe5d2 checks: Stop looking for imports
We can't reliably tell user what system packages are missing as the name
might be different on different systems. Addiotionally there's no reason
why not rely on the packaging to be correct.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-14 14:57:06 +02:00
Lubomír Sedlář 804a0049f6 ostree: Simplify configuration
It makes no sense to repeat the same configuration for multiple
architectures. Instead we should just list the architectures as another
key in the mapping. There is an option to specify multiple config dicts.

This preserves full backwards compatibility, the old config format is
still accepted.

Fixes: https://pagure.io/pungi/issue/678
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-14 14:53:22 +02:00
Lubomír Sedlář 9780f36e37 config: Reduce duplication in schema
This patch adds a helper function for easy creation of a structure where
either X or list of Xs is accepted.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-14 14:53:22 +02:00
Lubomír Sedlář c12bad295f config: Add option for dumping config schema
This makes it easier to compare changes to the schema in different
versions.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-14 14:53:22 +02:00
Lubomír Sedlář 22fdd59ca4 scm: Accept unicode as local path
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-14 14:36:47 +02:00
Lubomír Sedlář cae202c17b scm-wrapper: Allow running command after git clone
When a file should be obtained from a git repository, allow running an
arbitrary command (like `make`) after clone but before copying the files
out. This only works for the Git backend.

The downside is that a clone is needed and we can no longer use `git
archive` to speed things up.

Fixes: https://pagure.io/pungi/issue/5
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-10 15:32:24 +02:00
Lubomír Sedlář c9f34b6684 gather: Only parse pungi log once
No need to read the file three times, we can just get all the data in
one go.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-09 11:04:14 +02:00
Lubomír Sedlář cef8650c3e gather: Report missing comps packages
When a package mentioned in comps is not available in the package set,
print a warning about this. Additionally there is a config option that
allows to turn this warning into a fatal error.

Fixes: https://pagure.io/pungi/issue/50
Fixes: https://pagure.io/pungi/issue/683
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-09 11:04:14 +02:00
Lubomír Sedlář 1d7617f783 gather: Avoid reading whole log into memory
There's no reason for reading the whole log of depsolving into memory
just to split it into lines and process one line at a time.

We can just as well read it in chunks.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-09 11:04:14 +02:00
Lubomír Sedlář e2962dc547 repoclosure: Allow aborting compose when repoclosure fails
Alternatively the call to repoclosure can be turned off. This is
customizable per variant and architecture.

Fixes: https://pagure.io/pungi/issue/676
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-07 15:17:33 +02:00
Lubomír Sedlář 056ae31ef9 repoclosure: Fix logging errors
Instead of just printing the error directly to stderr, capture the
output and use proper logger. This makes sure the error is included in
the log file and also fixes `--quiet` option which was not properly
honored originally.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-07 15:17:33 +02:00
Ralph Bean 381d08a810 Make strict the only option.
Signed-off-by: Ralph Bean <rbean@redhat.com>
2017-08-07 14:56:59 +02:00
Ralph Bean 58fe997e29 Raise a ValueError with details if module not found in PDC.
Currently, this will fail two lines later when we try to access
`pdc_module['modulemd']` with an unhelpful `TypeError` since
`pdc_module` is `None`.

Signed-off-by: Ralph Bean <rbean@redhat.com>
2017-08-07 08:34:45 -04:00
Lubomír Sedlář 471e369d23 gather: Fix excluding debugsource packages from input list
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-27 08:50:21 +02:00
Till Maas b26547ae77 Use only one list of patterns/rules for debug packages
Signed-off-by: Till Maas <opensource@till.name>
2017-07-26 23:18:29 +02:00
Till Maas 938531e2b2 Do not match "*-debugsource-*" as debuginfo package
These kind of packages are not expected to exist in the wild.

Signed-off-by: Till Maas <opensource@till.name>
2017-07-26 23:18:24 +02:00
Till Maas 29bedf2ccc Use pungi.util.pkg_is_debug() instead of pungi.gather.is_debug()
There is no reason to have two functions for the same job.

Signed-off-by: Till Maas <opensource@till.name>
2017-07-26 23:18:15 +02:00
Qixiang Wan 0a3e5b27bf remove the dependency of rpmUtils
The rpmUtils module is provided yum-utils package, which is only
available for Python 2. There is no replacement for the functionality in
DNF.

There is a proposal to add this functionality to rpm itself, but it's
not really moving forward very much:
https://bugzilla.redhat.com/show_bug.cgi?id=1072972

As a short term solution let's copy the needed parts of rpmUtils.arch
module directly to pungi code base.

Fixes: https://pagure.io/pungi/issue/533
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-07-26 11:23:16 +02:00
Lubomír Sedlář d92390b80b Add support for debugsource packages
These packages should behave like regular debuginfo packages (at least
for now).

Fixes: https://pagure.io/pungi/issue/684
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-26 10:34:18 +02:00
Lubomír Sedlář 65078ef9cf gather: Don't pull multiple debuginfo packages
When there is a noarch subpackage, all compatible debuginfo would be
pulled in, which is not desirable.

Example: Server.x86_64 needs pkg.x86_64 and pkg-data.noarch. We only
want pkg-debuginfo.x86_64, but without this patch even
pkg-debuginfo.i686 would get in.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-20 15:00:09 +02:00
Jan Kaluza 5dd6b1b0e7 GatherSourceModule: return rpm_obj instead of the rpm_obj.name
Merges: https://pagure.io/pungi/pull-request/680
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2017-07-20 14:58:58 +02:00
Lubomír Sedlář 32ca02efd6 gather: Stop requiring comps file in nodeps
When there are no groups, we shouldn't try to read comps file (because
it may very well not be there).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-20 13:11:06 +02:00
Lubomír Sedlář 085a8ef7c7 checksum: Checksum each image only once
There is no point in reading the same image multiple times. This happens
for at least source ISOs.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-17 08:26:51 +02:00
Lubomír Sedlář 81cb0952ca checksum: Refactor creating checksum files
Instead of iterating over the images metadata and appending the checksum
to relevant files immediately, we should store them and write only once.

This avoid an issue when the same image is mentioned in the metadata
multiple times. This happens for source images that are listed under
each binary arch.

The unified isos script is updated to use the exact same logic and code.
This also uncovered a problem with the metadata for debuginfo unified
isos: their paths in metadata were incorrect, which lead to missing
checksums.

Fixes: https://pagure.io/pungi/issue/667
Fixes: https://pagure.io/pungi/issue/668
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-17 08:26:51 +02:00
Lubomír Sedlář a831d65c40 createrepo: Don't use existing metadata with deltas
When creating the final repo, we reuse metadata from arch repo used for
depsolving. This however breaks creating deltas with createrepo_c.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-14 14:29:16 +02:00
Lubomír Sedlář 4d117d17f8 util: Fix finding older compose
When there are composes with two digit respin, the code would prefer 9
over 10 as latest. Respin needs to be treated as a number.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-14 10:42:35 +02:00
Lubomír Sedlář 910f816be4 createrepo: Use correct paths for old package dirs
Createrepo expects to be pointed to a directory with the actual RPM
files, not the previous repo. This means that when hashed directories
are used, we need to pass in a lot of directories.

Fixes: https://pagure.io/pungi/issue/344
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-14 10:40:44 +02:00
Lubomír Sedlář e246aef5f6 ostree: Add URL to repo to message
The message announcing new ostree commit contains hash of the commit,
the ref it's for, but there is no information about where the repo
actually is.

This patch adds `repo_path` key into the message with URL of the repo
and `local_repo_path` with path to the repo on local filesystem.

Relates: https://pagure.io/pungi/issue/650
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-11 10:24:49 +02:00
Lubomír Sedlář 8c48dfb93a gather: nodeps should take packages from comps groups
When gather_method is set to nodeps, we should not ignore the comps
group that the method received. Instead it should find out which
packages are in those groups and take them into the compose.

In order for this to be of any reasonable use, the comps file needs to
include all dependencies for the packages.

Fixes: #653
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-11 10:21:13 +02:00
Ken Dreyer 5acfb90b23 createrepo: handle missing product ids scm dir
Prior to this change, if the entire product IDs SCM directory was missing, pungi would crash with an error.

For example, if "ceph-3" was missing from the SCM:

    OSError: [Errno 2] No such file or directory: '/tmp/tmpMb9O6r/product_ids/ceph-3'

This occurred even if product_id_allow_missing was set to True.

Make product_id_allow_missing cover this case as well, and gracefully
skip all product IDs.

We now see the following warning in the logs instead:

    [WARNING ] No product IDs in {'scm': 'git', 'repo': 'git://example.com/rcm/rcm-metadata.git', 'dir': 'product_ids/ceph-3'}

and the compose succeeds.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2017-07-07 09:59:17 -06:00
Lubomír Sedlář a21c8a555d notification: Allow specifying multiple scripts
The notification hooks can be useful for doing other things than just
announcing status on message bus. For this to be truly usable, we need
the ability to use multiple scripts.

This patch allows the command line option to be specified multiple
times. Each given script will be called. Even if the script fails, it
does not block the compose.

Additionally the output of the notification scripts is logged now to
make it possible to debug possible failure.

Relates: https://pagure.io/pungi/issue/650
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-29 08:30:56 +02:00
Qixiang Wan 18bd37ff2c pkgset: Allow populating packages from multiple koji tags
One of the use cases is https://pagure.io/odcs/issue/7.

Merges: #660
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-06-29 08:27:34 +02:00
Lubomír Sedlář ecbb43ab86 variants-wrapper: Remove main() function
Nothing is calling this code, and it's only really useful for manual
testing of parsing.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-27 09:36:48 +02:00
Lubomír Sedlář 69a6046bf8 multilib_yum: Remove main() function
Nothing is calling this file as an executable, so there is no reason to
have logic for parsing arguments.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-27 09:36:39 +02:00
Colin Walters af631f560e ostree: Don't automatically create a repo
Creating an OSTree repository is a notable event; the general
expectation is that rather than having lots of repositories,
one has branches inside a single repository.

For $reasons, Fedora is not currently doing this, but we will
change it to do so.

The reason I'm making this change is we discovered that
it looked like Fedora had somehow made a repo inside a repo,
presumably due to a configuration error.

https://lists.fedoraproject.org/archives/list/cloud@lists.fedoraproject.org/message/GBFSOLULGGZFGEFCIW6FG23NZZV5VH4K/
Signed-off-by: Colin Walters <walters@verbum.org>
2017-06-26 17:47:35 -04:00
Lubomír Sedlář bfc1cebbc4 osbs: Config validation should accept a list
There can be multiple images listed for a single variant, the config
validation should not reject it.

The syntax with a single config object is still accepted. The price for
that is less descriptive error message when there are errors.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-23 12:39:48 +02:00
Martin Curlej 079454c502 pkgset: Use release number of a module
Signed-off-by: Martin Curlej <mcurlej@redhat.com>

updated regex, added comment, pep8 fix
2017-06-21 15:53:39 +02:00
Dennis Gilmore 65b75e7049 Merge #646 `ostree-installer: Clean up output dir` 2017-06-20 15:31:09 +00:00
Lubomír Sedlář b2554ce663 gather: Log tag from which we pulled a package
For each tag we ask Koji about (there might be more than one in
modularity case), we create a log file with list of RPMs and details
about which tag they were pulled from. This makes it easier to find out
where the package is inherited from.

Fixes: #547
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-19 10:51:37 +02:00
Lubomír Sedlář 4fb28979cf gather: Display source repo of packages
When multiple repos are configured in pkgset_repos, the logs should
contain information on where exactly the package was pulled from. The
log file in question should be in
`logs/<arch>/pkgset_source.<arch>.log`.

Fixes: #545
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-16 12:39:04 +02:00
Lubomír Sedlář c337018294 pkgset: Use descriptive name for log file
When repos are used as pkgset source, the logs are stored in a file that
should have a better name than `fooo.<arch>.log`.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-16 09:57:49 +02:00
Lubomír Sedlář 6e6e250cec ostree-installer: Clean up output dir
The Koji task can be restarted and lorax will fail if the output
directory already exists. Let's start the work in runroot by removing
the output directory.

Relates: #641
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-15 08:17:07 +02:00
Jan Kaluza f7197ddbcc Allow gather source classes to return SimpleRpmWrapper objects from pkgset phase directly.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2017-06-14 08:51:37 +02:00
Lubomír Sedlář 81b71b9ed3 koji-wrapper: Handle failed subtasks
If a subtask fails, we can't ask about it's results as that would raise
an exception. We can safely assume that since the parent succeeded, any
failed child is actually allowed to fail.

Fixes: #641
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-12 18:15:29 +02:00