Commit Graph

394 Commits

Author SHA1 Message Date
Ralph Bean
5b6e468952 Allow extracting profiling information from pungi-gather.
`pungi-gather` (the tool that underlies both the `pkgset` and `gather`
phases) contains profiling code that will log statistics about how long
different function calls take.  However, pungi-koji did not contain a
way to pass the ``--profiler`` argument to enable this.

This change adds a new configuration option ``gather_profiler`` which,
when set to true, simply passes the argument to `pungi-koji`.  Hopefully
this can help shed some light on what is happening in some of our
longer-running composes.

Merges: https://pagure.io/pungi/pull-request/727
Signed-off-by: Ralph Bean <rbean@redhat.com>
2017-09-04 10:25:11 +02:00
Lubomír Sedlář
40796c04f4 createrepo: Only consider successful compose for deltas
If the compose failed, it may not have repos to compute deltas against,
and even if it has them, they were never shipped so no one will have the
older version of the package. We should instead go deeper in history and
pick a successful compose.

Relates: https://pagure.io/pungi/issue/715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-09-04 09:15:20 +02:00
Lubomír Sedlář
e21a27bdc9 createrepo: Allow selecting variants for delta RPMs
The configuration needs to be more granular than a single global option.
With this patch each tree can enable deltas separately.

Fixes: https://pagure.io/pungi/issue/715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-09-04 09:15:20 +02:00
Lubomír Sedlář
90be25c14c createrepo: Only create delta RPMs for binary repos
It does not make much sense to have deltas for source and debug repos.
No one benefits from it really and it takes a long time.

Relates: https://pagure.io/pungi/issue/715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-09-04 09:15:20 +02:00
Qixiang Wan
b81e94e808 image-build: add arch name(s) in image config file name
Pungi write image config file with name of <format>-<name>.cfg, if there
are two or more image configs present for different arches under the same
variant and with same format & name, the config file can be overwritten,
and result in invalid image conf file.

Example:

image_build = {
    '^Server$': [
        {
            'image-build': {
                'format': [('qcow2', 'qcow2'),],
                'name': 'fedora-guest-image',
                'target': 'guest-fedora-26-image',
                'version': '26',
                'ksurl': "git://git.example.com/ks.git?fedora#HEAD",
                'kickstart': "fedora-26-kvm.ks",
                'ksversion': 'f26',
                'distro': 'fedora-26',
                'disk-size': '10',
                'arches': ['x86_64'],
                'repo': ["http://example.com/linux/fedora/26/Everything/x86_64/os", ]
            }
        },
        {
           'image-build': {
                'format': [('qcow2', 'qcow2'),],
                'name': 'fedora-guest-image',
                'target': 'guest-fedora-26-image',
                'version': '26',
                'ksurl': "git://git.example.com/ks.git?fedora#HEAD",
                'kickstart': "fedora-26-kvm.ks",
                'ksversion': 'f26',
                'distro': 'fedora-26',
                'disk-size': '10',
                'arches': ['ppc64le'],
            }
        },
    ],
}

In this case, config file "qcow2_guest-fedora-26-image.cfg" will be
created for both x86_64 and ppc64le under the same variant dir, and
there is a high chance it will be over-written while Pungi creating the
koji task. We can add arch name(s) in config filename to avoid that.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-09-03 23:51:08 +08:00
Lubomír Sedlář
65910f2c33 Open files as binary where needed
In many cases we need to open files as binary to avoid errors on Py3
about writing binary data to file opened in text mode.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-28 13:47:18 +02:00
Lubomír Sedlář
fcbc3ed4ae buildinstall: No copy if task fails
Even a failed lorax task can leave behind some in-progress images. Pungi
needs to copy the files into compose/ subdirectory only if the task
finished successfully.

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1485021
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-28 13:43:32 +02:00
Lubomír Sedlář
a63e4746c9 Report warning when config sections are not used
It's possible a variant is excluded via tree_variants option and the
section does not match anything. It can be confusing to users why
nothing is happening. This patch lets Pungi log all unmatched patterns.

Fixes: https://pagure.io/pungi/issue/692
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-21 09:07:10 +02:00
Lubomír Sedlář
c3b49f7ffb pkgset: Download packages with dnf
When using repos as gather_source, we should use DNF backend even for
constructing initial package set and to download the packages from
source repos. Without this the repos source would not be usable on
Python 3.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-16 14:32:26 +02:00
Lubomír Sedlář
be4501c54b gather: Fix duplicated log line
Due to overwriting an existing variable the logs are getting duplicated
line about missing comps packages instead of announcement of gathering
being finished. Rename the variable to fix the problem.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-16 13:14:21 +02:00
Lubomír Sedlář
804a0049f6 ostree: Simplify configuration
It makes no sense to repeat the same configuration for multiple
architectures. Instead we should just list the architectures as another
key in the mapping. There is an option to specify multiple config dicts.

This preserves full backwards compatibility, the old config format is
still accepted.

Fixes: https://pagure.io/pungi/issue/678
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-14 14:53:22 +02:00
Lubomír Sedlář
c9f34b6684 gather: Only parse pungi log once
No need to read the file three times, we can just get all the data in
one go.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-09 11:04:14 +02:00
Lubomír Sedlář
cef8650c3e gather: Report missing comps packages
When a package mentioned in comps is not available in the package set,
print a warning about this. Additionally there is a config option that
allows to turn this warning into a fatal error.

Fixes: https://pagure.io/pungi/issue/50
Fixes: https://pagure.io/pungi/issue/683
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-09 11:04:14 +02:00
Lubomír Sedlář
1d7617f783 gather: Avoid reading whole log into memory
There's no reason for reading the whole log of depsolving into memory
just to split it into lines and process one line at a time.

We can just as well read it in chunks.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-09 11:04:14 +02:00
Lubomír Sedlář
e2962dc547 repoclosure: Allow aborting compose when repoclosure fails
Alternatively the call to repoclosure can be turned off. This is
customizable per variant and architecture.

Fixes: https://pagure.io/pungi/issue/676
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-07 15:17:33 +02:00
Lubomír Sedlář
056ae31ef9 repoclosure: Fix logging errors
Instead of just printing the error directly to stderr, capture the
output and use proper logger. This makes sure the error is included in
the log file and also fixes `--quiet` option which was not properly
honored originally.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-07 15:17:33 +02:00
Ralph Bean
381d08a810 Make strict the only option.
Signed-off-by: Ralph Bean <rbean@redhat.com>
2017-08-07 14:56:59 +02:00
Ralph Bean
58fe997e29 Raise a ValueError with details if module not found in PDC.
Currently, this will fail two lines later when we try to access
`pdc_module['modulemd']` with an unhelpful `TypeError` since
`pdc_module` is `None`.

Signed-off-by: Ralph Bean <rbean@redhat.com>
2017-08-07 08:34:45 -04:00
Lubomír Sedlář
65078ef9cf gather: Don't pull multiple debuginfo packages
When there is a noarch subpackage, all compatible debuginfo would be
pulled in, which is not desirable.

Example: Server.x86_64 needs pkg.x86_64 and pkg-data.noarch. We only
want pkg-debuginfo.x86_64, but without this patch even
pkg-debuginfo.i686 would get in.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-20 15:00:09 +02:00
Jan Kaluza
5dd6b1b0e7 GatherSourceModule: return rpm_obj instead of the rpm_obj.name
Merges: https://pagure.io/pungi/pull-request/680
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2017-07-20 14:58:58 +02:00
Lubomír Sedlář
32ca02efd6 gather: Stop requiring comps file in nodeps
When there are no groups, we shouldn't try to read comps file (because
it may very well not be there).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-20 13:11:06 +02:00
Lubomír Sedlář
085a8ef7c7 checksum: Checksum each image only once
There is no point in reading the same image multiple times. This happens
for at least source ISOs.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-17 08:26:51 +02:00
Lubomír Sedlář
81cb0952ca checksum: Refactor creating checksum files
Instead of iterating over the images metadata and appending the checksum
to relevant files immediately, we should store them and write only once.

This avoid an issue when the same image is mentioned in the metadata
multiple times. This happens for source images that are listed under
each binary arch.

The unified isos script is updated to use the exact same logic and code.
This also uncovered a problem with the metadata for debuginfo unified
isos: their paths in metadata were incorrect, which lead to missing
checksums.

Fixes: https://pagure.io/pungi/issue/667
Fixes: https://pagure.io/pungi/issue/668
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-17 08:26:51 +02:00
Lubomír Sedlář
a831d65c40 createrepo: Don't use existing metadata with deltas
When creating the final repo, we reuse metadata from arch repo used for
depsolving. This however breaks creating deltas with createrepo_c.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-14 14:29:16 +02:00
Lubomír Sedlář
910f816be4 createrepo: Use correct paths for old package dirs
Createrepo expects to be pointed to a directory with the actual RPM
files, not the previous repo. This means that when hashed directories
are used, we need to pass in a lot of directories.

Fixes: https://pagure.io/pungi/issue/344
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-14 10:40:44 +02:00
Lubomír Sedlář
e246aef5f6 ostree: Add URL to repo to message
The message announcing new ostree commit contains hash of the commit,
the ref it's for, but there is no information about where the repo
actually is.

This patch adds `repo_path` key into the message with URL of the repo
and `local_repo_path` with path to the repo on local filesystem.

Relates: https://pagure.io/pungi/issue/650
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-11 10:24:49 +02:00
Lubomír Sedlář
8c48dfb93a gather: nodeps should take packages from comps groups
When gather_method is set to nodeps, we should not ignore the comps
group that the method received. Instead it should find out which
packages are in those groups and take them into the compose.

In order for this to be of any reasonable use, the comps file needs to
include all dependencies for the packages.

Fixes: #653
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-11 10:21:13 +02:00
Ken Dreyer
5acfb90b23 createrepo: handle missing product ids scm dir
Prior to this change, if the entire product IDs SCM directory was missing, pungi would crash with an error.

For example, if "ceph-3" was missing from the SCM:

    OSError: [Errno 2] No such file or directory: '/tmp/tmpMb9O6r/product_ids/ceph-3'

This occurred even if product_id_allow_missing was set to True.

Make product_id_allow_missing cover this case as well, and gracefully
skip all product IDs.

We now see the following warning in the logs instead:

    [WARNING ] No product IDs in {'scm': 'git', 'repo': 'git://example.com/rcm/rcm-metadata.git', 'dir': 'product_ids/ceph-3'}

and the compose succeeds.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2017-07-07 09:59:17 -06:00
Qixiang Wan
18bd37ff2c pkgset: Allow populating packages from multiple koji tags
One of the use cases is https://pagure.io/odcs/issue/7.

Merges: #660
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-06-29 08:27:34 +02:00
Martin Curlej
079454c502 pkgset: Use release number of a module
Signed-off-by: Martin Curlej <mcurlej@redhat.com>

updated regex, added comment, pep8 fix
2017-06-21 15:53:39 +02:00
Dennis Gilmore
65b75e7049 Merge #646 ostree-installer: Clean up output dir 2017-06-20 15:31:09 +00:00
Lubomír Sedlář
b2554ce663 gather: Log tag from which we pulled a package
For each tag we ask Koji about (there might be more than one in
modularity case), we create a log file with list of RPMs and details
about which tag they were pulled from. This makes it easier to find out
where the package is inherited from.

Fixes: #547
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-19 10:51:37 +02:00
Lubomír Sedlář
c337018294 pkgset: Use descriptive name for log file
When repos are used as pkgset source, the logs are stored in a file that
should have a better name than `fooo.<arch>.log`.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-16 09:57:49 +02:00
Lubomír Sedlář
6e6e250cec ostree-installer: Clean up output dir
The Koji task can be restarted and lorax will fail if the output
directory already exists. Let's start the work in runroot by removing
the output directory.

Relates: #641
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-15 08:17:07 +02:00
Jan Kaluza
f7197ddbcc Allow gather source classes to return SimpleRpmWrapper objects from pkgset phase directly.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2017-06-14 08:51:37 +02:00
Lubomír Sedlář
700106facf arch: Move exclu(de|sive)arch check to a function
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-06 15:28:12 +02:00
Jan Kaluza
3601d6d1a8 gather-source: Check arch in module source
Skip the RPM if it is excluded on this arch or exclusive for different
arch.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2017-06-06 15:25:41 +02:00
Lubomír Sedlář
247a1a71ba Ensure all phases are stopped
If a phase is started successfully, it needs to be stopped as well. In
most cases when `stop` is called immediately after `start`, this is not
a problem.

Only when something else happens while a phase is runnning and this
something fails, Pungi will deadlock and never exit. This something
could be another phase or just main thread raising an exception.

Fixes: #625
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-02 09:52:18 +02:00
Jan Kaluza
118444a311 Generate proper modular metadata when there are different versions of the same package in the variant
Merges: #629
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2017-05-30 15:18:09 +02:00
Jan Kaluza
68351fa5a8 Add @retry decorator and use it to retry connection on PDC on IOError and in SCM's retry_run.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2017-05-29 11:33:23 +02:00
Lubomír Sedlář
7b880af0ad pkgset: Remove use of undefined variable
As long as the assertion is not violated, this would not be a problem as
the second argument is not evaluated. However if the condition was
broken, a NameError would be raised instead of AssertionError with a
nice message.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-05-05 07:54:32 +02:00
Jan Kaluza
d037d61521 Store RPM artifacts in resulting repository in modulemd metadata.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2017-05-04 21:12:30 +02:00
Lubomír Sedlář
c784dab4aa gather: Remove handling of impossible state
The get_system_release_packages function can never be called without a
variant, so it makes no sense to check for that condition.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-05-02 10:12:16 +02:00
Lubomír Sedlář
834445e7d0 gather: Clean up code
* add comments explaining what's going on
 * break too long lines
 * simplify the logic where possible
 * use with statement to work with files
 * remove commented out and unused code
 * introduce helpers to reduce code duplication

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-05-02 10:12:14 +02:00
Lubomír Sedlář
7028399403 util: Move get_buildroot_rpms to koji wrapper
This way the util module does not import the particular wrapper.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-25 15:22:41 +02:00
Lubomír Sedlář
c293a1e147 extra-files: Allow configuring used checksums
Instead of adding a new config option, we can just reuse the existing
`media_checksums` value. If the value is good for image checksums, it
should work for extra files as well.

Relates: #591
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-20 09:28:09 +02:00
Lubomír Sedlář
20005adb98 comps-wrapper: Return IDs instead of yum.comps.Group
We never need anything more than the ID, so passing around a big
complicated object is not necessary.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-13 10:28:26 +02:00
Lubomír Sedlář
f6d07c1651 Be explicit about generating release for images
The config now uses similar logic what previous commit did for OSTree.
Also we should report error when an unknown generator is used.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-13 10:17:59 +02:00
Lubomír Sedlář
98f40f6138 ostree: Autogenerate a version
If the value is a particular magic string, we will expand it to a proper
value.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-13 10:17:59 +02:00
Ralph Bean
01607602c6 Expand compatible arches when gathering from modules.
Fixes #596.

Signed-off-by: Ralph Bean <rbean@redhat.com>
2017-04-12 14:35:46 -04:00