Commit Graph

1496 Commits

Author SHA1 Message Date
Lubos Kocman 1a8f0bb0f9 gather_dnf.py: use self.logger for printing output
Signed-off-by: Lubos Kocman <lkocman@redhat.com>
2017-03-24 09:24:47 +01:00
Daniel Mach 320d2a8ae2 Boost add_initial_packages() by using query cache for 'build' greedy method.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:47 +01:00
Daniel Mach c02d801988 Boost prepopulate by using query cache.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:47 +01:00
Daniel Mach 321d471125 Additional profiling in Gather.add_initial_packages().
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:47 +01:00
Daniel Mach 5a2c4f5e0f Improve working with query cache.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:47 +01:00
Daniel Mach 06635a3917 Add profiler to pungi-gather.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:47 +01:00
Daniel Mach 61e76d5824 Improve depsolving speed by pre-computing provides query.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:47 +01:00
Michael Mraka 026e49e259 speedup filter(sourcerpm=XXX)
originally 6519 calls 18, i.e. 2% out of 15 mins

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:47 +01:00
Michael Mraka 835460ce8f speedup filter(name=XXX, version=YYY, release=ZZZ)
originally 3109 calls 54s, i.e. 6% out of 15 mins

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:47 +01:00
Michael Mraka 110fe7647c speedup filter(pkg=XXX, arch__neq=WWW)
originally 17976 calls 157s, i.e. 17% out of 15 mins

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:46 +01:00
Michael Mraka ec56ee75e2 speedup filter(name=XXX, version=YYY, release=ZZZ, arch__neq=WWW)
the second most demanding filter
it is called 8297 times and gets 189s out of 15min run which is 21%
spent here

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:46 +01:00
Michael Mraka d8fcaf95f7 speedup filter(sourcerpm=XXX, arch__neq=YYY)
this is most demanding filter
it is called 10826 times and gets 210.277010s out of 15min run which is
23% spent here

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:46 +01:00
Lubomír Sedlář 36f57b26b8 Add scripts to compare YUM and DNF gathering
There are scripts to re-run depsolving with any backend based on config
and log file.

There is a new script to compare logs from two runs and report
differences. This script will be installed system wide in final RPM.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:46 +01:00
Daniel Mach 99b68ca96b Simplify ks group handling.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:12:23 +01:00
Daniel Mach 7cb9e9dc20 Add support for mirrorlists in DNF gathering code.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:12:23 +01:00
Daniel Mach ddba26edd7 Don't run fulltree on packages added via fulltree.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:12:23 +01:00
Daniel Mach a5d302d036 Enable --fulltree, --selfhosting and --nodeps in pungi-gather.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:12:23 +01:00
Daniel Mach aff6ac906c Enable langpack gathering in DNF gather.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:12:23 +01:00
Daniel Mach 43d8d02cb1 Integrate DNF gathering code with Pungi.
Set 'dnf_gather = True' (temporary option) in config file to enable DNF
gathering.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:12:23 +01:00
Daniel Mach 9764acbc4d Cache pkg.sourcerpm to get additional >5% boost.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 08:51:38 +01:00
Daniel Mach 13ed3f2b1d Performance improvements of DNF gathering.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 08:51:38 +01:00
Daniel Mach caf11f55d4 New gathering (depsolving) module based on DNF.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 08:51:38 +01:00
Jan Kaluza 9fcd71f831 Add support for modular composes
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2017-03-22 15:55:52 +01:00
Lubomír Sedlář 4b90822115 Add a script for modifying ISO images
With this script it's possible to add additional files into an ISO file.
If the file happens to be ks.cfg, the boot configs are tweaked so that
the kickstart is actually used.

Resolves: #503
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-21 07:55:19 +01:00
Lubomír Sedlář ae5ee3d856 iso-wrapper: Refactor functions
Split the function for getting implant MD5 into two functions, so that
we can have access to other information retrieved by checkisomd5
command. A test case is added for both functions.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-21 07:55:19 +01:00
Lubomír Sedlář 306f7e69b0 iso-wrapper: Add utility for mounting images
This patch refactors logic for creating a temporary mount point,
mounting an image, running arbitrary code on it, unmounting the image
and removing the mount point. It immediately uses it in the buildinstall
phase.

Similar mounting is present in product_img phase as well, but due to
different usage pattern it's not changed yet.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-21 07:55:19 +01:00
Lubomír Sedlář 05a666fb3b buildinstall: Move tweaking configs into a function
This will allow us to more easily test the code and also reuse it in
other places.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-21 07:55:19 +01:00
Lubomír Sedlář 55035487de image-build: Correctly write can_fail option
Koji expects the value as comma separated list, otherwise it will ignore
it. This makes it possible for the image building to fail on per arch
basis.

Relates: #128
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-20 09:05:03 +01:00
Qixiang Wan 79e97dc845 pungi-koji: new cmd option '--latest-link-status'
Add a new option 'latest-link-status' to pungi-koji, if this is
specified, pungi will only create the latest symbol link to the compose
when compose's status matches the specified statuses. The status name is
case insensitive. If the option is not specified it will act as before.

Example:

pungi-koji --target-dir=_composes --config=data/dummy-pungi.conf \
--test --latest-link-status=finished --latest-link-status=finished_incomplete

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-20 11:15:30 +08:00
Lubomír Sedlář 23e53219eb live-cd: Print task ID on success
Relates: #570
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-15 17:37:00 +01:00
Lubomír Sedlář 69af0a6a65 ostree-installer: Print task ID on success
Relates: #570
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-15 17:37:00 +01:00
Lubomír Sedlář 263a990489 ostree: Print task ID on success
Relates: #570
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-15 17:37:00 +01:00
Lubomír Sedlář 5ee285cc24 live-media: Print task ID on success
Relates: #570
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-15 17:37:00 +01:00
Lubomír Sedlář b10d275745 image-build: Print task ID on success
Relates: #570
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-15 17:37:00 +01:00
Dennis Gilmore b09641a708 Merge #568 `buildinstall: Print debug info if unmount fails` 2017-03-15 16:07:33 +00:00
Lubomír Sedlář 94b1159829 ostree-installer: Fix logging directory
Currently if there are multiple ostree installers for the same
architecture, the logfiles all end up in the same location and overwrite
each other.

This patch moves the logs to logs/<arch>/<variant>/ostree_installer-X/
for a unique value of X so that there can be multiple runs even for the
same tree.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-15 15:54:45 +01:00
Lubomír Sedlář f0dc15bdc6 buildinstall: Print debug info if unmount fails
When even after retries the unmounting still fails, this patch runs `ls
-lA`, `fuser -vm` and `lsof +D` on the directory to give some idea of
what's blocking it.

Relates: #559
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-14 15:06:20 +01:00
Qixiang Wan d081a4eda8 pkgset: report all unsigned packages
Fixes: #552
Merges: #567
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-14 09:50:28 +01:00
Qixiang Wan ee21663c8b default createrepo_checksum to sha256
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-14 13:15:43 +08:00
Lubomír Sedlář a1214525f5 unified-iso: Log better error when linking fails
When linking files fails due to target path already existing, we should
print details about the conflict.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-13 14:45:13 +01:00
Lubomír Sedlář 08b9d275a8 unified-iso: Blacklist extra files metadata
The file is in each variant that has extra files, and would create a
conflict when linking the files.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-13 14:45:13 +01:00
Lubomír Sedlář 5cd1c22e2d buildinstall: Retry unmounting image
If the image can not be unmounted because the device is busy, we should
retry. There will be increasing pauses between the attempts. At most 10
attempts will be done before giving up.

Fixes: #559
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-10 08:29:54 +01:00
Lubomír Sedlář 9041ccccc4 Remove indices from documentation
They are not generated as we don't actually document code. The search
page is removed as well (because it does not seem to work).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Fixes: #462
2017-03-09 12:47:12 +01:00
Lubomír Sedlář cd805a1e6d iso-wrapper: Handle wrong implant md5
If the checkisomd5 command exits successfully but returns a wrong value,
we should catch and log that. In theory this should be impossible, but
we have seen it in production.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-07 09:58:14 +01:00
Lubomír Sedlář 1647f7612a image-build: Remove check for number of images
If task does not produce the expected number of images, it should fail
at Koji level. Also, this check does not really work when some
architectures are allowed to fail.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-06 19:31:08 +01:00
Lubomír Sedlář 63a3b6ccaa Merge #538 `consolidate repo option names` 2017-03-06 12:55:44 +00:00
Lubomír Sedlář fa3d94bae3 Merge #554 `osbs: write manifest for scratch osbs` 2017-03-06 12:52:47 +00:00
Lubomír Sedlář d3a2ceb8ce Extract only first version from specfile
When tagging a new version, rpm will give us the version twice, one for
main package and one for the -utils subpackage. We should only use the
first one (they should be the same anyway).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-06 09:08:13 +01:00
Qixiang Wan 2ae8710934 consolidate repo option names
live_images:            additional_repos        -> repo
ostree:                 source_repo_from        -> repo_from
                        extra_source_repos      -> repo
ostree_installer:       source_repo_from        -> repo_from

With the change, the phases have consolidate option names for variant
repos and external repos.

Old option names will continue to work, old names will be converted
to new names after validation automatically if new options are not
specified in config.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-06 09:26:42 +08:00
Qixiang Wan c93207addb checks: extend validator with 'alias'
When a property has 'alias' defined, and it's not present in instance,
if the alias property is present, add the property with value from alias
property before remove the alias property from instance.

Examples:

with schema:
{
        "$schema": "http://json-schema.org/draft-04/schema#",
        "title": "Pungi Configuration",
        "type": "object",
        "properties": {
            "release_name": {"type": "string", "alias": "product_name"},
        },
        "required": ["release_name"],
        "additionalProperties": False,
}

1. config = {"release_name": "dummy product"}:

   validate pass, config not changed after validation.

2. config = {"product_name": "dummy product"}:

   validate pass, config updated to the following after validation:

   config: {"release_name": "dummy product"}

3. config = {"name": "dummy product"}:

   validate fail, errror message is "Failed validation in : 'release_name' is a required property",
   and warning message is "WARNING: Unrecognized config option: name."

4. config = {"product_name": "dummy product", "release_name": "dummy product"}

   validate fail, error message is "Failed validation in : product_name is an alias of release_name, only one can be used."

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-06 09:26:42 +08:00