Commit Graph

382 Commits

Author SHA1 Message Date
Lubomír Sedlář
c293a1e147 extra-files: Allow configuring used checksums
Instead of adding a new config option, we can just reuse the existing
`media_checksums` value. If the value is good for image checksums, it
should work for extra files as well.

Relates: #591
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-20 09:28:09 +02:00
Lubomír Sedlář
610c4ec596 util: Report better error on resolving non-existing branch
When the config contains a git url pointing to a non-existing branch,
pungi will fail to get commit hash from that branch and die with a
confusing error message.

Fixes: #583
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-19 15:37:15 +02:00
Lubomír Sedlář
2bc719a33a util: Show choices for volid if all are too long
When we fail to generate a volume ID that fits in 32 characters, the
error message should include the options that were considered. It could
show that there might be a substitution that could fix the problem.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-18 16:13:55 +02:00
Qixiang Wan
63327e7d88 checks: Fix anyOf validator yield ValidationError on ConfigOptionWarning
We have some hooks yield ConfigOptionWarning. When it happens within
anyOf validator, anyOf validator yield ValidationError and reports the
config as incorrect. We need to overwrite it to pass not break.

Fixes: #598
Merges: #599
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-04-18 10:05:03 +02:00
Lubomír Sedlář
39c3f42f77 comps-wrapper: Sort langpacks by name
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-13 11:45:23 +02:00
Lubomír Sedlář
fbf189d9c7 comps-wrapper: Add tests
This tests pretty much all the code that parses, filters and then writes
comps back to a file.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-13 11:45:23 +02:00
Lubomír Sedlář
f6d07c1651 Be explicit about generating release for images
The config now uses similar logic what previous commit did for OSTree.
Also we should report error when an unknown generator is used.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-13 10:17:59 +02:00
Lubomír Sedlář
98f40f6138 ostree: Autogenerate a version
If the value is a particular magic string, we will expand it to a proper
value.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-13 10:17:59 +02:00
Lubomír Sedlář
dcc1750df2 gather: Clean up method deps
* Remove unused arguments
 * Remove duplication

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-11 15:46:34 +02:00
Lubomír Sedlář
0168388492 gather: Report error if there is no input
Running depsolving with no requested inputs will only lead to a hard to
decipher error. We should instead explicitly tell the user that there is
a problem.

Unit tests are added to add to test this functionality.

Relates: #585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-11 15:46:34 +02:00
Lubomír Sedlář
b5efb67ff1 init: Warn when variants mentions non-existing comps group
When variants XML lists a group that does not match any known group in
input comps, report a warning. This is not necessarily a problem in
itself, but having this information in the log can help debug problems.

Relates: #585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-11 15:46:34 +02:00
Qixiang Wan
2f5d6d7dcd unify repo and repo_from options
Config option 'repo' and 'repo_from' are used in several phases, merge
them with one option 'repo'. 'append' in schema is used for appending
the values from deprecated options to 'repo', so it won't break on any
existing config files that have the old options of 'repo_from' and
'source_repo_from' (which is an alias of 'repo_from').

And 'repo' schema is updated to support repo dict as the value or an
item in the values, a repo dict is just a dict contains repo options,
'baseurl' is required in the dict, like:

{"baseurl": "http://example.com/url/to/repo"}

or:

{"baseurl": "Serer"}

currently this is used in ostree phase to support extra repo options
like:

{"baseurl": "Server", "exclude": "systemd-container"}

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-29 10:12:32 +08:00
Qixiang Wan
0f508e2228 move translate_path from paths.py to util.py
So translate_path can be used in util.py, or it will result in
circular import error.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-29 10:12:32 +08:00
Qixiang Wan
d1763fca7e checks.py: support 'append' option
If 'append' is defined for a property, append the values from append
options to the property. Note: The property must support to be a list
of values.

For example:

with schema:

    schema = {
        "$schema": "http://json-schema.org/draft-04/schema#",
        "title": "Pungi Configuration",
        "type": "object",
        "definitions": {
            "list_of_strings": {
                "type": "array",
                "items": {"type": "string"},
            },
            "strings": {
                "anyOf": [
                    {"type": "string"},
                    {"$ref": "#/definitions/list_of_strings"},
                ]
            },
        },
        "properties": {
            "release_name": {"type": "string"},
            "repo": {"$ref": "#/definitions/strings", "append": "repo_from"}
        },
        "additionalProperties": False,
    }

and config:

    repo = "http://url/to/repo"
    repo_from = "Server"

config will be updated to:

    repo = ["http://url/to/repo", "Server"]

It supports multiple append options too, like:

    "repo": {
        "$ref": "#/definitions/strings",
        "append": ["repo_from", "source_repo_from"],
    }

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-29 10:12:32 +08:00
Qixiang Wan
2aacefd9cd checks.py: show warning message for alias option
Show warning message for any alias option find in config instance.
Example warning message:

WARNING: Config option 'product_name' is deprecated and now an alias to
'release_name', please use 'release_name' instead. In:
{'release_name': 'dummy product', 'product_name': 'dummy product'}

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-29 10:12:32 +08:00
Lubomír Sedlář
2b21e13aad Skip DNF tests if there are import problems
If there are import errors for DNF, multilib or other related package,
we can assume the tests are running on EPEL. The DNF tests should be
skipped in that case.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-27 09:42:58 +02:00
Lubomír Sedlář
76cf4a7540 Update to use python-multilib
We also rename the old multilib module used by dnf code to multilib_yum
to make it clear what is imported where.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:50 +01:00
Lubomír Sedlář
27b24a489f Handle noarch debuginfo packages
Pull debuginfo packages for noarch packages as well, but only when the
debuginfo is noarch as well.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:50 +01:00
Lubomír Sedlář
d23a2f4548 Fix multilib exclude pattern to not match noarch
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:50 +01:00
Lubomír Sedlář
35e72df99f Explicitly remove all temporary files
There is no guarantee __del__ will ever be called, and we were leaving a
ton of stuff in /tmp. With this patch we pass the temporary directories
explictly and make sure they are deleted at the end.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář
fe026bb588 Handle Requires(pre|post)
These requires are accessible from a separate attribute, but we want to
handle them the same ways as regular Requires.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář
d4f8e32a80 Fix handling globs in input packages
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář
cd80d6ef5a Stop caching provides of added packages
This only works in non-greedy mode. When greedy, the same provides can
be linked to multiple packages that should be pulled in.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář
e06e9165fc Add checking of flags for dnf depsolving
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář
ec67eac1cc Use enum for flags
This adds an extra dependency on python-enum34.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář
9fd8e6319f Handle fulltree_excludes list
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář
b53e4a84d3 Fix fulltree when greedy
When greedy, fulltree should pull in all native and multilib packages.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:48 +01:00
Lubomír Sedlář
0a44b2fd07 Filter source packages
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:48 +01:00
Lubomír Sedlář
99699b85aa Use same tests for yum and dnf depsolving
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:48 +01:00
Jan Kaluza
9fcd71f831 Add support for modular composes
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2017-03-22 15:55:52 +01:00
Lubomír Sedlář
4b90822115 Add a script for modifying ISO images
With this script it's possible to add additional files into an ISO file.
If the file happens to be ks.cfg, the boot configs are tweaked so that
the kickstart is actually used.

Resolves: #503
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-21 07:55:19 +01:00
Lubomír Sedlář
306f7e69b0 iso-wrapper: Add utility for mounting images
This patch refactors logic for creating a temporary mount point,
mounting an image, running arbitrary code on it, unmounting the image
and removing the mount point. It immediately uses it in the buildinstall
phase.

Similar mounting is present in product_img phase as well, but due to
different usage pattern it's not changed yet.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-21 07:55:19 +01:00
Lubomír Sedlář
05a666fb3b buildinstall: Move tweaking configs into a function
This will allow us to more easily test the code and also reuse it in
other places.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-21 07:55:19 +01:00
Dennis Gilmore
b09641a708 Merge #568 buildinstall: Print debug info if unmount fails 2017-03-15 16:07:33 +00:00
Lubomír Sedlář
94b1159829 ostree-installer: Fix logging directory
Currently if there are multiple ostree installers for the same
architecture, the logfiles all end up in the same location and overwrite
each other.

This patch moves the logs to logs/<arch>/<variant>/ostree_installer-X/
for a unique value of X so that there can be multiple runs even for the
same tree.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-15 15:54:45 +01:00
Lubomír Sedlář
f0dc15bdc6 buildinstall: Print debug info if unmount fails
When even after retries the unmounting still fails, this patch runs `ls
-lA`, `fuser -vm` and `lsof +D` on the directory to give some idea of
what's blocking it.

Relates: #559
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-14 15:06:20 +01:00
Qixiang Wan
d081a4eda8 pkgset: report all unsigned packages
Fixes: #552
Merges: #567
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-14 09:50:28 +01:00
Lubomír Sedlář
5cd1c22e2d buildinstall: Retry unmounting image
If the image can not be unmounted because the device is busy, we should
retry. There will be increasing pauses between the attempts. At most 10
attempts will be done before giving up.

Fixes: #559
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-10 08:29:54 +01:00
Lubomír Sedlář
cd805a1e6d iso-wrapper: Handle wrong implant md5
If the checkisomd5 command exits successfully but returns a wrong value,
we should catch and log that. In theory this should be impossible, but
we have seen it in production.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-07 09:58:14 +01:00
Lubomír Sedlář
63a3b6ccaa Merge #538 consolidate repo option names 2017-03-06 12:55:44 +00:00
Qixiang Wan
2ae8710934 consolidate repo option names
live_images:            additional_repos        -> repo
ostree:                 source_repo_from        -> repo_from
                        extra_source_repos      -> repo
ostree_installer:       source_repo_from        -> repo_from

With the change, the phases have consolidate option names for variant
repos and external repos.

Old option names will continue to work, old names will be converted
to new names after validation automatically if new options are not
specified in config.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-06 09:26:42 +08:00
Qixiang Wan
c93207addb checks: extend validator with 'alias'
When a property has 'alias' defined, and it's not present in instance,
if the alias property is present, add the property with value from alias
property before remove the alias property from instance.

Examples:

with schema:
{
        "$schema": "http://json-schema.org/draft-04/schema#",
        "title": "Pungi Configuration",
        "type": "object",
        "properties": {
            "release_name": {"type": "string", "alias": "product_name"},
        },
        "required": ["release_name"],
        "additionalProperties": False,
}

1. config = {"release_name": "dummy product"}:

   validate pass, config not changed after validation.

2. config = {"product_name": "dummy product"}:

   validate pass, config updated to the following after validation:

   config: {"release_name": "dummy product"}

3. config = {"name": "dummy product"}:

   validate fail, errror message is "Failed validation in : 'release_name' is a required property",
   and warning message is "WARNING: Unrecognized config option: name."

4. config = {"product_name": "dummy product", "release_name": "dummy product"}

   validate fail, error message is "Failed validation in : product_name is an alias of release_name, only one can be used."

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-06 09:26:42 +08:00
Qixiang Wan
40e2874676 osbs: write manifest for scratch osbs
FIXES: #485

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-06 09:26:01 +08:00
Dennis Gilmore
8b1fb287d3 Merge #539 comps: Filter comps groups for optional variants 2017-02-23 20:13:39 +00:00
Lubomír Sedlář
4ea1916a87 Enable customizing runroot task weight
For different cases where runroot is used it's now possible to set
custom weight. The usecase for this is to avoid one builder taking too
many tasks. Especially buildinstall is quite resource intensive, so one
builder taking multiple tasks at the same time leads to very slow
compose time.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-23 12:44:27 +01:00
Lubomír Sedlář
8bc65a8be5 comps: Filter comps groups for optional variants
The optional variant can defined by just adding has_optional=True into
variant xml. In such case it has no comps groups and Pungi would copy
the original file unmodified. This leads to extra packages being pulled
into the optional variant.

In this case the correct solution is to filter the comps and remove all
groups.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-23 12:44:20 +01:00
Lubomír Sedlář
857aee05c1 util: Add a utility for managing temporary files
In multiple situations we need to create temporary files or directories
that should not be preserved after compose is finished. Let's add
context managers that ensure these get cleaned up.

This fixes tests leaving garbage around in /tmp.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-20 09:14:25 +01:00
Daniel Mach
ad23efd323 Refactor depsolving tests
This adds a new test for Requires(pre) and (post). The general structure
of the test now makes it easy to use the same test scenarios for
different backend.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-13 12:32:07 +01:00
Lubomír Sedlář
9533fca96c koji_wrapper: Always use --profile option with koji
This patch adds this option to all invoked koji commands.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-30 13:00:25 +01:00
Dennis Gilmore
42cdf047cc add missing koji_profile from test compose setting
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2017-01-30 05:06:46 -06:00