Commit Graph

829 Commits

Author SHA1 Message Date
Qixiang Wan
63327e7d88 checks: Fix anyOf validator yield ValidationError on ConfigOptionWarning
We have some hooks yield ConfigOptionWarning. When it happens within
anyOf validator, anyOf validator yield ValidationError and reports the
config as incorrect. We need to overwrite it to pass not break.

Fixes: #598
Merges: #599
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-04-18 10:05:03 +02:00
Lubomír Sedlář
39c3f42f77 comps-wrapper: Sort langpacks by name
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-13 11:45:23 +02:00
Lubomír Sedlář
fbf189d9c7 comps-wrapper: Add tests
This tests pretty much all the code that parses, filters and then writes
comps back to a file.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-13 11:45:23 +02:00
Lubomír Sedlář
f6d07c1651 Be explicit about generating release for images
The config now uses similar logic what previous commit did for OSTree.
Also we should report error when an unknown generator is used.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-13 10:17:59 +02:00
Lubomír Sedlář
98f40f6138 ostree: Autogenerate a version
If the value is a particular magic string, we will expand it to a proper
value.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-13 10:17:59 +02:00
Lubomír Sedlář
dcc1750df2 gather: Clean up method deps
* Remove unused arguments
 * Remove duplication

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-11 15:46:34 +02:00
Lubomír Sedlář
0168388492 gather: Report error if there is no input
Running depsolving with no requested inputs will only lead to a hard to
decipher error. We should instead explicitly tell the user that there is
a problem.

Unit tests are added to add to test this functionality.

Relates: #585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-11 15:46:34 +02:00
Lubomír Sedlář
b5efb67ff1 init: Warn when variants mentions non-existing comps group
When variants XML lists a group that does not match any known group in
input comps, report a warning. This is not necessarily a problem in
itself, but having this information in the log can help debug problems.

Relates: #585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-11 15:46:34 +02:00
Qixiang Wan
2f5d6d7dcd unify repo and repo_from options
Config option 'repo' and 'repo_from' are used in several phases, merge
them with one option 'repo'. 'append' in schema is used for appending
the values from deprecated options to 'repo', so it won't break on any
existing config files that have the old options of 'repo_from' and
'source_repo_from' (which is an alias of 'repo_from').

And 'repo' schema is updated to support repo dict as the value or an
item in the values, a repo dict is just a dict contains repo options,
'baseurl' is required in the dict, like:

{"baseurl": "http://example.com/url/to/repo"}

or:

{"baseurl": "Serer"}

currently this is used in ostree phase to support extra repo options
like:

{"baseurl": "Server", "exclude": "systemd-container"}

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-29 10:12:32 +08:00
Qixiang Wan
0f508e2228 move translate_path from paths.py to util.py
So translate_path can be used in util.py, or it will result in
circular import error.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-29 10:12:32 +08:00
Qixiang Wan
d1763fca7e checks.py: support 'append' option
If 'append' is defined for a property, append the values from append
options to the property. Note: The property must support to be a list
of values.

For example:

with schema:

    schema = {
        "$schema": "http://json-schema.org/draft-04/schema#",
        "title": "Pungi Configuration",
        "type": "object",
        "definitions": {
            "list_of_strings": {
                "type": "array",
                "items": {"type": "string"},
            },
            "strings": {
                "anyOf": [
                    {"type": "string"},
                    {"$ref": "#/definitions/list_of_strings"},
                ]
            },
        },
        "properties": {
            "release_name": {"type": "string"},
            "repo": {"$ref": "#/definitions/strings", "append": "repo_from"}
        },
        "additionalProperties": False,
    }

and config:

    repo = "http://url/to/repo"
    repo_from = "Server"

config will be updated to:

    repo = ["http://url/to/repo", "Server"]

It supports multiple append options too, like:

    "repo": {
        "$ref": "#/definitions/strings",
        "append": ["repo_from", "source_repo_from"],
    }

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-29 10:12:32 +08:00
Qixiang Wan
2aacefd9cd checks.py: show warning message for alias option
Show warning message for any alias option find in config instance.
Example warning message:

WARNING: Config option 'product_name' is deprecated and now an alias to
'release_name', please use 'release_name' instead. In:
{'release_name': 'dummy product', 'product_name': 'dummy product'}

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-29 10:12:32 +08:00
Lubomír Sedlář
2b21e13aad Skip DNF tests if there are import problems
If there are import errors for DNF, multilib or other related package,
we can assume the tests are running on EPEL. The DNF tests should be
skipped in that case.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-27 09:42:58 +02:00
Lubomír Sedlář
76cf4a7540 Update to use python-multilib
We also rename the old multilib module used by dnf code to multilib_yum
to make it clear what is imported where.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:50 +01:00
Lubomír Sedlář
27b24a489f Handle noarch debuginfo packages
Pull debuginfo packages for noarch packages as well, but only when the
debuginfo is noarch as well.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:50 +01:00
Lubomír Sedlář
d23a2f4548 Fix multilib exclude pattern to not match noarch
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:50 +01:00
Lubomír Sedlář
35e72df99f Explicitly remove all temporary files
There is no guarantee __del__ will ever be called, and we were leaving a
ton of stuff in /tmp. With this patch we pass the temporary directories
explictly and make sure they are deleted at the end.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář
fe026bb588 Handle Requires(pre|post)
These requires are accessible from a separate attribute, but we want to
handle them the same ways as regular Requires.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář
d4f8e32a80 Fix handling globs in input packages
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář
cd80d6ef5a Stop caching provides of added packages
This only works in non-greedy mode. When greedy, the same provides can
be linked to multiple packages that should be pulled in.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář
e06e9165fc Add checking of flags for dnf depsolving
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář
ec67eac1cc Use enum for flags
This adds an extra dependency on python-enum34.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář
9fd8e6319f Handle fulltree_excludes list
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář
b53e4a84d3 Fix fulltree when greedy
When greedy, fulltree should pull in all native and multilib packages.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:48 +01:00
Lubomír Sedlář
0a44b2fd07 Filter source packages
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:48 +01:00
Lubomír Sedlář
99699b85aa Use same tests for yum and dnf depsolving
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:48 +01:00
Jan Kaluza
9fcd71f831 Add support for modular composes
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2017-03-22 15:55:52 +01:00
Lubomír Sedlář
4b90822115 Add a script for modifying ISO images
With this script it's possible to add additional files into an ISO file.
If the file happens to be ks.cfg, the boot configs are tweaked so that
the kickstart is actually used.

Resolves: #503
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-21 07:55:19 +01:00
Lubomír Sedlář
306f7e69b0 iso-wrapper: Add utility for mounting images
This patch refactors logic for creating a temporary mount point,
mounting an image, running arbitrary code on it, unmounting the image
and removing the mount point. It immediately uses it in the buildinstall
phase.

Similar mounting is present in product_img phase as well, but due to
different usage pattern it's not changed yet.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-21 07:55:19 +01:00
Lubomír Sedlář
05a666fb3b buildinstall: Move tweaking configs into a function
This will allow us to more easily test the code and also reuse it in
other places.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-21 07:55:19 +01:00
Dennis Gilmore
b09641a708 Merge #568 buildinstall: Print debug info if unmount fails 2017-03-15 16:07:33 +00:00
Lubomír Sedlář
94b1159829 ostree-installer: Fix logging directory
Currently if there are multiple ostree installers for the same
architecture, the logfiles all end up in the same location and overwrite
each other.

This patch moves the logs to logs/<arch>/<variant>/ostree_installer-X/
for a unique value of X so that there can be multiple runs even for the
same tree.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-15 15:54:45 +01:00
Lubomír Sedlář
f0dc15bdc6 buildinstall: Print debug info if unmount fails
When even after retries the unmounting still fails, this patch runs `ls
-lA`, `fuser -vm` and `lsof +D` on the directory to give some idea of
what's blocking it.

Relates: #559
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-14 15:06:20 +01:00
Qixiang Wan
d081a4eda8 pkgset: report all unsigned packages
Fixes: #552
Merges: #567
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-14 09:50:28 +01:00
Lubomír Sedlář
5cd1c22e2d buildinstall: Retry unmounting image
If the image can not be unmounted because the device is busy, we should
retry. There will be increasing pauses between the attempts. At most 10
attempts will be done before giving up.

Fixes: #559
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-10 08:29:54 +01:00
Lubomír Sedlář
cd805a1e6d iso-wrapper: Handle wrong implant md5
If the checkisomd5 command exits successfully but returns a wrong value,
we should catch and log that. In theory this should be impossible, but
we have seen it in production.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-07 09:58:14 +01:00
Lubomír Sedlář
63a3b6ccaa Merge #538 consolidate repo option names 2017-03-06 12:55:44 +00:00
Qixiang Wan
2ae8710934 consolidate repo option names
live_images:            additional_repos        -> repo
ostree:                 source_repo_from        -> repo_from
                        extra_source_repos      -> repo
ostree_installer:       source_repo_from        -> repo_from

With the change, the phases have consolidate option names for variant
repos and external repos.

Old option names will continue to work, old names will be converted
to new names after validation automatically if new options are not
specified in config.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-06 09:26:42 +08:00
Qixiang Wan
c93207addb checks: extend validator with 'alias'
When a property has 'alias' defined, and it's not present in instance,
if the alias property is present, add the property with value from alias
property before remove the alias property from instance.

Examples:

with schema:
{
        "$schema": "http://json-schema.org/draft-04/schema#",
        "title": "Pungi Configuration",
        "type": "object",
        "properties": {
            "release_name": {"type": "string", "alias": "product_name"},
        },
        "required": ["release_name"],
        "additionalProperties": False,
}

1. config = {"release_name": "dummy product"}:

   validate pass, config not changed after validation.

2. config = {"product_name": "dummy product"}:

   validate pass, config updated to the following after validation:

   config: {"release_name": "dummy product"}

3. config = {"name": "dummy product"}:

   validate fail, errror message is "Failed validation in : 'release_name' is a required property",
   and warning message is "WARNING: Unrecognized config option: name."

4. config = {"product_name": "dummy product", "release_name": "dummy product"}

   validate fail, error message is "Failed validation in : product_name is an alias of release_name, only one can be used."

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-06 09:26:42 +08:00
Qixiang Wan
40e2874676 osbs: write manifest for scratch osbs
FIXES: #485

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-06 09:26:01 +08:00
Dennis Gilmore
8b1fb287d3 Merge #539 comps: Filter comps groups for optional variants 2017-02-23 20:13:39 +00:00
Lubomír Sedlář
4ea1916a87 Enable customizing runroot task weight
For different cases where runroot is used it's now possible to set
custom weight. The usecase for this is to avoid one builder taking too
many tasks. Especially buildinstall is quite resource intensive, so one
builder taking multiple tasks at the same time leads to very slow
compose time.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-23 12:44:27 +01:00
Lubomír Sedlář
8bc65a8be5 comps: Filter comps groups for optional variants
The optional variant can defined by just adding has_optional=True into
variant xml. In such case it has no comps groups and Pungi would copy
the original file unmodified. This leads to extra packages being pulled
into the optional variant.

In this case the correct solution is to filter the comps and remove all
groups.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-23 12:44:20 +01:00
Lubomír Sedlář
857aee05c1 util: Add a utility for managing temporary files
In multiple situations we need to create temporary files or directories
that should not be preserved after compose is finished. Let's add
context managers that ensure these get cleaned up.

This fixes tests leaving garbage around in /tmp.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-20 09:14:25 +01:00
Daniel Mach
ad23efd323 Refactor depsolving tests
This adds a new test for Requires(pre) and (post). The general structure
of the test now makes it easy to use the same test scenarios for
different backend.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-13 12:32:07 +01:00
Lubomír Sedlář
9533fca96c koji_wrapper: Always use --profile option with koji
This patch adds this option to all invoked koji commands.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-30 13:00:25 +01:00
Dennis Gilmore
42cdf047cc add missing koji_profile from test compose setting
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2017-01-30 05:06:46 -06:00
Dennis Gilmore
c650f04d0b Revert "live-media: Call correct koji alias"
This was accidently merged and is not correct
This reverts commit 02beb35e67.

Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2017-01-30 04:44:17 -06:00
Lubomír Sedlář
dc7d3b36ab repoclosure: Don't run build deps check
Fixes: #521
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-30 10:16:03 +01:00
Dennis Gilmore
7feadb14ba Merge #518 Migrate test phase to dnf repoclosure 2017-01-28 16:49:10 +00:00
Lubomír Sedlář
02beb35e67 live-media: Call correct koji alias
We need call whatever command is configured in koji_profile. It's not
always `koji`.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-26 15:32:21 +01:00
Lubomír Sedlář
e3fe67be53 repoclosure: add option to use dnf backend
This adds a new option repoclosure_backend that changes what tool is
used for repoclosure.

Checking build dependencies is currently not supported, as `dnf` does
not have the corresponding option.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-26 13:39:12 +01:00
Lubomír Sedlář
95fc0fa4ab repoclosure: Add test for repoclosure in test phase
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-26 13:27:48 +01:00
Lubomír Sedlář
7a8fa87172 repoclosure: Remove useless wrapper class
We already get namespacing from being in a separate module.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-26 10:55:23 +01:00
Lubomír Sedlář
9af0cca9eb repoclosure: Add a test for the wrapper
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-26 10:55:23 +01:00
Lubomír Sedlář
8418b68fb0 image-build: Pass arches around as a list
Instead of joining the arches as a comma separated string and splitting
it again later. Ultimately we do need the original format to pass to
koji wrapper, but we can produce that value later.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-24 08:50:21 +01:00
Lubomír Sedlář
6c708549c8 image-build: Expand arches for can_fail
We need to work with a list of strings, not a comma-delimited single
string.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-24 08:50:21 +01:00
Qixiang Wan
40df2034a8 image_checksum: add file sizes to checksum files
Write file sizes of images in checksum files with comment lines,
checksum files are in BSD-style which supports comments by starting
a line with '#'.

Example:

$ cat RHEL-7.4-20170123.n.4/compose/Server/x86_64/iso/RHEL-Server-7.4-x86_64-20170123.n.4-CHECKSUM
 # RHEL-7.4-20170123.n.4-Server-x86_64-dvd1.iso: 3725590528 bytes
 # RHEL-7.4-20170123.n.4-Server-x86_64-boot.iso: 377487360 bytes
 SHA256 (RHEL-7.4-20170123.n.4-Server-x86_64-dvd1.iso) = fa3de37fe4b859a0285f16ea1123f44f15aec169aea84bf010aa3821bd58fc41
 SHA256 (RHEL-7.4-20170123.n.4-Server-x86_64-boot.iso) = 74bf68c54665328adb08b09daf773c67e633b5907e3e2797338ab3c1b58fdf48

(No space at the start of line, because git commit message drops lines
start with '#', added one space to avoid that.)

When there are multiple checksum types specified and checksums are
written to individual files, file size of the image will also be
written to every checksum files.

Fixes: #493

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-01-23 18:20:21 +08:00
Qixiang Wan
a9b275f13b osbs: optionally check GPG signatures
If gpgkey option is defined in config, set gpgcheck=1 and set
gpgkey=<value> in variant repo files.

Fixes: #487
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-01-17 13:08:58 +08:00
Qixiang Wan
5d241d316a ostree-installer: Allow multiple repos in ostree installer
Add new key 'repo' to allow specifying multiple repos as the source
repositories. And change 'source_repo_from' to allow specifying multiple
vairant names to use variant repos as source repositories.

Doc of 'source_repo_from' is updated to not mention URL is supported,
though we still support that in code. User should add url of repos in
'repo' key instead of 'source_repo_from'.

Fixes: #508

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-01-13 16:57:30 +08:00
Lubomír Sedlář
afaa0e025e unified-iso: Create isos with debuginfo packages
When creating unified ISOs, the script will now also create one iso per
architecture containing a repo with debuginfo packages.

There is no switch to turn this off. The images can simply not be
shipped if not wanted.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-11 14:36:25 +01:00
Qixiang Wan
6fbf1e8f59 Create temporary dirs under compose's workdir
Add compose.mkdtemp which creates and returns a unique temporary
directory under <compose_topdir>/work/{global,<arch>}/tmp[-<variant>]/

Change tempfile.mkdtemp in code to compose.mkdtemp, so temporary
dirs are created under the compose's workdir, this makes it more
easier to maintain the temporary dirs/files especially when compose
fails with amount of temporary files left there.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-01-11 15:25:47 +08:00
Dennis Gilmore
baca306edf Merge #495 osbs: Enable specifying extra repos 2017-01-04 14:44:13 +00:00
Dennis Gilmore
46f77403df Merge #475 Allow failure for some arches 2017-01-04 14:43:22 +00:00
Lubomír Sedlář
e260fe5581 Add script to generate unified ISOs
This a standalone script that will look into a compose and create
unified ISO for each architecture. The ISO contains RPM repositories for
all variants that have the arch.

Known issues:
 * The filename does not respect settings. This is tricky because the
   name could include variant name, which we don't have here (by design
   of unified ISO).
 * The same is true for volume id.

In order to test the feature without running actual compose, we need to
add essentially a big chunk of compose. Most of the files are empty, as
their content is never accessed.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-04 10:22:20 +01:00
Lubomír Sedlář
e7c8b2affd osbs: Validate config in tests
This makes sure the test configurations will be accepted in real usage.
It also enables us to remove some manual error checking that will be
performed by validator.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-04 09:47:26 +01:00
Lubomír Sedlář
61a4c43db0 osbs: Verify the .repo files contain correct URL
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-04 09:47:26 +01:00
Lubomír Sedlář
814bf4484b osbs: Enable specifying extra repos
The same way live_media and image_build accept additional external repos
or variants list, there is now a `repo` and `repo_from` configuration
key to add these.

Fixes: #486
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-04 09:47:19 +01:00
Patrick Uiterwijk
c64f75d1b4 Add test for krb_login with principal and keytab
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2016-12-14 12:58:23 +00:00
Patrick Uiterwijk
cb4b5cc54d Make sure that the profile name is parsed correctly
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2016-12-14 12:58:23 +00:00
Lubomír Sedlář
356b78d440 image-build: Allow failure only on some arches
This uses the --can-fail option in koji. Failing an optional image will
not abort whole task. If the whole task fails (or there is a problem on
the compose side), we abort unless all arches are optional.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-14 10:14:30 +01:00
Lubomír Sedlář
7f56b978ce live-media: Allow some arches to fail
This patch uses the `--can-fail` option of koji command line. If only
optional arches fail, the task will report as success. Failures on
compose box side are ignored if and only if all architectures are
optional.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-14 10:14:30 +01:00
Lubomír Sedlář
721932a573 image-build: Use install_tree from parent for nested variants
Buildinstall only runs for top-level variants. Addons, optionals and
integrated layered products must reuse install tree from their parent,
because otherwise there would be no boot.iso.

Fixes: #472
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-14 10:04:01 +01:00
Lubomír Sedlář
c338219ef0 config: Report unknown options as warnings
Given the way config files can include other files, it is entirely
possible to use the same config file for both Pungi and Distill. Pungi
will however complain about unknown options.

This patch reverts part of c38bb480 and moves deprecation handling back
into schema validation. The validation method then returns a list of
errors and a list of warnings.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-13 11:58:08 +01:00
Lubomír Sedlář
56147f2e4d pungi: Handle missing SRPM
Source packages can be excluded while binary packages should still go
into the compose. This patch makes it so that the mapping from binary
packages to source packages contains None in such case. The code is
already capable of handling that. A warning will be emitted for each
binary package without source.

This also allows us to remove some code from `createSourceHashes` that
is now unused.

A test for excluding source package is added as well.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-13 10:45:04 +01:00
Qixiang Wan
83428a06bf [ostree-installer] Add 'installer' sub-command to pungi-make-ostree
The new sub-command 'installer' is added to support build OSTree
installer image with pungi-make-ostree. It can take an optional argument
'--extra-config' to read some of configurations from a json file. The
content of the json file can contains the configuration which are
supported in OSTree installer phase, the difference is variant UID is
not supported as a repo url in this case. A valid json file can be like
the following:

{
    "source_repo_from": "http://www.example.com/repo/workstation/os",
    "installpkgs": [
        "fedora-productimg-workstation"
    ],
    "add_template": [
        "/path/to/installer/template/lorax-configure-repo.tmpl"
    ],
    "add_template_var": [
        "ostree_osname=fedora-workstation",
        "ostree_ref=fedora/25/x86_64/workstation"
    ],
    "add_arch_template": [
        "/path/to/installer/template/lorax-embed-repo.tmpl"
    ],
    "add_arch_template_var": [
        "ostree_repo=https://www.example.com/compose/ostree",
        "ostree_osname=fedora-workstation",
        "ostree_ref=fedora/25/x86_64/workstation"
    ]
}

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-12-12 19:30:35 +08:00
Qixiang Wan
e043604822 [ostree] Add 'tree' sub-command to pungi-make-ostree script
Update pungi-make-ostree to supourt sub-command 'tree', which is just
as the original feature of pungi-make-ostree to compose OSTree tree.
With the change we can add other sub commands later to build other
OSTree artifacts, like the installer image.

Inaddtional to the change, now the the 'tree' command can accept an
optional '--extra-config' parameter to update the original tree
configuration with extra configurations specified in a json file
before composing the OSTree tree.

Example:

pungi-make-ostree tree --repo=/ostree --treefile=/path/to/treefile \
--log-dir=/path/to/log --extra-config=/path/to/extra-config.json

The extra-config file can contains the same configuration as OSTree
phase, the difference is it doesn't understand variant UID as source
repo since it's not ran in the chain of phases. A valid configuration
can be like:

{
    "source_repo_from": "http://example.com/repo/x86_64/Server",
    "extra_source_repos": [
        {
            "name": "optional",
            "baseurl": "http://example.com/repo/x86_64/optional",
            "exclude": "systemd-container",
            "gpgcheck": False
        },
        {
            "name": "extra",
            "baseurl": "http://example.com/repo/x86_64/extra",
        }
    ],
    "keep_original_sources": True
}

The OSTree phase is updated to move out the task of updating treefile,
instead of that, it writes the extra configurations to a json file,
then 'pungi-make-ostree tree' will take it by option '--extra-config'.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-12-12 19:30:35 +08:00
Lubomír Sedlář
d3cad4795c metadata: Allow creating internal releases
The internal flag in productmd is meant to indicate that a compose is
not meant for publishing. This is potentially useful to allow filtering
in PDC or similar service.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-06 12:49:33 +01:00
Qixiang Wan
75934f20e5 Fix PhaseLoggerMixin in case of compose has _logger = None
80fa723 breaks pungi-config-validate that using ValidationCompose
with _logger set to None.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-11-30 14:41:02 +08:00
Lubomír Sedlář
dd814a5f4d ostree-installer: Use dvd-ostree as type in metadata
This depends on python-productmd >= 1.3 that understands this format.

Fixes: #417
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-28 10:32:50 +01:00
Lubomír Sedlář
2bd18b1b03 image-build: Reduce duplication
All tests should print complete diffs on failure, so there is no need to
define this in each test separately.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-28 09:28:46 +01:00
Lubomír Sedlář
5148adf233 createrepo: Add tests for adding product certificates
If the certificate is enabled and exists, it will currently be added to
all rpm repositories.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-23 11:17:04 +01:00
Lubomír Sedlář
96fc6fb11e createrepo: Add tests for retrieving product certificates
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-23 11:17:04 +01:00
Qixiang Wan
80fa723b1d Include phase name in log for some phases
Phases createiso, liveimages, image_build, ostree_installer and osbs are
done in parallel, logs from these phases are mixed and and it's not
obvious which log message belongs to which phase. This change adds phase
name in log message for these phases.

The new mixin 'PhaseLoggerMixin' is added to extend a Pungi phase with a
logging logger which copy handlers from compose's logger but with
formatter changed.

Fixes: #58
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-11-23 16:47:22 +08:00
Dennis Gilmore
9e52c68c82 Merge #465 media-split: Print sensible message for unlimited size 2016-11-17 23:14:44 +00:00
Dennis Gilmore
6c89f86c53 Merge #466 pungi: Include noarch debuginfo 2016-11-17 23:14:03 +00:00
Colin Walters
7bcbe30fd2 Expose lorax's --rootfs-size argument
This is going to be necessary for reworking the Atomic Host ISO,
see: https://pagure.io/fedora-lorax-templates/pull-request/6

Signed-off-by: Colin Walters <walters@verbum.org>
2016-11-17 10:17:39 -05:00
Lubomír Sedlář
deb8623bd1 pungi: Include noarch debuginfo
The check for arch compatibility should not be performed if the
debuginfo package is noarch. Such packages should be included
unconditionally.

Fixes: #450
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-16 14:50:03 +01:00
Lubomír Sedlář
6203541ac6 media-split: Print sensible message for unlimited size
If the media is bootable, we can not split it. The limit is not used in
that case and we may overflow it with an warning message. The warning
should correctly mention what is going on instead of printing a
non-sensical message about free space on media being some huge number.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-16 14:49:46 +01:00
Lubomír Sedlář
b7813d34ac pungi: Fix incorrectly skipped tests
The test was not correctly being skipped in Koji build, causing build
failure.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-16 09:43:19 +01:00
Lubomír Sedlář
3c1e2fba66 pungi: Fix tests on non-x86_64 arches
If the test did not specify an arch explicitly, it fell back on whatever
arch the current machine has. This was causing failures when building
RPM.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-16 09:20:10 +01:00
Dennis Gilmore
69d949aaf9 Merge #463 [ostree] Allow extra repos to get packages for composing OSTree repository 2016-11-15 21:06:05 +00:00
Dennis Gilmore
3953914bd1 Merge #461 extra-files: Nice error message on missing RPM 2016-11-15 21:01:42 +00:00
Dennis Gilmore
d4a78a1553 Merge #459 Fix building images and live media for addons 2016-11-15 21:00:50 +00:00
Dennis Gilmore
5dc8132fa4 Merge #456 Add tests for dependency solving 2016-11-15 20:58:40 +00:00
Qixiang Wan
68e121e421 [ostree] Allow extra repos to get packages for composing OSTree repository
Sometimes addtional repos are required to get necessary packages for
composing OSTree repository. For example, RHEL doesn't have an 'Everyting'
variant, so composing OSTree repository from any of the RHEL variants
won't work, addtional source repos need to be enabled to achieve that.
The new option "extra_source_repos" enable the ability of allowing extra
source repos.

And a new option 'keep_original_sources' is introduced to keep the
original repos found in tree config file, if this is enabled, Pungi
will not remove the existing source repos from the tree config file,
just add new repos of "source_repo_from" + "extra_source_repos" to
the existing repos.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-11-15 18:56:06 +08:00
Lubomír Sedlář
0cdef2d8e3 pungi: Run in-process for testing
Instead of spawning a separate process for each test, move the code to
run in the main process. This gives us correct coverage information and
makes the tests a lot faster.

The input is still provided in a kickstart file. The output also goes
into a log file that is later parsed.

There is some extra work needed with logging setup to make the logs go
to standard output so that nose can correctly capture them.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-10 13:28:02 +01:00
Lubomír Sedlář
88dbf8f849 extra-files: Nice error message on missing RPM
When a file is exported from an RPM in the compose, and there is no
matching RPM in the package set, we want a nice error message.

Fixes: #460
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-10 09:38:50 +01:00
Lubomír Sedlář
13871b64fb compose: Drop unused argument
The `get_variants()` method had a `recursive` argument with default
value of `False. However, this argument had no effect and the method
always returned all variants recursively.

We can just drop the argument. All callers are updated to not supply the
argument. Should any need for getting the top-level variants only arise,
they can be accessed as the `variants` attribute directly on the Compose
object.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 13:39:01 +01:00
Lubomír Sedlář
bd00920c62 compose: Search all nested variants
The code to search for install tree and repo for image-build and
live-media was only looking at top-level variants. Addons, optional or
integrated layered products could not have been found. This would lead
to error messages such as "There is no variant Server-optional to get
repo from when building live image for Client" even though the variant
exists.

Various tests are updated to exercise this edge case.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 13:18:34 +01:00
Lubomír Sedlář
99ba369901 ostree-installer: Capture all lorax logs
Fixes: #457
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 09:37:00 +01:00
Lubomír Sedlář
9a72ea8f6d lorax-wrapper: Put all log files into compose logs
Tell lorax to use a specific directory for log files so that we preserve
them despite koji not having any idea about them.

Fixes: #457
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 09:37:00 +01:00
Lubomír Sedlář
adaed6345c pungi: Fulltree should not apply for input multilib package
When the input explicitly lists a package as multilib, we should not
automatically add native version just because of fulltree.

The tests for this use case are now enabled and passing.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 09:36:23 +01:00
Lubomír Sedlář
8e5b197e3f pungi: Add tests for depsolving
These are tests written by Daniel Mach originally for Distill-NG. They
are ported to current Pungi. The test repositories are committed in the
tests/fixtures/ directory. This is the same data that is used for test
compose, but the actual RPM files are not present. Some tests are
adapted from dmach's fork of Pungi.

Some of the packages are marked with a comment saying they are
important. These are the packages that the test is specifically trying
to get included in the package set. There are also explicit tests for
packages that should not be included.

Two tests are skipped for now as there is a bug preventing them from
passing. This is related to fulltree being done for packages that are
explicitly multilib.

The depsolver is called by invoking a separate executable, so the
coverage data is wrong.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 09:36:23 +01:00
Qixiang Wan
fa3c5aff63 [ostree] Allow adding versioning metadata
Added new option '--version' to pungi-make-ostree, and this can be
enabled in ostree settings with 'version'. The version string will be
added as versioning metadata if this is specified.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-11-04 16:07:26 +08:00
Qixiang Wan
06ba3b8551 [ostree] New option to enable generating ostree summary file
Added new option '--update-summary' to pungi-make-ostree, and this can
be enabled in ostree settings with 'update_summary'. A summary file will
be generated (or re-generated if it was presented in an existing ostree
repo) when it is enabled.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-11-03 21:02:05 +08:00
Patrick Uiterwijk
a49510a9fb Update tests to use ostree write-commit-id
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2016-10-26 09:52:12 +00:00
Patrick Uiterwijk
754458823c ostree: Use the write-commitid-to feature rather than parsing ostree logs
This depends on an rpm-ostree release with
98332a3be4

Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2016-10-26 09:52:12 +00:00
Lubomír Sedlář
c5bd99da9e Merge #448 Remove executable permissions on test scripts 2016-10-26 07:31:37 +00:00
Lubomír Sedlář
bfd5a39ce6 checks: Check for createrepo_c
The createrepo package is needed always, but depending on configuration
we should also look for createrepo_c.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-25 16:00:37 +02:00
Lubomír Sedlář
f4cd25450b checks: Update tests to not require python modules
This way the tests are more independent of the system they are running
on.

Merges: #447
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-25 15:57:02 +02:00
Patrick Uiterwijk
59dbd9d35f Remove executable permissions on test scripts
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2016-10-25 10:42:16 +00:00
Lubomír Sedlář
c38bb4809b config: Don't abort on deprecated options
These options are in fact removed and have no effect anymore. This patch
changes the validation to print a warning that the option was removed
and what should be done instead. It no longer stops the whole compose.

The validation script still rejects configuration files with these
removed keys.

This change means we no longer check these removals with the JSON schema
(as that makes it hard to determine where exactly the problem is).

Fixes: #438
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-20 12:51:12 +02:00
Dennis Gilmore
a7dc4e80a7 Merge #433 createiso: Do not split bootable media 2016-10-19 16:37:25 +00:00
Lubomír Sedlář
84fcc0083f createiso: Do not split bootable media
If the media is bootable, we can not split it. Instead we will create an
ISO that is too big and issue a warning (aborting the whole compose
would be too much).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-19 18:36:19 +02:00
Dennis Gilmore
ff6c952094 Merge #430 createiso: Include layered product name in iso name 2016-10-19 16:36:07 +00:00
Dennis Gilmore
34d3be35dc Merge #426 iso-wrapper: Remove the class 2016-10-19 16:34:25 +00:00
Lubomír Sedlář
b46af7acbb createiso: Include layered product name in iso name
When creating an ISO for a layered product, the name of the integrated
product should be included in the file name.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-17 15:15:50 +02:00
Lubomír Sedlář
e58c78fd2f iso-wrapper: Remove the class
It really is just a group of independent functions, so we can simplify
it by removing the unused wrapper class. Instead of importing the
wrapper, instantiating it and calling its methods we can import the
module and call its functions directly.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-14 09:14:51 +02:00
Lubomír Sedlář
50e9294057 config: Validate variant regular expressions
When the mapping in configuration specifies incorrect regular expression
to match regular expressions, we should raise an error immediately and
not wait until the part of config is actually used.

This patch does not cover `live_media`, `image_build` and `osbs`
sections, as they use plain dicts and not the list of tuples format.

Fixes: #424
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-12 09:38:59 +02:00
Lubomír Sedlář
730764e2e7 ostree-installer: Reduce duplication in tests
The assertions were duplicated across multiple test cases. This patch
moves them into shared methods so that they are defined only once.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-06 09:42:31 +02:00
Lubomír Sedlář
32059c57c0 ostree-installer: Generate correct volume ID
The volume ID should be generated in Pungi with the same format as for
other images.

Fixes: #419
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-06 09:17:09 +02:00
Lubomír Sedlář
cc559b12f4 ostree-installer: Use ostree as type in filename
Instead of overloading the `dvd` value, use new value `ostree` that can
be overridden by a config change.

Fixes: https://pagure.io/pungi/issue/418
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-06 08:41:14 +02:00
Lubomír Sedlář
8880b1ea8a ostree: Use $basearch in repo file
Fixes: #413
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-05 18:41:55 +02:00
Dennis Gilmore
4e90f81fc1 use --new-chroot when making ostree's
to complement https://pagure.io/koji/pull-request/162 we need to adjust pungi
rpm-ostree uses bublewrap that does not work in mock. --new-chroot to mock
enables the use of systemd-nspawn instead of chroot resulting in working
rpm-ostree again

Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2016-09-28 20:58:17 -05:00
Lubomír Sedlář
7bf12636a0 Translate paths without double slash
If the path in `translate_paths` config ends with a slash, we would
create public path with double slash.

Fixes: https://pagure.io/pungi/issue/408
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-26 15:50:49 +02:00
Lubomír Sedlář
a042906717 init: Remove keep_original_comps option
The same information can be inferred from definitions in variants.xml:
if the variant has no groups defined, we include packages from all
groups. By the same logic we can also include all groups in the comps
file.

The config validation is updated to give a hint on how to remove the
option from the configuration.

Relates: #29
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-22 13:35:28 +02:00
Lubomír Sedlář
c118adc705 tests: Use unittest2 consistently
Without importing this module, the tests are going to fail on Python
2.6.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-22 10:28:01 +02:00
Lubomír Sedlář
2ac45a783d ostree_installer: Add --isfinal lorax argument
If the compose is supported, we need to tell lorax to use the option.

Fixes: #399
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-21 13:49:35 +02:00
Dennis Gilmore
3a68a93532 Merge #385 Test and clean up pungi.linker 2016-09-19 21:27:57 +00:00
Lubomír Sedlář
f97e9258aa checksums: Never skip checksumming phase
It does not make sense for this phase to be skipped. If there are any
images, we need to generate the checksums so that writing metadata
works. If there are no images, the phase does not do anything and is
therefore very fast.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-19 12:20:34 +02:00
Lubomír Sedlář
bdf968ff3f [linker] Remove dead code
There are big parts of code that can not be reached. Other parts expose
options that are not used anywhere. To keep things simple, all of that
is removed in this patch.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-19 12:19:43 +02:00
Lubomír Sedlář
444c3a21ca [linker] Add tests
These tests are adapted from internal ones.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-19 12:19:43 +02:00
Lubomír Sedlář
23e00fd87d ostree: Add tests for sending ostree messages
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-15 14:40:46 +02:00
Lubomír Sedlář
bf46048cbf createrepo: Add option to use xz compression
This patch adds a new config option createrepo_use_xz, which when set to
true will cause createrepo to compress sqlite databases with xz. The
default setting is False.

Fixes: #387
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-12 11:10:43 +02:00
Jeremy Cline
ee1ee0467b extra-files: Write a metadata file enumerating extra files
Introduces a new metadata file to track arbitrary files added during the
extra-files phase. This file is placed in the root of each tree and is
called ``extra_files.json``. It is a JSON file containing a single
object, which contains a "header" key with an object describing the
metadata, and a "data" key, which is an array of objects, where each
object represents a file. Each object contains the "file", "checksums",
and "size" keys. "file" is the relative path from the tree root to the
extra file. "checksums" is an object containing one or more checksums,
where the key is the digest type and the value of that key is the hex
digest. Finally, the size is the size of the file in bytes.

For example:
{
  "header": {"version": "1.0},
  "data": [
    {
      "file": "GPL",
      "checksums": {
        "sha256": "8177f97513213526df2cf6184d8ff986c675afb514d4e68a404010521b880643"
      },
      "size": 18092
    },
    {
      "file": "release-notes/notes.html",
      "checksums": {
        "sha256": "82b1ba8db522aadf101dca6404235fba179e559b95ea24ff39ee1e5d9a53bdcb"
      },
      "size": 1120
    }
  ]
}

Signed-off-by: Jeremy Cline <jeremy@jcline.org>
Fixes: #295
2016-09-07 13:02:48 +02:00
Dennis Gilmore
6aeab9ee9d Merge #381 Automatically generate missing image version 2016-09-06 15:19:55 +00:00
Lubomír Sedlář
223a015898 Automatically generate missing image version
If the configuration does not specify version for images or live media,
Pungi will create a default value based on `release_version`. If label
is used for the compose, the milestone from it will be appended to the
version (unless it's RC).

This change is backwards compatible: nothing changes when version is set
in configuration. If the version was missing before, building the
artifacts would fail. With this patch, default values will be supplied.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-01 15:13:04 +02:00
Lubomír Sedlář
f9a6c8418f Add JSON Schema for configuration
The schema is written in Python to reduce duplication. When
configuration is loaded, the validation checks if it's correct and fills
in default values.

There is a custom extension to the schema to report deprecated options.

The config dependencies are implemented as a separate pass. While it's
technically possible to express the dependencies in the schema itself,
the error messages are not very helpful and it makes the schema much
harder to read.

Phases no longer define `config_options`. New options should be added to
the schema. Since the default values are populated automatically during
validation, there is no need to duplicate them into the code.

The `pungi-config-validate` script is updated to use the schema and
report errors even for deeply nested fields.

The dependencies are updated: pungi now depends on `python-jsonschema`
(which is already available in Fedora).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-01 10:56:15 +02:00
Lubomír Sedlář
1881bf740d createiso: Report nice error when tag does not exist
Instead of failing with generic `AttributeError`, we can easily report
the actual problem.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-01 09:09:31 +02:00
Lubomír Sedlář
a31be0a3c4 Fix test data build script
It uses bash specific features, so it should not claim /bin/sh in
shebang. We also want to use `set -e` to catch possible errors instead
of claiming success every time.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-01 08:46:23 +02:00
Lubomír Sedlář
43fda1e9ef Use label to populate image release
When release for an image is specified as explicit `None`, we can
generate the value based on compose label. For example for `Alpha-1.2`
the release would be `1.2` instead of the date based one.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-25 15:24:36 +02:00
Lubomír Sedlář
b862fc5a50 [buildinstall] Fix cleaning output dir
Before the task is started, the output directory is checked and if it
exists and is not empty, the runroot task will be skipped. This is meant
for debugging when restarting the same compose. Under usual
circumstances, the directory will not be created in the first place.

The runroot task will start by removing the output directory. This way,
if koji restarts the task, lorax will not fail.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-18 09:11:30 +02:00
Lubomír Sedlář
c05282be44 [osbs] Convert build_id to int
The getBuild API call expects either an integer or string NVR, otherwise
it will crash.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-12 14:07:58 +02:00
Lubomír Sedlář
f37a14fb60 [createiso] Use shell script for runroot
Instead of installing pungi itself in the runroot, we can prepare the
commands to be run on compose box, write the shell script into work/
directory, which is mounted in the chroot, and execute that. This way
there is no business logic in runroot (except for finding lorax
templates).

The main advantage of this approach is that we don't need to pull any
extra dependencies into buildroot.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-10 13:39:50 +02:00
Lubomír Sedlář
ef99e28849 [test] Only check bootability for images on x86_64 and i386
Current check does not really work for other arches.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-10 13:02:56 +02:00
Dennis Gilmore
247149d4e1 Merge #339 Refactor failables, step 1 2016-08-03 07:36:34 +00:00
Lubomír Sedlář
463088d580 Refactor failables
This is a breaking change as big part of current failable_deliverables
options will be ignored.

There is no change for buildinstall and creatiso phase.

Failability for artifacts in other phases is now configured per
artifact. It already works correctly for ostree and ostree_installer
phases (even per-arch). For OSBS phase there is currently only a binary
switch as it does not handle multiple arches yet. When it gains that
support, the option should contain list of non-blocking architectures.

For live images, live media and image build phases each config block can
configure list of failable arches. If the list is not empty, it can
fail. Once we have a way to fail only some arches, the config will not
need to change.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-07-27 13:06:01 +02:00
Lubomír Sedlář
7d0ee41f23 Stop setting release in OSBS phase
Atomic Reactor does not honor this option. In the future we might need
to reintroduce this feature, but given that it does not work in the
current form it is better removed.

Fixes: #348
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-07-27 09:52:06 +02:00
Lubomír Sedlář
22e94caf3c [test] Correctly check bootable ISOs
ISO image without MBR and GPT can still be bootable if it has an El
Torito boot catalog. The test phase must accept such images.

This slightly defeats the point of the check: to verify the ISO is
hybrid. However, based on the metadata we have no way to actually tell
if the image is supposed to be hybrid.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-07-22 14:11:06 +02:00
Lubomír Sedlář
a72182c817 Simplify naming format placeholders
This patch makes it possible to use different style format placeholders.
Instead of the percent encoding it is now possible to use simple curly
braces.

    %(foo)s -> {foo}

The old format is still available.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-06-28 13:03:36 +02:00
Lubomír Sedlář
11bbbae2ed Treat variants without comps groups as having all of them
If the `<groups></groups>` section is not specified in the variants XML
file, all groups will be used in this variant. The section must be
omitted completely, not just empty. This is (and was) correct according
to the DTD, it just lead to crash before.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-06-28 09:01:30 +02:00
Lubomír Sedlář
8e4b1e278c [extra-files] Only copy files when there is a config
Instead of running the copy function for all variant.arch pairs
unconditionally, only do it if there is something to do. This makes the
log more understandable.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-06-23 07:25:52 +02:00
Lubomír Sedlář
74001d5aa5 [extra-files] Skip whole phase if not configured
This checks the configuration in one place rather than doing it for each
variant/arch combination.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-06-23 07:25:52 +02:00
Lubomír Sedlář
b67f6369db [extra-files] Copy files using existing function
Instead of calling cp as a system command. This will help debug problems
if something fails.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-06-23 07:25:52 +02:00
Lubomír Sedlář
d3b2fbe387 [extra-files] Add tests
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-06-23 07:25:52 +02:00
Lubomír Sedlář
b4fc97be03 [osbs] Add a phase to build images in OSBS
It will take RPM repo from a variant in this compose and a Dockerfile
from configured git and use it to build an image.

The build images are uploaded to some a Docker registry by OSBS and are
not directly part of compose (because there is no export function).
There is a new metadata file `osbs.json` that has some information that
can be used to find the image.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-06-22 10:02:03 +02:00
Lubomír Sedlář
9429c40b13 [createiso] Add test for adding source iso to metadata
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-06-06 16:04:54 +02:00
Dennis Gilmore
6800501217 Merge #325 Fix checking optional ISO images in test phase 2016-06-06 11:56:51 +00:00
Dennis Gilmore
469c275670 Merge #321 Add support for top-level variant IDs with dashes. 2016-06-06 11:56:27 +00:00
Lubomír Sedlář
b2a266c3e4 Fix checking optional ISO images in test phase
Instead of iterating over image manifest, loop through all variants and
arches and see if there are any images. This avoids a crash for variants
nested under other variants (layered products, optionals or addons).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-06-06 10:41:06 +02:00
Daniel Mach
f0aecf6744 Add support for top-level variant IDs with dashes.
This is for layered products that have a variant mapped
to multiple variants in a base product, for example:
 * Foo-Tools (id: FooTools, uid: Foo-Tools, name: Tools)
 * Bar-Tools (id: BarTools, uid: Bar-Tools, name: Tools)

Requires productmd >= 1.2.

Signed-off-by: Daniel Mach <dmach@redhat.com>
2016-06-02 10:16:16 -04:00
Daniel Mach
4582e635f6 Fix PYTHONPATH usage in test_compose.sh.
Signed-off-by: Daniel Mach <dmach@redhat.com>
2016-06-02 10:16:02 -04:00
Lubomír Sedlář
b62b468ccf [createiso] Enable customizing media reserve
Add a documented and tested config options for setting ISO parameters
instead of hardcoding magic values.

Fixes: #256
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-06-02 08:34:49 +02:00
Lubomír Sedlář
a8d7f8a63e [createiso] Add test for splitting media
This patch also fixes a bug where packages in hashed directories would
not be recognized as such and would not be placed after other files.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-06-02 08:34:49 +02:00
Lubomír Sedlář
97f9cc9abc [media-split] Add unit tests
The tests do not pass with current code - an error on too big file is
not raised.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-06-01 14:03:23 +02:00
Dennis Gilmore
264c7b1ddf Merge #309 Add compatibility for Python 2.6 2016-05-27 01:23:34 +00:00
Dennis Gilmore
0e0e333982 Merge #293 Add tests for generating discinfo and media.repo files 2016-05-27 01:23:22 +00:00
Dennis Gilmore
e2ee4b0cc1 Merge #287 Use koji profiles to list RPMs in buildroot 2016-05-27 01:23:03 +00:00
Lubomír Sedlář
022a4d36f5 [ostree-installer] Put images to os/ directory
Instead of putting everything to <variant>/<arch>/iso, move images/,
EFI/ and isolinux/ subdirs to <variant>/<arch>/os. The nicely named ISO
image is still in the original location.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-26 09:58:56 +02:00
Lubomír Sedlář
5698a9727e [ostree] Rename duplicated test
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-26 09:55:48 +02:00
Lubomír Sedlář
9eb85c481c [util] Use koji profile for getting RPMs from buildroot
Instead of having a separate config option, just use the koji profile.
According to release notes, this should have already been done.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-26 09:50:57 +02:00
Lubomír Sedlář
75d71ebbe1 [util] Add test for getting list of buildroot RPMs
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-26 09:50:57 +02:00
Lubomír Sedlář
d5512fd6ff Use unittest2 if available
The module backports features to Python 2.6 and 2.7. If it is available,
the tests will use it. If it is not available, it will fall back to
regular unittest. On Python 2.7, the tests pass anyway. On Python 2.6,
there are failures with Python 2.6.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-25 13:41:49 +02:00
Lubomír Sedlář
b634a18a7f Stop using str.format
On Python 2.6, it requires the format placeholder to have explicit index
of argument, so using % formatting is easier.

There are a couple places where the method is still used because the
same argument is used twice.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-25 13:41:49 +02:00
Lubomír Sedlář
b55f8abd29 Stop using functools.total_ordering
It is only available since 2.7, and only saves us from defining three
trivial methods.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-25 13:28:56 +02:00
Lubomír Sedlář
9ab9aaf8d9 The message attribute on exception is deprecated
One should rather use str(exception) to get the text message.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-25 13:28:56 +02:00
Lubomír Sedlář
6e10d8c713 [ostree] Rename duplicated test
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-25 12:30:17 +02:00
Lubomír Sedlář
c8341e1806 [ostree-installer] Allow using external repos as source
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-24 10:06:06 +02:00
Lubomír Sedlář
5d849cd050 [metadata] Add test for writing media.repo
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-23 18:34:04 +02:00
Lubomír Sedlář
2113b6475c [metadata] Add tests for discinfo files
This tests writing and reading them.

Also, it makes sure the description for layered products is correct:
there was a missing space.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-23 18:34:04 +02:00
Lubomír Sedlář
bb4afea4f1 [image-build] Allow using external install trees
This is needed for the twoweek Fedora atomic release to point to regular
nightly composes.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-23 18:30:14 +02:00
Dennis Gilmore
fcfdb36352 Merge #303 [ostree] Use unique work and log paths 2016-05-18 16:30:44 +00:00
Lubomír Sedlář
9dce30a78d [ostree] Use unique work and log paths
Each variant.arch combination can have multiple ostree repos configured,
so we need to make sure the filesystem paths don't clash.

The paths used now are:

    logs/<arch>/<variant>/ostree-<x>/
    work/ostree-<x>/config_repo

The x stands for a number identifying the task. It has no relation to
actual contents of the repo.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-18 08:03:31 +02:00
Lubomír Sedlář
b43e35f7f2 [arch] Add mock rpmUtils module
Use a mock instead of relying on the real module in tests. This fixes
test failures on RHEL-7 (caused by different configuration).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-17 12:10:45 +02:00
Lubomír Sedlář
eeacb5b9f1 [ostree] Put variant name in ostree log dir
This will prevent log files from being mangled when multiple ostree
repos are created.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-16 18:29:09 +02:00
Dennis Gilmore
e998bbe436 Merge #294 [ostree] Initialize empty repo 2016-05-16 13:39:55 +00:00
Lubomír Sedlář
d3900296af [util] Resolve git+https URLs
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-16 08:01:50 +02:00
Lubomír Sedlář
12652ad1fa [ostree] Initialize empty repo
Make sure the directory exists before calling Koji (because otherwise
the mounting will fail). Update the runroot script to initialize the
repo when there are no files in the target destination.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-10 08:27:43 +02:00
Lubomír Sedlář
4e3d87e658 [test] Add checks for created images
The performed checks:
 * If format is ISO, the file must have correct magic string
 * If it's bootable, there must be MBR or GPT

When a check fails on any failable deliverable, it will be logged and
the file removed from metadata (it will still remain on the disk). This
required a change to write the images.json file later (after test
phase).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-05 15:38:00 +02:00
Dennis Gilmore
c80b7c6894 Merge #273 Deduplicate configuration a bit 2016-04-29 12:26:51 +00:00
Dennis Gilmore
8615c96624 Merge #280 [createrepo] Use more verbose output 2016-04-29 12:26:15 +00:00
Dennis Gilmore
5c4d3a5196 [createiso] Add back running isohybrid on x86 disk images
rhbz#1331317 when we refactored how we make dvds in
df400002d8 we lost the ability to boot
the dvd as a disk image.

Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2016-04-29 09:00:44 +02:00
Lubomír Sedlář
b949bfd615 [createiso] Remove chdir()
Explicitly pass workdir to all invoked command. This is easier to follow
and test.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-29 08:25:49 +02:00
Lubomír Sedlář
d4effc1610 [pkgset] Fix caching RPMs
The test mock did not actually match what is provided by Kobo.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-28 09:05:03 +02:00
Lubomír Sedlář
4da77c5e5d [createrepo] Use more verbose output
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-27 18:35:43 +02:00
Lubomír Sedlář
377fcb6948 [pkgset] Add tests for writing filelists
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-26 14:58:09 +02:00
Lubomír Sedlář
d7012c442a [pkgset] Simplify finding RPM in koji buildroot
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-26 14:44:29 +02:00
Lubomír Sedlář
0c8eb6c0fb [pkgset] Clean up koji package set
* break too long lines
 * remove unneeded code
 * call parent methods with super()

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-26 14:43:43 +02:00
Lubomír Sedlář
ffcb3684de [pkgset] Add test for pkgset merging
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-22 14:06:06 +02:00
Lubomír Sedlář
2bb65408ee [pkgset] Add tests for KojiPackageSet
There is a lot of mock objects needed: we bypass calls to Koji, use a
mock FileCache that does not need valid RPMs on disk and avoid any
multithreading.

The test data in tests/fixtures/tagged-rpms.json comes from Koji. It is
filtered down to only a few packages to make it manageable.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-22 13:35:17 +02:00
Lubomír Sedlář
01422983cb [pkgset] Add tests for Koji source
This patch also fixes the bug where using a preconfigured Koji event
would not actually work and instead the latest event would be used.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-22 09:44:20 +02:00
Lubomír Sedlář
084994538b Add common global settings for images
The live_images, live_media and image_build phases have same options
that need to be specified on each image. This leads to a lot of
duplication in the config file. This patch adds global settings and
phase-level settings that allow to significantly reduce duplication.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-18 18:11:29 +02:00
Lubomír Sedlář
29a2b45fb4 [scm-wrapper] Report when file wrapper did not match anything
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-18 14:24:43 +02:00
Lubomír Sedlář
770ca9c202 [scm-wrapper] Copy files directly
Instead of spawning `cp x/* y` there is now Python code to the same
thing. This should help with debugging if something fails as the
traceback will be more informative (rather than saying a command
failed). As another benefit the tests get much simpler.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-18 14:24:43 +02:00
Lubomír Sedlář
bb9cbe88db [scm-wrapper] Add tests for SCM wrappers
All four methods are not somewhat tested.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-18 14:24:43 +02:00
Lubomír Sedlář
46ab1cb6f6 [ostree] Set each repo to point to current compose
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-15 08:17:40 +02:00
Lubomír Sedlář
e6079e4c85 [ostree-installer] Drop filename setting
This should just use the global default format. Other phases don't have
their own setting either.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-14 14:45:36 +02:00
Dennis Gilmore
ffd054d262 Merge #269 Improve logging of failable deliverables 2016-04-13 14:56:28 +00:00
Lubomír Sedlář
a83a34fd40 [ostree-installer] Fix example documentation
There is no `compose_date`, the field is called just `date`.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-13 15:12:28 +02:00
Lubomír Sedlář
18b6020ac5 Improve logging of failable deliverables
The pungi.global.log should show subvariant for each failed deliverable
(if available). When the compose finishes, there is a new log file in
logs/global/deliverables.json containing details about all deliverables
as triples of (variant, arch, subvariant).

 * `required` lists all deliverables that can not fail
 * `attempted` lists all failable deliverables that were started
 * `failed` is a subset of `attempted` and only contains deliverables
   that failed

If the compose fails, the lists may be incomplete.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-13 13:45:30 +02:00
Lubomír Sedlář
ad32b73918 [ostree-installer] Install ostree in runroot
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-12 12:47:02 +02:00
Lubomír Sedlář
a696eb4f3a [ostree-installer] Clone repo with templates
We can't assume the templates will just be available. This patch adds a
configuration option to point to the git repo with them. It will be
cloned at compose box and relative paths to templates will be made
absolute respective to this clone.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-11 10:25:10 +02:00
Lubomír Sedlář
f17b5e13ad [ostree-installer] Copy all lorax outputs
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-08 12:48:42 +02:00
Lubomír Sedlář
1bf4bec32c [ostree] Log to stdout as well
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-08 11:37:16 +02:00
Lubomír Sedlář
8799d4cd85 [ostree-installer] Use separate directory for logs
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-08 11:36:06 +02:00
Dennis Gilmore
09222a87a0 Merge #260 Maybe fix ostree? 2016-04-08 09:26:11 +00:00
Lubomír Sedlář
81d84a030b [ostree-installer] Put lorax output into work dir
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-08 09:40:41 +02:00
Lubomír Sedlář
8c381e33aa [ostree] Add test check for modified repo baseurl
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-07 15:42:41 +02:00
Lubomír Sedlář
e705cdd165 [ostree] Move cloning repo back to compose box
and pass absolute path to the config into runroot. This could avoid the
problem of not being able to clone the repo in runroot.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-07 15:28:15 +02:00
Lubomír Sedlář
b7948f2d65 [ostree] Mount ostree directory in koji
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-07 14:04:50 +02:00
Lubomír Sedlář
ed12a64a84 [ostree] Enable marking ostree phase as failable
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-06 16:12:29 +02:00
Lubomír Sedlář
3e1a6edecc [createiso] Simplify code, test phase
A couple arguments passed from phase to worker threads were not
duplicated. Only one copy is passed now.

A test case was added both for the phase itself and for worker thread as
well.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-06 13:25:49 +02:00
Lubomír Sedlář
df400002d8 [createiso] Move runroot work to separate script
Instead of running a long command line in the runroot (or locally), move
all that work into a separate script that will be installed. This means
chroot will need to install pungi.

Everything should work as it did before. The only exception to this is
that there is logic to find lorax templates instead of harcoding the
location. This is done using a separate script.

Related: #230
Fixes: #231
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-06 13:25:49 +02:00
Lubomír Sedlář
5b1b6c1c4f [ostree] Use explicit work directory
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-05 12:46:59 +02:00
Lubomír Sedlář
96e7ddd3ea [ostree] Rename atomic to ostree
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-05 09:14:00 +02:00
Lubomír Sedlář
ad87ea2d28 [ostree] Move cloning config repo to chroot
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-05 09:13:55 +02:00
Lubomír Sedlář
5ff2d9f957 [ostree] Fix call to kobo.shortcuts.run
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-04 15:39:38 +02:00
Lubomír Sedlář
1ebb9d1773 [checksum] Add arch to file name
This allows the checksum file to include the architecture.

Fixes: #243
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-04 09:38:48 +02:00
Dennis Gilmore
428af17cb1 Merge #242 Fix wrong file permissions 2016-04-01 13:19:04 +00:00
Lubomír Sedlář
e5f37016e0 Fix atomic/ostree config validations
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-01 07:42:49 +02:00
Lubomír Sedlář
490514e263 [checks] Add a check for too restrictive umask
If umask is set to something too high (>0022), a warning will be
printed. It does not abort the compose though.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-31 10:10:18 +02:00
Lubomír Sedlář
69316d827a [util] Remove umask manipulation from makedirs
This causes a race condition: umask is process-wide setting, so any file
created by another thread while the umask is set to 0 will have wrong
permissions. This also removes the possible problem of not resetting the
umask if exception is raised.

Fixes: #239
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-31 10:10:10 +02:00
Dennis Gilmore
4ec8ad5de8 Merge #240 Filter variants and architectures 2016-03-29 15:36:49 +00:00
Lubomír Sedlář
d383e6c5c0 Filter variants and architectures
There already were config options tree_arches and tree_variants, but the
filtering was done at the compose level and there was not interaction
between the two options. This led to problems when a variant would have
all its arches filtered out.

This patch moves the filtering to the variant loader. It adds better
logging, so whenever a variant is filtered (for any reason), it will be
explicitly stated in the logs.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-29 16:15:30 +02:00
Lubomír Sedlář
98f7ef739e Refactor checking for failable deliverables
This uses a new common function that will ensure consistent logging
without duplicating the functionality all over the place.

A couple tests are updated to verify that correct data is logged.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-24 10:12:20 +01:00
Lubomír Sedlář
bb0267bc7c [buildinstall] Do not crash on failure
When the koji tasks fails but produces some output (e.g. .discinfo),
copying the artifacts would crash the compose. That should not happen if
the variant/arch is marked as failable.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-24 09:17:38 +01:00
Lubomír Sedlář
652987f2cc Reuse helper in all tests
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-23 12:00:06 +01:00
Lubomír Sedlář
8d224b206b [atomic] Add atomic_installer phase
This phase runs lorax with extra templates in Koji runroot task, links
the boot.iso to proper location in compose directory and adds the
installer iso to image manifest. This phase runs concurrently with live
media etc.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-23 12:00:06 +01:00
Lubomír Sedlář
536c6c85b7 [ostree] Add ostree phase
This phase runs the script to make ostree repository in koji runroot
task. It runs right after regular yum repos are created.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-23 10:58:35 +01:00
Lubomír Sedlář
282058dafe [atomic] Add a script to create ostree repo
This is a wrapper over ostree and rpm-ostree. It is intended to be run
in either Koji or Mock chroot.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-23 10:28:08 +01:00
Dennis Gilmore
6230b0ff3e Merge #232 Improve logging by adding subvariants 2016-03-22 14:15:26 +00:00
Lubomír Sedlář
ecbf08c6f8 Add compose type to release for images
Move getting of the precomputed value to a single place in Compose
class. The value now has format `date[.type_suffix].respin`.

Resolves: rhbz#1319924
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-22 08:57:40 +01:00
Lubomír Sedlář
ea7f98ef4e [image-build] Use subvariants in logging output
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-17 10:19:28 +01:00
Lubomír Sedlář
f1974d3f03 [live-media] Use subvariants in logging
Print the subvariant together with variant UID and arches. This should
make the global log easier to understand.

The output of koji command should now be split into multiple files in
logs/{arch}/livemedia-{variant}-{subvariant}.{arch}.log instead of
having them all in one file.

The tests were updated to use more sensible value for the subvariant
field.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-17 09:04:50 +01:00
Lubomír Sedlář
aab22b6f9f [checksums] Add label to file name
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-16 08:49:44 +01:00
Lubomír Sedlář
aa38eb1fd7 [buildinstall] Use customized dvd disc type
This should generate the same volume id in buildinstall and createiso
phases, with isolinux.cfg being consistent as well.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-14 08:20:43 +01:00
Lubomír Sedlář
be8605ac91 Remove check for disc type
This will get validated in productmd anyway, no need to duplicate the
check in Pungi itself.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-11 08:55:21 +01:00
Lubomír Sedlář
c1b8fbe4f9 Update tests to match the subvariant
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-10 16:59:34 -06:00
Lubomír Sedlář
07f5da5d4f [checks] Fix tests to never use real real arch
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-10 18:54:14 +01:00
Lubomír Sedlář
1efd4a9873 [init] Remove duplicated checks for comps
Check once before iterating through the variants. This greatly
simplifies the tests as each function now has one less code path.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-10 15:30:05 +01:00
Lubomír Sedlář
41cdf54c05 [init] Don't overwrite the same log file
When creating the comps repo, use a separate log file for each variant
instead of overwriting the same global file again and again.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-10 15:29:40 +01:00
Lubomír Sedlář
2ca002d602 [init] Add config option for keeping original comps
The configuration can specify a list of variants for which the original
comps file will be copied without any modification.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-10 15:22:21 +01:00
Lubomír Sedlář
6f9f84c4d4 [init] Add tests for the init phase
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-10 15:22:09 +01:00
Lubomír Sedlář
3174f06fd5 [checks] Test printing in all cases
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-10 12:47:49 +01:00
Lubomír Sedlář
0f3bfbbddf [checks] Relax check for genisoimage
The rules are similar to isohybrid, but there are no checks for
architecture.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-10 11:23:45 +01:00
Lubomír Sedlář
2d9dd0e535 [checks] Relax check for isohybrid command
It is used only in createiso and productimg phases.

For productimg, it needs to be present only when the compose is bootable
and productimg phase is explicitly enabled.

For createiso, it is only needed if runroot is not enabled.

Additionally, if we detect pungi running on arch for which syslinux is
not available, a warning is printed, but the compose is allowed to
continue (and possibly crash later).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-10 11:20:10 +01:00
Lubomír Sedlář
62b73944da [checks] Add tests for dependency checking
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-10 10:43:04 +01:00
Lubomír Sedlář
337d2a999c [checks] Don't always require jigdo
If the configuration specifically requests no jigdos, there is no point
in checking for the binary existence.

This is not 100% reliable. The jigdo option defaults to True, so if the
option is not specified the binary is required even if there are no
images configured.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-10 09:08:19 +01:00
Lubomír Sedlář
7e2cd28fb8 [live-media] Support release set to None globally
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-09 13:08:22 +01:00
Dennis Gilmore
0e7ea91d9d Merge #206 Add option to customize disc type 2016-03-08 13:40:43 +00:00
Dennis Gilmore
182c39669d Merge #204 [paths] Document and test translate_path 2016-03-08 13:38:53 +00:00
Dennis Gilmore
f898c7ca7d Merge #203 Simplify common live media options 2016-03-08 13:38:39 +00:00
Lubomír Sedlář
44d7b31a80 [live-images] Add customizing disc type
Relates: #105
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-08 13:04:34 +01:00
Lubomír Sedlář
0c9ad96a31 [buildinstall] Add customizing disc type
This patch adds configuration option to change disc type used in file
name. So far this can only be changed for link to images/boot.iso.

Resolves: #109
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-08 13:04:28 +01:00
Lubomír Sedlář
ad104f994a [buildinstall] Rename method to not mention symlinks
It is either hardlinking or copying, so the name is quite misleading.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-08 11:26:14 +01:00
Lubomír Sedlář
8e20f216f1 [paths] Document and test translate_path
The documentation was inaccurate.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-07 15:18:57 +01:00
Lubomír Sedlář
27375788c2 [createrepo] Compute delta RPMS against old compose
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-07 14:49:17 +01:00
Lubomír Sedlář
19c3707aee [util] Add function to search for old composes
This was already implemented as part of pkgset phase. It is now moved to
the util module and covered with tests.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-07 14:49:17 +01:00
Lubomír Sedlář
86bb816417 [live-media] Add global settings
These can be overriden for a particular image, but in general case they
can simplify the config.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-04 08:15:25 +01:00
Lubomír Sedlář
595845e104 [live-media] Rename test case
Originally it was copied from image build phase, and was not renamed.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-04 08:15:25 +01:00
Lubomír Sedlář
223344e04a [createrepo] Add tests
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-02 16:11:25 +01:00
Lubomír Sedlář
d619f14dbf [createrepo-wrapper] Fix --deltas argument
There is no value, the flag is there or it is not there.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-02 16:11:25 +01:00
Lubomír Sedlář
c93772ade8 [createrepo-wrapper] Add tests
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-02 16:11:25 +01:00
Lubomír Sedlář
a813e926dc [koji-wrapper] Retry watching on connection errors
With this patch Pungi should be more tolerant of network failures when
running a blocking command (creating live media or live images).

If the connection drops and the output indicates network problems, Pungi
will try to watch the task with `koji watch-task`. This will be retried
until it finishes (successfully or with some other failure). There is an
increasing timeout after each retry. Currently the maximum number of
retries is not limited.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-02 14:01:17 +01:00
Lubomír Sedlář
4a4ef23e3c [image-build] Fix resolving git urls
After config was modified to work with sections, the resolving broke.
This patch fixes it and adds a test to catch this problem in the future.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-29 08:23:26 +01:00
Lubomír Sedlář
73a560d63c [live-images] No manifest for appliances
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-26 19:48:52 +01:00
Dennis Gilmore
e6596d818c Merge #190 [live-images] Rename log file 2016-02-26 13:54:01 +00:00
Dennis Gilmore
8670452b6b Merge #189 [buildinstall] Use -dvd- in volume ids instead of -boot- 2016-02-26 13:53:43 +00:00
Lubomír Sedlář
abb27ac7d7 [live-images] Rename log file
The log file started with createiso-, which is utterly confusing. With
this patch, the logs are named with liveimage- prefix.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-26 12:05:58 +01:00
Lubomír Sedlář
452da86649 [buildinstall] Use -dvd- in volume ids instead of -boot-
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-26 11:42:41 +01:00
Lubomír Sedlář
636ac79186 [buildinstall] Hardlink boot isos
Instead of creating a symlink, try to hardlink the image, and copy it if
hardlinking fails.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-26 10:58:03 +01:00
Lubomír Sedlář
60d820d4fd [util] Resolve branches in git urls
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-25 08:39:01 +01:00
Lubomír Sedlář
6e8970e648 [live-images] Fix crash when repo_from is not a list
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-24 19:33:28 +01:00
Lubomír Sedlář
48979a4979 [tests] Fix wrong checks in buildinstall tests
There were asserts that actually did not test anything. They should have
checked that koji is called with correct arguments.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-23 15:30:49 +01:00
Lubomír Sedlář
c61abac137 [tests] Use temporary files for buildinstall
Instead of mocking open just write to the temporary file and then check
its content.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-23 15:30:49 +01:00
Lubomír Sedlář
3ff1d3e21f [tests] Do not mock open for koji wrapper tests
Instead create a temporary file and write into it, then check its
contents.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-23 15:30:49 +01:00
Lubomír Sedlář
ca7c78d98c [live-images] Set type to raw-xz for appliances
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-23 09:41:03 +01:00
Lubomír Sedlář
456fbb8812 [live-images] Correctly create format
Extracting extensions from the file name is not reliable as there is no
way to determine where extensions start. There can very well be a
version.release separated by dot. To bypass this, just use hardcoded
list of possible formats.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-23 09:35:50 +01:00
Lubomír Sedlář
e9292fc942 [tests] Dummy compose is no longer private
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-22 19:11:04 +01:00
Lubomír Sedlář
385b52041c [tests] Move buildinstall tests to new infrastructure
This test case now uses the same mocks as other tests, it only restricts
the variants used.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-22 19:11:04 +01:00
Lubomír Sedlář
90954ddf0e [tests] Use real paths module in testing
This avoids problems of mock paths having different API that the real
ones.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-22 19:11:04 +01:00
Lubomír Sedlář
8661d294ab [tests] Move dummy testing compose into separate module
This greatly reduces duplication, as a single copy is now shared among
all test cases.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-22 19:10:12 +01:00
Lubomír Sedlář
7734ddf57e [live-images] Add images to manifest
The type is either "live" or "appliance", the format is set based on
filename extension.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-22 14:27:51 +01:00
Lubomír Sedlář
f80c97e3ec [live-images] Fix path processing
* Add option to keep filenames generated by Koji.
 * Put results of spin-appliance into image dir
 * On failure images are no longer deleted.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-22 14:27:26 +01:00
Lubomír Sedlář
ab44f3539e [live-images] Filter non-image results
Only keep results that have known extension.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-22 12:27:17 +01:00
Lubomír Sedlář
1044fb01f8 [live-images] Rename repos_from to repo_from
It should be consistent with live media and image build.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-22 08:48:12 +01:00
Lubomír Sedlář
33d5440a8a [koji-wrapper] Add test for passing release to image-build
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-22 08:42:24 +01:00
Lubomír Sedlář
7c81c5aa9c [live-images] Automatically populate release with date and respin
This is the same feature that is already available for image-build and
live-media.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-22 08:38:19 +01:00
Lubomír Sedlář
ec03a8685a [live-media] Respect release set in configuration
Apparently just processing the config is not sufficient we don't pass
the value to Koji.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-22 08:32:27 +01:00
Lubomír Sedlář
32a966fc91 [live-images] Build all images specified in config
Not just the first one.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-22 08:21:51 +01:00
Lubomír Sedlář
0fb8049fdd Update tests
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-19 08:51:03 +01:00
Lubomír Sedlář
37ac0d7bfe [live-media] Update tests to use $basearch
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-18 16:17:30 +01:00
Lubomír Sedlář
1b6688d22c [buildinstall] Don't run lorax for empty variants
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-18 16:12:44 +01:00
Tomas Mlcoch
5bffca5037 Support signing of rpm wrapped live images
With this patch, you can specify a command for
signing of koji builds. For example:

    signing_key_password_file = '~/file_with_password_for_key_fedora-24'
    signing_key_id = '81b46521'
    signing_command = '~/git/releng/scripts/sigulsign_unsigned.py -vv --password=%(signing_key_password)s fedora-24'

'signing_key_password_file' is a path to a file which contains
a password that will be formatted into 'signing_command' string
via '%(signing_key_password)s' string format syntax (if used).
Because pungi config is usualy stored in git and part of compose
logs we don't want password to be included directly in the config.
Note: If '-' is used instead of a filename, then you will be asked
for the password interactivelly right after pungi starts.

'signing_key_id' is ID of the key that will be used for the signing.
This ID will be used when crafting koji paths to signed files
(kojipkgs.fedoraproject.org/packages/NAME/VER/REL/data/signed/KEYID/..).

'signing_command' a command that will be run with a build as a single
argument. This command mustn't require any user interaction.
If you need to pass a password for a signing key to the command,
do this via command line option of the command with use of string
formatting syntax '%(signing_key_password)s' (see details
about 'signing_key_password_file').

Signed-off-by: Tomáš Mlčoch <tmlcoch@redhat.com>
2016-02-18 13:36:38 +01:00
Lubomír Sedlář
bf5196af4a [live-images] Don't tweak kickstarts
Instead of downloading the kickstart file in Pungi and modifying it,
just pass it to Koji.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-17 14:35:28 +01:00
Lubomír Sedlář
454363fba8 Allow specifying empty variants
The variants.xml file can list a variant with is_empty="true" and no
groups. If such variant is found, not package gathering will be run for
it, and no repos will be created.

This only makes sense for a variant that will have some other
deliverables like live media or images.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-16 15:08:15 +01:00
Lubomír Sedlář
826afdc162 Keep empty query string in resolved git url
Pagure: #152
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-16 10:42:59 +01:00
Lubomír Sedlář
e1895bff26 [koji-wrapper] Get task id from failed runroot
Even when runroot task fails, it can emit task_id.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-12 12:38:45 +01:00
Lubomír Sedlář
93015d679c [live-media] Pass ksurl to koji
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 17:09:30 +01:00
Dennis Gilmore
96489a1e78 Merge #146 [live-media] Properly calculate iso dir 2016-02-11 13:07:30 +00:00
Lubomír Sedlář
8ef2e7e308 [live-media] Properly calculate iso dir
The method expects to be called with a arch, not just variant.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 13:58:42 +01:00
Lubomír Sedlář
e9a42ad54f [image-build] Fix tests
This patch updates tests so that they pass, and fixes a couple places in
image build that did not work correctly with the nested configuration.
The documentation is reformatted so that it horizontal scrolling is less
likely.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 13:22:26 +01:00
Lubomír Sedlář
39c688f7ef [koji-wrapper] Add tests for get_create_image_cmd
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:23:54 +01:00
Lubomír Sedlář
5138a9b6e6 [live-images] Add support for spin-appliance
The config can specify type=appliance so that koji uses spin-appliance
instead of spin-livecd.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 12:08:48 +01:00
Lubomír Sedlář
ac29bf05ee [live-media] Koji option is ksfile, not kickstart
In config, it is still called kickstart to be consistent with how
image-build calls it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 10:40:04 +01:00
Lubomír Sedlář
6ec151cde2 [live-media] Use install tree from another variant
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 10:24:12 +01:00
Lubomír Sedlář
0c4bd42f80 [live-media] Put images into iso dir
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 10:11:38 +01:00
Lubomír Sedlář
91faa0fafe [image-build] Koji expects arches as a comma separated string
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-11 07:50:02 +01:00
Dennis Gilmore
8bf1b09641 Merge #139 Log more details when any deliverable fails 2016-02-10 21:03:32 +00:00
Lubomír Sedlář
b35264cfba [live-media] Version is required argument
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-10 18:29:29 +01:00
Lubomír Sedlář
aec57dc72b [koji-wrapper] Only parse output on success
If Koji fails runroot task for some reason, the output will most likely
not have the required format and will crash Pungi.

Pagure: #140
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-10 09:08:21 +01:00
Lubomír Sedlář
8e5e893e5c [koji-wrapper] Add tests for runroot wrapper
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-08 13:27:18 +01:00
Lubomír Sedlář
f57665f963 [buildinstall] Improve logging
This patch fixes how logs are stored if lorax is used as buildinstall
method. The logs for each variant are in a separate file now. If failure
is allowed, the global log will now show why it failed.

A couple tests are added as well.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-08 13:17:08 +01:00
Lubomír Sedlář
3baa386ed4 [image-build] Fix failable tests
A missing input value was causing tests to not check the expected
condition (even though they were still passing).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-08 10:54:24 +01:00
Dennis Gilmore
2b897ec6ea Merge #135 Add live media support 2016-02-02 10:15:56 +00:00
Lubomír Sedlář
439622d576 [live-media] Add live media phase
This phase builds live media in Koji using the Live Media Creator. It
runs in parallel with current live images, create ISO and image build
phases.

The documentation is updated to explain how to configure this.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-01 13:28:49 +01:00
Lubomír Sedlář
a5a0f3d69f [koji-wrapper] Add support for spin-livemedia
This patch adds support for live media creator in Koji. The intended
workflow is to create a command , run it and finally collect built
artifacts.

    get_live_media_cmd()
    run_blocking_cmd()
    get_image_paths()

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-29 09:02:02 +01:00
Lubomír Sedlář
ae30c07553 [koji-wrapper] Use more descriptive method names
The methods mentioning image build are generic and can work for other
task types.

get_image_build_paths -> get_image_paths
run_create_image_cmd -> run_blocking_cmd

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-29 09:02:02 +01:00
Lubomír Sedlář
a698c7cc0a [image-build] Allow running image build scratch tasks
Issue #130

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-26 12:47:59 +01:00
Lubomír Sedlář
dc4d502f2a [image-build] Allow dynamic release for images
When the release is explicitly set to None, generate a value from date
and respin. The documentation is updated to explain how it works.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-21 14:32:29 +01:00
Dennis Gilmore
c77e78c4f1 Merge #123 Live images: add repo from another variant 2016-01-20 15:37:36 +00:00
Dennis Gilmore
09aebd3db9 Merge #125 [image-build] Stop creating wrong arch dirs 2016-01-20 15:37:27 +00:00
Lubomír Sedlář
8b4ad1ea02 Toggle multilib per variant
There is now a single option `multilib`, that maps variants and arches
to multilib methods. This replaces old `multilib_methods` option.

Multilib arches are implicitly deduced instead of using the
`multilib_arches` option.

The test compose is updated to only enable multilib on Server and its
addons.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-20 12:53:08 +01:00
Lubomír Sedlář
6d7fff5d1b [live-images] Code cleanup
This patch removes some duplicated variables that get passed on to the
build thread. It also moves creation of the command for generating image
manifest closer to where it is used. Finally it adds tests for the
thread.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-19 08:26:59 +01:00
Lubomír Sedlář
9059852ab5 [live-images] Add repos from other variants
This patch adds option for adding repositories from other variants to
the koji task for building images.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-19 08:26:59 +01:00
Lubomír Sedlář
e016015cf3 [image-build] Stop creating wrong arch dirs
There is no such arch as $arch or %(arch)s.

Pagure: #124
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-19 08:24:02 +01:00
Lubomír Sedlář
5f0675dd66 Store which deliverables failed
When compose is finished successfully, and there are some failed
deliverables, modify the final status to FINISHED_INCOMPLETE and log
what failed for which variants/arches.

This means the failures are logged twice, first time immediately after
it failed, second time in the summary at the end.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-15 08:41:08 +01:00
Lubomír Sedlář
5688f1cbae [image-build] Optionally do not break whole compose
The configuration can now specify image-build as a deliverable that can
fail but not abort the whole compose.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-12 08:39:30 +01:00
Lubomír Sedlář
d2ea1dd288 [image-build] Use repo from another variant
The config can now additionally specify other variants whose repos will
be passed on to koji. The previous way of specifying extra repos is
still available, as is the automatic adding of repo for current variant.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 16:42:05 +01:00
Lubomír Sedlář
7f815d80f5 [image-build] Take install tree from another variant
Add a configuration option to use install tree from another variant.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 16:42:05 +01:00
Lubomír Sedlář
195b13d434 Add missing formats to volumeid and image name
Also, when the format uses something unrecognized, die with more
descriptive error message.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-11 16:28:10 +01:00
Lubomír Sedlář
9c1418eb0a [image-build] Use single koji task per variant
Given a list of arches, koji can build multiple images in one go
(automatically starting children tasks for each one).

This causes a bunch of changes:
 * The configuration no longer allows changing config based on
   architecture, only variants are allowed. It is however possible to
   filter which arches are used for the building in the variant.
 * The configuration files for koji image-build are stored in
   work/image-build/$variant (not split based on arch).

This patch also changes the option name that is passed to koji
image-build: the repos should be specified under key `repo` (without the
trailing slash).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-08 14:33:54 +01:00
Lubomír Sedlář
a6b673dbdd Add option to specify non-failing stuff
There is a new configuration option that allows listing what can fail
without aborting the whole compose. So far, only buildinstall, createiso
and liveimages phases react to this option.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-17 11:15:47 +01:00
Lubomír Sedlář
0e237db5f6 Allow customizing image name and volume id
The computation of image name has been moved to a separate function.
This allowed simplification in how work dir for isos is computed, but
the result is not changed.

The live image phase has some special casing for names of RPM wrapped
ISOs. This is moved to the actual phase. Since this is (and has been)
undocumented, there should not be many users of this special case.

The documentation is updated to describe how image names and volume ids
are determined and how the process can be customized.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-14 14:39:21 +01:00
Lubomír Sedlář
719ec458f4 Fix notifier tests
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-14 14:35:00 +01:00
Dennis Gilmore
10a90f97b4 Merge #75 Start of development guide 2015-12-08 17:41:00 +00:00
Dennis Gilmore
81e90eb780 Merge #88 Resolve HEAD in ksurl to actual hash 2015-12-08 17:37:09 +00:00
Lubomír Sedlář
fe95221f10 Resolve HEAD in ksurl to actual hash
When the image build configuration specifies kickstart URL as a HEAD of
a git repo, pungi now figures out what the actual hash of that commit is
and uses that hash instead. This might make logs clearer and should
prevent potential problems if someone pushes to that repo during
composing.

Documentation is updated to mention this.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-08 14:29:18 +01:00
Lubomír Sedlář
716b2d521a Add support for customizing lorax options
The config file can now specify options for lorax per variant and arch.
This is needed for Fedora to set x86_64 images as bootable on mac.

The old config option `buildinstall_upgrade_image` was deprecated, as
the same can be specified via new `lorax_options`.

Documentation and tests are updated.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-08 12:47:21 +01:00
Lubomír Sedlář
4a96e8e313 Run lorax in separate dirs for each variant
Lorax fails when run on an existing non-empty directory. This patch runs
it in separate subdirectiories base on variant. Running with
buildinstall should not be changed at all.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-07 09:45:33 +01:00
Dennis Gilmore
bdb1bcb35c Merge #84 Allow specifying --installpkgs for lorax 2015-12-04 19:41:28 +00:00
Dennis Gilmore
4a26b99a24 Merge #83 Fix recently discovered bugs 2015-12-04 19:39:49 +00:00
Lubomír Sedlář
cdec198b6d Start lorax for each variant separately
The buildinstall phase now starts lorax for each variant separately.
Running with buildinstall should not be affected in any way.

Only variants of type=variant are considered. The side-effect of this is
that if an architecture is only used by variants of other types, lorax
will not be called at all.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-03 15:19:12 +01:00
Lubomír Sedlář
94f519d01c Update lorax wrapper to use --installpkgs
The lorax wrapper class now understands the --installpkgs argument and
can use it to pass multiple package names to the command.

There is a simple test to make sure the commands includes all specified
options. A bug is fixed where the bug URL would not be correctly
included.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-03 15:19:12 +01:00
Lubomír Sedlář
cfda99e6fe Add basic tests for buildinstall phase
The test only checks that commands are created with correct arguments
and that a proper number of threads is started. There is no validation
for what the threads are actually doing.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-03 15:19:12 +01:00
Lubomír Sedlář
b6d9b5632e Fix generating checksum files
This patch modifies how checksums are stored - it uses BSD-style
checksums.

The filename with the checksum can now be customized depending on actual
compose run and metadata. This required adding another option to the
checksumming phase. Documentation is updated and includes example for
creating names used in Fedora.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-03 10:54:09 +01:00
Lubomír Sedlář
539736a11e Use lowercase hashed directories
On case-insensitive filesystems it is not such a good idea to have
directories that only differ in case. Packages should be always split
into lowercased directories.

The test data is modified to include some packages starting with
uppercase letters. The example in code can be verified by running
`nosetests --with-doctest`.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-03 09:08:15 +01:00
Lubomír Sedlář
bd8d814230 Move messaging into cli options
The messaging is not really part of compose settings. It is an
infrastructure part. As such, it should really be set up as part of
pungi invocation, not compose configuration.

The documentation is updated to reflect this. Some updates to the
documentation are done as well: listing messages about ISOs and minor
formatting updates.

The test_compose.sh script can now accept additional command line
options and pass them on to pungi-koji to simplify testing.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-11-27 10:35:08 +01:00
Lubomír Sedlář
ba8ae15eb1 Allow running scripts with any python in PATH
Instead of hardcoding /usr/bin/python in shebangs, use /usr/bin/env.
This allows Pungi to work with dependencies installed in virtualenv.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-11-27 08:38:47 +01:00
Dennis Gilmore
6f00f20b3d Merge #68 Add support for sending messages 2015-11-25 14:56:51 +00:00
Lubomír Sedlář
066855a039 Add ability to send messages about progress
With this patch, Pungi can invoke an arbitrary command on various
moments of the compose process. The invoked command can the decide on
what message to send (and using what messaging system).

The actual command is specified in the config file.

There is a script provided that sends the messages via fedmsg.

The documentation is updated to have details about the new config option
as well as the interface for the messaging script.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-11-25 15:46:50 +01:00
Lubomír Sedlář
129e654690 Compute checksums in ImageChecksumPhase
The phase goes through all images declared in image manifest, computes
their checksums, stores them in appropriate files and updates the
manifest so that it includes the actual checksums.

The documentation contains details about new configuration options.

The test suite now needs Python's mock package.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-11-24 09:36:22 +01:00
Daniel Mach
0d8ad9a111 gather: Implement hashed directories.
Set 'hashed_directories = True' config option to enable the feature.
2015-07-24 04:40:55 -04:00
Daniel Mach
1f313b39ad createiso: Add createiso_skip options to skip createiso on any variant/arch. 2015-07-23 10:58:35 -04:00
Daniel Mach
e80879a4fe Add armhfp arch tests. 2015-07-22 08:27:56 -04:00
Daniel Mach
72302bd98e Rename product_* to release_*.
CHANGE: Rename product_* config options to release_* for consistency with productmd.
ACTION: Rename product_name, product_short, product_version, product_is_layered to release_* in config files.
        Rename //variant/product to //variant/release in variants XML.
2015-07-09 06:58:30 -04:00
Daniel Mach
6286e87d3b Config option create_optional_isos now defaults to False.
CHANGE: create_optional_isos config option now defaults to False.
ACTION: You can safely remove 'create_optional_isos = False' from config files.
2015-07-08 09:00:30 -04:00
Daniel Mach
90918dc34d Change createrepo config options defaults.
CHANGE: createrepo_c is config option now defaults to True.
ACTION: You can safely remove 'createrepo_c = True' from config files.
        Set 'createrepo_c = False' if you need legacy createrepo.

CHANGE: createrepo_checksum config option is now mandatory.
ACTION: Add 'createrepo_checksum = "sha256"' (or "sha") to config files.
2015-07-08 08:57:50 -04:00
Daniel Mach
81e935fe0e Fix test data, improve Makefile.
Previous test data was insufficient for proper testing.
Test compose runs and depsolving tests require precisely
set NVRs, dependencies, sub-packages, etc.
Using rpmfluff for these would be an overkill, it's better
to create RPMs directly from specs.
2015-06-25 08:02:57 -04:00
Adam Miller
793e9f3d06 Add some basic testing, dummy rpm creation, and a testing README 2015-06-10 09:53:31 -05:00
Daniel Mach
f5c6d44000 Initial changes for Pungi 4.0. 2015-02-10 06:47:16 -05:00
Daniel Mach
f9f9e6a151 Add test_arch.py. 2013-10-27 15:44:08 -04:00
Daniel Mach
30c0f358d9 Resolve multilib packages. Controlled by the --multilib option. 2013-08-20 11:22:24 -05:00
jkeating@harpoon.lab.boston.redhat.com
023ad1b0fb Shuffle things around so that dist-utils can be used. 2006-11-08 16:34:59 -05:00
jkeating@harpoon.lab.boston.redhat.com
ca72e5dcc1 Change some tabs to spaces 2006-11-06 16:19:47 -05:00
jkeating@harpoon.lab.boston.redhat.com
e4d95b0da5 Add our out splittree until patches are accepted upstream 2006-11-01 16:29:38 -05:00
jkeating@localhost.localdomain
faca390f6e Fonts in comps don't need to be default. 2006-10-23 23:56:24 -04:00
jkeating@harpoon.lab.boston.redhat.com
d3e8cff9ba Update the plan with new info and project layout.
Add a tests dir with some testing files.
2006-10-23 17:47:02 -04:00