Commit Graph

1750 Commits

Author SHA1 Message Date
Lubomír Sedlář
fa3d94bae3 Merge #554 osbs: write manifest for scratch osbs 2017-03-06 12:52:47 +00:00
Lubomír Sedlář
d3a2ceb8ce Extract only first version from specfile
When tagging a new version, rpm will give us the version twice, one for
main package and one for the -utils subpackage. We should only use the
first one (they should be the same anyway).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-06 09:08:13 +01:00
Qixiang Wan
2ae8710934 consolidate repo option names
live_images:            additional_repos        -> repo
ostree:                 source_repo_from        -> repo_from
                        extra_source_repos      -> repo
ostree_installer:       source_repo_from        -> repo_from

With the change, the phases have consolidate option names for variant
repos and external repos.

Old option names will continue to work, old names will be converted
to new names after validation automatically if new options are not
specified in config.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-06 09:26:42 +08:00
Qixiang Wan
c93207addb checks: extend validator with 'alias'
When a property has 'alias' defined, and it's not present in instance,
if the alias property is present, add the property with value from alias
property before remove the alias property from instance.

Examples:

with schema:
{
        "$schema": "http://json-schema.org/draft-04/schema#",
        "title": "Pungi Configuration",
        "type": "object",
        "properties": {
            "release_name": {"type": "string", "alias": "product_name"},
        },
        "required": ["release_name"],
        "additionalProperties": False,
}

1. config = {"release_name": "dummy product"}:

   validate pass, config not changed after validation.

2. config = {"product_name": "dummy product"}:

   validate pass, config updated to the following after validation:

   config: {"release_name": "dummy product"}

3. config = {"name": "dummy product"}:

   validate fail, errror message is "Failed validation in : 'release_name' is a required property",
   and warning message is "WARNING: Unrecognized config option: name."

4. config = {"product_name": "dummy product", "release_name": "dummy product"}

   validate fail, error message is "Failed validation in : product_name is an alias of release_name, only one can be used."

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-06 09:26:42 +08:00
Qixiang Wan
40e2874676 osbs: write manifest for scratch osbs
FIXES: #485

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-06 09:26:01 +08:00
Lubomír Sedlář
345308464f 4.1.13 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-03 14:34:45 +01:00
Lubomír Sedlář
069ee8d3df Merge #555 Make MANIFEST.in stricter 2017-03-03 11:56:41 +00:00
Qixiang Wan
a52561da6d Make MANIFEST.in stricter
Global exclude python bytecode files.
Global exclude vim & emace temporary files.
Only include specs dir under tests/data, there will be other dirs
generated for testing, we don't want them.

FIXES: #427

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-03 14:01:50 +08:00
Qixiang Wan
f43d3584c5 Remove one line of log print
The line of log printing can result in "No handlers could be found for
logger" warning in quiet mode when custom paths module is enabled,
because file handler is setup after that, since this line is only
printed to screen (when pungi-koji is not ran in quiet mode) and doesn't
contain important info, it doesn't hurt to remove it.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-02 11:46:33 +08:00
Lubomír Sedlář
a49bf3d74e gather: Filter comps group on depsolving input of optional
This is a follow up of 8bc65a8be5. When
resolving packages that go into optional variant, we only want to skip
filtering groups when working on a top-level variant.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-27 12:54:12 +01:00
Dennis Gilmore
8b1fb287d3 Merge #539 comps: Filter comps groups for optional variants 2017-02-23 20:13:39 +00:00
Lubomír Sedlář
4ea1916a87 Enable customizing runroot task weight
For different cases where runroot is used it's now possible to set
custom weight. The usecase for this is to avoid one builder taking too
many tasks. Especially buildinstall is quite resource intensive, so one
builder taking multiple tasks at the same time leads to very slow
compose time.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-23 12:44:27 +01:00
Lubomír Sedlář
8bc65a8be5 comps: Filter comps groups for optional variants
The optional variant can defined by just adding has_optional=True into
variant xml. In such case it has no comps groups and Pungi would copy
the original file unmodified. This leads to extra packages being pulled
into the optional variant.

In this case the correct solution is to filter the comps and remove all
groups.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-23 12:44:20 +01:00
Lubomír Sedlář
d9ab899920 Rename main logger
The name is not displayed anywhere. Changing it to lowercase will
however avoid possible problem if we create a sublogger somewhere else
in the codebase with name based on module name. Such logger would not
have any handlers defined.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-23 11:08:31 +01:00
Lubomír Sedlář
9e020c2782 ostree: Silence logger in tests
Instead of creating and configuring the logger at module import time, we
can only get the logger if it's actually needed and configure it from
the main script.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-23 11:08:31 +01:00
Lubomír Sedlář
3162fea60d ostree: Fix crash when extra repos are missing
The default needs to be an empty list, not `None`.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-22 15:48:06 +01:00
Lubomír Sedlář
857aee05c1 util: Add a utility for managing temporary files
In multiple situations we need to create temporary files or directories
that should not be preserved after compose is finished. Let's add
context managers that ensure these get cleaned up.

This fixes tests leaving garbage around in /tmp.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-20 09:14:25 +01:00
Qixiang Wan
a57bc13e30 Add --quiet option to pungi-koji
log will not be printed to screen when quiet mode is enabled.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-02-20 15:00:03 +08:00
Qixiang Wan
d496eeb090 handle opening empty images.json while re-running pungi-koji in debug mode
If there is an empty images.json file exists, re-running pungi-koji
in debug-mode will fail with exception. We can just ignore the exception
in such cases, pungi-koji will create a correct images.json later.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-02-16 17:00:14 +08:00
Qixiang Wan
8d41a004c3 minor change: remove an always true condition
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-02-14 16:36:47 +08:00
Daniel Mach
ad23efd323 Refactor depsolving tests
This adds a new test for Requires(pre) and (post). The general structure
of the test now makes it easy to use the same test scenarios for
different backend.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-13 12:32:07 +01:00
Lubomír Sedlář
e00776a413 multilib: Remove FileMultilibMethod class
In order to do something meaningful, the class needs to be instantiated
with arguments pointing the blacklist and whitelist.

The `file` multilib method used via `pungi-koji` or `pungi` directly has
no way to pass those in.

The only way this class can be useful would be if someone actually
imported the class directly in their own code. Pungi is not meant to be
used as a library though, so this is not really a supported use case.

Not to mention that the `select` method always returned `False`.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-10 12:32:45 +01:00
Lubomír Sedlář
c3cf09a2f7 pkgset: Use additional packages for initial pull
We call pungi to get the packages from external repos if
pkgset_source=repos. In this case we need to look at additional packages
as well.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Fixes: #528
2017-02-09 13:50:09 +01:00
Lubomír Sedlář
b5cfeaa6ca metadata: Fix .treeinfo paths for addons
Currently .treeinfo sections for addons are missing paths to packages
and repository.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-07 15:50:33 +01:00
Lubomír Sedlář
9533fca96c koji_wrapper: Always use --profile option with koji
This patch adds this option to all invoked koji commands.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-30 13:00:25 +01:00
Dennis Gilmore
42cdf047cc add missing koji_profile from test compose setting
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2017-01-30 05:06:46 -06:00
Dennis Gilmore
5a498f80b6 use koji --profile when calling koji for livemedia
take the profile from the koji config and apply it to the koji cli
when building livemedia

Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2017-01-30 04:44:31 -06:00
Dennis Gilmore
c650f04d0b Revert "live-media: Call correct koji alias"
This was accidently merged and is not correct
This reverts commit 02beb35e67.

Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2017-01-30 04:44:17 -06:00
Lubomír Sedlář
dc7d3b36ab repoclosure: Don't run build deps check
Fixes: #521
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-30 10:16:03 +01:00
Dennis Gilmore
7feadb14ba Merge #518 Migrate test phase to dnf repoclosure 2017-01-28 16:49:10 +00:00
Lubomír Sedlář
02beb35e67 live-media: Call correct koji alias
We need call whatever command is configured in koji_profile. It's not
always `koji`.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-26 15:32:21 +01:00
Lubomír Sedlář
e3fe67be53 repoclosure: add option to use dnf backend
This adds a new option repoclosure_backend that changes what tool is
used for repoclosure.

Checking build dependencies is currently not supported, as `dnf` does
not have the corresponding option.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-26 13:39:12 +01:00
Lubomír Sedlář
95fc0fa4ab repoclosure: Add test for repoclosure in test phase
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-26 13:27:48 +01:00
Lubomír Sedlář
56932f9067 repoclosure: Remove duplicated code
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-26 10:55:23 +01:00
Lubomír Sedlář
7a8fa87172 repoclosure: Remove useless wrapper class
We already get namespacing from being in a separate module.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-26 10:55:23 +01:00
Lubomír Sedlář
5693bf9925 repoclosure: Remove unused code
There are a lot of options that are not used by any code path in Pungi.
There is not reason to provide access to them.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-26 10:55:23 +01:00
Lubomír Sedlář
9af0cca9eb repoclosure: Add a test for the wrapper
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-26 10:55:23 +01:00
Lubomír Sedlář
8418b68fb0 image-build: Pass arches around as a list
Instead of joining the arches as a comma separated string and splitting
it again later. Ultimately we do need the original format to pass to
koji wrapper, but we can produce that value later.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-24 08:50:21 +01:00
Lubomír Sedlář
6c708549c8 image-build: Expand arches for can_fail
We need to work with a list of strings, not a comma-delimited single
string.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-24 08:50:21 +01:00
Qixiang Wan
40df2034a8 image_checksum: add file sizes to checksum files
Write file sizes of images in checksum files with comment lines,
checksum files are in BSD-style which supports comments by starting
a line with '#'.

Example:

$ cat RHEL-7.4-20170123.n.4/compose/Server/x86_64/iso/RHEL-Server-7.4-x86_64-20170123.n.4-CHECKSUM
 # RHEL-7.4-20170123.n.4-Server-x86_64-dvd1.iso: 3725590528 bytes
 # RHEL-7.4-20170123.n.4-Server-x86_64-boot.iso: 377487360 bytes
 SHA256 (RHEL-7.4-20170123.n.4-Server-x86_64-dvd1.iso) = fa3de37fe4b859a0285f16ea1123f44f15aec169aea84bf010aa3821bd58fc41
 SHA256 (RHEL-7.4-20170123.n.4-Server-x86_64-boot.iso) = 74bf68c54665328adb08b09daf773c67e633b5907e3e2797338ab3c1b58fdf48

(No space at the start of line, because git commit message drops lines
start with '#', added one space to avoid that.)

When there are multiple checksum types specified and checksums are
written to individual files, file size of the image will also be
written to every checksum files.

Fixes: #493

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-01-23 18:20:21 +08:00
Lubomír Sedlář
4f011fbd45 Add documentation and example for greedy_method
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-19 09:24:54 +01:00
Dennis Gilmore
59dd4dbcd8 replace ${basearch} when updating the ref
pungi-make-ostree has to run on the target arch so that rpm
scriptlets can be ran. as a reult we can ask rpm what the
basearch is for the running environment. For notifications
we have to pass in the arch we are running for.

Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2017-01-18 21:21:26 -06:00
Patrick Uiterwijk
4edf567bd4 Add some debugging about ref updating
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2017-01-18 14:09:15 -06:00
Lubomír Sedlář
d3e701e10f 4.1.12 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-18 08:07:42 +01:00
Lubomír Sedlář
2c76313382 unified-iso: Fall back to default config
When the configuration can not be read from the compose, we should use
default values instead of aborting. This allows us to work with composes
produced by Distill.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-17 08:15:13 +01:00
Qixiang Wan
a9b275f13b osbs: optionally check GPG signatures
If gpgkey option is defined in config, set gpgcheck=1 and set
gpgkey=<value> in variant repo files.

Fixes: #487
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-01-17 13:08:58 +08:00
Qixiang Wan
5d241d316a ostree-installer: Allow multiple repos in ostree installer
Add new key 'repo' to allow specifying multiple repos as the source
repositories. And change 'source_repo_from' to allow specifying multiple
vairant names to use variant repos as source repositories.

Doc of 'source_repo_from' is updated to not mention URL is supported,
though we still support that in code. User should add url of repos in
'repo' key instead of 'source_repo_from'.

Fixes: #508

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-01-13 16:57:30 +08:00
Lubomír Sedlář
aa5487d544 Update tox.ini
Add more useless pycodestyle checks, and explain what the codes mean.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-11 14:37:41 +01:00
Lubomír Sedlář
afaa0e025e unified-iso: Create isos with debuginfo packages
When creating unified ISOs, the script will now also create one iso per
architecture containing a repo with debuginfo packages.

There is no switch to turn this off. The images can simply not be
shipped if not wanted.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-11 14:36:25 +01:00
Qixiang Wan
6fbf1e8f59 Create temporary dirs under compose's workdir
Add compose.mkdtemp which creates and returns a unique temporary
directory under <compose_topdir>/work/{global,<arch>}/tmp[-<variant>]/

Change tempfile.mkdtemp in code to compose.mkdtemp, so temporary
dirs are created under the compose's workdir, this makes it more
easier to maintain the temporary dirs/files especially when compose
fails with amount of temporary files left there.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-01-11 15:25:47 +08:00