Commit Graph

902 Commits

Author SHA1 Message Date
Lubomír Sedlář cbfb556f7c extra-iso: Use correct efiboot.img file
If the file is generated, Pungi will modify it at the end of
buildinstall phase. We need to point ISO creation to the modified file,
otherwise the checksum in .treeinfo will be incorrect.

It's sufficient to just update the graft point to correct file, copying
is not needed.

JIRA: COMPOSE-2976
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-12 13:41:36 +02:00
Lubomír Sedlář a00ea3d4a6 extra-iso: Fix treeinfo
Remove boot.iso references and add [media] section.

JIRA: COMPOSE-2974
JIRA: COMPOSE-2975
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-11 13:19:07 +02:00
Lubomír Sedlář bb6e68a853 createiso: Move code for tweaking treeinfo into a function
The function loads existing treeinfo, removes reference to boot.iso and
adds [media] section. This is the basic tweak that should happen for all
ISOs. Additional changes depend on the actual content.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-11 13:19:07 +02:00
Lubomír Sedlář 740df1bc6c extra-iso: Generate jigdo by default
It can be turned off by the same option that is used in createiso phase.

JIRA: COMPOSE-2962
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-10 16:06:46 +02:00
Lubomír Sedlář 3fea217b9c hybrid: Remove modules not listed by fus
It's possible we ask to include module X, but it's in lookaside and as
such it should not be in the output. Therefore we need to remove it from
the variant.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-08 14:41:04 +02:00
Lubomír Sedlář 1a161982c0 gather: Make devel modules configurable
By default nothing should change. This patch adds a new config option
that enables all this new craziness:

 * it turns of applying module filters at pkgset level
 * it creates new modules and adds them to the compose

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-08 14:41:04 +02:00
Lubomír Sedlář 06b32b5d80 pkgset: Stop prefilling RPM artifacts
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-08 14:41:04 +02:00
Lubomír Sedlář 08d65bdde6 gather: Create devel module for each normal module
The module has same S:V:C, but the name is suffixed with `-devel`. The
module should contain all packages from the module koji tag that were
not included in the actual module.

The devel module has the same dependencies as the regular module, but
also additionally depends on the original module. The API and profiles
are cleared in the new module.

In the metadata it shows the same koji tag.

The test if package goes to the module is refactored to a function to
make work with the negated case a bit easier.

There may be unneeded multilib packages in the -devel module, because
there might be buildtime dependencies between things that we don't see.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>

Clear API and profiles

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-08 14:40:13 +02:00
Lubomír Sedlář 410d0125bc pkgset: Save package set for each module
This can be used in figuring out which packages go into the module and
which not.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-08 14:36:43 +02:00
Lubomír Sedlář 1bff5ccfa2 fus: List lookaside repos first
This works around a bug where fus prioritizes first repo and thus could
include packages even if they were in lookaside.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-08 14:36:43 +02:00
Lubomír Sedlář aab2fc4519 extra_iso: Add list of variants to metadata
The main variant is already available, this patch adds information about
additional variants that are included in the image.

JIRA: COMPOSE-2917
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-05 12:01:43 +02:00
Lubomír Sedlář e2e8df3f09 gather: Hide pid of fus process
There's an environment variable that can suppress printing of PID for
every debug message. It provides us with no real information and only
makes the log larger.

JIRA: COMPOSE-2973
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-03 10:23:39 +02:00
Lubomír Sedlář 4537001ff6 fus: Strip protocol from repo path
Only local paths are supported currently. As such, `file://` can be
stripped, and for anything else we should raise an exception.

JIRA: COMPOSE-2996
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-03 09:47:00 +02:00
Jan Kaluza 37c89dfde6 Add 'pkgset_koji_builds' option to include extra builds in a compose
This PR adds new pkgset_koji_builds configuration option.

This option allows setting list of extra Koji build NVRs which will be
included in a compose. This is useful in two cases:

a) It allows generating standard composes with few packages update to
certain version to test how the compose behaves when the package is
updated for real.

b) It allows generating compose consisting only from particular builds
when pkgset_koji_tag = '' or None. This is useful when one want to
regenerate the compose with packages which are not tagged in single Koji
tag. This is very useful for ODCS when reproducing old composes.

Merges: https://pagure.io/pungi/pull-request/1049
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-09-21 14:06:49 +02:00
Lubomír Sedlář 2a65b8fb7d ostree: Reduce duplication in tests
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-19 13:07:03 +02:00
Lubomír Sedlář 36c347eb79 ostree: Use --touch-if-changed
There are three different cases:

 * we expect commitid and it's there
 * we expect commitid and it's missing
 * we don't expect commitid

This patch helps differentiate between the second two. In former one we
should report an error and mark the phase as failed. The latter is
perfectly fine and no error should be reported

Fixes: https://pagure.io/pungi/issue/1046
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-19 13:07:03 +02:00
Lubomír Sedlář a53dc6f1bb gather: Expand multilib lists for hybrid method
If there are globs in the blacklist or whitelist, we need to expand it
to full package name.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-05 15:08:31 +02:00
Lubomír Sedlář 1350684c31 Index arch modulemd by full NSVC
There can be multiple modules with the same name and stream. They should
all have the same version, but will have different contexts. Fus takes
only N:S as input, but should pull in all matching modules. We just need
to give it correct data in the repo.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-05 13:10:53 +02:00
Lubomír Sedlář fbb739ef17 pkgset: Apply whitelist to modules in the tag
This patch changes the behaviour when both module tag and NSV?C? is
specified. The NSVC are used as a whitelist and only matching modules
will be included in the compose.

Additionally this patch adds filtering based on inheritance: when
finding the latest module for each N:S combination, only the top tag in
which the module is tagged is used. Even if a newer build is available
somewhere deeper in the inheritance, it's not going to be used.

Example inheritance and tagged modules

    f29-compose (foo:1:2018:cafe)
    └─ f29-candidate (foo:1:2019:cafe)

The compose will use 2018 version, because it's in the topmost tag.

JIRA: COMPOSE-2685
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-03 12:19:50 +02:00
Lubomír Sedlář 3419762830 extra_iso: Set unified flag in metadata
This will avoid conflict in productmd that two images share the same
attributes.

JIRA: COMPOSE-2908
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-31 08:46:21 +02:00
Lubomír Sedlář 828557b4d6 pkgset: Respect koji event when searching for modules
In the search result, we should ignore any module build that finished
after the event that we are working with.

Fixes: https://pagure.io/pungi/issue/999
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-30 14:58:45 +02:00
Jan Kaluza d2f392fac8 Use dogpile.cache to cache the listTaggedRPMS calls if possible
If the same tag is queried with the same event, Pungi can cache the
response and call the API again. Particularly for small composes this
can save up significant amount of time.

Merges: https://pagure.io/pungi/pull-request/1022
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-08-30 13:21:01 +02:00
Lubomír Sedlář 92968fe52d gather: Keep original rpms.json in debug mode
If we're running in debug mode and the file is already present, it
should not be modified. This means that in order to rerun the actual
gather phase the file needs to be manually deleted first. However the
much more common use is to skip gather phase (because only images should
be re-run). In that case the manifest will be preserved correctly.

JIRA: COMPOSE-2756
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-29 15:52:45 +02:00
Lubomír Sedlář 7c5020e82d Reduce duplication in tests
Set the dummy compose into non-debug mode in a single place, and only
change it when needed.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-29 15:52:45 +02:00
Lubomír Sedlář 5346e000f0 Create non-bootable ISO for variant without buildinstall
If the configuration specifically excludes a variant from buildinstall,
but does not also disable ISO creation, we should just create a
non-bootable ISO instead of reporting a warning.

JIRA: COMPOSE-2887
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-28 08:56:03 +02:00
Lubomír Sedlář 45fa9d3273 Clean up after yum tests
The tests create empty directories in current working directory, but
they never clean them up. Instead we can switch to a temporary location
that will be cleaned up automatically.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-27 09:16:40 +02:00
Lubomír Sedlář 4b05e7b9fd extra-iso: Rename test data file
The file is not being included in the source tarball, which breaks
internal jenkins and will eventually cause the test to fail at RPM build
time.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-15 14:57:43 +02:00
Lubomír Sedlář ac0d5e4ede createiso: Use correct python version
On some systems we need to use python3, in other places it can be
another version. Instead of guessing, let's look at shebang line in
lorax executable and use the same.

JIRA: COMPOSE-2852
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-15 08:37:12 +02:00
Lubomír Sedlář b85cd7ff9f ostree: Update tests for working with YAML file
Merges: https://pagure.io/pungi/pull-request/1019
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-14 16:13:23 +02:00
Lubomír Sedlář 2fa1f09827 createrepo: Allow passing arbitrary arguments
This could be used to enable zchunk generation, which can require up to
4 different options. Instead of hardcoding every single one, let's just
allow more direct access to the executed command.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-08 16:15:51 +02:00
Lubomír Sedlář b12deab153 gather: Get modular packages from fus
Fus returns also RPMs in modules, but until latest version it only
worked if the package was in the same repo as the metadata. This changed
in latest version and now Pungi does not need to expand the list
anymore.

JIRA: COMPOSE-2779
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-03 08:00:37 +02:00
Lubomír Sedlář 5926858b58 util: Remove escaping spaces from volume ID
This was not configurable for users, and the default was always used,
which meant no escaping. Might as well just remove the dead code.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-03 07:57:09 +02:00
Lubomír Sedlář add9835b56 Allow removing non-alnum chars from volid
This mimics similar change in lorax.

JIRA: RCM-36970
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-03 07:57:09 +02:00
Lubomír Sedlář 10bdb370ba extra-isos: Include treeinfo pointing to all variants
This will be used by Anaconda to consume multiple repos for
installation.

JIRA: RCM-36970
JIRA: COMPOSE-2753
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>

Fixup
2018-08-03 07:57:09 +02:00
Lubomír Sedlář b772d4a773 gather: Add langpacks in hybrid solver
Comps file specifies a pattern for some packages. If that package is
installed, all packages matching the pattern are added as well. This can
be added to fus as another pass.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-23 09:08:33 -04:00
Lubomír Sedlář 16ac225013 comps: Add get_langpacks function
This also simplifies the test data by removing a lot of the langpacks.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-23 09:08:33 -04:00
Lubomír Sedlář 7d099012aa Fix tests for DNF 3
The configuration for modules has changed.

Lookaside handling has changed, and there are now test failures. This
is a not a bug in DNF, so we need to fix it on our side.

Relates: https://bugzilla.redhat.com/show_bug.cgi?id=1603123
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-19 14:22:10 +02:00
Lubomír Sedlář 0b9652f2d6 tests: Use unittest2 when available
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-18 12:55:34 +02:00
Jan Kaluza 6080b45178 Fix Koji search for modules with dash in stream
MBS replaces the dash by underscore. We have to match that.

Merges: https://pagure.io/pungi/pull-request/1005
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-07-18 10:31:19 +02:00
Lubomír Sedlář 23c454ff67 buildinstall: Make output world readable
This makes it possible to run a compose as non-root user, plus removes
the need for workarounds to publish the results directly.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-17 10:12:52 +02:00
Lubomír Sedlář e3de4dcccf Report failed failable deliverables as errors
It does not abort the compose, but it's still an error. It should be
marked as such in the log. This will allow easier searching for the
failures.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-17 09:31:50 +02:00
Lubomír Sedlář ba0193ca28 gather: Add a hybrid depsolver backend
This patch adds a new gather method called `hybrid`, which uses a `fus`
binary, which must exist somewhere on the `$PATH`. It will call it
multiple times to add multilib packages.

The solver can handle packages, modules and comps groups as input.
However comps groups are never passed in. Pungi will expand it to a list
of packages to avoid issues with comps handling in fus. It ignores
optional packages, and if the group mentions a package that does not
exist, nothing else from the group is included.

Multilib is also handled outside of fus. Pungi will run it, parse the
packages from output, determines multilib packages and adds them as
input. Then it runs the solver again. This is done until nothing new is
added. Usually two passes should be enough.

Source packages and debuginfo are added as a final step. All debuginfo
packages from any included source are added. If the source or debuginfo
package is included in any lookaside repo, it will be skipped.

The tool expects to get a platform stream that should be provided for
modules to depend on. Pungi looks into the modules and gets the platform
from there. If there are more requests, an error is raised.

There is some missing functionality and options that are ignored.
Particularly these are:

 * gather_fulltree
 * gather_selfhosting
 * greedy_method

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-16 10:21:41 +02:00
Lubomír Sedlář d2849d3826 osbs: Generate unique repo names
And include variant in repo file name. The whole path is unique already,
but not the filename itself.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-09 13:21:58 +02:00
Sinny Kumari 1759c1ba80 Expand version field during image_build using version_generator
We can specify !VERSION_FROM_VERSION in version field during
image_build to expand it to correct release number without any label
information.

Also implemented !RELEASE_FROM_DATE_RESPIN to provide correct
release number. This helps to keep Atomic Host media files name
produced by image_build during bodhi updates compose run
consistent with nightly run.

Fixes: https://pagure.io/pungi/issue/987
Merges: https://pagure.io/pungi/pull-request/995

Signed-off-by: Sinny Kumari <sinny@redhat.com>
2018-07-09 12:56:00 +02:00
Lubomír Sedlář 98e7106f3e pkgset: Apply module filters on pkgset level
If a module says to filter a package out, we can do it immediately when
getting the build information from Koji.

This avoids a possible problem of something pulling the module package
in as a dependency, but it should also make the package set slightly
smaller.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-25 15:15:42 +02:00
Lubomír Sedlář 470c0ab3be init: Validate whitespace in comps groups
If a package name contains leading or trailing whitespace, it will
eventually lead to issues: pungi will try to include that group, but
since it does not exist, the packages will not make it in.

The root cause is hard to find. Better report an error immediately.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-25 11:58:52 +02:00
Lubomír Sedlář 096075848b createrepo: Include empty modules
A module without any RPMs is still valid and should be included. It
should also be in the metadata.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-22 12:54:39 +02:00
Lubomír Sedlář 38f1a8509e createiso: Break hardlinks by copying files
If a file has multiple hard links, genisoimage will put the wrong number
on the ISO. This patch can work around it by copying hard-linked files
into a temporary staging directory.

JIRA: COMPOSE-2610
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-20 12:40:43 +02:00
Martin Curlej d8c03f6239 pkgset: Query Koji instead of PDC
PDC is deprecated in upstream. The usecase for getting list of modules
by NS, NSV or NSVC can however be satisfied by querying modules imported
into Koji.

This makes it possible to deprecate PDC configuration.

Merges: https://pagure.io/pungi/pull-request/985
Signed-off-by: Martin Curlej <mcurlej@redhat.com>
2018-06-18 15:17:04 +02:00
Lubomír Sedlář 4a61de1e8a variants: Reject values with whitespace
If there is leading or trailing whitespace in a comps group name, it
will not be included in the compose and there will even be no error
message. Whitespace on module name results in a failure.

To avoid these errors, validating the variants file will now also check
that there is no whitespace in significant places, and abort the compose
if there a problem.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-12 12:48:14 +02:00
Lubomír Sedlář eed97f357c init: Test that init phase correctly clones defaults
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-07 10:25:35 +02:00
Lubomír Sedlář 13f5018ce1 init: Add tests for cloning module defaults
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-07 10:25:35 +02:00
Lubomír Sedlář cd493bc4c1 init: Add validation for module defaults
After cloning the repository with defaults, open each file and try to
check if there is more than one definition for the same module. It's not
a problem for the compose process, but consumers of the compose would
get confused and possibly explode. Better alert people early.

Conceptually this should be part of the test phase, but that would mean
waiting for the compose to finish before reporting the error. The
earlier the better.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-07 10:25:35 +02:00
Lubomír Sedlář c10a4ca337 ostree-installer: Skip comps repo if there are no comps
When the compose is not using comps, we can't pass the comps to lorax,
since it doesn't exist and it would cause a crash.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-07 09:27:14 +02:00
Lubomír Sedlář 663a07068e kojiwrapper: Call chmod recursively
Related: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-04 13:11:37 +02:00
Lubomír Sedlář 71f2c61020 Add test for getting licenses from a repo
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-04 08:26:26 +02:00
Lubomír Sedlář 102fec83b3 kojiwrapper: Don't mark runroot as successful by chmod
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-31 14:50:03 +02:00
Lubomír Sedlář cbcebe90e1 Allow extracting koji event from another compose
When multiple composes are chained, they should reuse the same event.
However it is tricky as the value would have to be passed by hand. This
patch makes it possible to read the value from another compose (the
first one in the chain).

JIRA: COMPOSE-2571
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-31 11:11:21 +02:00
Lubomír Sedlář 92b5ad2e05 kojiwrapper: Make result of runroot world readable
The commands in runroot run as root every time. If they create files
that are not readable to other users, the reset of compose could have
problems with it if it does not run as root too. Particularly updates
composes in Bodhi run under apache user.

Relates: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-29 10:31:42 +02:00
Lubomír Sedlář e53da69db3 osbs: Add nvr to metadata
All the components are there already separately, but having the full NVR
should simplify searching the metadata with grep.

JIRA: COMPOSE-2519
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-28 13:16:01 +02:00
Patrick Uiterwijk f1cd1ae562 Always get old compose with release type suffix
Signed-off-by: Patrick Uiterwijk <patrick@puiterwijk.org>
Merges: https://pagure.io/pungi/pull-request/792
2018-05-25 09:49:14 +02:00
Patrick Uiterwijk 288d9ecc90 Make ostree_installer check if buildinstall is skipped correctly
The _skipped attribute is only set after the buildinstall phase is started.

Fixes: #909
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2018-05-24 13:25:34 +02:00
Lubomír Sedlář fdea7878f1 comps-wrapper: Make tests pass on EL6
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-22 12:34:39 +02:00
Ondrej Nosek c85d80f3c2 Handling multiple modules with the same NSV - PDC
Signed-off-by: Ondrej Nosek <onosek@redhat.com>

JIRA: COMPOSE-2510
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-05-21 11:24:46 +02:00
Lubomír Sedlář b4e746aa71 createrepo: Allow disabling SQLite database
This is an optimization for Yum. DNF does not care at all.

The behaviour is configurable, but the default depends on gather
backend, as that is what users should be using to consume the packages
from the repo.

Fixes: https://pagure.io/pungi/issue/951
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-21 08:13:29 +02:00
Lubomír Sedlář 1afb709404 init: Drop database from comps repo
It should not be needed there, since the repo is empty anyway. A test is
added for the variant specific comps repo.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-21 08:13:29 +02:00
Lubomír Sedlář 5324c6441f arch: Make i386 map to i686 instead of athlon
This is used for mapping basearch to binary arch.

One use for this is when running depsolving to tell yum/dnf what arch to
work with. With this change it will get i686 instead of athlon when
working on i386 basearch.

The other use case is finding arches compatible with given basearch. The
change will remove athlon from the list for i386.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-16 15:22:31 +02:00
Lubomír Sedlář 2862ae0b28 ostree-installer: Use Python function to copy
This should give us better error reporting. The `copy_all` function
should preserve permissions on all files.

Relates: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-16 12:35:32 +02:00
Lubomír Sedlář b2f995d516 Add a phase for creating extra ISOs
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-16 12:20:01 +02:00
Ondrej Nosek b2190c1c3f Validation of parameter skip_phases
It checks config file value 'skip_phases' for valid phases names.
Also checks command-line attribute 'skip-phase' of 'bin/pungi-koji'.

JIRA: COMPOSE-2493

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-05-15 13:30:00 +02:00
Lubomír Sedlář 905064775a arch: Remove mocks in tests
This was added back when rpmutils module was used. Since we migrated to
Python 3 we have included the code directly in Pungi (because it was
coming from yum which is not going to Py3 any time soon). The mocks can
be dropped.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-09 13:49:29 +02:00
Lubomír Sedlář 7c237c2c63 ostree-installer: Allow overwriting buildinstall
Relates: https://pagure.io/pungi/issue/909
Relates: https://pagure.io/pungi/issue/695
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-07 15:22:01 +02:00
Lubomír Sedlář 543154d597 ostree-installer: Work with skipped buildinstall
Fixes: https://pagure.io/pungi/issue/909
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-07 12:29:20 +02:00
Lubomír Sedlář 076be762ec createrepo: Use less verbose logs
There is really no need to write out megabytes of logs that are not
really interesting. This should also help the parallelization. With the
verbose log createrepo fills the output buffer and needs to wait for the
busy python program to read it first.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-07 08:38:23 +02:00
Lubomír Sedlář bd9a0ceda2 pkgset: Create global repo in parallel to merging pkgsets
The work in creating the repo is done in a separate process. This can
easily use two threads.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-07 08:38:23 +02:00
Lubomír Sedlář dc557b809a ostree-installer: Copy files without owner
The files created in koji runroot will be owned by root. If the compose
is done under different user, there could be a problem with copying the
files preserving the owner. Let's just copy them without that.

Fixes: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-04 14:18:24 +02:00
Lubomír Sedlář 99b6e44a30 createiso: Skip if buildinstall fails
If the ISO is meant to be bootable but lorax fails, there's no point in
creating the ISO as it will not behave as expected.

Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1574585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-04 08:45:45 +02:00
Lubomír Sedlář 490f079c44 Update tests for libmodulemd 1.4.0
It got a little picky about allowed inputs. We need full RPM NEVRA in
all tests.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-02 15:36:52 +02:00
Lubomír Sedlář ed8fffb6d1 koji-wrapper: Log failed subtasks
If the parent task is successful, there can still be failed child tasks
for failable arches. We need to log those and potentially mark the
compose as incomplete.

Fixes: https://pagure.io/pungi/issue/874
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-02 08:27:44 +02:00
Lubomír Sedlář b8555b7869 modules: Allow context in variants XML
JIRA: COMPOSE-2508
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-26 09:26:37 +02:00
Lubomír Sedlář eaf58f7d40 gather: Use another variant as lookaside
Create a temporary repository and add it as another lookaside in the
compose.

JIRA: COMPOSE-2426
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-23 10:51:56 +02:00
Lubomír Sedlář ea0964eeef buildinstall: Use metadata if skipped
If the phase is skipped, it could mean that we are doing a debug run and
we don't want to mess up the .treefile by missing arch specific images.
The other alternative is that the phase was really skipped, in which
case there will be no files generated and we already handle that fine.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-23 10:42:33 +02:00
Jan Kaluza 9915c7d644 Allow reusing pkgset FileCache from old composes.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-04-23 09:57:03 +02:00
Lubomír Sedlář 58afece033 Only use comps repo if we really have comps
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-17 15:32:02 +02:00
Lubomír Sedlář 267ff86f04 pkgset: Use modules PDC API
Instead of the deprecated and confusing unreleasedvariants.

JIRA: COMPOSE-2363
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-17 12:16:18 +02:00
Patrick Uiterwijk 527394707d Access ci_base date via compose
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2018-04-15 02:16:23 +02:00
Lubomír Sedlář a217eea24e Allow filtering comps for different variants
We already filter for arches, so this just extends the filter to work
with variants as well.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-13 14:56:06 +02:00
Ondrej Nosek cb3d36be5d Variant as a lookaside - configuration
Relates: COMPOSE-2425

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-04-13 14:14:43 +02:00
Lubomír Sedlář 15ccd309fa Remove comps from arch repo
The comps could potentially be different in different variants, so
instead we can create the comps repo for every variant separately and
use two repos instead of one (packages in one repository, comps in
another one).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-12 13:52:36 +02:00
Petr Šabata 0e7f770fb7 Include module defaults in the repodata
If the compose configuration includes the module_defaults_dir (an
scm_dict), clone the directory, read the module defaults contained
therein and include relevant defaults in the combined modulemd file.

Only defaults for modules present in the variant are included.

This requires libmodulemd 1.2.0+.

Merges: https://pagure.io/pungi/pull-request/891
Signed-off-by: Petr Šabata <contyk@redhat.com>
2018-04-11 16:29:17 +02:00
Lubomír Sedlář 8f1beeb54b init: Always filter comps file
Even for Everything we want to filter the comps file to make sure we
remove the stuff that is not compatible with current arch. All groups
are still preserved in that case.

This allows us to do the filtering once in init phase than just use the
prepared file in comps source.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-10 08:21:56 +02:00
Lubomír Sedlář aab3b04b08 gather: Use comps for given variant
The file for variant is filtered to only contain relevant parts.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-10 08:21:56 +02:00
Lubomír Sedlář de231064b7 comps: Add tests for CompsFilter
All use cases that are actually used by pungi-koji are tested. There is
missing coverage for

 * keeping only items with matching arch
 * not reindenting the file

These aren't currently used and should be removed in the future, but
there may be other tools depending on the comps_filter executable.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-09 07:48:28 +02:00
Ondrej Nosek 7798174b30 Tests fail if unittest2 library is missing
Merges: https://pagure.io/pungi/pull-request/894
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-04-09 07:47:17 +02:00
Lubomír Sedlář f38770c67d pkgset: Construct UID for PDC modules
We can't rely on the UID to be correctly joined with colons. There may
be historical data that still uses dashes.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-06 10:47:22 +02:00
Lubomír Sedlář d9e2101b08 buildinstall: Add extra repos
A new configuration option is added that allows users to point lorax to
extra repositories. This can be handy if some tools to create the
bootable image are not part of the product itself.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
JIRA: COMPOSE-2253
2018-04-05 07:56:20 +02:00
Lubomír Sedlář 1436ea2b03 tests: Use dummy modulesdir for DNF
Otherwise it tries to ensure it exists, and since the default is
/etc/dnf/modules.d, it's causing problems if the directory does not
exist and user does not have permissions to create it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-04 12:23:43 +02:00
Ondrej Nosek 1f0739831c Update tests for Python 2.6
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-03-29 16:34:52 +02:00
Ondrej Nosek e3aa2f769b Write module metadata
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-03-29 09:47:55 +02:00
Jan Kaluza c6d507582a Support multilib in GatherSourceModule
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-03-28 13:38:46 +02:00
Patrick Uiterwijk f814651d91 If sigkeys is specified, require at least one
Merges: https://pagure.io/pungi/pull-request/880
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2018-03-21 15:34:58 +01:00
Jan Kaluza 9be2d6a920 Allow setting <kojitag/> in <modules/> in variants.xml to get the modules from this Koji tag.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-03-21 14:33:45 +01:00
Jan Kaluza 1574f306c7 Move Modulemd import to pungi/__init__.py to remove duplicated code.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-03-20 12:59:17 +01:00
Jan Kaluza 3f71cdd384 Use Modulemd.Module for 'variant.arch_mmds' instead of yaml dump
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
Merges: https://pagure.io/pungi/pull-request/872
2018-03-19 14:47:03 +01:00
Jan Kaluza fedce5dff1 Remove the filtered RPMs from module metadata even in case all RPMs are filtered out.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-03-15 11:13:39 +01:00
Lubomír Sedlář 5c902592ae pkgset: Allow empty list of modules
This should indicate that it's a modular variant, but there is no
modular content yet. We don't want to treat that as Everything.

The end result will be an empty repository.

Fixes: https://pagure.io/pungi/issue/871
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-14 14:49:09 +01:00
Lubomír Sedlář 95bb147015 buildinstall: Add option to disable it
Fixes: https://pagure.io/pungi/issue/854
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-13 15:54:52 +01:00
Jan Kaluza 340ae4d286 Use libmodulemd instead of modulemd Python module
Merges: https://pagure.io/pungi/pull-request/851
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-03-12 13:07:33 +01:00
Lubomír Sedlář 56e00505e0 gather: Fix package set whitelist
We need to include all relevant arches, not just the base one (including
noarch and src). However the list can be shortened by only listing
NEVRs, because that should be unique.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-09 15:53:31 +01:00
Lubomír Sedlář c83316da31 pkgset: Merge initial package set without checks
For the first pass we don't need to filter out exclusive architectures,
and we don't need to exclude source packages without any binary
packages. We just want to merge the two package sets as fast as
possible.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-08 14:29:23 +01:00
Lubomír Sedlář a03a46a078 gather: Honor package whitelist
Basically everything not on the list is excluded. This has to be applied
before we filter only the latest versions (otherwise we could lose
packages that are on the whitelist).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-08 14:29:23 +01:00
Lubomír Sedlář 41d0139b39 Write package whitelist for each variant
If we have a package set for the variant (which happens if there are
modules), include a list of all NEVRAs in the pungi kickstart.

This can be used to make sure only packages from correct tag get into
the compose. If two packages with same name but different version get
into the compose, this can help get even older version into a particular
variant.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-08 14:29:23 +01:00
Lubomír Sedlář bd852f4059 Remove comps groups from purely modular variants
The comps source should not return all groups when there are only
modules defined. This fixes part of the problem: non-modular packages
will not go in by default.

The second part is the comps file in the created repository. It will be
filtered to not contain any groups (because packages from there will not
be in the repo).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-02 08:46:23 +01:00
Lubomír Sedlář 3201648c37 gather: Allow filtering debuginfo packages
This already works on YUM backend, and this patch makes it work for DNF
as well.

Both native and multilib debuginfo and debugsource packages will be
excluded. This matches the yum behavior.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-02-28 10:07:42 +01:00
Ondrej Nosek 660c04368b Move ostree phase and pipelines for running phases
Signed-off-by: Ondrej Nosek <onosek@redhat.com>

Related: https://pagure.io/pungi/issue/778
2018-02-26 13:43:44 +01:00
Ondrej Nosek 5c081cb545 Other repo for OstreeInstaller
OstreeInstaller phase will be moved to a different timeslot
and therefore needs different repo not to depend on Gather
phase which runs at the same time.

Related: https://pagure.io/pungi/issue/778

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-02-26 13:10:36 +01:00
Adam Williamson a1d559fb93 Correct fix for volume ID substition sorting by length
The previous attempt - caed78e - is not really correct. It sorts
the dict item tuples according to the alphabetical sort order of
the first item of each tuple (reversed). This will always work
when both substitutions *start* with the same characters, as in
the case of two strings that start with the same characters but
have a different length, the shorter one sorts alphabetically
first, and we reverse that. But it is not safe if the shorter
substitution doesn't start with the same characters, as in the
case I put in the tests: we should sort 'zzzaaaaaazzz' before
'aaaaaa' (and hence apply the 'zzzaaaaaazzz' substitution to a
volume ID that contains that string and not the 'aaaaaa' one),
but the previous commit did not.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-02-23 14:48:04 -08:00
Ondrej Nosek caed78e11a Ordering processing for volume ID substitutions
Related: https://pagure.io/pungi/issue/840

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-02-22 16:46:58 +01:00
Jan Kaluza ef058d1f9b Fail early if input packages are unsigned
Use 'get_packages_to_gather' to fail early if these packages are not
signed with right key. This prevents us from having to wait for the
repo to be created and depsolving to finish. Unsigned dependencies will
still be reported later than previously.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-02-13 12:35:55 +01:00
Jan Kaluza c75f4a1e96 Allow composing from tag with unsigned packages
There can be packages in the tag that will not end up in the compose.
Instead of failing immediately with error, this patch delays the check
until after depsolving finishes and only checks packages that will
really be included.

This is not an issue for nodeps compose, as that already pulls in only
packages that will be composed and nothing else.

Merges: https://pagure.io/pungi/pull-request/843
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-02-13 12:35:17 +01:00
Ondrej Nosek c7cc200246 Ostree can use pkgset repos
Related: https://pagure.io/pungi/issue/778

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-02-08 15:23:15 +01:00
Lubomír Sedlář 364d7f5229 Support multiple sources in one variant
With this patch the gather_source option is no longer used. Instead, all
sources are always used. If they return at least some input packages,
then a configured method is used and the returned lists of packages from
all sources are merged.

The method used for gathering can be configured for each variant and
gather source separately.

Additional packages are only added to the comps source.

Each gathering step is logged separately. All the logs are preserved for
later inspection.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-02-01 14:31:09 +01:00
Ondrej Nosek f301158974 ostree/utils: Drop timestamps from generated repo names - tests
Issue: https://pagure.io/pungi/issue/811

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-01-17 16:48:22 +01:00
Lubomír Sedlář 03293c725b config: Make pkgset_koji_tag optional
There are valid use cases for not specifying this option: specifically a
modular compose will get the tags to use from modules listed in the
variants file.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-01-12 15:50:21 +01:00
Ondrej Nosek 51cd359057 ostree: Add force_new_commit option - test added
Relates: https://pagure.io/pungi/issue/811

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-01-11 15:08:07 +01:00
Ondrej Nosek d4d264eb33 Improve logging for unsigned packages
Relates: https://pagure.io/pungi/issue/820

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-01-03 11:08:00 +01:00
Ondrej Nosek 2152e7ea26 Fall back to mount if guestmount is not available
Relates: https://pagure.io/pungi/issue/803

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-01-02 17:13:47 +01:00
Ondrej Nosek 116e7ca3bd El-Torito boot information on s390x
Relates: https://github.com/rhinstaller/lorax/pull/236

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2017-12-14 10:08:30 +01:00
Ondrej Nosek 59c162d46f Remove strace from buildinstall runroot
The package is not really needed.

Relates: https://pagure.io/pungi/issue/799

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2017-12-11 17:36:28 +01:00
Lubomír Sedlář e15a49defd tests: Use correct python version for config validation test
When tests are running using Py3, the script should be called with Py3
as well.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-12-06 13:26:43 +01:00
Lubomír Sedlář d5a0316877 Use dnf backend for repoclosure on PY3
When yum is not available, it makes no sense to offer it as an option.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-12-06 13:26:43 +01:00
Lubomír Sedlář 001ae1d7b4 Use modifyrepo_c if possible
When configured to use createrepo_c, we should also use modifyrepo_c.
That allows us to relax the check for createrepo package.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-12-06 13:26:43 +01:00
Ondrej Nosek 9642c1171c add ability to specify ostree ref in OSTREE phase - update
Additionally ostree_ref (if parameter is given) should be placed into treefile.

Relates: https://pagure.io/pungi/issue/777

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2017-12-04 18:07:36 +01:00
Jan Kaluza 953fb4c54c buildinstall: Allow using external dire for runroot task
A new `buildinstall_topdir` option allows using buildinstall even when
the compose is created on a different volume that Koji is using.

The files are created in this external directory and then copies into
the usual location.

Merges: https://pagure.io/pungi/pull-request/807
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2017-11-30 10:55:52 +01:00
Ondrej Nosek 91ee1fb854 Remove deprecated options
Options that are currently marked as deprecated do not have any effect
anymore (other than printing warning). We should remove them and update
the message so that we can mark options as deprecated even when they
still work.

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2017-11-16 11:12:34 +01:00
Patrick Uiterwijk 8181c5be48 Turn COMPOSE_ID version generator into DATE_RESPIN
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2017-11-10 11:14:11 +01:00
Lubomír Sedlář 388be481ea iso-wrapper: Remove hacks for sorting
We can use a key function instead of relying to the deprecated cmp. This
makes the code work on Python 2.6 and on recent versions it makes it
faster.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-11-08 15:35:29 +01:00
Patrick Uiterwijk 1dbd0248d4 Implement version.compose_id version generator
Signed-off-by: Patrick Uiterwijk <patrick@puiterwijk.org>
2017-11-06 16:14:44 +01:00
Patrick Uiterwijk 1a10a1fe83 Optionally do old_compose per release type
This would make sure that e.g. "updates" composes don't try to use "updates-testing" as an
old_compose_path, which would create practically useless deltarpms and for no repodata
reuse at all.

Signed-off-by: Patrick Uiterwijk <patrick@puiterwijk.org>
2017-11-06 15:17:33 +01:00
Lubomír Sedlář cdfa3cb45f image-build: Drop suffixes from configuration
The file extension in configuration is only used to tell Pungi which
files from the task results should be downloaded. The user has to get it
right or the phase will fail. Each format has a single valid suffix.

Pungi should not require users to specify the suffix, since it can just
as well just know the right value.

The old configuration will continue working, only the extension will be
ignored.

Fixes: https://pagure.io/pungi/issue/753
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-11-01 12:47:36 +01:00
Lubomír Sedlář 026ba10987 kojiwrapper: Deal with multiple values for image-build
When the config for image-build command contains multiple values, they
should be joined with commas into a single value.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-11-01 12:47:36 +01:00
Lubomír Sedlář 390dff52ae notification: Add more info into the messages
Fixes: https://pagure.io/pungi/issue/771
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-31 11:11:04 +01:00
Lubomír Sedlář 28c3bc6268 notification: Fix running on Python 3
The input is given as unicode value, not a bytestring. We need universal
newlines to handle the conversion as needed.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-31 11:10:27 +01:00
Ondrej Nosek e0308a74ff remove remaining hard coded createrepo threads
To be more precise, new createrepo parameter "workers" was customized and
new default value was set. This fixes issue #752.

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2017-10-30 10:37:10 +01:00
Lubomír Sedlář 6d6cf6e233 tests: Fix remaining missing assertions
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 09:29:28 +01:00
Lubomír Sedlář f21e3a2d6d tests: Work with older unittest2
RHEL has an older version of the library which does not backport all the
assertions that we used. In order for the tests to pass there we need to
use names that exist everywhere.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 09:16:18 +01:00
Lubomír Sedlář 6f21576921 tests: Skip testing pdc logs if dependencies are not installed
On EPEL 6 modulemd is not available, so the test fails with an import
error. We should just skip this particular tests, as the rest of the
functionality is still perfectly fine.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 08:21:21 +01:00
Dong Wang 463fb961bc Log PDC communications and info for modular composes
Fixes: https://pagure.io/pungi/issue/664
Merges: https://pagure.io/pungi/pull-request/723
Signed-off-by: Dong Wang <dowang@redhat.com>
2017-10-27 16:20:21 +02:00
Lubomír Sedlář daf162503c Reject yum gather backend on Python 3
It will not run, and having a nice error message is better than a
cryptic crash.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-24 15:15:31 +02:00
Lubomír Sedlář 797b13b34a Convert configparser values to string
On Python 3, configparser will reject non string values, and
theoretically we could have some in the configuration.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-24 15:15:31 +02:00
Lubomír Sedlář 2efc4d8561 Explicitly decode test files as UTF-8
This way the test does not fail if run in a non-UTF-8 locale.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-24 15:15:31 +02:00
Lubomír Sedlář ed9d7f69a6 Use universal_newlines when running other commands
This will automatically convert the output to unicode/str and we will
not have to worry about decoding ourselves.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-24 15:15:31 +02:00
Lubomír Sedlář ed22e07ef9 Port to Python 3
This should make all tests pass on both Python 2 and Python 3.

Unittest2 is required on Py 2.6 and Py 3.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-24 15:15:31 +02:00
Lubomír Sedlář be39dc3caf buildinstall: Expose template arguments for lorax
This would be useful for modularity. The templates can be added now and
variables set via the existing `lorax_option`.

It's not possible to use custom templates not shipped with lorax, as
passing the path to a random directory is a little bit more tricky.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-23 16:08:09 +02:00
Jan Kaluza 894a5a11a4 Add support for new modules naming policy with colon delimiter
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2017-10-20 10:42:09 +02:00
Jan Kaluza 6208dae869 pkgset: Cherry-pick packages from Koji when we know already what packages will end up in compose
Merges: https://pagure.io/pungi/pull-request/763
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2017-10-19 13:43:59 +02:00
Lubomír Sedlář 2819311d6e config: Allow comps_file for any gather_source
It only affects gathering packages when gather_source is set to comps,
but it could still make sense to include the file in the repository even
if the not used for gathering.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-18 15:24:24 +02:00
Lubomír Sedlář c89f033457 live-images: Honor global settings for target
Instead of the old default value and custom handling introduce new
option `live_images_target` (for consistency with other phases) and use
the usual inheritance rules.

Fixes: https://pagure.io/pungi/issue/749
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-09 13:46:54 +02:00
Lubomír Sedlář 842e2e810e unified-isos: Stop erasing metadata on failure
When saving new metadata fails, the images.json file would be left
empty. Instead we should not touch it at all.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-06 08:37:56 +02:00
Lubomír Sedlář 9ab3840085 Add directory name for checksum file
Fixes: https://pagure.io/pungi/issue/745
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-05 16:08:27 +02:00
Lubomír Sedlář dec00fe2f4 createrepo: Allow customizing number of threads
The default is now to use one thread per CPU.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-03 15:21:37 +02:00
Dong Wang adaab46bf7 Fix pungi-koji --version
Signed-off-by: dowang <dowang@redhat.com>
2017-09-18 17:04:05 +08:00
Lubomír Sedlář 9dbf231080 util: Fix timezone offset
We need to negate the value: the values are in seconds west of UTC, but
ISO 8601 wants the offset to be negative for times behind UTC (i.e. to
the west).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-09-12 09:41:53 +02:00
Lubomír Sedlář ed0a8249b1 gather: Add tests for excluding packages from lookaside
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-09-12 08:38:08 +02:00
Lubomír Sedlář 4b40a1258a gather: Capture broken deps in test
This will help test more aspects of depsolving.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-09-12 08:38:08 +02:00
Lubomír Sedlář 6a425ee891 Fix formatting timezone offset
Displaying the offset in seconds makes very little sense. We should
adhere to ISO 8601 format of `+HH:MM` which is much easier to
understand.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-09-11 10:24:33 +02:00
Lubomír Sedlář 40796c04f4 createrepo: Only consider successful compose for deltas
If the compose failed, it may not have repos to compute deltas against,
and even if it has them, they were never shipped so no one will have the
older version of the package. We should instead go deeper in history and
pick a successful compose.

Relates: https://pagure.io/pungi/issue/715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-09-04 09:15:20 +02:00
Lubomír Sedlář e21a27bdc9 createrepo: Allow selecting variants for delta RPMs
The configuration needs to be more granular than a single global option.
With this patch each tree can enable deltas separately.

Fixes: https://pagure.io/pungi/issue/715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-09-04 09:15:20 +02:00
Lubomír Sedlář 90be25c14c createrepo: Only create delta RPMs for binary repos
It does not make much sense to have deltas for source and debug repos.
No one benefits from it really and it takes a long time.

Relates: https://pagure.io/pungi/issue/715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-09-04 09:15:20 +02:00
Qixiang Wan b81e94e808 image-build: add arch name(s) in image config file name
Pungi write image config file with name of <format>-<name>.cfg, if there
are two or more image configs present for different arches under the same
variant and with same format & name, the config file can be overwritten,
and result in invalid image conf file.

Example:

image_build = {
    '^Server$': [
        {
            'image-build': {
                'format': [('qcow2', 'qcow2'),],
                'name': 'fedora-guest-image',
                'target': 'guest-fedora-26-image',
                'version': '26',
                'ksurl': "git://git.example.com/ks.git?fedora#HEAD",
                'kickstart': "fedora-26-kvm.ks",
                'ksversion': 'f26',
                'distro': 'fedora-26',
                'disk-size': '10',
                'arches': ['x86_64'],
                'repo': ["http://example.com/linux/fedora/26/Everything/x86_64/os", ]
            }
        },
        {
           'image-build': {
                'format': [('qcow2', 'qcow2'),],
                'name': 'fedora-guest-image',
                'target': 'guest-fedora-26-image',
                'version': '26',
                'ksurl': "git://git.example.com/ks.git?fedora#HEAD",
                'kickstart': "fedora-26-kvm.ks",
                'ksversion': 'f26',
                'distro': 'fedora-26',
                'disk-size': '10',
                'arches': ['ppc64le'],
            }
        },
    ],
}

In this case, config file "qcow2_guest-fedora-26-image.cfg" will be
created for both x86_64 and ppc64le under the same variant dir, and
there is a high chance it will be over-written while Pungi creating the
koji task. We can add arch name(s) in config filename to avoid that.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-09-03 23:51:08 +08:00
Lubomír Sedlář 65910f2c33 Open files as binary where needed
In many cases we need to open files as binary to avoid errors on Py3
about writing binary data to file opened in text mode.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-28 13:47:18 +02:00
Lubomír Sedlář fcbc3ed4ae buildinstall: No copy if task fails
Even a failed lorax task can leave behind some in-progress images. Pungi
needs to copy the files into compose/ subdirectory only if the task
finished successfully.

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1485021
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-28 13:43:32 +02:00
Lubomír Sedlář a63e4746c9 Report warning when config sections are not used
It's possible a variant is excluded via tree_variants option and the
section does not match anything. It can be confusing to users why
nothing is happening. This patch lets Pungi log all unmatched patterns.

Fixes: https://pagure.io/pungi/issue/692
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-21 09:07:10 +02:00
Lubomír Sedlář 23ca2fe5d2 checks: Stop looking for imports
We can't reliably tell user what system packages are missing as the name
might be different on different systems. Addiotionally there's no reason
why not rely on the packaging to be correct.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-14 14:57:06 +02:00
Lubomír Sedlář 804a0049f6 ostree: Simplify configuration
It makes no sense to repeat the same configuration for multiple
architectures. Instead we should just list the architectures as another
key in the mapping. There is an option to specify multiple config dicts.

This preserves full backwards compatibility, the old config format is
still accepted.

Fixes: https://pagure.io/pungi/issue/678
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-14 14:53:22 +02:00
Lubomír Sedlář c12bad295f config: Add option for dumping config schema
This makes it easier to compare changes to the schema in different
versions.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-14 14:53:22 +02:00
Lubomír Sedlář cae202c17b scm-wrapper: Allow running command after git clone
When a file should be obtained from a git repository, allow running an
arbitrary command (like `make`) after clone but before copying the files
out. This only works for the Git backend.

The downside is that a clone is needed and we can no longer use `git
archive` to speed things up.

Fixes: https://pagure.io/pungi/issue/5
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-10 15:32:24 +02:00
Lubomír Sedlář f9640ae0b4 scm-wrapper: Test correct file lists are returned
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-10 15:32:24 +02:00
Lubomír Sedlář 470b3e4923 tests: Fix test_compose.sh paths
The paths need to be absolute so that subprocesses started during the
compose with a modified cwd will still work.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-10 09:28:36 +02:00
Lubomír Sedlář c9f34b6684 gather: Only parse pungi log once
No need to read the file three times, we can just get all the data in
one go.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-09 11:04:14 +02:00
Lubomír Sedlář 1d7617f783 gather: Avoid reading whole log into memory
There's no reason for reading the whole log of depsolving into memory
just to split it into lines and process one line at a time.

We can just as well read it in chunks.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-09 11:04:14 +02:00
Lubomír Sedlář e2962dc547 repoclosure: Allow aborting compose when repoclosure fails
Alternatively the call to repoclosure can be turned off. This is
customizable per variant and architecture.

Fixes: https://pagure.io/pungi/issue/676
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-07 15:17:33 +02:00
Lubomír Sedlář 99204bb695 tests: Make test-compose cwd independent
With this patch it is possible to run the test compose script from the
top level directory. This makes it slightly easier to use.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-07 15:17:33 +02:00
Lubomír Sedlář c66f2228b5 unified-iso: Only link to non-empty variants
Instead of adding images to metadata and then creating hardlinks in a
separate step, do it immediately while we still have accurate
information about what variants it should be linked to.

Fixes: https://pagure.io/pungi/issue/670
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-31 15:15:17 +02:00
Lubomír Sedlář 471e369d23 gather: Fix excluding debugsource packages from input list
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-27 08:50:21 +02:00
Lubomír Sedlář e1eacd456e gather: Add debugsource package to tests
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-27 08:31:39 +02:00
Qixiang Wan 0a3e5b27bf remove the dependency of rpmUtils
The rpmUtils module is provided yum-utils package, which is only
available for Python 2. There is no replacement for the functionality in
DNF.

There is a proposal to add this functionality to rpm itself, but it's
not really moving forward very much:
https://bugzilla.redhat.com/show_bug.cgi?id=1072972

As a short term solution let's copy the needed parts of rpmUtils.arch
module directly to pungi code base.

Fixes: https://pagure.io/pungi/issue/533
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-07-26 11:23:16 +02:00
Lubomír Sedlář 81cb0952ca checksum: Refactor creating checksum files
Instead of iterating over the images metadata and appending the checksum
to relevant files immediately, we should store them and write only once.

This avoid an issue when the same image is mentioned in the metadata
multiple times. This happens for source images that are listed under
each binary arch.

The unified isos script is updated to use the exact same logic and code.
This also uncovered a problem with the metadata for debuginfo unified
isos: their paths in metadata were incorrect, which lead to missing
checksums.

Fixes: https://pagure.io/pungi/issue/667
Fixes: https://pagure.io/pungi/issue/668
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-17 08:26:51 +02:00
Lubomír Sedlář a831d65c40 createrepo: Don't use existing metadata with deltas
When creating the final repo, we reuse metadata from arch repo used for
depsolving. This however breaks creating deltas with createrepo_c.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-14 14:29:16 +02:00
Lubomír Sedlář 4d117d17f8 util: Fix finding older compose
When there are composes with two digit respin, the code would prefer 9
over 10 as latest. Respin needs to be treated as a number.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-14 10:42:35 +02:00
Lubomír Sedlář 910f816be4 createrepo: Use correct paths for old package dirs
Createrepo expects to be pointed to a directory with the actual RPM
files, not the previous repo. This means that when hashed directories
are used, we need to pass in a lot of directories.

Fixes: https://pagure.io/pungi/issue/344
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-14 10:40:44 +02:00
Lubomír Sedlář e246aef5f6 ostree: Add URL to repo to message
The message announcing new ostree commit contains hash of the commit,
the ref it's for, but there is no information about where the repo
actually is.

This patch adds `repo_path` key into the message with URL of the repo
and `local_repo_path` with path to the repo on local filesystem.

Relates: https://pagure.io/pungi/issue/650
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-11 10:24:49 +02:00
Lubomír Sedlář 8c48dfb93a gather: nodeps should take packages from comps groups
When gather_method is set to nodeps, we should not ignore the comps
group that the method received. Instead it should find out which
packages are in those groups and take them into the compose.

In order for this to be of any reasonable use, the comps file needs to
include all dependencies for the packages.

Fixes: #653
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-11 10:21:13 +02:00
Lubomír Sedlář a21c8a555d notification: Allow specifying multiple scripts
The notification hooks can be useful for doing other things than just
announcing status on message bus. For this to be truly usable, we need
the ability to use multiple scripts.

This patch allows the command line option to be specified multiple
times. Each given script will be called. Even if the script fails, it
does not block the compose.

Additionally the output of the notification scripts is logged now to
make it possible to debug possible failure.

Relates: https://pagure.io/pungi/issue/650
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-29 08:30:56 +02:00
Qixiang Wan 18bd37ff2c pkgset: Allow populating packages from multiple koji tags
One of the use cases is https://pagure.io/odcs/issue/7.

Merges: #660
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-06-29 08:27:34 +02:00
Lubomír Sedlář 68098bec37 ostree: Update tests for no ostree init
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-27 07:46:35 +02:00
Lubomír Sedlář bfc1cebbc4 osbs: Config validation should accept a list
There can be multiple images listed for a single variant, the config
validation should not reject it.

The syntax with a single config object is still accepted. The price for
that is less descriptive error message when there are errors.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-23 12:39:48 +02:00
Martin Curlej 079454c502 pkgset: Use release number of a module
Signed-off-by: Martin Curlej <mcurlej@redhat.com>

updated regex, added comment, pep8 fix
2017-06-21 15:53:39 +02:00
Dennis Gilmore 65b75e7049 Merge #646 `ostree-installer: Clean up output dir` 2017-06-20 15:31:09 +00:00
Lubomír Sedlář b2554ce663 gather: Log tag from which we pulled a package
For each tag we ask Koji about (there might be more than one in
modularity case), we create a log file with list of RPMs and details
about which tag they were pulled from. This makes it easier to find out
where the package is inherited from.

Fixes: #547
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-19 10:51:37 +02:00
Lubomír Sedlář 6e6e250cec ostree-installer: Clean up output dir
The Koji task can be restarted and lorax will fail if the output
directory already exists. Let's start the work in runroot by removing
the output directory.

Relates: #641
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-15 08:17:07 +02:00
Lubomír Sedlář a54b68d08b tests: use unittest2 if available
This fixes a test failure on RHEL 6.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-13 08:55:41 +02:00
Lubomír Sedlář 0cdf996e6e util: Retry resolving git branches
When there's a temporary network issue, Pungi will fail to turn a branch
into a commit hash. This would abort the whole compose. Instead we
should just retry a few times.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-06 15:31:39 +02:00
Lubomír Sedlář 700106facf arch: Move exclu(de|sive)arch check to a function
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-06 15:28:12 +02:00
Lubomír Sedlář 1db1abbb82 koji-wrapper: Stop mangling env variables
When koji is authenticated with a keytab, by setting the private
directory we erased rest of existing environment. In non-keytab path,
the environment variables got removed as well.

This patch makes sure that the environment will not be modified more
than necessary (by setting KRB5CCNAME if needed).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-05 12:46:45 +02:00
Dennis Gilmore 3ddfd4d172 Merge #621 `gather: Don't resolve dependencies in lookaside` 2017-06-02 14:01:46 +00:00
Lubomír Sedlář 247a1a71ba Ensure all phases are stopped
If a phase is started successfully, it needs to be stopped as well. In
most cases when `stop` is called immediately after `start`, this is not
a problem.

Only when something else happens while a phase is runnning and this
something fails, Pungi will deadlock and never exit. This something
could be another phase or just main thread raising an exception.

Fixes: #625
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-02 09:52:18 +02:00
Lubomír Sedlář 8c237b78c2 comps-wrapper: Report unknown package types
When there is a typo in the comps file, instead of crashing with a
non-descript KeyError we should raise a nice error with details about
the problem.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-01 09:59:41 +02:00
Lubomír Sedlář 63c0bbc5c9 iso-wrapper: Capture debug information for mounting
Occasionally we have seen the mount command fail. The default error
message says to set some environment variables and try again. We can
just always set the environment and only print the output on failure.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-05-29 13:18:07 +02:00
Lubomír Sedlář 3130d837c0 gather: Don't resolve dependencies in lookaside
When looking at a package in a lookaside repo, it does not make much
sense to process its dependencies. We should just assume that the
lookaside can satisfy them.

In the worst case, this could result in packages being pulled into the
compose just so that they could satisfy a dep of something in lookaside.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-05-29 12:55:38 +02:00
Lubomír Sedlář f27f3ce4ba koji-wrapper: Run all blocking commands with fresh ccache
If keytab is used for authentication, other commands than runroot can
possibly fail due to the credentials cache being overwritten.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-05-29 12:53:29 +02:00
Jan Kaluza d037d61521 Store RPM artifacts in resulting repository in modulemd metadata.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2017-05-04 21:12:30 +02:00
Lubomír Sedlář c784dab4aa gather: Remove handling of impossible state
The get_system_release_packages function can never be called without a
variant, so it makes no sense to check for that condition.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-05-02 10:12:16 +02:00
Lubomír Sedlář 834445e7d0 gather: Clean up code
* add comments explaining what's going on
 * break too long lines
 * simplify the logic where possible
 * use with statement to work with files
 * remove commented out and unused code
 * introduce helpers to reduce code duplication

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-05-02 10:12:14 +02:00
Lubomír Sedlář feb87077b3 gather: Add tests for gather phase
And also all helper functions in the module.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-05-02 10:11:27 +02:00
Lubomír Sedlář be0c1e1964 tests: Avoid creating unused temporary files
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-27 15:59:19 +02:00
Lubomír Sedlář 24c32831a1 tests: Clean up persistent temporary data
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-27 15:56:03 +02:00
Lubomír Sedlář 059449e140 koji-wrapper: Run koji runroot with fresh credentials cache
If the koji profile we are using is configured to use keytab, we should
run koji executable with a fresh credentials cache. Otherwise we risk a
race condition as multiple processes will trample over the same
directory in /tmp/krbcc_0.

This is currently only implemented for calling `koji runroot`. We might
need to do it for other commands as well (currently there is a sleep to
avoid the race condition for other commands).

Fixes: https://pagure.io/releng/issue/6715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-25 15:45:00 +02:00
Lubomír Sedlář 7028399403 util: Move get_buildroot_rpms to koji wrapper
This way the util module does not import the particular wrapper.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-25 15:22:41 +02:00
Lubomír Sedlář bab2a125d4 osbs: Make git_branch required option
In order to avoid conflicting tags, OSBS allows only one build for a
repo/branch pair at the same time. To avoid race conditions, we should
make sure we always pass in the branch. This commit makes it a required
option.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-24 10:05:25 +02:00
Lubomír Sedlář c293a1e147 extra-files: Allow configuring used checksums
Instead of adding a new config option, we can just reuse the existing
`media_checksums` value. If the value is good for image checksums, it
should work for extra files as well.

Relates: #591
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-20 09:28:09 +02:00
Lubomír Sedlář 610c4ec596 util: Report better error on resolving non-existing branch
When the config contains a git url pointing to a non-existing branch,
pungi will fail to get commit hash from that branch and die with a
confusing error message.

Fixes: #583
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-19 15:37:15 +02:00
Lubomír Sedlář 2bc719a33a util: Show choices for volid if all are too long
When we fail to generate a volume ID that fits in 32 characters, the
error message should include the options that were considered. It could
show that there might be a substitution that could fix the problem.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-18 16:13:55 +02:00
Qixiang Wan 63327e7d88 checks: Fix anyOf validator yield ValidationError on ConfigOptionWarning
We have some hooks yield ConfigOptionWarning. When it happens within
anyOf validator, anyOf validator yield ValidationError and reports the
config as incorrect. We need to overwrite it to pass not break.

Fixes: #598
Merges: #599
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-04-18 10:05:03 +02:00
Lubomír Sedlář 39c3f42f77 comps-wrapper: Sort langpacks by name
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-13 11:45:23 +02:00
Lubomír Sedlář fbf189d9c7 comps-wrapper: Add tests
This tests pretty much all the code that parses, filters and then writes
comps back to a file.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-13 11:45:23 +02:00
Lubomír Sedlář f6d07c1651 Be explicit about generating release for images
The config now uses similar logic what previous commit did for OSTree.
Also we should report error when an unknown generator is used.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-13 10:17:59 +02:00
Lubomír Sedlář 98f40f6138 ostree: Autogenerate a version
If the value is a particular magic string, we will expand it to a proper
value.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-13 10:17:59 +02:00
Lubomír Sedlář dcc1750df2 gather: Clean up method deps
* Remove unused arguments
 * Remove duplication

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-11 15:46:34 +02:00
Lubomír Sedlář 0168388492 gather: Report error if there is no input
Running depsolving with no requested inputs will only lead to a hard to
decipher error. We should instead explicitly tell the user that there is
a problem.

Unit tests are added to add to test this functionality.

Relates: #585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-11 15:46:34 +02:00
Lubomír Sedlář b5efb67ff1 init: Warn when variants mentions non-existing comps group
When variants XML lists a group that does not match any known group in
input comps, report a warning. This is not necessarily a problem in
itself, but having this information in the log can help debug problems.

Relates: #585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-11 15:46:34 +02:00
Qixiang Wan 2f5d6d7dcd unify repo and repo_from options
Config option 'repo' and 'repo_from' are used in several phases, merge
them with one option 'repo'. 'append' in schema is used for appending
the values from deprecated options to 'repo', so it won't break on any
existing config files that have the old options of 'repo_from' and
'source_repo_from' (which is an alias of 'repo_from').

And 'repo' schema is updated to support repo dict as the value or an
item in the values, a repo dict is just a dict contains repo options,
'baseurl' is required in the dict, like:

{"baseurl": "http://example.com/url/to/repo"}

or:

{"baseurl": "Serer"}

currently this is used in ostree phase to support extra repo options
like:

{"baseurl": "Server", "exclude": "systemd-container"}

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-29 10:12:32 +08:00
Qixiang Wan 0f508e2228 move translate_path from paths.py to util.py
So translate_path can be used in util.py, or it will result in
circular import error.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-29 10:12:32 +08:00
Qixiang Wan d1763fca7e checks.py: support 'append' option
If 'append' is defined for a property, append the values from append
options to the property. Note: The property must support to be a list
of values.

For example:

with schema:

    schema = {
        "$schema": "http://json-schema.org/draft-04/schema#",
        "title": "Pungi Configuration",
        "type": "object",
        "definitions": {
            "list_of_strings": {
                "type": "array",
                "items": {"type": "string"},
            },
            "strings": {
                "anyOf": [
                    {"type": "string"},
                    {"$ref": "#/definitions/list_of_strings"},
                ]
            },
        },
        "properties": {
            "release_name": {"type": "string"},
            "repo": {"$ref": "#/definitions/strings", "append": "repo_from"}
        },
        "additionalProperties": False,
    }

and config:

    repo = "http://url/to/repo"
    repo_from = "Server"

config will be updated to:

    repo = ["http://url/to/repo", "Server"]

It supports multiple append options too, like:

    "repo": {
        "$ref": "#/definitions/strings",
        "append": ["repo_from", "source_repo_from"],
    }

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-29 10:12:32 +08:00
Qixiang Wan 2aacefd9cd checks.py: show warning message for alias option
Show warning message for any alias option find in config instance.
Example warning message:

WARNING: Config option 'product_name' is deprecated and now an alias to
'release_name', please use 'release_name' instead. In:
{'release_name': 'dummy product', 'product_name': 'dummy product'}

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-29 10:12:32 +08:00
Lubomír Sedlář 2b21e13aad Skip DNF tests if there are import problems
If there are import errors for DNF, multilib or other related package,
we can assume the tests are running on EPEL. The DNF tests should be
skipped in that case.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-27 09:42:58 +02:00
Lubomír Sedlář 76cf4a7540 Update to use python-multilib
We also rename the old multilib module used by dnf code to multilib_yum
to make it clear what is imported where.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:50 +01:00
Lubomír Sedlář 27b24a489f Handle noarch debuginfo packages
Pull debuginfo packages for noarch packages as well, but only when the
debuginfo is noarch as well.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:50 +01:00
Lubomír Sedlář d23a2f4548 Fix multilib exclude pattern to not match noarch
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:50 +01:00
Lubomír Sedlář 35e72df99f Explicitly remove all temporary files
There is no guarantee __del__ will ever be called, and we were leaving a
ton of stuff in /tmp. With this patch we pass the temporary directories
explictly and make sure they are deleted at the end.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář fe026bb588 Handle Requires(pre|post)
These requires are accessible from a separate attribute, but we want to
handle them the same ways as regular Requires.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář d4f8e32a80 Fix handling globs in input packages
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář cd80d6ef5a Stop caching provides of added packages
This only works in non-greedy mode. When greedy, the same provides can
be linked to multiple packages that should be pulled in.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář e06e9165fc Add checking of flags for dnf depsolving
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář ec67eac1cc Use enum for flags
This adds an extra dependency on python-enum34.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář 9fd8e6319f Handle fulltree_excludes list
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář b53e4a84d3 Fix fulltree when greedy
When greedy, fulltree should pull in all native and multilib packages.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:48 +01:00
Lubomír Sedlář 0a44b2fd07 Filter source packages
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:48 +01:00
Lubomír Sedlář 99699b85aa Use same tests for yum and dnf depsolving
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:48 +01:00
Jan Kaluza 9fcd71f831 Add support for modular composes
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2017-03-22 15:55:52 +01:00
Lubomír Sedlář 4b90822115 Add a script for modifying ISO images
With this script it's possible to add additional files into an ISO file.
If the file happens to be ks.cfg, the boot configs are tweaked so that
the kickstart is actually used.

Resolves: #503
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-21 07:55:19 +01:00
Lubomír Sedlář 306f7e69b0 iso-wrapper: Add utility for mounting images
This patch refactors logic for creating a temporary mount point,
mounting an image, running arbitrary code on it, unmounting the image
and removing the mount point. It immediately uses it in the buildinstall
phase.

Similar mounting is present in product_img phase as well, but due to
different usage pattern it's not changed yet.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-21 07:55:19 +01:00
Lubomír Sedlář 05a666fb3b buildinstall: Move tweaking configs into a function
This will allow us to more easily test the code and also reuse it in
other places.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-21 07:55:19 +01:00
Dennis Gilmore b09641a708 Merge #568 `buildinstall: Print debug info if unmount fails` 2017-03-15 16:07:33 +00:00
Lubomír Sedlář 94b1159829 ostree-installer: Fix logging directory
Currently if there are multiple ostree installers for the same
architecture, the logfiles all end up in the same location and overwrite
each other.

This patch moves the logs to logs/<arch>/<variant>/ostree_installer-X/
for a unique value of X so that there can be multiple runs even for the
same tree.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-15 15:54:45 +01:00
Lubomír Sedlář f0dc15bdc6 buildinstall: Print debug info if unmount fails
When even after retries the unmounting still fails, this patch runs `ls
-lA`, `fuser -vm` and `lsof +D` on the directory to give some idea of
what's blocking it.

Relates: #559
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-14 15:06:20 +01:00
Qixiang Wan d081a4eda8 pkgset: report all unsigned packages
Fixes: #552
Merges: #567
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-14 09:50:28 +01:00
Lubomír Sedlář 5cd1c22e2d buildinstall: Retry unmounting image
If the image can not be unmounted because the device is busy, we should
retry. There will be increasing pauses between the attempts. At most 10
attempts will be done before giving up.

Fixes: #559
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-10 08:29:54 +01:00
Lubomír Sedlář cd805a1e6d iso-wrapper: Handle wrong implant md5
If the checkisomd5 command exits successfully but returns a wrong value,
we should catch and log that. In theory this should be impossible, but
we have seen it in production.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-07 09:58:14 +01:00
Lubomír Sedlář 63a3b6ccaa Merge #538 `consolidate repo option names` 2017-03-06 12:55:44 +00:00
Qixiang Wan 2ae8710934 consolidate repo option names
live_images:            additional_repos        -> repo
ostree:                 source_repo_from        -> repo_from
                        extra_source_repos      -> repo
ostree_installer:       source_repo_from        -> repo_from

With the change, the phases have consolidate option names for variant
repos and external repos.

Old option names will continue to work, old names will be converted
to new names after validation automatically if new options are not
specified in config.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-06 09:26:42 +08:00
Qixiang Wan c93207addb checks: extend validator with 'alias'
When a property has 'alias' defined, and it's not present in instance,
if the alias property is present, add the property with value from alias
property before remove the alias property from instance.

Examples:

with schema:
{
        "$schema": "http://json-schema.org/draft-04/schema#",
        "title": "Pungi Configuration",
        "type": "object",
        "properties": {
            "release_name": {"type": "string", "alias": "product_name"},
        },
        "required": ["release_name"],
        "additionalProperties": False,
}

1. config = {"release_name": "dummy product"}:

   validate pass, config not changed after validation.

2. config = {"product_name": "dummy product"}:

   validate pass, config updated to the following after validation:

   config: {"release_name": "dummy product"}

3. config = {"name": "dummy product"}:

   validate fail, errror message is "Failed validation in : 'release_name' is a required property",
   and warning message is "WARNING: Unrecognized config option: name."

4. config = {"product_name": "dummy product", "release_name": "dummy product"}

   validate fail, error message is "Failed validation in : product_name is an alias of release_name, only one can be used."

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-06 09:26:42 +08:00
Qixiang Wan 40e2874676 osbs: write manifest for scratch osbs
FIXES: #485

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-06 09:26:01 +08:00
Dennis Gilmore 8b1fb287d3 Merge #539 `comps: Filter comps groups for optional variants` 2017-02-23 20:13:39 +00:00
Lubomír Sedlář 4ea1916a87 Enable customizing runroot task weight
For different cases where runroot is used it's now possible to set
custom weight. The usecase for this is to avoid one builder taking too
many tasks. Especially buildinstall is quite resource intensive, so one
builder taking multiple tasks at the same time leads to very slow
compose time.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-23 12:44:27 +01:00
Lubomír Sedlář 8bc65a8be5 comps: Filter comps groups for optional variants
The optional variant can defined by just adding has_optional=True into
variant xml. In such case it has no comps groups and Pungi would copy
the original file unmodified. This leads to extra packages being pulled
into the optional variant.

In this case the correct solution is to filter the comps and remove all
groups.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-23 12:44:20 +01:00
Lubomír Sedlář 857aee05c1 util: Add a utility for managing temporary files
In multiple situations we need to create temporary files or directories
that should not be preserved after compose is finished. Let's add
context managers that ensure these get cleaned up.

This fixes tests leaving garbage around in /tmp.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-20 09:14:25 +01:00
Daniel Mach ad23efd323 Refactor depsolving tests
This adds a new test for Requires(pre) and (post). The general structure
of the test now makes it easy to use the same test scenarios for
different backend.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-13 12:32:07 +01:00
Lubomír Sedlář 9533fca96c koji_wrapper: Always use --profile option with koji
This patch adds this option to all invoked koji commands.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-30 13:00:25 +01:00
Dennis Gilmore 42cdf047cc add missing koji_profile from test compose setting
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2017-01-30 05:06:46 -06:00
Dennis Gilmore c650f04d0b Revert "live-media: Call correct koji alias"
This was accidently merged and is not correct
This reverts commit 02beb35e67.

Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2017-01-30 04:44:17 -06:00
Lubomír Sedlář dc7d3b36ab repoclosure: Don't run build deps check
Fixes: #521
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-30 10:16:03 +01:00
Dennis Gilmore 7feadb14ba Merge #518 `Migrate test phase to dnf repoclosure` 2017-01-28 16:49:10 +00:00
Lubomír Sedlář 02beb35e67 live-media: Call correct koji alias
We need call whatever command is configured in koji_profile. It's not
always `koji`.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-26 15:32:21 +01:00
Lubomír Sedlář e3fe67be53 repoclosure: add option to use dnf backend
This adds a new option repoclosure_backend that changes what tool is
used for repoclosure.

Checking build dependencies is currently not supported, as `dnf` does
not have the corresponding option.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-26 13:39:12 +01:00
Lubomír Sedlář 95fc0fa4ab repoclosure: Add test for repoclosure in test phase
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-26 13:27:48 +01:00
Lubomír Sedlář 7a8fa87172 repoclosure: Remove useless wrapper class
We already get namespacing from being in a separate module.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-26 10:55:23 +01:00
Lubomír Sedlář 9af0cca9eb repoclosure: Add a test for the wrapper
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-26 10:55:23 +01:00
Lubomír Sedlář 8418b68fb0 image-build: Pass arches around as a list
Instead of joining the arches as a comma separated string and splitting
it again later. Ultimately we do need the original format to pass to
koji wrapper, but we can produce that value later.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-24 08:50:21 +01:00
Lubomír Sedlář 6c708549c8 image-build: Expand arches for can_fail
We need to work with a list of strings, not a comma-delimited single
string.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-24 08:50:21 +01:00
Qixiang Wan 40df2034a8 image_checksum: add file sizes to checksum files
Write file sizes of images in checksum files with comment lines,
checksum files are in BSD-style which supports comments by starting
a line with '#'.

Example:

$ cat RHEL-7.4-20170123.n.4/compose/Server/x86_64/iso/RHEL-Server-7.4-x86_64-20170123.n.4-CHECKSUM
 # RHEL-7.4-20170123.n.4-Server-x86_64-dvd1.iso: 3725590528 bytes
 # RHEL-7.4-20170123.n.4-Server-x86_64-boot.iso: 377487360 bytes
 SHA256 (RHEL-7.4-20170123.n.4-Server-x86_64-dvd1.iso) = fa3de37fe4b859a0285f16ea1123f44f15aec169aea84bf010aa3821bd58fc41
 SHA256 (RHEL-7.4-20170123.n.4-Server-x86_64-boot.iso) = 74bf68c54665328adb08b09daf773c67e633b5907e3e2797338ab3c1b58fdf48

(No space at the start of line, because git commit message drops lines
start with '#', added one space to avoid that.)

When there are multiple checksum types specified and checksums are
written to individual files, file size of the image will also be
written to every checksum files.

Fixes: #493

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-01-23 18:20:21 +08:00
Qixiang Wan a9b275f13b osbs: optionally check GPG signatures
If gpgkey option is defined in config, set gpgcheck=1 and set
gpgkey=<value> in variant repo files.

Fixes: #487
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-01-17 13:08:58 +08:00
Qixiang Wan 5d241d316a ostree-installer: Allow multiple repos in ostree installer
Add new key 'repo' to allow specifying multiple repos as the source
repositories. And change 'source_repo_from' to allow specifying multiple
vairant names to use variant repos as source repositories.

Doc of 'source_repo_from' is updated to not mention URL is supported,
though we still support that in code. User should add url of repos in
'repo' key instead of 'source_repo_from'.

Fixes: #508

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-01-13 16:57:30 +08:00
Lubomír Sedlář afaa0e025e unified-iso: Create isos with debuginfo packages
When creating unified ISOs, the script will now also create one iso per
architecture containing a repo with debuginfo packages.

There is no switch to turn this off. The images can simply not be
shipped if not wanted.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-11 14:36:25 +01:00
Qixiang Wan 6fbf1e8f59 Create temporary dirs under compose's workdir
Add compose.mkdtemp which creates and returns a unique temporary
directory under <compose_topdir>/work/{global,<arch>}/tmp[-<variant>]/

Change tempfile.mkdtemp in code to compose.mkdtemp, so temporary
dirs are created under the compose's workdir, this makes it more
easier to maintain the temporary dirs/files especially when compose
fails with amount of temporary files left there.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-01-11 15:25:47 +08:00
Dennis Gilmore baca306edf Merge #495 `osbs: Enable specifying extra repos` 2017-01-04 14:44:13 +00:00
Dennis Gilmore 46f77403df Merge #475 `Allow failure for some arches` 2017-01-04 14:43:22 +00:00
Lubomír Sedlář e260fe5581 Add script to generate unified ISOs
This a standalone script that will look into a compose and create
unified ISO for each architecture. The ISO contains RPM repositories for
all variants that have the arch.

Known issues:
 * The filename does not respect settings. This is tricky because the
   name could include variant name, which we don't have here (by design
   of unified ISO).
 * The same is true for volume id.

In order to test the feature without running actual compose, we need to
add essentially a big chunk of compose. Most of the files are empty, as
their content is never accessed.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-04 10:22:20 +01:00
Lubomír Sedlář e7c8b2affd osbs: Validate config in tests
This makes sure the test configurations will be accepted in real usage.
It also enables us to remove some manual error checking that will be
performed by validator.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-04 09:47:26 +01:00
Lubomír Sedlář 61a4c43db0 osbs: Verify the .repo files contain correct URL
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-04 09:47:26 +01:00
Lubomír Sedlář 814bf4484b osbs: Enable specifying extra repos
The same way live_media and image_build accept additional external repos
or variants list, there is now a `repo` and `repo_from` configuration
key to add these.

Fixes: #486
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-04 09:47:19 +01:00
Patrick Uiterwijk c64f75d1b4 Add test for krb_login with principal and keytab
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2016-12-14 12:58:23 +00:00
Patrick Uiterwijk cb4b5cc54d Make sure that the profile name is parsed correctly
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2016-12-14 12:58:23 +00:00
Lubomír Sedlář 356b78d440 image-build: Allow failure only on some arches
This uses the --can-fail option in koji. Failing an optional image will
not abort whole task. If the whole task fails (or there is a problem on
the compose side), we abort unless all arches are optional.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-14 10:14:30 +01:00
Lubomír Sedlář 7f56b978ce live-media: Allow some arches to fail
This patch uses the `--can-fail` option of koji command line. If only
optional arches fail, the task will report as success. Failures on
compose box side are ignored if and only if all architectures are
optional.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-14 10:14:30 +01:00
Lubomír Sedlář 721932a573 image-build: Use install_tree from parent for nested variants
Buildinstall only runs for top-level variants. Addons, optionals and
integrated layered products must reuse install tree from their parent,
because otherwise there would be no boot.iso.

Fixes: #472
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-14 10:04:01 +01:00
Lubomír Sedlář c338219ef0 config: Report unknown options as warnings
Given the way config files can include other files, it is entirely
possible to use the same config file for both Pungi and Distill. Pungi
will however complain about unknown options.

This patch reverts part of c38bb480 and moves deprecation handling back
into schema validation. The validation method then returns a list of
errors and a list of warnings.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-13 11:58:08 +01:00
Lubomír Sedlář 56147f2e4d pungi: Handle missing SRPM
Source packages can be excluded while binary packages should still go
into the compose. This patch makes it so that the mapping from binary
packages to source packages contains None in such case. The code is
already capable of handling that. A warning will be emitted for each
binary package without source.

This also allows us to remove some code from `createSourceHashes` that
is now unused.

A test for excluding source package is added as well.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-13 10:45:04 +01:00
Qixiang Wan 83428a06bf [ostree-installer] Add 'installer' sub-command to pungi-make-ostree
The new sub-command 'installer' is added to support build OSTree
installer image with pungi-make-ostree. It can take an optional argument
'--extra-config' to read some of configurations from a json file. The
content of the json file can contains the configuration which are
supported in OSTree installer phase, the difference is variant UID is
not supported as a repo url in this case. A valid json file can be like
the following:

{
    "source_repo_from": "http://www.example.com/repo/workstation/os",
    "installpkgs": [
        "fedora-productimg-workstation"
    ],
    "add_template": [
        "/path/to/installer/template/lorax-configure-repo.tmpl"
    ],
    "add_template_var": [
        "ostree_osname=fedora-workstation",
        "ostree_ref=fedora/25/x86_64/workstation"
    ],
    "add_arch_template": [
        "/path/to/installer/template/lorax-embed-repo.tmpl"
    ],
    "add_arch_template_var": [
        "ostree_repo=https://www.example.com/compose/ostree",
        "ostree_osname=fedora-workstation",
        "ostree_ref=fedora/25/x86_64/workstation"
    ]
}

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-12-12 19:30:35 +08:00
Qixiang Wan e043604822 [ostree] Add 'tree' sub-command to pungi-make-ostree script
Update pungi-make-ostree to supourt sub-command 'tree', which is just
as the original feature of pungi-make-ostree to compose OSTree tree.
With the change we can add other sub commands later to build other
OSTree artifacts, like the installer image.

Inaddtional to the change, now the the 'tree' command can accept an
optional '--extra-config' parameter to update the original tree
configuration with extra configurations specified in a json file
before composing the OSTree tree.

Example:

pungi-make-ostree tree --repo=/ostree --treefile=/path/to/treefile \
--log-dir=/path/to/log --extra-config=/path/to/extra-config.json

The extra-config file can contains the same configuration as OSTree
phase, the difference is it doesn't understand variant UID as source
repo since it's not ran in the chain of phases. A valid configuration
can be like:

{
    "source_repo_from": "http://example.com/repo/x86_64/Server",
    "extra_source_repos": [
        {
            "name": "optional",
            "baseurl": "http://example.com/repo/x86_64/optional",
            "exclude": "systemd-container",
            "gpgcheck": False
        },
        {
            "name": "extra",
            "baseurl": "http://example.com/repo/x86_64/extra",
        }
    ],
    "keep_original_sources": True
}

The OSTree phase is updated to move out the task of updating treefile,
instead of that, it writes the extra configurations to a json file,
then 'pungi-make-ostree tree' will take it by option '--extra-config'.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-12-12 19:30:35 +08:00
Lubomír Sedlář d3cad4795c metadata: Allow creating internal releases
The internal flag in productmd is meant to indicate that a compose is
not meant for publishing. This is potentially useful to allow filtering
in PDC or similar service.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-06 12:49:33 +01:00
Qixiang Wan 75934f20e5 Fix PhaseLoggerMixin in case of compose has _logger = None
80fa723 breaks pungi-config-validate that using ValidationCompose
with _logger set to None.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-11-30 14:41:02 +08:00
Lubomír Sedlář dd814a5f4d ostree-installer: Use dvd-ostree as type in metadata
This depends on python-productmd >= 1.3 that understands this format.

Fixes: #417
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-28 10:32:50 +01:00
Lubomír Sedlář 2bd18b1b03 image-build: Reduce duplication
All tests should print complete diffs on failure, so there is no need to
define this in each test separately.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-28 09:28:46 +01:00
Lubomír Sedlář 5148adf233 createrepo: Add tests for adding product certificates
If the certificate is enabled and exists, it will currently be added to
all rpm repositories.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-23 11:17:04 +01:00
Lubomír Sedlář 96fc6fb11e createrepo: Add tests for retrieving product certificates
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-23 11:17:04 +01:00
Qixiang Wan 80fa723b1d Include phase name in log for some phases
Phases createiso, liveimages, image_build, ostree_installer and osbs are
done in parallel, logs from these phases are mixed and and it's not
obvious which log message belongs to which phase. This change adds phase
name in log message for these phases.

The new mixin 'PhaseLoggerMixin' is added to extend a Pungi phase with a
logging logger which copy handlers from compose's logger but with
formatter changed.

Fixes: #58
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-11-23 16:47:22 +08:00
Dennis Gilmore 9e52c68c82 Merge #465 `media-split: Print sensible message for unlimited size` 2016-11-17 23:14:44 +00:00
Dennis Gilmore 6c89f86c53 Merge #466 `pungi: Include noarch debuginfo` 2016-11-17 23:14:03 +00:00
Colin Walters 7bcbe30fd2 Expose lorax's --rootfs-size argument
This is going to be necessary for reworking the Atomic Host ISO,
see: https://pagure.io/fedora-lorax-templates/pull-request/6

Signed-off-by: Colin Walters <walters@verbum.org>
2016-11-17 10:17:39 -05:00
Lubomír Sedlář deb8623bd1 pungi: Include noarch debuginfo
The check for arch compatibility should not be performed if the
debuginfo package is noarch. Such packages should be included
unconditionally.

Fixes: #450
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-16 14:50:03 +01:00
Lubomír Sedlář 6203541ac6 media-split: Print sensible message for unlimited size
If the media is bootable, we can not split it. The limit is not used in
that case and we may overflow it with an warning message. The warning
should correctly mention what is going on instead of printing a
non-sensical message about free space on media being some huge number.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-16 14:49:46 +01:00
Lubomír Sedlář b7813d34ac pungi: Fix incorrectly skipped tests
The test was not correctly being skipped in Koji build, causing build
failure.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-16 09:43:19 +01:00
Lubomír Sedlář 3c1e2fba66 pungi: Fix tests on non-x86_64 arches
If the test did not specify an arch explicitly, it fell back on whatever
arch the current machine has. This was causing failures when building
RPM.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-16 09:20:10 +01:00
Dennis Gilmore 69d949aaf9 Merge #463 `[ostree] Allow extra repos to get packages for composing OSTree repository` 2016-11-15 21:06:05 +00:00
Dennis Gilmore 3953914bd1 Merge #461 `extra-files: Nice error message on missing RPM` 2016-11-15 21:01:42 +00:00
Dennis Gilmore d4a78a1553 Merge #459 `Fix building images and live media for addons` 2016-11-15 21:00:50 +00:00
Dennis Gilmore 5dc8132fa4 Merge #456 `Add tests for dependency solving` 2016-11-15 20:58:40 +00:00
Qixiang Wan 68e121e421 [ostree] Allow extra repos to get packages for composing OSTree repository
Sometimes addtional repos are required to get necessary packages for
composing OSTree repository. For example, RHEL doesn't have an 'Everyting'
variant, so composing OSTree repository from any of the RHEL variants
won't work, addtional source repos need to be enabled to achieve that.
The new option "extra_source_repos" enable the ability of allowing extra
source repos.

And a new option 'keep_original_sources' is introduced to keep the
original repos found in tree config file, if this is enabled, Pungi
will not remove the existing source repos from the tree config file,
just add new repos of "source_repo_from" + "extra_source_repos" to
the existing repos.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-11-15 18:56:06 +08:00
Lubomír Sedlář 0cdef2d8e3 pungi: Run in-process for testing
Instead of spawning a separate process for each test, move the code to
run in the main process. This gives us correct coverage information and
makes the tests a lot faster.

The input is still provided in a kickstart file. The output also goes
into a log file that is later parsed.

There is some extra work needed with logging setup to make the logs go
to standard output so that nose can correctly capture them.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-10 13:28:02 +01:00
Lubomír Sedlář 88dbf8f849 extra-files: Nice error message on missing RPM
When a file is exported from an RPM in the compose, and there is no
matching RPM in the package set, we want a nice error message.

Fixes: #460
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-10 09:38:50 +01:00
Lubomír Sedlář 13871b64fb compose: Drop unused argument
The `get_variants()` method had a `recursive` argument with default
value of `False. However, this argument had no effect and the method
always returned all variants recursively.

We can just drop the argument. All callers are updated to not supply the
argument. Should any need for getting the top-level variants only arise,
they can be accessed as the `variants` attribute directly on the Compose
object.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 13:39:01 +01:00
Lubomír Sedlář bd00920c62 compose: Search all nested variants
The code to search for install tree and repo for image-build and
live-media was only looking at top-level variants. Addons, optional or
integrated layered products could not have been found. This would lead
to error messages such as "There is no variant Server-optional to get
repo from when building live image for Client" even though the variant
exists.

Various tests are updated to exercise this edge case.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 13:18:34 +01:00
Lubomír Sedlář 99ba369901 ostree-installer: Capture all lorax logs
Fixes: #457
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 09:37:00 +01:00
Lubomír Sedlář 9a72ea8f6d lorax-wrapper: Put all log files into compose logs
Tell lorax to use a specific directory for log files so that we preserve
them despite koji not having any idea about them.

Fixes: #457
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 09:37:00 +01:00
Lubomír Sedlář adaed6345c pungi: Fulltree should not apply for input multilib package
When the input explicitly lists a package as multilib, we should not
automatically add native version just because of fulltree.

The tests for this use case are now enabled and passing.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 09:36:23 +01:00
Lubomír Sedlář 8e5b197e3f pungi: Add tests for depsolving
These are tests written by Daniel Mach originally for Distill-NG. They
are ported to current Pungi. The test repositories are committed in the
tests/fixtures/ directory. This is the same data that is used for test
compose, but the actual RPM files are not present. Some tests are
adapted from dmach's fork of Pungi.

Some of the packages are marked with a comment saying they are
important. These are the packages that the test is specifically trying
to get included in the package set. There are also explicit tests for
packages that should not be included.

Two tests are skipped for now as there is a bug preventing them from
passing. This is related to fulltree being done for packages that are
explicitly multilib.

The depsolver is called by invoking a separate executable, so the
coverage data is wrong.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 09:36:23 +01:00
Qixiang Wan fa3c5aff63 [ostree] Allow adding versioning metadata
Added new option '--version' to pungi-make-ostree, and this can be
enabled in ostree settings with 'version'. The version string will be
added as versioning metadata if this is specified.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-11-04 16:07:26 +08:00
Qixiang Wan 06ba3b8551 [ostree] New option to enable generating ostree summary file
Added new option '--update-summary' to pungi-make-ostree, and this can
be enabled in ostree settings with 'update_summary'. A summary file will
be generated (or re-generated if it was presented in an existing ostree
repo) when it is enabled.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-11-03 21:02:05 +08:00
Patrick Uiterwijk a49510a9fb Update tests to use ostree write-commit-id
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2016-10-26 09:52:12 +00:00
Patrick Uiterwijk 754458823c ostree: Use the write-commitid-to feature rather than parsing ostree logs
This depends on an rpm-ostree release with
98332a3be4

Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2016-10-26 09:52:12 +00:00
Lubomír Sedlář c5bd99da9e Merge #448 `Remove executable permissions on test scripts` 2016-10-26 07:31:37 +00:00
Lubomír Sedlář bfd5a39ce6 checks: Check for createrepo_c
The createrepo package is needed always, but depending on configuration
we should also look for createrepo_c.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-25 16:00:37 +02:00
Lubomír Sedlář f4cd25450b checks: Update tests to not require python modules
This way the tests are more independent of the system they are running
on.

Merges: #447
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-25 15:57:02 +02:00
Patrick Uiterwijk 59dbd9d35f Remove executable permissions on test scripts
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2016-10-25 10:42:16 +00:00
Lubomír Sedlář c38bb4809b config: Don't abort on deprecated options
These options are in fact removed and have no effect anymore. This patch
changes the validation to print a warning that the option was removed
and what should be done instead. It no longer stops the whole compose.

The validation script still rejects configuration files with these
removed keys.

This change means we no longer check these removals with the JSON schema
(as that makes it hard to determine where exactly the problem is).

Fixes: #438
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-20 12:51:12 +02:00
Dennis Gilmore a7dc4e80a7 Merge #433 `createiso: Do not split bootable media` 2016-10-19 16:37:25 +00:00
Lubomír Sedlář 84fcc0083f createiso: Do not split bootable media
If the media is bootable, we can not split it. Instead we will create an
ISO that is too big and issue a warning (aborting the whole compose
would be too much).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-19 18:36:19 +02:00
Dennis Gilmore ff6c952094 Merge #430 `createiso: Include layered product name in iso name` 2016-10-19 16:36:07 +00:00
Dennis Gilmore 34d3be35dc Merge #426 `iso-wrapper: Remove the class` 2016-10-19 16:34:25 +00:00
Lubomír Sedlář b46af7acbb createiso: Include layered product name in iso name
When creating an ISO for a layered product, the name of the integrated
product should be included in the file name.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-17 15:15:50 +02:00
Lubomír Sedlář e58c78fd2f iso-wrapper: Remove the class
It really is just a group of independent functions, so we can simplify
it by removing the unused wrapper class. Instead of importing the
wrapper, instantiating it and calling its methods we can import the
module and call its functions directly.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-14 09:14:51 +02:00
Lubomír Sedlář 50e9294057 config: Validate variant regular expressions
When the mapping in configuration specifies incorrect regular expression
to match regular expressions, we should raise an error immediately and
not wait until the part of config is actually used.

This patch does not cover `live_media`, `image_build` and `osbs`
sections, as they use plain dicts and not the list of tuples format.

Fixes: #424
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-12 09:38:59 +02:00
Lubomír Sedlář 730764e2e7 ostree-installer: Reduce duplication in tests
The assertions were duplicated across multiple test cases. This patch
moves them into shared methods so that they are defined only once.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-06 09:42:31 +02:00
Lubomír Sedlář 32059c57c0 ostree-installer: Generate correct volume ID
The volume ID should be generated in Pungi with the same format as for
other images.

Fixes: #419
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-06 09:17:09 +02:00
Lubomír Sedlář cc559b12f4 ostree-installer: Use ostree as type in filename
Instead of overloading the `dvd` value, use new value `ostree` that can
be overridden by a config change.

Fixes: https://pagure.io/pungi/issue/418
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-06 08:41:14 +02:00
Lubomír Sedlář 8880b1ea8a ostree: Use $basearch in repo file
Fixes: #413
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-05 18:41:55 +02:00
Dennis Gilmore 4e90f81fc1 use --new-chroot when making ostree's
to complement https://pagure.io/koji/pull-request/162 we need to adjust pungi
rpm-ostree uses bublewrap that does not work in mock. --new-chroot to mock
enables the use of systemd-nspawn instead of chroot resulting in working
rpm-ostree again

Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2016-09-28 20:58:17 -05:00
Lubomír Sedlář 7bf12636a0 Translate paths without double slash
If the path in `translate_paths` config ends with a slash, we would
create public path with double slash.

Fixes: https://pagure.io/pungi/issue/408
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-26 15:50:49 +02:00
Lubomír Sedlář a042906717 init: Remove keep_original_comps option
The same information can be inferred from definitions in variants.xml:
if the variant has no groups defined, we include packages from all
groups. By the same logic we can also include all groups in the comps
file.

The config validation is updated to give a hint on how to remove the
option from the configuration.

Relates: #29
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-22 13:35:28 +02:00