Commit Graph

175 Commits

Author SHA1 Message Date
Lubomír Sedlář
63a8b7b6c9 Remove package whitelist
This was a workaround to make some packages from the global repo
invisible for depsolving. This is now handled by packages being in
different repos. We can select which repos are enabled at which point.
This achieves the same result, but much faster.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
2d0ffb56ca ostree-installer: Remove usage of arch_repo
Pass all pkgset repos as input to the task.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
26ddd46acb ostree: Remove arch_repo path usage
Pass all pkgset repos as input to the task.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
ab9122be2a buildinstall: Remove arch_repo usage
Simply use all existing package set repos as input for the runroot
task. The command line gets a bit long, but the actual behaviour should
remain the same.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
3824eab15b createrepo: Remove arch_repo usage
There is no longer a single repo with all packages. This means that the
metadata has to be loaded from another location.

When taking packages from Koji, we can assume that the non-modular
package tag will be processed last. The repo for this tag will be used.
This has better chance of being useful than using a random module.

For repo sources, there is only one package set anyway, so this change
makes no difference.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
6efaae19fd Remove debug mode
This was already discouraged to not be used, and is a bad idea in
current setup anyway. Removing this can simplify the code.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-24 13:36:23 +02:00
Lubomír Sedlář
32a6415e58 config: Keep known options defined on CLI
If the validation or dumping script is given some options, they should
only be removed if they are not valid. We have to remove the invalid
ones, otherwise that would cause a warning about unknown options.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-25 15:13:16 +02:00
Lubomír Sedlář
d063217d6f config-dump: Report better error for non-existing sources
JIRA: COMPOSE-3624
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-25 14:42:50 +02:00
Lubomír Sedlář
acd3c19618 config-validate: Allow defining variables
When trying to validate a template that should later be filled in with
`pungi-config-dump`, there will be errors about undefined variables.
These are meant to be set when the template is populated.

This patch adds support for `-e`, `--define` argument to the validation
script that can be used to suppress these errors.

Alternatively a JSON file is read from the directory with config file
that can contain values for the variables.

The `--define` option is changed in both validation and dumping to allow
empty string as an accepted value.

JIRA: COMPOSE-3599
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-21 12:47:50 +02:00
Lubomír Sedlář
32624c59b1 config: Report validation warning if variants fail to load
In a real compose this would be a blocker issue and the compose would be
aborted, but for validation it may make sense to continue. Instead of
crashing, let's report a clear warning.

JIRA: COMPOSE-3606
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-21 10:02:25 +02:00
Haibo Lin
ce066707c1 config: Deprecate release_is_layered option
Fixes: https://pagure.io/pungi/issue/977
Merges: https://pagure.io/pungi/pull-request/1204
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-06-06 13:45:49 +02:00
Lubomír Sedlář
217fcd6c02 config-dump: Allow dumping config for multi compose
Clone the main config, all referenced parts and included variants or
comps files.

This requires a new argument: destination directory to which to write
the copied files.

JIRA: COMPOSE-3289
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-27 08:42:42 +02:00
Lubomír Sedlář
2dd30008ae buildinstall: Copy files in thread
Instead of running the copy from the main script explicitly, make it
part of the thread.

This should make things very slightly faster, and the code is much
simpler.

Fixes: https://pagure.io/pungi/issue/959
JIRA: COMPOSE-2604
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-13 11:04:37 +02:00
Lubomír Sedlář
dc69281025 Fall back to C locale if UTF8 version does not exist
This can happen on RHEL 6 or 7. All current Fedora version have it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-13 10:25:00 +02:00
Lubomír Sedlář
c1a03c259b Switch locale to C.UTF-8
Relates: https://pagure.io/dusty/failed-composes/issue/1642
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-09 08:01:13 +02:00
Jiri Konecny
0cd089802f
patch-iso supports multiple graft directories
Pungi patch iso now supports multiple graft directories. This should
make usage more comfortable.

Signed-off-by: Jiri Konecny <jkonecny@redhat.com>
2019-04-12 15:27:54 +02:00
Lubomír Sedlář
b165866f39 config-dump: Fix crash when used without --define
The default value is None, which is not iterable.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-03 13:09:12 +02:00
Lubomír Sedlář
fd0117f38c config-dump: Allow defining variables on CLI
This can be used to fill particular values to a predefined template
without editing any file disk.

JIRA: COMPOSE-3316
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-02 09:54:25 +02:00
Lubomír Sedlář
c6a86c444a gather: Apply repo path substitutions for DNF backend
Use a method to add repos that will apply $arch and $basearch
substitution automatically. Yum backend only applies $basearch, so if
compatibility is needed, that one should be used.

Drop code for handling mirrorlist, since Pungi does not ever use it, and
being used externally is not really supported.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-01 09:49:53 +02:00
Lubomír Sedlář
31ef7736aa orchestrator: Monitor status of parts
When a phase is started or stopped, add a line to the to output. This
should help users keep track of what is happening in case the part takes
a long time to run.

JIRA: COMPOSE-3287
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-25 14:20:04 +01:00
Lubomír Sedlář
45cdbb2faf orchestrator: Send messages about the main compose
Only start/finish messages will be sent if a handler is configured.

JIRA: COMPOSE-3288
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-21 09:29:54 +01:00
Lubomír Sedlář
a206a73db1 config: Allow validating configuration in JSON
JIRA: COMPOSE-3318
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-04 14:36:28 +01:00
Lubomír Sedlář
8b84aa384a config-dump: Allow freezing koji event
Accept similar argument to pungi-koji: either direct event ID, or a path
to a compose from which the event will be extracted.

It would be nice if we could reuse the path to compose given as source,
but there may be more that one compose, so ultimately we need a way to
overwrite it anyway.

JIRA: COMPOSE-3278
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-26 10:52:45 +01:00
Lubomír Sedlář
7693e562b1 osbs: Process data about pushing images to registries
This patch does not do any actual pushing. It will only extract data
about push targets from the main configuration and store it together
with exact Koji NVR in a well-defined location, and also send the data
to message bus for another service to handle.

JIRA: COMPOSE-3228
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-12 15:12:50 +01:00
Frédéric Pierret (fepitre)
86314fdc83 pungi-legacy: expose lorax's --rootfs-size argument
Merges: https://pagure.io/pungi/pull-request/1112
Fixes: https://pagure.io/pungi/issue/1107
Signed-off-by: Frédéric Pierret (fepitre) <frederic.pierret@qubes-os.org>
2019-01-14 10:41:37 +01:00
Lubomír Sedlář
90c60f8e64 Add script to orchestrate multiple composes
It may make sense to break a big compose into smaller chunks that can be
done independently. This script allows describing the smaller parts,
runs them with correct dependencies and arranges the result to look like
a single big compose.

All parts use the same koji event, that is either obtained from Koji, or
from command line argument.

JIRA: COMPOSE-2654
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-01-08 09:40:36 +01:00
Lubomír Sedlář
df35f26910 Add script to merge and dump multiple configuration files
The script can either take config from an existing compose, or load
files by path. By default it will perform full validation (and add
default values and resolved git references). This can be turned off.

The final JSON is printed to stdout.

JIRA: COMPOSE-3066
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-26 07:59:31 +01:00
Lubomír Sedlář
ca7d6256e5 Move resolving git reference to config validation
Instead of multiple places handling the same thing duplicating the
logic, it's better to do it once upfront. This allows easy caching of
the results.

Additional advantage of this approach is that the config dump will
include resolved URLs. The original reference will still be available in
the copy of the original config.

JIRA: COMPOSE-3065
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-21 13:25:35 +01:00
Mohan Boddu
4864a0f58e Copy config files into logs/global/config-copy/ directory
Fixes: https://pagure.io/pungi/issue/994
Merges: https://pagure.io/pungi/pull-request/1013
Signed-off-by: Mohan Boddu <mboddu@bhujji.com>
2018-11-19 14:42:38 +01:00
Lubomír Sedlář
7e923d3823 Remove timestamp from config dump
This makes it hard to programatically access the dump.

The timestamp was included originally to make sure that restarting a
compose in debug mode will preserve all configs. However we don't really
support that usecase anyway.

JIRA: COMPOSE-3063
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-19 13:53:34 +01:00
Lubomír Sedlář
605c9ca435 Allow reading configuration from JSON
JIRA: COMPOSE-3064
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-19 12:02:19 +01:00
Lubomír Sedlář
85bf5535bc init: Keep parent groups in addon comps environments
The environment in comps for a variant can refer to groups in parent
variant (either for addons, or because of other configuration). We
should not remove the groups in this case.

This requires changes in two places:
 * teaching `comps_filter` about groups that should not be removed
 * fixing writing comps so that it does not actually change the data as
   well

JIRA: COMPOSE-2612
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-02 12:43:56 +01:00
Lubomír Sedlář
93bc843682 ostree: Fix handler crash without commit ID
If there is no new commit in the repo, we should not wait for a
signature, as there is nothing to sign.

Fixes: https://pagure.io/pungi/issue/1046
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-19 13:07:03 +02:00
Lubomír Sedlář
833ba64c51 ostree: Wait for updated ref as well as signature
Fixes: https://pagure.io/pungi/issue/1036
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-03 09:54:12 +02:00
Lubomír Sedlář
04f68a018f config: Report variants validity issues
If the variants file is invalid, we want to report the errors. If we
fail to load the file at all, that is fine.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-12 12:50:12 +02:00
Lubomír Sedlář
004ef31917 gather: Ignore comps in lookaside repo
This fixes a case where extra packages are pulled in.

The scenario is:

 * there's a lookaside repo which contains group G which has package P
 * we want to pull group G into the compose, but our definition of G
   does not contain P
 * lookaside repo does not contain package P
 * current package set has P

DNF depsolver will then merge the two definitions and try to get all the
packages. For most cases this is not a problem, since the package is in
the lookaside repo and will not be pulled into the compose. But in the
example above since P is not in lookaside, Pungi will put it into
current compose.

This is also ugly in the depsolving log says it includes package P
because it was in input. But checking comps file does not show it.

The result of this change is that some packages might disappear from
current composes. This can only happen if there is a lookaside which
defines groups with similar IDs, which should be very rare. In cases
where this would be a problem the fix is to explicitly add wanted
packages either to comps or to additional packages.

Fixes: https://pagure.io/pungi/issue/978
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-08 13:32:24 +02:00
Lubomír Sedlář
cbcebe90e1 Allow extracting koji event from another compose
When multiple composes are chained, they should reuse the same event.
However it is tricky as the value would have to be passed by hand. This
patch makes it possible to read the value from another compose (the
first one in the chain).

JIRA: COMPOSE-2571
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-31 11:11:21 +02:00
Lubomír Sedlář
49e8aa0c7e Silence config warnings in quiet mode
It's a bit unfortunate we can't use the logger for printing these, but
there is cyclic dependency: to set up logger, we need Compose object (to
generate path to log file), but to create Compose object we need a valid
config.

JIRA: COMPOSE-2577
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-30 15:03:07 +02:00
Lubomír Sedlář
b2f995d516 Add a phase for creating extra ISOs
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-16 12:20:01 +02:00
Ondrej Nosek
b2190c1c3f Validation of parameter skip_phases
It checks config file value 'skip_phases' for valid phases names.
Also checks command-line attribute 'skip-phase' of 'bin/pungi-koji'.

JIRA: COMPOSE-2493

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-05-15 13:30:00 +02:00
Patrick Uiterwijk
ab2faa85b3 Capture sigterm and mark the compose as DOOMED
Merges: https://pagure.io/pungi/pull-request/946
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2018-05-15 12:36:20 +02:00
Patrick Uiterwijk
8f3c06bd14 Make wait-for-signed-ostree repeat the fedmsg in case the signer crashed
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2018-05-09 23:17:59 +02:00
Lubomír Sedlář
543154d597 ostree-installer: Work with skipped buildinstall
Fixes: https://pagure.io/pungi/issue/909
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-07 12:29:20 +02:00
Lubomír Sedlář
99b6e44a30 createiso: Skip if buildinstall fails
If the ISO is meant to be bootable but lorax fails, there's no point in
creating the ISO as it will not behave as expected.

Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1574585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-04 08:45:45 +02:00
Lubomír Sedlář
f298121bbc Update compose status when config validation fails
If the config is syntactically correct, but it contains some issues
discovered in `validate` method of any phase, we need to raise an
exception, not exit immediately.

JIRA: COMPOSE-2431
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-27 07:41:45 +02:00
Lubomír Sedlář
8c22236ad4 gather: Print profiling information to stderr
On stdout it gets mixed with listing of gathered packages. This has no
effect in a real pungi-koji run, where both streams are merged into a
single file which is then parsed, but in manual debugging runs it's a
little obnoxious.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-25 14:21:09 +02:00
Lubomír Sedlář
1e972885f5 validation: Populate dict of all variants
Until now only list of top level variants was enough, but for variant
as lookaside we need a dict including even child variants.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-20 10:03:01 +02:00
Lubomír Sedlář
a217eea24e Allow filtering comps for different variants
We already filter for arches, so this just extends the filter to work
with variants as well.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-13 14:56:06 +02:00
Lubomír Sedlář
de231064b7 comps: Add tests for CompsFilter
All use cases that are actually used by pungi-koji are tested. There is
missing coverage for

 * keeping only items with matching arch
 * not reindenting the file

These aren't currently used and should be removed in the future, but
there may be other tools depending on the comps_filter executable.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-09 07:48:28 +02:00
Lubomír Sedlář
7ea4c33d87 comps: Move filtering into wrapper module
The code should not live directly in the executable, that makes it very
hard to test.

Other than the move there is no functional change.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-09 07:48:28 +02:00