It may make sense to break a big compose into smaller chunks that can be
done independently. This script allows describing the smaller parts,
runs them with correct dependencies and arranges the result to look like
a single big compose.
All parts use the same koji event, that is either obtained from Koji, or
from command line argument.
JIRA: COMPOSE-2654
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The script can either take config from an existing compose, or load
files by path. By default it will perform full validation (and add
default values and resolved git references). This can be turned off.
The final JSON is printed to stdout.
JIRA: COMPOSE-3066
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of multiple places handling the same thing duplicating the
logic, it's better to do it once upfront. This allows easy caching of
the results.
Additional advantage of this approach is that the config dump will
include resolved URLs. The original reference will still be available in
the copy of the original config.
JIRA: COMPOSE-3065
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This makes it hard to programatically access the dump.
The timestamp was included originally to make sure that restarting a
compose in debug mode will preserve all configs. However we don't really
support that usecase anyway.
JIRA: COMPOSE-3063
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The environment in comps for a variant can refer to groups in parent
variant (either for addons, or because of other configuration). We
should not remove the groups in this case.
This requires changes in two places:
* teaching `comps_filter` about groups that should not be removed
* fixing writing comps so that it does not actually change the data as
well
JIRA: COMPOSE-2612
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If there is no new commit in the repo, we should not wait for a
signature, as there is nothing to sign.
Fixes: https://pagure.io/pungi/issue/1046
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the variants file is invalid, we want to report the errors. If we
fail to load the file at all, that is fine.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This fixes a case where extra packages are pulled in.
The scenario is:
* there's a lookaside repo which contains group G which has package P
* we want to pull group G into the compose, but our definition of G
does not contain P
* lookaside repo does not contain package P
* current package set has P
DNF depsolver will then merge the two definitions and try to get all the
packages. For most cases this is not a problem, since the package is in
the lookaside repo and will not be pulled into the compose. But in the
example above since P is not in lookaside, Pungi will put it into
current compose.
This is also ugly in the depsolving log says it includes package P
because it was in input. But checking comps file does not show it.
The result of this change is that some packages might disappear from
current composes. This can only happen if there is a lookaside which
defines groups with similar IDs, which should be very rare. In cases
where this would be a problem the fix is to explicitly add wanted
packages either to comps or to additional packages.
Fixes: https://pagure.io/pungi/issue/978
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When multiple composes are chained, they should reuse the same event.
However it is tricky as the value would have to be passed by hand. This
patch makes it possible to read the value from another compose (the
first one in the chain).
JIRA: COMPOSE-2571
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It's a bit unfortunate we can't use the logger for printing these, but
there is cyclic dependency: to set up logger, we need Compose object (to
generate path to log file), but to create Compose object we need a valid
config.
JIRA: COMPOSE-2577
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It checks config file value 'skip_phases' for valid phases names.
Also checks command-line attribute 'skip-phase' of 'bin/pungi-koji'.
JIRA: COMPOSE-2493
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
If the ISO is meant to be bootable but lorax fails, there's no point in
creating the ISO as it will not behave as expected.
Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1574585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the config is syntactically correct, but it contains some issues
discovered in `validate` method of any phase, we need to raise an
exception, not exit immediately.
JIRA: COMPOSE-2431
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
On stdout it gets mixed with listing of gathered packages. This has no
effect in a real pungi-koji run, where both streams are merged into a
single file which is then parsed, but in manual debugging runs it's a
little obnoxious.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Until now only list of top level variants was enough, but for variant
as lookaside we need a dict including even child variants.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
All use cases that are actually used by pungi-koji are tested. There is
missing coverage for
* keeping only items with matching arch
* not reindenting the file
These aren't currently used and should be removed in the future, but
there may be other tools depending on the comps_filter executable.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The code should not live directly in the executable, that makes it very
hard to test.
Other than the move there is no functional change.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Basically everything not on the list is excluded. This has to be applied
before we filter only the latest versions (otherwise we could lose
packages that are on the whitelist).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When buildinstall fails, there will be no lorax generated files copied
into the compose directory. However they may still be mentioned in the
.treefile in work/ subdirectory where lorax runs.
To avoid possible issues, we should use the lorax created .treeinfo only
if the run was successful.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead use the definition from python-six. Once we drop Py 2 support
completely, we'll just swap underscores with dots.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should make all tests pass on both Python 2 and Python 3.
Unittest2 is required on Py 2.6 and Py 3.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Displaying the offset in seconds makes very little sense. We should
adhere to ISO 8601 format of `+HH:MM` which is much easier to
understand.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Timestamps in the main log are currently printed in local time. This
patch adds UTC offset to the top of the log so it's possible to find out
exactly when events happen.
Fixes: https://pagure.io/pungi/issue/710
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Save the config files which are imported by the following syntax:
from some_config import *
Note: This requires kobo >= 0.6
Signed-off-by: Qixiang Wan <qwan@redhat.com>
If the variant is not empty, it buildinstall will run lorax on it and
create some files that would otherwise be overwritten by the ostree
installer.
The validation script is updated to load variants file as it needs a
list of variants to find out if the config is wrong.
Fixes: https://pagure.io/pungi/issue/695
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
In many cases we need to open files as binary to avoid errors on Py3
about writing binary data to file opened in text mode.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The config can change the default compose type. This can still be
overwritten by a CLI argument. A `--production` option is now added to
CLI (because that was the default before).
Fixes: https://pagure.io/pungi/issue/694
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
In once case we can completely drop try-except statements: it only
logs the exception details but that will be logged again anyway.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is updated in all files for consistency, even the modules that will
never be ported to Py 3 completely due to dependency on Yum.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When using repos as gather_source, we should use DNF backend even for
constructing initial package set and to download the packages from
source repos. Without this the repos source would not be usable on
Python 3.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is needed for correct trimming of addons and optional. A package
pulled in as a dependency but that matches something on fulltree exclude
list should have this flag. It will then be moved from addon to base
variant and therefore excluded from optional.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This script can be used as a notification handler. For most messages it
does nothing, but when it sees a new commit in an ostree repo, it will
wait for a signature of the new commit to appear.
This is useful for building images later so that they include the
signature as well.
Fixes: https://pagure.io/pungi/issue/650
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
* Remove explicit option requirement. Argparse can take care of that
while also making this information visible in help output.
* Simplify writing resulting comps.
* Remove unused code.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the environment has arch attribute, the environment should be removed
from the file on all other arches. This mirrors similar behaviour for
groups and packages.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
