Commit Graph

106 Commits

Author SHA1 Message Date
Lubomír Sedlář
c3b49f7ffb pkgset: Download packages with dnf
When using repos as gather_source, we should use DNF backend even for
constructing initial package set and to download the packages from
source repos. Without this the repos source would not be usable on
Python 3.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-16 14:32:26 +02:00
Lubomír Sedlář
a8cd78faf9 gather: Add fulltree-exclude flag to DNF backend
This is needed for correct trimming of addons and optional. A package
pulled in as a dependency but that matches something on fulltree exclude
list should have this flag. It will then be moved from addon to base
variant and therefore excluded from optional.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-16 13:06:55 +02:00
Lubomír Sedlář
c12bad295f config: Add option for dumping config schema
This makes it easier to compare changes to the schema in different
versions.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-14 14:53:22 +02:00
Lubomír Sedlář
f7a9c77626 ostree: Add notification handler to wait for signature
This script can be used as a notification handler. For most messages it
does nothing, but when it sees a new commit in an ostree repo, it will
wait for a signature of the new commit to appear.

This is useful for building images later so that they include the
signature as well.

Fixes: https://pagure.io/pungi/issue/650
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-11 10:24:49 +02:00
Lubomír Sedlář
afffb27f94 comps_wrapper: Code clean up
* Remove explicit option requirement. Argparse can take care of that
  while also making this information visible in help output.
* Simplify writing resulting comps.
* Remove unused code.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-29 15:51:48 +02:00
Pavel Holica
9a3d04c305 comps_filter: Filter environments by arch
If the environment has arch attribute, the environment should be removed
from the file on all other arches. This mirrors similar behaviour for
groups and packages.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-29 14:45:59 +02:00
Lubomír Sedlář
a21c8a555d notification: Allow specifying multiple scripts
The notification hooks can be useful for doing other things than just
announcing status on message bus. For this to be truly usable, we need
the ability to use multiple scripts.

This patch allows the command line option to be specified multiple
times. Each given script will be called. Even if the script fails, it
does not block the compose.

Additionally the output of the notification scripts is logged now to
make it possible to debug possible failure.

Relates: https://pagure.io/pungi/issue/650
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-29 08:30:56 +02:00
Lubomír Sedlář
9193a6902e pungi: Port to argparse
And fix PEP8 violations while modifying the file.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-27 09:37:03 +02:00
Lubomír Sedlář
87c485f2d0 comps_filter: Port to argparse
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-27 09:37:03 +02:00
Lubomír Sedlář
01026858f7 pungi-koji: Port to argparse
Optparse is deprecated since Python 2.7, long live argparse.

This also allows us to remove some of the manual error checking and make
the library check required or conflicting options.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-27 09:36:23 +02:00
Lubomír Sedlář
d586368515 Add dropped livemedia phase
After the patch to ensure that all phases are stopped the livemedia
phase was apparently dropped by accident.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-19 08:58:53 +02:00
Lubomír Sedlář
247a1a71ba Ensure all phases are stopped
If a phase is started successfully, it needs to be stopped as well. In
most cases when `stop` is called immediately after `start`, this is not
a problem.

Only when something else happens while a phase is runnning and this
something fails, Pungi will deadlock and never exit. This something
could be another phase or just main thread raising an exception.

Fixes: #625
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-02 09:52:18 +02:00
Lubomír Sedlář
35e72df99f Explicitly remove all temporary files
There is no guarantee __del__ will ever be called, and we were leaving a
ton of stuff in /tmp. With this patch we pass the temporary directories
explictly and make sure they are deleted at the end.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář
ec67eac1cc Use enum for flags
This adds an extra dependency on python-enum34.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:49 +01:00
Lubomír Sedlář
a155674269 Set persistdir to a temporary location
We don't want to even touch any system directory.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:48 +01:00
Lubomír Sedlář
4f4fd845d9 Fix PEP8 complaints
* Remove dead imports
* Remove dead code
* Fix extra empty lines
* Fix formatting
* Break too long lines

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:48 +01:00
Daniel Mach
06635a3917 Add profiler to pungi-gather.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:47 +01:00
Daniel Mach
7cb9e9dc20 Add support for mirrorlists in DNF gathering code.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:12:23 +01:00
Daniel Mach
a5d302d036 Enable --fulltree, --selfhosting and --nodeps in pungi-gather.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:12:23 +01:00
Daniel Mach
aff6ac906c Enable langpack gathering in DNF gather.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:12:23 +01:00
Daniel Mach
43d8d02cb1 Integrate DNF gathering code with Pungi.
Set 'dnf_gather = True' (temporary option) in config file to enable DNF
gathering.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:12:23 +01:00
Daniel Mach
caf11f55d4 New gathering (depsolving) module based on DNF.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 08:51:38 +01:00
Lubomír Sedlář
4b90822115 Add a script for modifying ISO images
With this script it's possible to add additional files into an ISO file.
If the file happens to be ks.cfg, the boot configs are tweaked so that
the kickstart is actually used.

Resolves: #503
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-21 07:55:19 +01:00
Qixiang Wan
79e97dc845 pungi-koji: new cmd option '--latest-link-status'
Add a new option 'latest-link-status' to pungi-koji, if this is
specified, pungi will only create the latest symbol link to the compose
when compose's status matches the specified statuses. The status name is
case insensitive. If the option is not specified it will act as before.

Example:

pungi-koji --target-dir=_composes --config=data/dummy-pungi.conf \
--test --latest-link-status=finished --latest-link-status=finished_incomplete

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-20 11:15:30 +08:00
Lubomír Sedlář
d9ab899920 Rename main logger
The name is not displayed anywhere. Changing it to lowercase will
however avoid possible problem if we create a sublogger somewhere else
in the codebase with name based on module name. Such logger would not
have any handlers defined.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-23 11:08:31 +01:00
Lubomír Sedlář
857aee05c1 util: Add a utility for managing temporary files
In multiple situations we need to create temporary files or directories
that should not be preserved after compose is finished. Let's add
context managers that ensure these get cleaned up.

This fixes tests leaving garbage around in /tmp.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-20 09:14:25 +01:00
Qixiang Wan
a57bc13e30 Add --quiet option to pungi-koji
log will not be printed to screen when quiet mode is enabled.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-02-20 15:00:03 +08:00
Lubomír Sedlář
e260fe5581 Add script to generate unified ISOs
This a standalone script that will look into a compose and create
unified ISO for each architecture. The ISO contains RPM repositories for
all variants that have the arch.

Known issues:
 * The filename does not respect settings. This is tricky because the
   name could include variant name, which we don't have here (by design
   of unified ISO).
 * The same is true for volume id.

In order to test the feature without running actual compose, we need to
add essentially a big chunk of compose. Most of the files are empty, as
their content is never accessed.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-04 10:22:20 +01:00
Lubomír Sedlář
c338219ef0 config: Report unknown options as warnings
Given the way config files can include other files, it is entirely
possible to use the same config file for both Pungi and Distill. Pungi
will however complain about unknown options.

This patch reverts part of c38bb480 and moves deprecation handling back
into schema validation. The validation method then returns a list of
errors and a list of warnings.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-13 11:58:08 +01:00
Lubomír Sedlář
e66d8dd190 Add CLI option to create ci compose
Fixes: #476
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-30 12:30:09 +01:00
Qixiang Wan
80fa723b1d Include phase name in log for some phases
Phases createiso, liveimages, image_build, ostree_installer and osbs are
done in parallel, logs from these phases are mixed and and it's not
obvious which log message belongs to which phase. This change adds phase
name in log message for these phases.

The new mixin 'PhaseLoggerMixin' is added to extend a Pungi phase with a
logging logger which copy handlers from compose's logger but with
formatter changed.

Fixes: #58
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-11-23 16:47:22 +08:00
Dennis Gilmore
d4a78a1553 Merge #459 Fix building images and live media for addons 2016-11-15 21:00:50 +00:00
Lubomír Sedlář
13871b64fb compose: Drop unused argument
The `get_variants()` method had a `recursive` argument with default
value of `False. However, this argument had no effect and the method
always returned all variants recursively.

We can just drop the argument. All callers are updated to not supply the
argument. Should any need for getting the top-level variants only arise,
they can be accessed as the `variants` attribute directly on the Compose
object.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 13:39:01 +01:00
Lubomír Sedlář
8e5b197e3f pungi: Add tests for depsolving
These are tests written by Daniel Mach originally for Distill-NG. They
are ported to current Pungi. The test repositories are committed in the
tests/fixtures/ directory. This is the same data that is used for test
compose, but the actual RPM files are not present. Some tests are
adapted from dmach's fork of Pungi.

Some of the packages are marked with a comment saying they are
important. These are the packages that the test is specifically trying
to get included in the package set. There are also explicit tests for
packages that should not be included.

Two tests are skipped for now as there is a bug preventing them from
passing. This is related to fulltree being done for packages that are
explicitly multilib.

The depsolver is called by invoking a separate executable, so the
coverage data is wrong.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 09:36:23 +01:00
Qixiang Wan
1345eb87a4 pungi-koji: add option for not creating latest symbol link
Fixes: #53
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-10-25 12:45:19 +08:00
Dennis Gilmore
18128623b4 Merge #443 config: Don't abort on deprecated options 2016-10-20 20:26:17 +00:00
Lubomír Sedlář
c38bb4809b config: Don't abort on deprecated options
These options are in fact removed and have no effect anymore. This patch
changes the validation to print a warning that the option was removed
and what should be done instead. It no longer stops the whole compose.

The validation script still rejects configuration files with these
removed keys.

This change means we no longer check these removals with the JSON schema
(as that makes it hard to determine where exactly the problem is).

Fixes: #438
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-20 12:51:12 +02:00
Lubomír Sedlář
74aa41f8bd Send notification when compose fails to start
This is tricky as this early in the process we don't know the compose
ID. The new message gives the full command line that was called.

Relates: #439
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-20 09:39:18 +02:00
Lubomír Sedlář
b5e4ea848a Dump images.json after checksumming
Once we have the checksums, nothing else will change in the metadata. We
should make sure it is on disk as soon as possible.

Fixes: #436
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-19 15:02:33 +02:00
Lubomír Sedlář
ac635a0937 Remove duplicated version from pungi script
We can get the version using same logic as we do for pungi-koji. This
means one less place to bump the version during release.

Fixes: https://pagure.io/pungi/issue/406
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-29 11:23:30 +02:00
Lubomír Sedlář
d6dc269ef4 Remove FSF address from comments
The address is no longer correct. We can just as well simply point to
the web page describing the license.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-23 10:26:43 +02:00
Lubomír Sedlář
9a3b0f089e 4.1.9 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-21 14:07:21 +02:00
Lubomír Sedlář
bfe1068b8e Recreate JSON dump of configuration
We need both full dump and the copy of original file. Having just the
copy is not sufficient because while it preserves comments and
formatting, there can be include statements for other files and those
are not copied.

With this patch we create the dump in the same way as before, and
additionally create a copy of of the config file specified on command
line.

Fixes: #398
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-21 08:31:43 +02:00
Chenxiong Qi
912f81ca16 Dump original pungi conf
Fix #383

Signed-off-by: Chenxiong Qi <cqi@redhat.com>
2016-09-19 17:31:43 +08:00
Pat Riehecky
10b93796bd Allow user to set a ~/.pungirc for some defaults
Merges: #64
Signed-off-by: Pat Riehecky <riehecky@fnal.gov>
2016-09-08 15:47:50 +02:00
Lubomír Sedlář
f9a6c8418f Add JSON Schema for configuration
The schema is written in Python to reduce duplication. When
configuration is loaded, the validation checks if it's correct and fills
in default values.

There is a custom extension to the schema to report deprecated options.

The config dependencies are implemented as a separate pass. While it's
technically possible to express the dependencies in the schema itself,
the error messages are not very helpful and it makes the schema much
harder to read.

Phases no longer define `config_options`. New options should be added to
the schema. Since the default values are populated automatically during
validation, there is no need to duplicate them into the code.

The `pungi-config-validate` script is updated to use the schema and
report errors even for deeply nested fields.

The dependencies are updated: pungi now depends on `python-jsonschema`
(which is already available in Fedora).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-01 10:56:15 +02:00
Lubomír Sedlář
d3b9ae79a5 [createiso] Remove unused script
The pungi-createiso script should have been removed in f37a14fb, but was
forgotten. The script is obsoleted by a new method in which ISOs are
created. It's now broken as it calls methods that no longer exist.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-30 10:28:18 +02:00
Dennis Gilmore
cd747ac065 Merge #361 Fix config validation script 2016-08-16 20:53:50 +00:00
Lubomír Sedlář
a72a38b278 Add full Pungi version to log output
This should help with debugging by providing better information on which
Pungi version created the compose. In development, the version will show
output of git describe, in production it asks which version is installed
in site-packages/. The egg-info directory must be installed for this to
work.

It is no longer necessary to synchronize version in `setup.py` with
`pungi/__init__.py`.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-16 08:12:40 +02:00
Lubomír Sedlář
3de6d094be Fix config validation script
It got broken in commit 8323a735 which added a requirement for a compose
to have valid paths. This patch adds that with somewhat dummy values.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-12 14:08:11 +02:00