Commit Graph

49 Commits

Author SHA1 Message Date
Lubomír Sedlář
a21c8a555d notification: Allow specifying multiple scripts
The notification hooks can be useful for doing other things than just
announcing status on message bus. For this to be truly usable, we need
the ability to use multiple scripts.

This patch allows the command line option to be specified multiple
times. Each given script will be called. Even if the script fails, it
does not block the compose.

Additionally the output of the notification scripts is logged now to
make it possible to debug possible failure.

Relates: https://pagure.io/pungi/issue/650
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-29 08:30:56 +02:00
Lubomír Sedlář
01026858f7 pungi-koji: Port to argparse
Optparse is deprecated since Python 2.7, long live argparse.

This also allows us to remove some of the manual error checking and make
the library check required or conflicting options.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-27 09:36:23 +02:00
Lubomír Sedlář
d586368515 Add dropped livemedia phase
After the patch to ensure that all phases are stopped the livemedia
phase was apparently dropped by accident.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-19 08:58:53 +02:00
Lubomír Sedlář
247a1a71ba Ensure all phases are stopped
If a phase is started successfully, it needs to be stopped as well. In
most cases when `stop` is called immediately after `start`, this is not
a problem.

Only when something else happens while a phase is runnning and this
something fails, Pungi will deadlock and never exit. This something
could be another phase or just main thread raising an exception.

Fixes: #625
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-02 09:52:18 +02:00
Qixiang Wan
79e97dc845 pungi-koji: new cmd option '--latest-link-status'
Add a new option 'latest-link-status' to pungi-koji, if this is
specified, pungi will only create the latest symbol link to the compose
when compose's status matches the specified statuses. The status name is
case insensitive. If the option is not specified it will act as before.

Example:

pungi-koji --target-dir=_composes --config=data/dummy-pungi.conf \
--test --latest-link-status=finished --latest-link-status=finished_incomplete

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-20 11:15:30 +08:00
Lubomír Sedlář
d9ab899920 Rename main logger
The name is not displayed anywhere. Changing it to lowercase will
however avoid possible problem if we create a sublogger somewhere else
in the codebase with name based on module name. Such logger would not
have any handlers defined.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-23 11:08:31 +01:00
Qixiang Wan
a57bc13e30 Add --quiet option to pungi-koji
log will not be printed to screen when quiet mode is enabled.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-02-20 15:00:03 +08:00
Lubomír Sedlář
c338219ef0 config: Report unknown options as warnings
Given the way config files can include other files, it is entirely
possible to use the same config file for both Pungi and Distill. Pungi
will however complain about unknown options.

This patch reverts part of c38bb480 and moves deprecation handling back
into schema validation. The validation method then returns a list of
errors and a list of warnings.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-13 11:58:08 +01:00
Lubomír Sedlář
e66d8dd190 Add CLI option to create ci compose
Fixes: #476
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-30 12:30:09 +01:00
Qixiang Wan
80fa723b1d Include phase name in log for some phases
Phases createiso, liveimages, image_build, ostree_installer and osbs are
done in parallel, logs from these phases are mixed and and it's not
obvious which log message belongs to which phase. This change adds phase
name in log message for these phases.

The new mixin 'PhaseLoggerMixin' is added to extend a Pungi phase with a
logging logger which copy handlers from compose's logger but with
formatter changed.

Fixes: #58
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-11-23 16:47:22 +08:00
Lubomír Sedlář
13871b64fb compose: Drop unused argument
The `get_variants()` method had a `recursive` argument with default
value of `False. However, this argument had no effect and the method
always returned all variants recursively.

We can just drop the argument. All callers are updated to not supply the
argument. Should any need for getting the top-level variants only arise,
they can be accessed as the `variants` attribute directly on the Compose
object.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 13:39:01 +01:00
Qixiang Wan
1345eb87a4 pungi-koji: add option for not creating latest symbol link
Fixes: #53
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-10-25 12:45:19 +08:00
Dennis Gilmore
18128623b4 Merge #443 config: Don't abort on deprecated options 2016-10-20 20:26:17 +00:00
Lubomír Sedlář
c38bb4809b config: Don't abort on deprecated options
These options are in fact removed and have no effect anymore. This patch
changes the validation to print a warning that the option was removed
and what should be done instead. It no longer stops the whole compose.

The validation script still rejects configuration files with these
removed keys.

This change means we no longer check these removals with the JSON schema
(as that makes it hard to determine where exactly the problem is).

Fixes: #438
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-20 12:51:12 +02:00
Lubomír Sedlář
74aa41f8bd Send notification when compose fails to start
This is tricky as this early in the process we don't know the compose
ID. The new message gives the full command line that was called.

Relates: #439
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-20 09:39:18 +02:00
Lubomír Sedlář
b5e4ea848a Dump images.json after checksumming
Once we have the checksums, nothing else will change in the metadata. We
should make sure it is on disk as soon as possible.

Fixes: #436
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-19 15:02:33 +02:00
Lubomír Sedlář
bfe1068b8e Recreate JSON dump of configuration
We need both full dump and the copy of original file. Having just the
copy is not sufficient because while it preserves comments and
formatting, there can be include statements for other files and those
are not copied.

With this patch we create the dump in the same way as before, and
additionally create a copy of of the config file specified on command
line.

Fixes: #398
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-21 08:31:43 +02:00
Chenxiong Qi
912f81ca16 Dump original pungi conf
Fix #383

Signed-off-by: Chenxiong Qi <cqi@redhat.com>
2016-09-19 17:31:43 +08:00
Lubomír Sedlář
f9a6c8418f Add JSON Schema for configuration
The schema is written in Python to reduce duplication. When
configuration is loaded, the validation checks if it's correct and fills
in default values.

There is a custom extension to the schema to report deprecated options.

The config dependencies are implemented as a separate pass. While it's
technically possible to express the dependencies in the schema itself,
the error messages are not very helpful and it makes the schema much
harder to read.

Phases no longer define `config_options`. New options should be added to
the schema. Since the default values are populated automatically during
validation, there is no need to duplicate them into the code.

The `pungi-config-validate` script is updated to use the schema and
report errors even for deeply nested fields.

The dependencies are updated: pungi now depends on `python-jsonschema`
(which is already available in Fedora).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-01 10:56:15 +02:00
Lubomír Sedlář
a72a38b278 Add full Pungi version to log output
This should help with debugging by providing better information on which
Pungi version created the compose. In development, the version will show
output of git describe, in production it asks which version is installed
in site-packages/. The egg-info directory must be installed for this to
work.

It is no longer necessary to synchronize version in `setup.py` with
`pungi/__init__.py`.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-16 08:12:40 +02:00
Lubomír Sedlář
b4fc97be03 [osbs] Add a phase to build images in OSBS
It will take RPM repo from a variant in this compose and a Dockerfile
from configured git and use it to build an image.

The build images are uploaded to some a Docker registry by OSBS and are
not directly part of compose (because there is no export function).
There is a new metadata file `osbs.json` that has some information that
can be used to find the image.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-06-22 10:02:03 +02:00
Lubomír Sedlář
95cfbfb3fe Setup global log file before logging anything
This should fix the issue with only printing information about automatic
toggling of `supported` flag to standard output and not to a file.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-06-14 08:48:36 +02:00
Dennis Gilmore
d13f655300 pungi-koji: fix up latest symlink creation
If release_version does not have a . in it the current logic gives us
and empty result. in those cases, such as 23 or Rawhide use the value
of release_version as is.

Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2016-05-26 09:46:44 +02:00
Lubomír Sedlář
4e3d87e658 [test] Add checks for created images
The performed checks:
 * If format is ISO, the file must have correct magic string
 * If it's bootable, there must be MBR or GPT

When a check fails on any failable deliverable, it will be logged and
the file removed from metadata (it will still remain on the disk). This
required a change to write the images.json file later (after test
phase).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-05 15:38:00 +02:00
Lubomír Sedlář
96e7ddd3ea [ostree] Rename atomic to ostree
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-05 09:14:00 +02:00
Lubomír Sedlář
490514e263 [checks] Add a check for too restrictive umask
If umask is set to something too high (>0022), a warning will be
printed. It does not abort the compose though.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-31 10:10:18 +02:00
Lubomír Sedlář
8d224b206b [atomic] Add atomic_installer phase
This phase runs lorax with extra templates in Koji runroot task, links
the boot.iso to proper location in compose directory and adds the
installer iso to image manifest. This phase runs concurrently with live
media etc.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-23 12:00:06 +01:00
Lubomír Sedlář
536c6c85b7 [ostree] Add ostree phase
This phase runs the script to make ostree repository in koji runroot
task. It runs right after regular yum repos are created.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-23 10:58:35 +01:00
Lubomír Sedlář
337d2a999c [checks] Don't always require jigdo
If the configuration specifically requests no jigdos, there is no point
in checking for the binary existence.

This is not 100% reliable. The jigdo option defaults to True, so if the
option is not specified the binary is required even if there are no
images configured.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-10 09:08:19 +01:00
Tomas Mlcoch
5bffca5037 Support signing of rpm wrapped live images
With this patch, you can specify a command for
signing of koji builds. For example:

    signing_key_password_file = '~/file_with_password_for_key_fedora-24'
    signing_key_id = '81b46521'
    signing_command = '~/git/releng/scripts/sigulsign_unsigned.py -vv --password=%(signing_key_password)s fedora-24'

'signing_key_password_file' is a path to a file which contains
a password that will be formatted into 'signing_command' string
via '%(signing_key_password)s' string format syntax (if used).
Because pungi config is usualy stored in git and part of compose
logs we don't want password to be included directly in the config.
Note: If '-' is used instead of a filename, then you will be asked
for the password interactivelly right after pungi starts.

'signing_key_id' is ID of the key that will be used for the signing.
This ID will be used when crafting koji paths to signed files
(kojipkgs.fedoraproject.org/packages/NAME/VER/REL/data/signed/KEYID/..).

'signing_command' a command that will be run with a build as a single
argument. This command mustn't require any user interaction.
If you need to pass a password for a signing key to the command,
do this via command line option of the command with use of string
formatting syntax '%(signing_key_password)s' (see details
about 'signing_key_password_file').

Signed-off-by: Tomáš Mlčoch <tmlcoch@redhat.com>
2016-02-18 13:36:38 +01:00
Lubomír Sedlář
454363fba8 Allow specifying empty variants
The variants.xml file can list a variant with is_empty="true" and no
groups. If such variant is found, not package gathering will be run for
it, and no repos will be created.

This only makes sense for a variant that will have some other
deliverables like live media or images.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-16 15:08:15 +01:00
Lubomír Sedlář
439622d576 [live-media] Add live media phase
This phase builds live media in Koji using the Live Media Creator. It
runs in parallel with current live images, create ISO and image build
phases.

The documentation is updated to explain how to configure this.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-01 13:28:49 +01:00
Dennis Gilmore
10a90f97b4 Merge #75 Start of development guide 2015-12-08 17:41:00 +00:00
Lubomír Sedlář
bd8d814230 Move messaging into cli options
The messaging is not really part of compose settings. It is an
infrastructure part. As such, it should really be set up as part of
pungi invocation, not compose configuration.

The documentation is updated to reflect this. Some updates to the
documentation are done as well: listing messages about ISOs and minor
formatting updates.

The test_compose.sh script can now accept additional command line
options and pass them on to pungi-koji to simplify testing.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-11-27 10:35:08 +01:00
Lubomír Sedlář
ba8ae15eb1 Allow running scripts with any python in PATH
Instead of hardcoding /usr/bin/python in shebangs, use /usr/bin/env.
This allows Pungi to work with dependencies installed in virtualenv.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-11-27 08:38:47 +01:00
Dennis Gilmore
6f00f20b3d Merge #68 Add support for sending messages 2015-11-25 14:56:51 +00:00
Lubomír Sedlář
066855a039 Add ability to send messages about progress
With this patch, Pungi can invoke an arbitrary command on various
moments of the compose process. The invoked command can the decide on
what message to send (and using what messaging system).

The actual command is specified in the config file.

There is a script provided that sends the messages via fedmsg.

The documentation is updated to have details about the new config option
as well as the interface for the messaging script.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-11-25 15:46:50 +01:00
Lubomír Sedlář
359eb444e5 Move image checksum collecting to separate phase
The checksums are still generated in the ImageBuild and CreatISO phases,
but collecting of them into single file is moved from pungi-koji script
into a separate phase.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-11-24 09:36:13 +01:00
Tomas Mlcoch
5c8694b286 Merge #57 Initialize phases in their usage order 2015-09-09 08:00:21 -04:00
Tomas Mlcoch
ecc6b0f62a Initialize phases in their usage order 2015-09-09 08:00:21 -04:00
Tomas Mlcoch
4d93221005 Add missing config check for phases: createrepo, extrafiles, liveimages and imate_build 2015-09-10 07:12:08 -04:00
Lubos Kocman
8e90a2a32a Add image-build support
Signed-off-by: Lubos Kocman <lkocman@redhat.com>
2015-09-01 08:03:34 +00:00
Pavol Babincak
853c27d1be pungi-koji: Don't print traceback if error occurred.
If compose directory was created and logger setup properly traceback is
written in traceback.log. Otherwise traceback is printed on stdout.
2015-08-25 08:11:14 -04:00
Daniel Mach
72302bd98e Rename product_* to release_*.
CHANGE: Rename product_* config options to release_* for consistency with productmd.
ACTION: Rename product_name, product_short, product_version, product_is_layered to release_* in config files.
        Rename //variant/product to //variant/release in variants XML.
2015-07-09 06:58:30 -04:00
Lubos Kocman
7d714e49f3 pungi-koji: use logger instead of print when it's available 2015-06-07 18:28:15 +00:00
Lubos Kocman
008d80186e pungi-koji: fix bad module path to verify_label() 2015-06-07 18:27:58 +00:00
Dennis Gilmore
b6825f3471 fix up productmd import in the executable
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2015-04-24 17:58:59 -05:00
Dennis Gilmore
5197fed16a add missing ) causing a syntax error
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2015-03-14 14:20:31 -05:00
Dennis Gilmore
a3158ec144 rename binaries
rename the pungi binary to pungi-koji since it does is tasks in koji
rename pungi-gather to pungi as it is the standalone old pungi binary

there is scripts that expect pungi to be the old pungi, the new binary
is not yet in use, pungi-koji semes to make sense, open to better ideas
2015-03-12 13:15:29 -05:00