Commit Graph

92 Commits

Author SHA1 Message Date
Lubomír Sedlář
a155674269 Set persistdir to a temporary location
We don't want to even touch any system directory.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:48 +01:00
Lubomír Sedlář
4f4fd845d9 Fix PEP8 complaints
* Remove dead imports
* Remove dead code
* Fix extra empty lines
* Fix formatting
* Break too long lines

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:48 +01:00
Daniel Mach
06635a3917 Add profiler to pungi-gather.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:47 +01:00
Daniel Mach
7cb9e9dc20 Add support for mirrorlists in DNF gathering code.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:12:23 +01:00
Daniel Mach
a5d302d036 Enable --fulltree, --selfhosting and --nodeps in pungi-gather.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:12:23 +01:00
Daniel Mach
aff6ac906c Enable langpack gathering in DNF gather.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:12:23 +01:00
Daniel Mach
43d8d02cb1 Integrate DNF gathering code with Pungi.
Set 'dnf_gather = True' (temporary option) in config file to enable DNF
gathering.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:12:23 +01:00
Daniel Mach
caf11f55d4 New gathering (depsolving) module based on DNF.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 08:51:38 +01:00
Lubomír Sedlář
4b90822115 Add a script for modifying ISO images
With this script it's possible to add additional files into an ISO file.
If the file happens to be ks.cfg, the boot configs are tweaked so that
the kickstart is actually used.

Resolves: #503
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-21 07:55:19 +01:00
Qixiang Wan
79e97dc845 pungi-koji: new cmd option '--latest-link-status'
Add a new option 'latest-link-status' to pungi-koji, if this is
specified, pungi will only create the latest symbol link to the compose
when compose's status matches the specified statuses. The status name is
case insensitive. If the option is not specified it will act as before.

Example:

pungi-koji --target-dir=_composes --config=data/dummy-pungi.conf \
--test --latest-link-status=finished --latest-link-status=finished_incomplete

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-20 11:15:30 +08:00
Lubomír Sedlář
d9ab899920 Rename main logger
The name is not displayed anywhere. Changing it to lowercase will
however avoid possible problem if we create a sublogger somewhere else
in the codebase with name based on module name. Such logger would not
have any handlers defined.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-23 11:08:31 +01:00
Lubomír Sedlář
857aee05c1 util: Add a utility for managing temporary files
In multiple situations we need to create temporary files or directories
that should not be preserved after compose is finished. Let's add
context managers that ensure these get cleaned up.

This fixes tests leaving garbage around in /tmp.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-20 09:14:25 +01:00
Qixiang Wan
a57bc13e30 Add --quiet option to pungi-koji
log will not be printed to screen when quiet mode is enabled.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-02-20 15:00:03 +08:00
Lubomír Sedlář
e260fe5581 Add script to generate unified ISOs
This a standalone script that will look into a compose and create
unified ISO for each architecture. The ISO contains RPM repositories for
all variants that have the arch.

Known issues:
 * The filename does not respect settings. This is tricky because the
   name could include variant name, which we don't have here (by design
   of unified ISO).
 * The same is true for volume id.

In order to test the feature without running actual compose, we need to
add essentially a big chunk of compose. Most of the files are empty, as
their content is never accessed.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-04 10:22:20 +01:00
Lubomír Sedlář
c338219ef0 config: Report unknown options as warnings
Given the way config files can include other files, it is entirely
possible to use the same config file for both Pungi and Distill. Pungi
will however complain about unknown options.

This patch reverts part of c38bb480 and moves deprecation handling back
into schema validation. The validation method then returns a list of
errors and a list of warnings.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-13 11:58:08 +01:00
Lubomír Sedlář
e66d8dd190 Add CLI option to create ci compose
Fixes: #476
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-30 12:30:09 +01:00
Qixiang Wan
80fa723b1d Include phase name in log for some phases
Phases createiso, liveimages, image_build, ostree_installer and osbs are
done in parallel, logs from these phases are mixed and and it's not
obvious which log message belongs to which phase. This change adds phase
name in log message for these phases.

The new mixin 'PhaseLoggerMixin' is added to extend a Pungi phase with a
logging logger which copy handlers from compose's logger but with
formatter changed.

Fixes: #58
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-11-23 16:47:22 +08:00
Dennis Gilmore
d4a78a1553 Merge #459 Fix building images and live media for addons 2016-11-15 21:00:50 +00:00
Lubomír Sedlář
13871b64fb compose: Drop unused argument
The `get_variants()` method had a `recursive` argument with default
value of `False. However, this argument had no effect and the method
always returned all variants recursively.

We can just drop the argument. All callers are updated to not supply the
argument. Should any need for getting the top-level variants only arise,
they can be accessed as the `variants` attribute directly on the Compose
object.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 13:39:01 +01:00
Lubomír Sedlář
8e5b197e3f pungi: Add tests for depsolving
These are tests written by Daniel Mach originally for Distill-NG. They
are ported to current Pungi. The test repositories are committed in the
tests/fixtures/ directory. This is the same data that is used for test
compose, but the actual RPM files are not present. Some tests are
adapted from dmach's fork of Pungi.

Some of the packages are marked with a comment saying they are
important. These are the packages that the test is specifically trying
to get included in the package set. There are also explicit tests for
packages that should not be included.

Two tests are skipped for now as there is a bug preventing them from
passing. This is related to fulltree being done for packages that are
explicitly multilib.

The depsolver is called by invoking a separate executable, so the
coverage data is wrong.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 09:36:23 +01:00
Qixiang Wan
1345eb87a4 pungi-koji: add option for not creating latest symbol link
Fixes: #53
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-10-25 12:45:19 +08:00
Dennis Gilmore
18128623b4 Merge #443 config: Don't abort on deprecated options 2016-10-20 20:26:17 +00:00
Lubomír Sedlář
c38bb4809b config: Don't abort on deprecated options
These options are in fact removed and have no effect anymore. This patch
changes the validation to print a warning that the option was removed
and what should be done instead. It no longer stops the whole compose.

The validation script still rejects configuration files with these
removed keys.

This change means we no longer check these removals with the JSON schema
(as that makes it hard to determine where exactly the problem is).

Fixes: #438
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-20 12:51:12 +02:00
Lubomír Sedlář
74aa41f8bd Send notification when compose fails to start
This is tricky as this early in the process we don't know the compose
ID. The new message gives the full command line that was called.

Relates: #439
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-20 09:39:18 +02:00
Lubomír Sedlář
b5e4ea848a Dump images.json after checksumming
Once we have the checksums, nothing else will change in the metadata. We
should make sure it is on disk as soon as possible.

Fixes: #436
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-19 15:02:33 +02:00
Lubomír Sedlář
ac635a0937 Remove duplicated version from pungi script
We can get the version using same logic as we do for pungi-koji. This
means one less place to bump the version during release.

Fixes: https://pagure.io/pungi/issue/406
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-29 11:23:30 +02:00
Lubomír Sedlář
d6dc269ef4 Remove FSF address from comments
The address is no longer correct. We can just as well simply point to
the web page describing the license.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-23 10:26:43 +02:00
Lubomír Sedlář
9a3b0f089e 4.1.9 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-21 14:07:21 +02:00
Lubomír Sedlář
bfe1068b8e Recreate JSON dump of configuration
We need both full dump and the copy of original file. Having just the
copy is not sufficient because while it preserves comments and
formatting, there can be include statements for other files and those
are not copied.

With this patch we create the dump in the same way as before, and
additionally create a copy of of the config file specified on command
line.

Fixes: #398
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-21 08:31:43 +02:00
Chenxiong Qi
912f81ca16 Dump original pungi conf
Fix #383

Signed-off-by: Chenxiong Qi <cqi@redhat.com>
2016-09-19 17:31:43 +08:00
Pat Riehecky
10b93796bd Allow user to set a ~/.pungirc for some defaults
Merges: #64
Signed-off-by: Pat Riehecky <riehecky@fnal.gov>
2016-09-08 15:47:50 +02:00
Lubomír Sedlář
f9a6c8418f Add JSON Schema for configuration
The schema is written in Python to reduce duplication. When
configuration is loaded, the validation checks if it's correct and fills
in default values.

There is a custom extension to the schema to report deprecated options.

The config dependencies are implemented as a separate pass. While it's
technically possible to express the dependencies in the schema itself,
the error messages are not very helpful and it makes the schema much
harder to read.

Phases no longer define `config_options`. New options should be added to
the schema. Since the default values are populated automatically during
validation, there is no need to duplicate them into the code.

The `pungi-config-validate` script is updated to use the schema and
report errors even for deeply nested fields.

The dependencies are updated: pungi now depends on `python-jsonschema`
(which is already available in Fedora).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-01 10:56:15 +02:00
Lubomír Sedlář
d3b9ae79a5 [createiso] Remove unused script
The pungi-createiso script should have been removed in f37a14fb, but was
forgotten. The script is obsoleted by a new method in which ISOs are
created. It's now broken as it calls methods that no longer exist.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-30 10:28:18 +02:00
Dennis Gilmore
cd747ac065 Merge #361 Fix config validation script 2016-08-16 20:53:50 +00:00
Lubomír Sedlář
a72a38b278 Add full Pungi version to log output
This should help with debugging by providing better information on which
Pungi version created the compose. In development, the version will show
output of git describe, in production it asks which version is installed
in site-packages/. The egg-info directory must be installed for this to
work.

It is no longer necessary to synchronize version in `setup.py` with
`pungi/__init__.py`.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-16 08:12:40 +02:00
Lubomír Sedlář
3de6d094be Fix config validation script
It got broken in commit 8323a735 which added a requirement for a compose
to have valid paths. This patch adds that with somewhat dummy values.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-12 14:08:11 +02:00
Dennis Gilmore
fc23ca5fdf 4.1.8 release
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2016-08-10 07:21:40 -05:00
Lubomír Sedlář
f37a14fb60 [createiso] Use shell script for runroot
Instead of installing pungi itself in the runroot, we can prepare the
commands to be run on compose box, write the shell script into work/
directory, which is mounted in the chroot, and execute that. This way
there is no business logic in runroot (except for finding lorax
templates).

The main advantage of this approach is that we don't need to pull any
extra dependencies into buildroot.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-10 13:39:50 +02:00
Dennis Gilmore
00e11b2f51 4.1.7 release
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2016-06-23 05:14:21 -05:00
Lubomír Sedlář
b4fc97be03 [osbs] Add a phase to build images in OSBS
It will take RPM repo from a variant in this compose and a Dockerfile
from configured git and use it to build an image.

The build images are uploaded to some a Docker registry by OSBS and are
not directly part of compose (because there is no export function).
There is a new metadata file `osbs.json` that has some information that
can be used to find the image.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-06-22 10:02:03 +02:00
Lubomír Sedlář
95cfbfb3fe Setup global log file before logging anything
This should fix the issue with only printing information about automatic
toggling of `supported` flag to standard output and not to a file.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-06-14 08:48:36 +02:00
Dennis Gilmore
d13f655300 pungi-koji: fix up latest symlink creation
If release_version does not have a . in it the current logic gives us
and empty result. in those cases, such as 23 or Rawhide use the value
of release_version as is.

Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2016-05-26 09:46:44 +02:00
Dennis Gilmore
8de7d685d9 4.1.6 release
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2016-05-24 16:36:07 -05:00
Dennis Gilmore
00f2e24bce 4.1.5 release
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2016-05-16 20:56:48 -05:00
Lubomír Sedlář
4e3d87e658 [test] Add checks for created images
The performed checks:
 * If format is ISO, the file must have correct magic string
 * If it's bootable, there must be MBR or GPT

When a check fails on any failable deliverable, it will be logged and
the file removed from metadata (it will still remain on the disk). This
required a change to write the images.json file later (after test
phase).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-05 15:38:00 +02:00
Dennis Gilmore
d8272bd0fa 4.1.4 release
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2016-04-29 15:36:41 -05:00
Dennis Gilmore
4f76425c9f 4.1.3 release
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2016-04-08 07:34:57 -05:00
Dennis Gilmore
36e7279a5e 4.1.2 release
Signed-off-by: Dennis Gilmore <dennis@ra.ausil.us>
2016-04-06 17:44:40 -05:00
Lubomír Sedlář
df400002d8 [createiso] Move runroot work to separate script
Instead of running a long command line in the runroot (or locally), move
all that work into a separate script that will be installed. This means
chroot will need to install pungi.

Everything should work as it did before. The only exception to this is
that there is logic to find lorax templates instead of harcoding the
location. This is done using a separate script.

Related: #230
Fixes: #231
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-06 13:25:49 +02:00
Lubomír Sedlář
96e7ddd3ea [ostree] Rename atomic to ostree
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-05 09:14:00 +02:00