Commit Graph

2716 Commits

Author SHA1 Message Date
Pat Riehecky
10b93796bd Allow user to set a ~/.pungirc for some defaults
Merges: #64
Signed-off-by: Pat Riehecky <riehecky@fnal.gov>
2016-09-08 15:47:50 +02:00
Lubomír Sedlář
1a16d94dda metadata: Improve error reporting on failed checksum
Instead of using the _doChecksum function, it now uses a function from
kobo which has nicer API. If there is an error, the message now includes
more details.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-07 13:05:11 +02:00
Jeremy Cline
ee1ee0467b extra-files: Write a metadata file enumerating extra files
Introduces a new metadata file to track arbitrary files added during the
extra-files phase. This file is placed in the root of each tree and is
called ``extra_files.json``. It is a JSON file containing a single
object, which contains a "header" key with an object describing the
metadata, and a "data" key, which is an array of objects, where each
object represents a file. Each object contains the "file", "checksums",
and "size" keys. "file" is the relative path from the tree root to the
extra file. "checksums" is an object containing one or more checksums,
where the key is the digest type and the value of that key is the hex
digest. Finally, the size is the size of the file in bytes.

For example:
{
  "header": {"version": "1.0},
  "data": [
    {
      "file": "GPL",
      "checksums": {
        "sha256": "8177f97513213526df2cf6184d8ff986c675afb514d4e68a404010521b880643"
      },
      "size": 18092
    },
    {
      "file": "release-notes/notes.html",
      "checksums": {
        "sha256": "82b1ba8db522aadf101dca6404235fba179e559b95ea24ff39ee1e5d9a53bdcb"
      },
      "size": 1120
    }
  ]
}

Signed-off-by: Jeremy Cline <jeremy@jcline.org>
Fixes: #295
2016-09-07 13:02:48 +02:00
Dennis Gilmore
6aeab9ee9d Merge #381 Automatically generate missing image version 2016-09-06 15:19:55 +00:00
Lubomír Sedlář
223a015898 Automatically generate missing image version
If the configuration does not specify version for images or live media,
Pungi will create a default value based on `release_version`. If label
is used for the compose, the milestone from it will be appended to the
version (unless it's RC).

This change is backwards compatible: nothing changes when version is set
in configuration. If the version was missing before, building the
artifacts would fail. With this patch, default values will be supplied.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-01 15:13:04 +02:00
Lubomír Sedlář
f9a6c8418f Add JSON Schema for configuration
The schema is written in Python to reduce duplication. When
configuration is loaded, the validation checks if it's correct and fills
in default values.

There is a custom extension to the schema to report deprecated options.

The config dependencies are implemented as a separate pass. While it's
technically possible to express the dependencies in the schema itself,
the error messages are not very helpful and it makes the schema much
harder to read.

Phases no longer define `config_options`. New options should be added to
the schema. Since the default values are populated automatically during
validation, there is no need to duplicate them into the code.

The `pungi-config-validate` script is updated to use the schema and
report errors even for deeply nested fields.

The dependencies are updated: pungi now depends on `python-jsonschema`
(which is already available in Fedora).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-01 10:56:15 +02:00
Lubomír Sedlář
5534fda192 Allow arbitrary arguments in make test
This can be helpful to enable additional nose plugin.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-01 10:49:36 +02:00
Lubomír Sedlář
1881bf740d createiso: Report nice error when tag does not exist
Instead of failing with generic `AttributeError`, we can easily report
the actual problem.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-01 09:09:31 +02:00
Lubomír Sedlář
a31be0a3c4 Fix test data build script
It uses bash specific features, so it should not claim /bin/sh in
shebang. We also want to use `set -e` to catch possible errors instead
of claiming success every time.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-01 08:46:23 +02:00
Lubomír Sedlář
f43b8736b7 [osbs] Add NVRA of created image into main log
Currently the main log only says the phase started and finished. With
this patch each created image will be mentioned by its name, version,
release and arch.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-30 10:37:17 +02:00
Lubomír Sedlář
d3b9ae79a5 [createiso] Remove unused script
The pungi-createiso script should have been removed in f37a14fb, but was
forgotten. The script is obsoleted by a new method in which ISOs are
created. It's now broken as it calls methods that no longer exist.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-30 10:28:18 +02:00
Lubomír Sedlář
f507b14954 Update doc about generating release value
It is taken from compose label if provided, and only then falls back to
date, type and respin.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-26 10:05:25 +02:00
Lubomír Sedlář
43fda1e9ef Use label to populate image release
When release for an image is specified as explicit `None`, we can
generate the value based on compose label. For example for `Alpha-1.2`
the release would be `1.2` instead of the date based one.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-25 15:24:36 +02:00
Lubomír Sedlář
d90dba9312 doc: Fix example for image_build
The example configuration can now be copy-pasted into real configuration
file and will parse without any issues.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-25 14:21:03 +02:00
Lubomír Sedlář
7fd5403495 Ignore module imports not at top of file
In multiple places we need to set up import paths before importing
modules. This Flake8 warning is just noise.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-22 10:55:48 +02:00
Dennis Gilmore
76f3f909ec Merge #367 Remove unused imports 2016-08-18 13:16:53 +00:00
Lubomír Sedlář
b862fc5a50 [buildinstall] Fix cleaning output dir
Before the task is started, the output directory is checked and if it
exists and is not empty, the runroot task will be skipped. This is meant
for debugging when restarting the same compose. Under usual
circumstances, the directory will not be created in the first place.

The runroot task will start by removing the output directory. This way,
if koji restarts the task, lorax will not fail.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-18 09:11:30 +02:00
Lubomír Sedlář
f0f236ac5e Remove unused imports
After duplicated functions were removed, some imports were left behind.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-17 08:20:01 +02:00
Dennis Gilmore
1238aaa2e6 Merge #360 [osbs] Convert build_id to int 2016-08-16 20:54:10 +00:00
Dennis Gilmore
cd747ac065 Merge #361 Fix config validation script 2016-08-16 20:53:50 +00:00
Dennis Gilmore
492cdec719 Merge #365 Make image test at end of compose less strict 2016-08-16 20:53:23 +00:00
Lubomír Sedlář
91b2f6e941 [test] Make image test at end of compose less strict
Originally, the idea was to remove non-blocking images that failed the
check. They were only removed from the manifest, which creates some
confusion as to what is going on. With this patch, a failed check on
non-blocking deliverable will only print an error message.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-16 10:18:12 +02:00
Lubomír Sedlář
b4765459f3 [iso] Fix check on failable ISO
When ISO is marked as failable and check fails, the compose would still
be aborted. This patch fixes that

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-16 10:10:16 +02:00
Lubomír Sedlář
a72a38b278 Add full Pungi version to log output
This should help with debugging by providing better information on which
Pungi version created the compose. In development, the version will show
output of git describe, in production it asks which version is installed
in site-packages/. The egg-info directory must be installed for this to
work.

It is no longer necessary to synchronize version in `setup.py` with
`pungi/__init__.py`.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-16 08:12:40 +02:00
Lubomír Sedlář
3de6d094be Fix config validation script
It got broken in commit 8323a735 which added a requirement for a compose
to have valid paths. This patch adds that with somewhat dummy values.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-12 14:08:11 +02:00
Lubomír Sedlář
c05282be44 [osbs] Convert build_id to int
The getBuild API call expects either an integer or string NVR, otherwise
it will crash.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-12 14:07:58 +02:00
Lubomír Sedlář
1a4e1b211c [image-build] Get failable config from correct place
The key should be nested in `image-build` block.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-11 14:41:04 +02:00
Dennis Gilmore
fc23ca5fdf 4.1.8 release
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2016-08-10 07:21:40 -05:00
Lubomír Sedlář
f37a14fb60 [createiso] Use shell script for runroot
Instead of installing pungi itself in the runroot, we can prepare the
commands to be run on compose box, write the shell script into work/
directory, which is mounted in the chroot, and execute that. This way
there is no business logic in runroot (except for finding lorax
templates).

The main advantage of this approach is that we don't need to pull any
extra dependencies into buildroot.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-10 13:39:50 +02:00
Dennis Gilmore
6a682f64fe Merge #357 Improve error messages for gathering packages 2016-08-10 11:36:18 +00:00
Lubomír Sedlář
ef99e28849 [test] Only check bootability for images on x86_64 and i386
Current check does not really work for other arches.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-10 13:02:56 +02:00
Lubomír Sedlář
761a0ef40e Improve error messages for gathering packages
When package has incompatible arch for a tree, the same error message
could be raised from two different places. This patch adds more
information to the message, so that it is easier to find the actual
problem.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-10 09:32:39 +02:00
Dennis Gilmore
247149d4e1 Merge #339 Refactor failables, step 1 2016-08-03 07:36:34 +00:00
Lubomír Sedlář
463088d580 Refactor failables
This is a breaking change as big part of current failable_deliverables
options will be ignored.

There is no change for buildinstall and creatiso phase.

Failability for artifacts in other phases is now configured per
artifact. It already works correctly for ostree and ostree_installer
phases (even per-arch). For OSBS phase there is currently only a binary
switch as it does not handle multiple arches yet. When it gains that
support, the option should contain list of non-blocking architectures.

For live images, live media and image build phases each config block can
configure list of failable arches. If the list is not empty, it can
fail. Once we have a way to fail only some arches, the config will not
need to change.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-07-27 13:06:01 +02:00
Lubomír Sedlář
7d0ee41f23 Stop setting release in OSBS phase
Atomic Reactor does not honor this option. In the future we might need
to reintroduce this feature, but given that it does not work in the
current form it is better removed.

Fixes: #348
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-07-27 09:52:06 +02:00
Dennis Gilmore
664c5e54a3 Merge #351 Remove ambiguous imports 2016-07-26 13:54:44 +00:00
Lubomír Sedlář
22e94caf3c [test] Correctly check bootable ISOs
ISO image without MBR and GPT can still be bootable if it has an El
Torito boot catalog. The test phase must accept such images.

This slightly defeats the point of the check: to verify the ISO is
hybrid. However, based on the metadata we have no way to actually tell
if the image is supposed to be hybrid.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-07-22 14:11:06 +02:00
Lubomír Sedlář
d51550da84 Remove ambiguous imports
Make all imports either use full package name starting with `pungi` or
use explicitly relative import. This will avoid issues when importing a
module that shadows another module on PYTHONPATH.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-07-20 13:36:42 +02:00
Lubomír Sedlář
f0dca7687e Merge #347 Remove duplicate definition of find_old_composes. 2016-07-20 08:43:47 +00:00
Dennis Gilmore
9861e80c80 Merge #342 Simplify naming format placeholders 2016-07-01 00:04:17 +00:00
Dennis Gilmore
5ae58b6d46 Merge #345 createrepo: use separate logs for different pkg_type 2016-07-01 00:03:46 +00:00
Ralph Bean
ad44ef4695 Remove duplicate definition of find_old_composes...
This lives now in pungi.util.  It was copied there long ago and all the
places that reference it reference it in that location.

I think this code in pungi.phases.pkgset is unused and can be removed.

The other function moved to the common module.

Signed-off-by: Ralph Bean <rbean@redhat.com>
2016-06-30 16:43:34 -04:00
Qixiang Wan
c32080e7f9 [createrepo] fix 'createrepo_deltas' option
'createrepo_deltas = False' in configuration doesn't work

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-06-30 18:44:25 +08:00
Lubomír Sedlář
a1b86b93ea createrepo: use separate logs for different pkg_type
Instead of overwriting the same log file, make sure we keep all the logs
for debuginfo, source and binary packages.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-06-29 07:46:33 +02:00
Lubomír Sedlář
a72182c817 Simplify naming format placeholders
This patch makes it possible to use different style format placeholders.
Instead of the percent encoding it is now possible to use simple curly
braces.

    %(foo)s -> {foo}

The old format is still available.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-06-28 13:03:36 +02:00
Lubomír Sedlář
11bbbae2ed Treat variants without comps groups as having all of them
If the `<groups></groups>` section is not specified in the variants XML
file, all groups will be used in this variant. The section must be
omitted completely, not just empty. This is (and was) correct according
to the DTD, it just lead to crash before.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-06-28 09:01:30 +02:00
Lubomír Sedlář
8323a735e7 Always generate rpms.json file
Even if gather phase is skipped, it will still generate a rpms.json
metadata file. It will have no payload, but the header structure is
there.

Createrepo phase had to be updated to ignore variants that have no RPMs
listed in metadata.

There is one more fix for writing treeinfo files: if repomd.xml files
are not generated, it will no longer crash.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-06-27 08:50:19 +02:00
Dennis Gilmore
00e11b2f51 4.1.7 release
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2016-06-23 05:14:21 -05:00
Lubomír Sedlář
e1ab519542 [scm] Add logging for exporting local files
This will avoid confusion when the file can not be found, but previous
export from remote location worked well. In such case the log said:

    Exporting from SCM...
    Boom, no files found...

This commit changes it into:

    Exporting from SCM...
    Exporting file from current working dir...
    Boom, no files found...

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-06-23 08:07:23 +02:00
Lubomír Sedlář
8e4b1e278c [extra-files] Only copy files when there is a config
Instead of running the copy function for all variant.arch pairs
unconditionally, only do it if there is something to do. This makes the
log more understandable.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-06-23 07:25:52 +02:00