Commit Graph

332 Commits

Author SHA1 Message Date
Lubomír Sedlář
2fa1f09827 createrepo: Allow passing arbitrary arguments
This could be used to enable zchunk generation, which can require up to
4 different options. Instead of hardcoding every single one, let's just
allow more direct access to the executed command.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-08 16:15:51 +02:00
Lubomír Sedlář
add9835b56 Allow removing non-alnum chars from volid
This mimics similar change in lorax.

JIRA: RCM-36970
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-03 07:57:09 +02:00
Lubomír Sedlář
878eaeaaf6 Stop importing PDCClient
We no longer use it anywhere, so drop the import.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-16 15:52:42 +02:00
Lubomír Sedlář
c8bd967a57 4.1.26 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-16 10:37:23 +02:00
Lubomír Sedlář
ba0193ca28 gather: Add a hybrid depsolver backend
This patch adds a new gather method called `hybrid`, which uses a `fus`
binary, which must exist somewhere on the `$PATH`. It will call it
multiple times to add multilib packages.

The solver can handle packages, modules and comps groups as input.
However comps groups are never passed in. Pungi will expand it to a list
of packages to avoid issues with comps handling in fus. It ignores
optional packages, and if the group mentions a package that does not
exist, nothing else from the group is included.

Multilib is also handled outside of fus. Pungi will run it, parse the
packages from output, determines multilib packages and adds them as
input. Then it runs the solver again. This is done until nothing new is
added. Usually two passes should be enough.

Source packages and debuginfo are added as a final step. All debuginfo
packages from any included source are added. If the source or debuginfo
package is included in any lookaside repo, it will be skipped.

The tool expects to get a platform stream that should be provided for
modules to depend on. Pungi looks into the modules and gets the platform
from there. If there are more requests, an error is raised.

There is some missing functionality and options that are ignored.
Particularly these are:

 * gather_fulltree
 * gather_selfhosting
 * greedy_method

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-07-16 10:21:41 +02:00
Ken Dreyer
a509064696 doc: closing parentheses for require_all_comps_packages
"(the default)" was missing the closing parentheses character.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2018-07-11 09:37:29 -06:00
Sinny Kumari
1759c1ba80 Expand version field during image_build using version_generator
We can specify !VERSION_FROM_VERSION in version field during
image_build to expand it to correct release number without any label
information.

Also implemented !RELEASE_FROM_DATE_RESPIN to provide correct
release number. This helps to keep Atomic Host media files name
produced by image_build during bodhi updates compose run
consistent with nightly run.

Fixes: https://pagure.io/pungi/issue/987
Merges: https://pagure.io/pungi/pull-request/995

Signed-off-by: Sinny Kumari <sinny@redhat.com>
2018-07-09 12:56:00 +02:00
Lubomír Sedlář
38f1a8509e createiso: Break hardlinks by copying files
If a file has multiple hard links, genisoimage will put the wrong number
on the ISO. This patch can work around it by copying hard-linked files
into a temporary staging directory.

JIRA: COMPOSE-2610
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-20 12:40:43 +02:00
Martin Curlej
d8c03f6239 pkgset: Query Koji instead of PDC
PDC is deprecated in upstream. The usecase for getting list of modules
by NS, NSV or NSVC can however be satisfied by querying modules imported
into Koji.

This makes it possible to deprecate PDC configuration.

Merges: https://pagure.io/pungi/pull-request/985
Signed-off-by: Martin Curlej <mcurlej@redhat.com>
2018-06-18 15:17:04 +02:00
Lubomír Sedlář
8b9508b027 Update virtualenv instructions
* Koji can now be installed from PyPI and it will also pull in rpm. This
  needs rpm-devel install on the system. This means we can get kobo from
  PyPI as well.
* Using print function should make it work for python 3 virtualenvs.
* Newer we need kobo in 0.6.0 at least, but newer is fine too.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-01 12:11:55 +02:00
Ondrej Nosek
420b744193 4.1.25 release
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-05-22 14:12:31 +02:00
Lubomír Sedlář
a4bbf475f1 pkgset: Add option to ignore noarch in ExclusiveArch
The `add_noarch` option of `get_valid_arches` is broken and doesn't
really do anything (noarch is always present in the result).

This causes packages that have ExclusiveArch including noarch to
actually not be excluded. They should be.

Changing this globally could have a very big impact. Therefore we can
hide it behind a configuration option so that it's opt-in.

JIRA: COMPOSE-2457
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-21 12:36:16 +02:00
Lubomír Sedlář
b4e746aa71 createrepo: Allow disabling SQLite database
This is an optimization for Yum. DNF does not care at all.

The behaviour is configurable, but the default depends on gather
backend, as that is what users should be using to consume the packages
from the repo.

Fixes: https://pagure.io/pungi/issue/951
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-21 08:13:29 +02:00
Lubomír Sedlář
b2f995d516 Add a phase for creating extra ISOs
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-16 12:20:01 +02:00
Lubomír Sedlář
7c237c2c63 ostree-installer: Allow overwriting buildinstall
Relates: https://pagure.io/pungi/issue/909
Relates: https://pagure.io/pungi/issue/695
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-07 15:22:01 +02:00
Lubomír Sedlář
182a5a399b 4.1.24 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-02 14:16:44 +02:00
Lubomír Sedlář
e419100d5f pkgset: Allow different inheritance for modules
With one compose combining traditional and modular content there might
be different requirements for tag inheritance. This patch adds a new
option that controls whether builds in modular tags should be inherited.
It defaults to False, which is the right option for current MBS
behavior.

JIRA: COMPOSE-2148
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-26 14:42:30 +02:00
Ondrej Nosek
cb3d36be5d Variant as a lookaside - configuration
Relates: COMPOSE-2425

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-04-13 14:14:43 +02:00
Petr Šabata
8238011bdd Update configuration docs with module_defaults_dir
Signed-off-by: Petr Šabata <contyk@redhat.com>
2018-04-11 16:29:18 +02:00
Lubomír Sedlář
6daee968ae docs: Describe comps processing
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-10 08:21:56 +02:00
Lubomír Sedlář
064f922117 docs: Fix typo
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-10 08:21:56 +02:00
Robert Marshall
6454f30714 Add unittest2 and rpmdevools to contributing doc
- Add rpmdevtools and python-unittest2 to the list of packages
  required to set up a testing environment

Signed-off-by: Robert Marshall <rmarshall@redhat.com>
2018-04-06 19:27:22 -04:00
Lubomír Sedlář
d9e2101b08 buildinstall: Add extra repos
A new configuration option is added that allows users to point lorax to
extra repositories. This can be handy if some tools to create the
bootable image are not part of the product itself.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
JIRA: COMPOSE-2253
2018-04-05 07:56:20 +02:00
Ondrej Nosek
f3b5a66614 4.1.23 release
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-03-29 14:11:58 +02:00
Ondrej Nosek
9b46377fb6 Update documentation section 'contributing'
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-03-29 11:23:57 +02:00
Lubomír Sedlář
95bb147015 buildinstall: Add option to disable it
Fixes: https://pagure.io/pungi/issue/854
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-13 15:54:52 +01:00
Ondrej Nosek
660c04368b Move ostree phase and pipelines for running phases
Signed-off-by: Ondrej Nosek <onosek@redhat.com>

Related: https://pagure.io/pungi/issue/778
2018-02-26 13:43:44 +01:00
Lubomír Sedlář
364d7f5229 Support multiple sources in one variant
With this patch the gather_source option is no longer used. Instead, all
sources are always used. If they return at least some input packages,
then a configured method is used and the returned lists of packages from
all sources are merged.

The method used for gathering can be configured for each variant and
gather source separately.

Additional packages are only added to the comps source.

Each gathering step is logged separately. All the logs are preserved for
later inspection.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-02-01 14:31:09 +01:00
Ondrej Nosek
f1f6ca74f1 4.1.22 release
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-01-24 10:11:30 +01:00
Lubomír Sedlář
03293c725b config: Make pkgset_koji_tag optional
There are valid use cases for not specifying this option: specifically a
modular compose will get the tags to use from modules listed in the
variants file.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-01-12 15:50:21 +01:00
Ondrej Nosek
51cd359057 ostree: Add force_new_commit option - test added
Relates: https://pagure.io/pungi/issue/811

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-01-11 15:08:07 +01:00
Colin Walters
161b7f974b ostree: Add force_new_commit option
Followup from discussion in: https://pagure.io/pungi/issue/811

It's likely now that for Fedora Atomic Host we'll use this, to work
around other issues, after we fix the FAW change detection.

Signed-off-by: Colin Walters <walters@verbum.org>
2018-01-11 15:03:36 +01:00
Ondrej Nosek
59c162d46f Remove strace from buildinstall runroot
The package is not really needed.

Relates: https://pagure.io/pungi/issue/799

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2017-12-11 17:36:28 +01:00
Ondrej Nosek
ba3adf9bc4 Merge #818 doc: move "Phases" up, "Contributing" down 2017-12-11 08:54:32 +00:00
Ken Dreyer
6c3245bdee doc: fix "Miscellaneous" spelling in Config section
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2017-12-08 13:46:14 -07:00
Ken Dreyer
d66abc3aea doc: move "Phases" up, "Contributing" down
The "Phases" section breaks down what Pungi does in detail. Place it
towards the top of the documentation table of contents.

The "Contributing" and "Testing" sections are relevant to developers,
not all users, so move them to the end.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2017-12-08 13:43:39 -07:00
Lubomír Sedlář
25907f61ce 4.1.21 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-12-06 13:43:10 +01:00
Lubomír Sedlář
d5a0316877 Use dnf backend for repoclosure on PY3
When yum is not available, it makes no sense to offer it as an option.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-12-06 13:26:43 +01:00
Ondrej Nosek
a6c65e026a add ability to specify ostree ref in OSTREE phase
It allows specify what ref we want this compose to commit to.
New parameter 'ostree_ref' overrides the default value from the treefile json.

Relates: https://pagure.io/pungi/issue/777

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2017-12-04 09:27:19 +01:00
Jan Kaluza
953fb4c54c buildinstall: Allow using external dire for runroot task
A new `buildinstall_topdir` option allows using buildinstall even when
the compose is created on a different volume that Koji is using.

The files are created in this external directory and then copies into
the usual location.

Merges: https://pagure.io/pungi/pull-request/807
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2017-11-30 10:55:52 +01:00
Lubomír Sedlář
b068514471 Add documentation for modular composes
List all available configuration options and go into some detail on how
modules should be specified. This would probably deserver a more
thorough explanation, but it's still bit in a flux.

Fixes: https://pagure.io/pungi/issue/767
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-11-21 08:47:19 +01:00
Ondrej Nosek
91ee1fb854 Remove deprecated options
Options that are currently marked as deprecated do not have any effect
anymore (other than printing warning). We should remove them and update
the message so that we can mark options as deprecated even when they
still work.

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2017-11-16 11:12:34 +01:00
Patrick Uiterwijk
8181c5be48 Turn COMPOSE_ID version generator into DATE_RESPIN
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2017-11-10 11:14:11 +01:00
Lubomír Sedlář
385002fe94 4.1.20 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-11-01 15:01:02 +01:00
Lubomír Sedlář
cdfa3cb45f image-build: Drop suffixes from configuration
The file extension in configuration is only used to tell Pungi which
files from the task results should be downloaded. The user has to get it
right or the phase will fail. Each format has a single valid suffix.

Pungi should not require users to specify the suffix, since it can just
as well just know the right value.

The old configuration will continue working, only the extension will be
ignored.

Fixes: https://pagure.io/pungi/issue/753
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-11-01 12:47:36 +01:00
Ondrej Nosek
e0308a74ff remove remaining hard coded createrepo threads
To be more precise, new createrepo parameter "workers" was customized and
new default value was set. This fixes issue #752.

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2017-10-30 10:37:10 +01:00
Ondrej Nosek
4ff3190935 Update documentation section "Contributing to Pungi".
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2017-10-25 10:28:08 +02:00
Lubomír Sedlář
daf162503c Reject yum gather backend on Python 3
It will not run, and having a nice error message is better than a
cryptic crash.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-24 15:15:31 +02:00
Patrick Uiterwijk
3088df8e60 checks: Use list of release types from productmd
Let's not duplicate the list. Productmd exports a list of valid values,
so Pungi should just pick and use that.

Closes: https://pagure.io/pungi/pull-request/773
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-24 15:05:08 +02:00
Lubomír Sedlář
be39dc3caf buildinstall: Expose template arguments for lorax
This would be useful for modularity. The templates can be added now and
variables set via the existing `lorax_option`.

It's not possible to use custom templates not shipped with lorax, as
passing the path to a random directory is a little bit more tricky.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-23 16:08:09 +02:00
Lubomír Sedlář
68fdef451c pkgset: Allow unsigned packages by empty key
Currently `None` has to be included to allow using unsigned packages.
ODCS has trouble with including non-string value in the list though, so
we can treat empty string the same way (it's not a valid key ID anyway).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-16 13:48:27 +02:00
Lubomír Sedlář
c89f033457 live-images: Honor global settings for target
Instead of the old default value and custom handling introduce new
option `live_images_target` (for consistency with other phases) and use
the usual inheritance rules.

Fixes: https://pagure.io/pungi/issue/749
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-09 13:46:54 +02:00
Dennis Gilmore
b51a639277 Merge #743 Make ostree installer before cloud images 2017-10-06 00:18:46 +00:00
Lubomír Sedlář
9ab3840085 Add directory name for checksum file
Fixes: https://pagure.io/pungi/issue/745
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-05 16:08:27 +02:00
Lubomír Sedlář
dec00fe2f4 createrepo: Allow customizing number of threads
The default is now to use one thread per CPU.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-03 15:21:37 +02:00
Lubomír Sedlář
19436c66b2 Make ostree installer before cloud images
Fixes: https://pagure.io/pungi/issue/742
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-09-20 08:33:40 +02:00
Lubomír Sedlář
44c523339c 4.1.19 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-09-18 13:17:18 +02:00
Lubomír Sedlář
04836cfa9f docs: Mention how input package list are interpreted
It's really just an RPM name, not any random provide or source package
name.

Fixes: https://pagure.io/pungi/issue/725
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-09-18 12:06:03 +02:00
Ralph Bean
5b6e468952 Allow extracting profiling information from pungi-gather.
`pungi-gather` (the tool that underlies both the `pkgset` and `gather`
phases) contains profiling code that will log statistics about how long
different function calls take.  However, pungi-koji did not contain a
way to pass the ``--profiler`` argument to enable this.

This change adds a new configuration option ``gather_profiler`` which,
when set to true, simply passes the argument to `pungi-koji`.  Hopefully
this can help shed some light on what is happening in some of our
longer-running composes.

Merges: https://pagure.io/pungi/pull-request/727
Signed-off-by: Ralph Bean <rbean@redhat.com>
2017-09-04 10:25:11 +02:00
Lubomír Sedlář
e21a27bdc9 createrepo: Allow selecting variants for delta RPMs
The configuration needs to be more granular than a single global option.
With this patch each tree can enable deltas separately.

Fixes: https://pagure.io/pungi/issue/715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-09-04 09:15:20 +02:00
Lubomír Sedlář
2e3a9385a3 config: Allow setting default compose type
The config can change the default compose type. This can still be
overwritten by a CLI argument. A `--production` option is now added to
CLI (because that was the default before).

Fixes: https://pagure.io/pungi/issue/694
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-25 10:07:33 +02:00
Lubomír Sedlář
c14c28a157 docs: Abort update script on error
If building documentation fails, there's no point in trying to update
the repo.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-22 10:28:38 +02:00
Lubomír Sedlář
c9d95f5f6f 4.1.18 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-22 10:22:27 +02:00
Lubomír Sedlář
804a0049f6 ostree: Simplify configuration
It makes no sense to repeat the same configuration for multiple
architectures. Instead we should just list the architectures as another
key in the mapping. There is an option to specify multiple config dicts.

This preserves full backwards compatibility, the old config format is
still accepted.

Fixes: https://pagure.io/pungi/issue/678
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-14 14:53:22 +02:00
Lubomír Sedlář
7e03133c8f docs: Add documentation for scm_dict
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-10 15:32:24 +02:00
Lubomír Sedlář
cef8650c3e gather: Report missing comps packages
When a package mentioned in comps is not available in the package set,
print a warning about this. Additionally there is a config option that
allows to turn this warning into a fatal error.

Fixes: https://pagure.io/pungi/issue/50
Fixes: https://pagure.io/pungi/issue/683
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-09 11:04:14 +02:00
Lubomír Sedlář
e2962dc547 repoclosure: Allow aborting compose when repoclosure fails
Alternatively the call to repoclosure can be turned off. This is
customizable per variant and architecture.

Fixes: https://pagure.io/pungi/issue/676
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-07 15:17:33 +02:00
Qixiang Wan
0a3e5b27bf remove the dependency of rpmUtils
The rpmUtils module is provided yum-utils package, which is only
available for Python 2. There is no replacement for the functionality in
DNF.

There is a proposal to add this functionality to rpm itself, but it's
not really moving forward very much:
https://bugzilla.redhat.com/show_bug.cgi?id=1072972

As a short term solution let's copy the needed parts of rpmUtils.arch
module directly to pungi code base.

Fixes: https://pagure.io/pungi/issue/533
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-07-26 11:23:16 +02:00
Lubomír Sedlář
c0bac63f4d 4.1.17 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-17 09:58:35 +02:00
Lubomír Sedlář
6cf912f555 docs: Convert phases diagram to PNG
Sphinx and Latex are unable to include SVG apparently:
https://github.com/sphinx-doc/sphinx/issues/1907

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-17 09:58:35 +02:00
Lubomír Sedlář
543c184e0d docs: Minor improvements to documentation
* add description for `gather_method`
 * fix typo in `createrepo_deltas`

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-13 10:03:54 +02:00
Qixiang Wan
18bd37ff2c pkgset: Allow populating packages from multiple koji tags
One of the use cases is https://pagure.io/odcs/issue/7.

Merges: #660
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-06-29 08:27:34 +02:00
Lubomír Sedlář
65bc6969e2 docs: Add a basic info about gathering packages
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-21 09:55:29 +02:00
Lubomír Sedlář
3e64238fca docs: Kobo can be installed via pip now
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-20 11:11:16 +02:00
Dennis Gilmore
72081a3a35 Merge #649 docs: Add overview of what each phase does 2017-06-19 17:22:43 +00:00
Lubomír Sedlář
9ab3dc89b2 docs: Add overview of what each phase does
Fixes: #520
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-19 10:58:01 +02:00
Lubomír Sedlář
2a8d7f8843 docs: Document config file format
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-19 10:51:31 +02:00
Lubomír Sedlář
bef573d222 docs: Move logo to _static subdir
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-19 10:51:23 +02:00
Lubomír Sedlář
d3dd7aa7ce 4.1.16 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-09 13:12:19 +02:00
Lubomír Sedlář
e549c732a4 4.1.15 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-05-05 08:16:11 +02:00
Lubomír Sedlář
68a1051f14 docs: Add a logo on the About page
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-26 10:46:57 +02:00
Lubomír Sedlář
a2c3274585 docs: Document origin of the name
Fixes: #600
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-26 10:46:57 +02:00
Lubomír Sedlář
bab2a125d4 osbs: Make git_branch required option
In order to avoid conflicting tags, OSBS allows only one build for a
repo/branch pair at the same time. To avoid race conditions, we should
make sure we always pass in the branch. This commit makes it a required
option.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-24 10:05:25 +02:00
Lubomír Sedlář
21952955ad docs: Update createrepo_checksum allowed values
Relates: #591
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-20 15:22:58 +02:00
Lubomír Sedlář
c293a1e147 extra-files: Allow configuring used checksums
Instead of adding a new config option, we can just reuse the existing
`media_checksums` value. If the value is good for image checksums, it
should work for extra files as well.

Relates: #591
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-20 09:28:09 +02:00
Lubomír Sedlář
532b6b1fbc doc: Document options for media checksums
Pungi does not really care about the values as long as hashlib supports
them.

Relates: #591
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-20 09:28:09 +02:00
Lubomír Sedlář
f6d07c1651 Be explicit about generating release for images
The config now uses similar logic what previous commit did for OSTree.
Also we should report error when an unknown generator is used.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-13 10:17:59 +02:00
Lubomír Sedlář
0f4b6b1947 docs: Add examples for generated versions
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-13 10:17:59 +02:00
Qixiang Wan
2f5d6d7dcd unify repo and repo_from options
Config option 'repo' and 'repo_from' are used in several phases, merge
them with one option 'repo'. 'append' in schema is used for appending
the values from deprecated options to 'repo', so it won't break on any
existing config files that have the old options of 'repo_from' and
'source_repo_from' (which is an alias of 'repo_from').

And 'repo' schema is updated to support repo dict as the value or an
item in the values, a repo dict is just a dict contains repo options,
'baseurl' is required in the dict, like:

{"baseurl": "http://example.com/url/to/repo"}

or:

{"baseurl": "Serer"}

currently this is used in ostree phase to support extra repo options
like:

{"baseurl": "Server", "exclude": "systemd-container"}

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-29 10:12:32 +08:00
Qixiang Wan
0f508e2228 move translate_path from paths.py to util.py
So translate_path can be used in util.py, or it will result in
circular import error.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-29 10:12:32 +08:00
Lubomír Sedlář
9784961568 4.1.14 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-27 18:04:26 +02:00
Lubomír Sedlář
5dd2b3947d Rename option to switch gather backend
Calling it gather_backend is similar to repoclosure_backend we already
have. It's also more obvious what it does.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:50 +01:00
Lubomír Sedlář
76cf4a7540 Update to use python-multilib
We also rename the old multilib module used by dnf code to multilib_yum
to make it clear what is imported where.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:50 +01:00
Lubomír Sedlář
77673e1eac Improve documentation
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:50 +01:00
Qixiang Wan
ee21663c8b default createrepo_checksum to sha256
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-14 13:15:43 +08:00
Lubomír Sedlář
9041ccccc4 Remove indices from documentation
They are not generated as we don't actually document code. The search
page is removed as well (because it does not seem to work).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Fixes: #462
2017-03-09 12:47:12 +01:00
Qixiang Wan
2ae8710934 consolidate repo option names
live_images:            additional_repos        -> repo
ostree:                 source_repo_from        -> repo_from
                        extra_source_repos      -> repo
ostree_installer:       source_repo_from        -> repo_from

With the change, the phases have consolidate option names for variant
repos and external repos.

Old option names will continue to work, old names will be converted
to new names after validation automatically if new options are not
specified in config.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-06 09:26:42 +08:00
Lubomír Sedlář
345308464f 4.1.13 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-03 14:34:45 +01:00
Lubomír Sedlář
4ea1916a87 Enable customizing runroot task weight
For different cases where runroot is used it's now possible to set
custom weight. The usecase for this is to avoid one builder taking too
many tasks. Especially buildinstall is quite resource intensive, so one
builder taking multiple tasks at the same time leads to very slow
compose time.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-23 12:44:27 +01:00
Lubomír Sedlář
e00776a413 multilib: Remove FileMultilibMethod class
In order to do something meaningful, the class needs to be instantiated
with arguments pointing the blacklist and whitelist.

The `file` multilib method used via `pungi-koji` or `pungi` directly has
no way to pass those in.

The only way this class can be useful would be if someone actually
imported the class directly in their own code. Pungi is not meant to be
used as a library though, so this is not really a supported use case.

Not to mention that the `select` method always returned `False`.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-10 12:32:45 +01:00
Lubomír Sedlář
e3fe67be53 repoclosure: add option to use dnf backend
This adds a new option repoclosure_backend that changes what tool is
used for repoclosure.

Checking build dependencies is currently not supported, as `dnf` does
not have the corresponding option.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-26 13:39:12 +01:00
Lubomír Sedlář
4f011fbd45 Add documentation and example for greedy_method
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-19 09:24:54 +01:00
Lubomír Sedlář
d3e701e10f 4.1.12 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-18 08:07:42 +01:00
Qixiang Wan
a9b275f13b osbs: optionally check GPG signatures
If gpgkey option is defined in config, set gpgcheck=1 and set
gpgkey=<value> in variant repo files.

Fixes: #487
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-01-17 13:08:58 +08:00
Qixiang Wan
5d241d316a ostree-installer: Allow multiple repos in ostree installer
Add new key 'repo' to allow specifying multiple repos as the source
repositories. And change 'source_repo_from' to allow specifying multiple
vairant names to use variant repos as source repositories.

Doc of 'source_repo_from' is updated to not mention URL is supported,
though we still support that in code. User should add url of repos in
'repo' key instead of 'source_repo_from'.

Fixes: #508

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-01-13 16:57:30 +08:00
Lubomír Sedlář
ce218288db osbs: Add better example to documentation
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-05 09:32:22 +01:00
Dennis Gilmore
baca306edf Merge #495 osbs: Enable specifying extra repos 2017-01-04 14:44:13 +00:00
Lubomír Sedlář
814bf4484b osbs: Enable specifying extra repos
The same way live_media and image_build accept additional external repos
or variants list, there is now a `repo` and `repo_from` configuration
key to add these.

Fixes: #486
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-04 09:47:19 +01:00
Lubomír Sedlář
17a5f2841c Update documentation
Also remove the TODO comment from live images phase: the appliances are
already submitted one task per single arch, so this change is not
necessary.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-14 10:14:30 +01:00
Lubomír Sedlář
d3cad4795c metadata: Allow creating internal releases
The internal flag in productmd is meant to indicate that a compose is
not meant for publishing. This is potentially useful to allow filtering
in PDC or similar service.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-12-06 12:49:33 +01:00
Colin Walters
7bcbe30fd2 Expose lorax's --rootfs-size argument
This is going to be necessary for reworking the Atomic Host ISO,
see: https://pagure.io/fedora-lorax-templates/pull-request/6

Signed-off-by: Colin Walters <walters@verbum.org>
2016-11-17 10:17:39 -05:00
Dennis Gilmore
13c1c4da69 4.1.11 release
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2016-11-15 15:12:02 -06:00
Qixiang Wan
68e121e421 [ostree] Allow extra repos to get packages for composing OSTree repository
Sometimes addtional repos are required to get necessary packages for
composing OSTree repository. For example, RHEL doesn't have an 'Everyting'
variant, so composing OSTree repository from any of the RHEL variants
won't work, addtional source repos need to be enabled to achieve that.
The new option "extra_source_repos" enable the ability of allowing extra
source repos.

And a new option 'keep_original_sources' is introduced to keep the
original repos found in tree config file, if this is enabled, Pungi
will not remove the existing source repos from the tree config file,
just add new repos of "source_repo_from" + "extra_source_repos" to
the existing repos.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-11-15 18:56:06 +08:00
Lubomír Sedlář
4427769f6a Update ostree phase documentation
The message and the new option were not documented.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-04 09:35:08 +01:00
Qixiang Wan
fa3c5aff63 [ostree] Allow adding versioning metadata
Added new option '--version' to pungi-make-ostree, and this can be
enabled in ostree settings with 'version'. The version string will be
added as versioning metadata if this is specified.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-11-04 16:07:26 +08:00
Qixiang Wan
06ba3b8551 [ostree] New option to enable generating ostree summary file
Added new option '--update-summary' to pungi-make-ostree, and this can
be enabled in ostree settings with 'update_summary'. A summary file will
be generated (or re-generated if it was presented in an existing ostree
repo) when it is enabled.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-11-03 21:02:05 +08:00
Lubomír Sedlář
74aa41f8bd Send notification when compose fails to start
This is tricky as this early in the process we don't know the compose
ID. The new message gives the full command line that was called.

Relates: #439
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-20 09:39:18 +02:00
Lubomír Sedlář
fd08efaa03 doc: Fix a typo in progress notification example
Fixes: #435
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-19 15:04:21 +02:00
Dennis Gilmore
b2b5dd919b 4.1.10 release
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2016-10-08 12:15:32 -05:00
Lubomír Sedlář
cc559b12f4 ostree-installer: Use ostree as type in filename
Instead of overloading the `dvd` value, use new value `ostree` that can
be overridden by a config change.

Fixes: https://pagure.io/pungi/issue/418
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-06 08:41:14 +02:00
Lubomír Sedlář
160df7f89a Update contributing guide
The instructions for setting up virtualenv were out dated.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-23 10:19:34 +02:00
Lubomír Sedlář
a042906717 init: Remove keep_original_comps option
The same information can be inferred from definitions in variants.xml:
if the variant has no groups defined, we include packages from all
groups. By the same logic we can also include all groups in the comps
file.

The config validation is updated to give a hint on how to remove the
option from the configuration.

Relates: #29
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-22 13:35:28 +02:00
Lubomír Sedlář
9a3b0f089e 4.1.9 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-21 14:07:21 +02:00
Lubomír Sedlář
bf46048cbf createrepo: Add option to use xz compression
This patch adds a new config option createrepo_use_xz, which when set to
true will cause createrepo to compress sqlite databases with xz. The
default setting is False.

Fixes: #387
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-12 11:10:43 +02:00
Jeremy Cline
ee1ee0467b extra-files: Write a metadata file enumerating extra files
Introduces a new metadata file to track arbitrary files added during the
extra-files phase. This file is placed in the root of each tree and is
called ``extra_files.json``. It is a JSON file containing a single
object, which contains a "header" key with an object describing the
metadata, and a "data" key, which is an array of objects, where each
object represents a file. Each object contains the "file", "checksums",
and "size" keys. "file" is the relative path from the tree root to the
extra file. "checksums" is an object containing one or more checksums,
where the key is the digest type and the value of that key is the hex
digest. Finally, the size is the size of the file in bytes.

For example:
{
  "header": {"version": "1.0},
  "data": [
    {
      "file": "GPL",
      "checksums": {
        "sha256": "8177f97513213526df2cf6184d8ff986c675afb514d4e68a404010521b880643"
      },
      "size": 18092
    },
    {
      "file": "release-notes/notes.html",
      "checksums": {
        "sha256": "82b1ba8db522aadf101dca6404235fba179e559b95ea24ff39ee1e5d9a53bdcb"
      },
      "size": 1120
    }
  ]
}

Signed-off-by: Jeremy Cline <jeremy@jcline.org>
Fixes: #295
2016-09-07 13:02:48 +02:00
Dennis Gilmore
6aeab9ee9d Merge #381 Automatically generate missing image version 2016-09-06 15:19:55 +00:00
Lubomír Sedlář
223a015898 Automatically generate missing image version
If the configuration does not specify version for images or live media,
Pungi will create a default value based on `release_version`. If label
is used for the compose, the milestone from it will be appended to the
version (unless it's RC).

This change is backwards compatible: nothing changes when version is set
in configuration. If the version was missing before, building the
artifacts would fail. With this patch, default values will be supplied.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-01 15:13:04 +02:00
Lubomír Sedlář
f9a6c8418f Add JSON Schema for configuration
The schema is written in Python to reduce duplication. When
configuration is loaded, the validation checks if it's correct and fills
in default values.

There is a custom extension to the schema to report deprecated options.

The config dependencies are implemented as a separate pass. While it's
technically possible to express the dependencies in the schema itself,
the error messages are not very helpful and it makes the schema much
harder to read.

Phases no longer define `config_options`. New options should be added to
the schema. Since the default values are populated automatically during
validation, there is no need to duplicate them into the code.

The `pungi-config-validate` script is updated to use the schema and
report errors even for deeply nested fields.

The dependencies are updated: pungi now depends on `python-jsonschema`
(which is already available in Fedora).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-01 10:56:15 +02:00
Lubomír Sedlář
f507b14954 Update doc about generating release value
It is taken from compose label if provided, and only then falls back to
date, type and respin.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-26 10:05:25 +02:00
Lubomír Sedlář
d90dba9312 doc: Fix example for image_build
The example configuration can now be copy-pasted into real configuration
file and will parse without any issues.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-25 14:21:03 +02:00
Dennis Gilmore
fc23ca5fdf 4.1.8 release
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2016-08-10 07:21:40 -05:00
Dennis Gilmore
247149d4e1 Merge #339 Refactor failables, step 1 2016-08-03 07:36:34 +00:00
Lubomír Sedlář
463088d580 Refactor failables
This is a breaking change as big part of current failable_deliverables
options will be ignored.

There is no change for buildinstall and creatiso phase.

Failability for artifacts in other phases is now configured per
artifact. It already works correctly for ostree and ostree_installer
phases (even per-arch). For OSBS phase there is currently only a binary
switch as it does not handle multiple arches yet. When it gains that
support, the option should contain list of non-blocking architectures.

For live images, live media and image build phases each config block can
configure list of failable arches. If the list is not empty, it can
fail. Once we have a way to fail only some arches, the config will not
need to change.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-07-27 13:06:01 +02:00
Lubomír Sedlář
7d0ee41f23 Stop setting release in OSBS phase
Atomic Reactor does not honor this option. In the future we might need
to reintroduce this feature, but given that it does not work in the
current form it is better removed.

Fixes: #348
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-07-27 09:52:06 +02:00
Dennis Gilmore
00e11b2f51 4.1.7 release
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2016-06-23 05:14:21 -05:00
Lubomír Sedlář
b4fc97be03 [osbs] Add a phase to build images in OSBS
It will take RPM repo from a variant in this compose and a Dockerfile
from configured git and use it to build an image.

The build images are uploaded to some a Docker registry by OSBS and are
not directly part of compose (because there is no export function).
There is a new metadata file `osbs.json` that has some information that
can be used to find the image.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-06-22 10:02:03 +02:00
Lubomír Sedlář
b62b468ccf [createiso] Enable customizing media reserve
Add a documented and tested config options for setting ISO parameters
instead of hardcoding magic values.

Fixes: #256
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-06-02 08:34:49 +02:00
Lubomír Sedlář
5cc7dc204f Add missing documentation
There a couple config options that are not even mentioned in the
documentation. This patch adds them with a basic introduction of what
they do.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-31 13:34:04 +02:00
Dennis Gilmore
8de7d685d9 4.1.6 release
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2016-05-24 16:36:07 -05:00
Lubomír Sedlář
c8341e1806 [ostree-installer] Allow using external repos as source
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-24 10:06:06 +02:00
Lubomír Sedlář
385f0e950c Add type to base product for layered releases
Default value is "ga", can be overwritten by `base_product_type` config
option.

Fixes: #305
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-23 14:12:23 +02:00
Dennis Gilmore
00f2e24bce 4.1.5 release
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2016-05-16 20:56:48 -05:00
Dennis Gilmore
d8272bd0fa 4.1.4 release
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2016-04-29 15:36:41 -05:00
Lubomír Sedlář
084994538b Add common global settings for images
The live_images, live_media and image_build phases have same options
that need to be specified on each image. This leads to a lot of
duplication in the config file. This patch adds global settings and
phase-level settings that allow to significantly reduce duplication.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-18 18:11:29 +02:00
Lubomír Sedlář
e6079e4c85 [ostree-installer] Drop filename setting
This should just use the global default format. Other phases don't have
their own setting either.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-14 14:45:36 +02:00
Lubomír Sedlář
a83a34fd40 [ostree-installer] Fix example documentation
There is no `compose_date`, the field is called just `date`.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-13 15:12:28 +02:00
Lubomír Sedlář
a696eb4f3a [ostree-installer] Clone repo with templates
We can't assume the templates will just be available. This patch adds a
configuration option to point to the git repo with them. It will be
cloned at compose box and relative paths to templates will be made
absolute respective to this clone.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-11 10:25:10 +02:00
Dennis Gilmore
4f76425c9f 4.1.3 release
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2016-04-08 07:34:57 -05:00
Dennis Gilmore
36e7279a5e 4.1.2 release
Signed-off-by: Dennis Gilmore <dennis@ra.ausil.us>
2016-04-06 17:44:40 -05:00
Lubomír Sedlář
ed12a64a84 [ostree] Enable marking ostree phase as failable
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-04-06 16:12:29 +02:00