Commit Graph

174 Commits

Author SHA1 Message Date
Ralph Bean
5b6e468952 Allow extracting profiling information from pungi-gather.
`pungi-gather` (the tool that underlies both the `pkgset` and `gather`
phases) contains profiling code that will log statistics about how long
different function calls take.  However, pungi-koji did not contain a
way to pass the ``--profiler`` argument to enable this.

This change adds a new configuration option ``gather_profiler`` which,
when set to true, simply passes the argument to `pungi-koji`.  Hopefully
this can help shed some light on what is happening in some of our
longer-running composes.

Merges: https://pagure.io/pungi/pull-request/727
Signed-off-by: Ralph Bean <rbean@redhat.com>
2017-09-04 10:25:11 +02:00
Lubomír Sedlář
e21a27bdc9 createrepo: Allow selecting variants for delta RPMs
The configuration needs to be more granular than a single global option.
With this patch each tree can enable deltas separately.

Fixes: https://pagure.io/pungi/issue/715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-09-04 09:15:20 +02:00
Lubomír Sedlář
2e3a9385a3 config: Allow setting default compose type
The config can change the default compose type. This can still be
overwritten by a CLI argument. A `--production` option is now added to
CLI (because that was the default before).

Fixes: https://pagure.io/pungi/issue/694
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-25 10:07:33 +02:00
Lubomír Sedlář
c14c28a157 docs: Abort update script on error
If building documentation fails, there's no point in trying to update
the repo.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-22 10:28:38 +02:00
Lubomír Sedlář
c9d95f5f6f 4.1.18 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-22 10:22:27 +02:00
Lubomír Sedlář
804a0049f6 ostree: Simplify configuration
It makes no sense to repeat the same configuration for multiple
architectures. Instead we should just list the architectures as another
key in the mapping. There is an option to specify multiple config dicts.

This preserves full backwards compatibility, the old config format is
still accepted.

Fixes: https://pagure.io/pungi/issue/678
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-14 14:53:22 +02:00
Lubomír Sedlář
7e03133c8f docs: Add documentation for scm_dict
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-10 15:32:24 +02:00
Lubomír Sedlář
cef8650c3e gather: Report missing comps packages
When a package mentioned in comps is not available in the package set,
print a warning about this. Additionally there is a config option that
allows to turn this warning into a fatal error.

Fixes: https://pagure.io/pungi/issue/50
Fixes: https://pagure.io/pungi/issue/683
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-09 11:04:14 +02:00
Lubomír Sedlář
e2962dc547 repoclosure: Allow aborting compose when repoclosure fails
Alternatively the call to repoclosure can be turned off. This is
customizable per variant and architecture.

Fixes: https://pagure.io/pungi/issue/676
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-07 15:17:33 +02:00
Qixiang Wan
0a3e5b27bf remove the dependency of rpmUtils
The rpmUtils module is provided yum-utils package, which is only
available for Python 2. There is no replacement for the functionality in
DNF.

There is a proposal to add this functionality to rpm itself, but it's
not really moving forward very much:
https://bugzilla.redhat.com/show_bug.cgi?id=1072972

As a short term solution let's copy the needed parts of rpmUtils.arch
module directly to pungi code base.

Fixes: https://pagure.io/pungi/issue/533
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-07-26 11:23:16 +02:00
Lubomír Sedlář
c0bac63f4d 4.1.17 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-17 09:58:35 +02:00
Lubomír Sedlář
6cf912f555 docs: Convert phases diagram to PNG
Sphinx and Latex are unable to include SVG apparently:
https://github.com/sphinx-doc/sphinx/issues/1907

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-17 09:58:35 +02:00
Lubomír Sedlář
543c184e0d docs: Minor improvements to documentation
* add description for `gather_method`
 * fix typo in `createrepo_deltas`

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-13 10:03:54 +02:00
Qixiang Wan
18bd37ff2c pkgset: Allow populating packages from multiple koji tags
One of the use cases is https://pagure.io/odcs/issue/7.

Merges: #660
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-06-29 08:27:34 +02:00
Lubomír Sedlář
65bc6969e2 docs: Add a basic info about gathering packages
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-21 09:55:29 +02:00
Lubomír Sedlář
3e64238fca docs: Kobo can be installed via pip now
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-20 11:11:16 +02:00
Dennis Gilmore
72081a3a35 Merge #649 docs: Add overview of what each phase does 2017-06-19 17:22:43 +00:00
Lubomír Sedlář
9ab3dc89b2 docs: Add overview of what each phase does
Fixes: #520
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-19 10:58:01 +02:00
Lubomír Sedlář
2a8d7f8843 docs: Document config file format
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-19 10:51:31 +02:00
Lubomír Sedlář
bef573d222 docs: Move logo to _static subdir
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-19 10:51:23 +02:00
Lubomír Sedlář
d3dd7aa7ce 4.1.16 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-09 13:12:19 +02:00
Lubomír Sedlář
e549c732a4 4.1.15 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-05-05 08:16:11 +02:00
Lubomír Sedlář
68a1051f14 docs: Add a logo on the About page
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-26 10:46:57 +02:00
Lubomír Sedlář
a2c3274585 docs: Document origin of the name
Fixes: #600
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-26 10:46:57 +02:00
Lubomír Sedlář
bab2a125d4 osbs: Make git_branch required option
In order to avoid conflicting tags, OSBS allows only one build for a
repo/branch pair at the same time. To avoid race conditions, we should
make sure we always pass in the branch. This commit makes it a required
option.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-24 10:05:25 +02:00
Lubomír Sedlář
21952955ad docs: Update createrepo_checksum allowed values
Relates: #591
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-20 15:22:58 +02:00
Lubomír Sedlář
c293a1e147 extra-files: Allow configuring used checksums
Instead of adding a new config option, we can just reuse the existing
`media_checksums` value. If the value is good for image checksums, it
should work for extra files as well.

Relates: #591
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-20 09:28:09 +02:00
Lubomír Sedlář
532b6b1fbc doc: Document options for media checksums
Pungi does not really care about the values as long as hashlib supports
them.

Relates: #591
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-20 09:28:09 +02:00
Lubomír Sedlář
f6d07c1651 Be explicit about generating release for images
The config now uses similar logic what previous commit did for OSTree.
Also we should report error when an unknown generator is used.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-13 10:17:59 +02:00
Lubomír Sedlář
0f4b6b1947 docs: Add examples for generated versions
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-13 10:17:59 +02:00
Qixiang Wan
2f5d6d7dcd unify repo and repo_from options
Config option 'repo' and 'repo_from' are used in several phases, merge
them with one option 'repo'. 'append' in schema is used for appending
the values from deprecated options to 'repo', so it won't break on any
existing config files that have the old options of 'repo_from' and
'source_repo_from' (which is an alias of 'repo_from').

And 'repo' schema is updated to support repo dict as the value or an
item in the values, a repo dict is just a dict contains repo options,
'baseurl' is required in the dict, like:

{"baseurl": "http://example.com/url/to/repo"}

or:

{"baseurl": "Serer"}

currently this is used in ostree phase to support extra repo options
like:

{"baseurl": "Server", "exclude": "systemd-container"}

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-29 10:12:32 +08:00
Qixiang Wan
0f508e2228 move translate_path from paths.py to util.py
So translate_path can be used in util.py, or it will result in
circular import error.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-29 10:12:32 +08:00
Lubomír Sedlář
9784961568 4.1.14 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-27 18:04:26 +02:00
Lubomír Sedlář
5dd2b3947d Rename option to switch gather backend
Calling it gather_backend is similar to repoclosure_backend we already
have. It's also more obvious what it does.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:50 +01:00
Lubomír Sedlář
76cf4a7540 Update to use python-multilib
We also rename the old multilib module used by dnf code to multilib_yum
to make it clear what is imported where.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:50 +01:00
Lubomír Sedlář
77673e1eac Improve documentation
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-24 09:24:50 +01:00
Qixiang Wan
ee21663c8b default createrepo_checksum to sha256
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-14 13:15:43 +08:00
Lubomír Sedlář
9041ccccc4 Remove indices from documentation
They are not generated as we don't actually document code. The search
page is removed as well (because it does not seem to work).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Fixes: #462
2017-03-09 12:47:12 +01:00
Qixiang Wan
2ae8710934 consolidate repo option names
live_images:            additional_repos        -> repo
ostree:                 source_repo_from        -> repo_from
                        extra_source_repos      -> repo
ostree_installer:       source_repo_from        -> repo_from

With the change, the phases have consolidate option names for variant
repos and external repos.

Old option names will continue to work, old names will be converted
to new names after validation automatically if new options are not
specified in config.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-06 09:26:42 +08:00
Lubomír Sedlář
345308464f 4.1.13 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-03 14:34:45 +01:00
Lubomír Sedlář
4ea1916a87 Enable customizing runroot task weight
For different cases where runroot is used it's now possible to set
custom weight. The usecase for this is to avoid one builder taking too
many tasks. Especially buildinstall is quite resource intensive, so one
builder taking multiple tasks at the same time leads to very slow
compose time.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-23 12:44:27 +01:00
Lubomír Sedlář
e00776a413 multilib: Remove FileMultilibMethod class
In order to do something meaningful, the class needs to be instantiated
with arguments pointing the blacklist and whitelist.

The `file` multilib method used via `pungi-koji` or `pungi` directly has
no way to pass those in.

The only way this class can be useful would be if someone actually
imported the class directly in their own code. Pungi is not meant to be
used as a library though, so this is not really a supported use case.

Not to mention that the `select` method always returned `False`.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-10 12:32:45 +01:00
Lubomír Sedlář
e3fe67be53 repoclosure: add option to use dnf backend
This adds a new option repoclosure_backend that changes what tool is
used for repoclosure.

Checking build dependencies is currently not supported, as `dnf` does
not have the corresponding option.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-26 13:39:12 +01:00
Lubomír Sedlář
4f011fbd45 Add documentation and example for greedy_method
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-19 09:24:54 +01:00
Lubomír Sedlář
d3e701e10f 4.1.12 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-18 08:07:42 +01:00
Qixiang Wan
a9b275f13b osbs: optionally check GPG signatures
If gpgkey option is defined in config, set gpgcheck=1 and set
gpgkey=<value> in variant repo files.

Fixes: #487
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-01-17 13:08:58 +08:00
Qixiang Wan
5d241d316a ostree-installer: Allow multiple repos in ostree installer
Add new key 'repo' to allow specifying multiple repos as the source
repositories. And change 'source_repo_from' to allow specifying multiple
vairant names to use variant repos as source repositories.

Doc of 'source_repo_from' is updated to not mention URL is supported,
though we still support that in code. User should add url of repos in
'repo' key instead of 'source_repo_from'.

Fixes: #508

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-01-13 16:57:30 +08:00
Lubomír Sedlář
ce218288db osbs: Add better example to documentation
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-05 09:32:22 +01:00
Dennis Gilmore
baca306edf Merge #495 osbs: Enable specifying extra repos 2017-01-04 14:44:13 +00:00
Lubomír Sedlář
814bf4484b osbs: Enable specifying extra repos
The same way live_media and image_build accept additional external repos
or variants list, there is now a `repo` and `repo_from` configuration
key to add these.

Fixes: #486
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-04 09:47:19 +01:00