Commit Graph

2301 Commits

Author SHA1 Message Date
Lubomír Sedlář
3efcfede6a image-build: Support repo/install_tree as path
If a repo or install tree is specified as an absolute path on the local
filesystem, we should either translate it using the configured mappings,
or if no mapping matches, it should be return unchanged. A variant name
can not start with a slash, so attempting that translation does not make
much sense.

JIRA: COMPOSE-3290
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-25 08:18:47 +01:00
Lubomír Sedlář
8c82cfc1c7 osbs: Remove format requirement for registry
Pungi does not really care if it's an object, list or string. Anything
should be allowed.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-22 15:10:45 +01:00
Lubomír Sedlář
f33973ee65 Make the Apple/HFS compatibility configurable
The default is the original behaviour. On F30+ a new option should be
added to config to make it work.

Over time as users move to this option (which requires a new enough
version of lorax), the default should be switched and then the option
removed.

Resolves: https://pagure.io/pungi/issue/1126
Merges: https://pagure.io/pungi/pull-request/1128
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-21 16:11:01 +01:00
Dan Horák
071d11a883 update iso creation for ppc64le
Don't produce ISO image with Apple/HFS compatibility stuff on ppc64le.

Signed-off-by: Dan Horák <dan@danny.cz>
2019-02-21 16:11:01 +01:00
Lubomír Sedlář
59c9c150ef orchestrator: Use prefix for config substitutions
This will avoid possible conflicts. The names for parts are chosen by
users, and if we decide to add more options in the future, this can
become handy.

JIRA: COMPOSE-3285
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-21 09:42:14 +01:00
Ken Dreyer
5a251ff24c README: add link to documentation
Make it easier for README users to discover the Pungi docs on Pagure.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2019-02-18 14:36:48 -07:00
Lubomír Sedlář
e21c49d9e2 4.1.33 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-13 10:37:48 +01:00
Lubomír Sedlář
6c6d4759f5 isos: Check maximum expected size
This patch allows the configuration to express maximum expected size for
ISOs created in createiso and extra_isos phases. If the image is larger
than this limit, a warning is emitted in test phase. The compose itself
is not affected in any way.

JIRA: COMPOSE-2824
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-12 15:15:30 +01:00
Lubomír Sedlář
7693e562b1 osbs: Process data about pushing images to registries
This patch does not do any actual pushing. It will only extract data
about push targets from the main configuration and store it together
with exact Koji NVR in a well-defined location, and also send the data
to message bus for another service to handle.

JIRA: COMPOSE-3228
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-12 15:12:50 +01:00
Lubomír Sedlář
23bf01bb45 hybrid: Apply filters to debuginfo and source packages
For binary packages the filters are handled at the depsolver level.
However sources and debuginfo is added later in the process, so the
filters have to be explicitly applied.

JIRA: COMPOSE-3114
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-12 12:20:46 +01:00
Lubomír Sedlář
e573246a2a hybrid: Get platform from lookaside repos
If the hybrid solver is used in a situation where there are modules in
lookaside repo, but not in the compose itself, it will fail to detect
any platform. Since we are already opening the module repodata, we can
retrieve platforms from all modules in there as well.

If there are conflicts (e.g. multiple modules depending on different
platforms), an error will be reported.

JIRA: COMPOSE-3277
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-08 14:41:56 +01:00
Jan Kaluza
c3aa297d8c Return RPMs added to -devel module in GatherSourceModule.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2019-02-08 14:24:16 +01:00
Jan Kaluza
31bafa29c5 Allow setting wildcard as a module name in variants to include all the modules.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2019-02-07 13:59:57 +01:00
Lubomír Sedlář
940a581bd9 gather: Link files in order for dependant variants
If there are variants that depend on another, they should be processed
in order to make sure packages from the base variant are linked first.
That way the srpm cache is populated and any package in layered variant
but with source in base will have access to correct epoch information.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-04 08:24:22 +01:00
Lubomír Sedlář
8065239e04 buildinstall: Pick correct config with rootfs_size
If there are multiple matching configuration blocks, we should take the
value that is defined in the last one, but only if it actually is there.
That's how it works for other options.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-01 14:24:20 +01:00
Lubomír Sedlář
f59034b22d hybrid: Add packages from prepopulate to input
This is really an oversight. The list of packages from prepopulate list
should be treated the same as if the packagees were coming from comps or
additional packages.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-01-29 16:16:44 +01:00
Lubomír Sedlář
da1ea83561 ostree_installer: Pass --buildarch to lorax
This should tell lorax what arch to use and avoids fragile detection
based on contents of source repo.

It uses the same logic buildinstall phase uses to get the buildarch.

Related: https://pagure.io/teamsilverblue/issue/67
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-01-28 13:41:49 +01:00
Lubomír Sedlář
f92c71683f orchestrator: Add missing function arguments
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-01-21 09:13:03 +01:00
Lubomír Sedlář
7320bf4943 orchestrator: Compatibility with Python 2.6
This is why we can't have nice things.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-01-14 10:57:05 +01:00
Frédéric Pierret (fepitre)
86314fdc83 pungi-legacy: expose lorax's --rootfs-size argument
Merges: https://pagure.io/pungi/pull-request/1112
Fixes: https://pagure.io/pungi/issue/1107
Signed-off-by: Frédéric Pierret (fepitre) <frederic.pierret@qubes-os.org>
2019-01-14 10:41:37 +01:00
Miro Hrončok
3fe4beb20c Only require enum34 on Legacy Python
This resolves https://bugzilla.redhat.com/show_bug.cgi?id=1665501

Signed-off-by: Why Do You <enforce@this.qm>
2019-01-14 09:16:14 +01:00
Lubomír Sedlář
6a32b3b741 ostree: Add test for expanding basearch for message
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-01-10 07:53:46 +01:00
Patrick Uiterwijk
c9c5fcac94 Make sure ${basearch} is also replaced with config['ostree_ref']
Signed-off-by: Patrick Uiterwijk <patrick@puiterwijk.org>
2019-01-09 21:39:33 +01:00
Lubomír Sedlář
dbc0d5736c 4.1.32 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-01-08 12:17:29 +01:00
Lubomír Sedlář
90c60f8e64 Add script to orchestrate multiple composes
It may make sense to break a big compose into smaller chunks that can be
done independently. This script allows describing the smaller parts,
runs them with correct dependencies and arranges the result to look like
a single big compose.

All parts use the same koji event, that is either obtained from Koji, or
from command line argument.

JIRA: COMPOSE-2654
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-01-08 09:40:36 +01:00
Lubomír Sedlář
ba260c24e8 buildinstall: Expose lorax's --rootfs-size argument
We already make this possible for the ostree installer, but it was
missing from the traditional one. The default behaviour is to let lorax
decide, but if user knows better, they can overwrite in configuration.

JIRA: COMPOSE-3188
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-01-07 11:00:43 +01:00
Frédéric Pierret (fepitre)
682f959ee0 Support for pungi-legacy with productmd format
Merges: https://pagure.io/pungi/pull-request/1099
Signed-off-by: Frédéric Pierret (fepitre) <frederic.pierret@qubes-os.org>
2019-01-04 14:11:23 +01:00
Ondrej Nosek
70f46dfb62 Unify update-docs.sh script with rpkg
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2019-01-04 13:32:17 +01:00
Lubomír Sedlář
610bf4b125 Remove createrepo references from doc and spec
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-01-03 10:37:59 +01:00
Frédéric Pierret (fepitre)
b7fa03dffd CreaterepoWrapper: add 'basedir' and 'compress-type' args for createrepo_c
Signed-off-by: Frédéric Pierret (fepitre) <frederic.pierret@qubes-os.org>
2019-01-03 10:37:59 +01:00
Frédéric Pierret (fepitre)
f41c32e413 gather.py: use createrepo_c for creating repodata instead of obsolete createrepo python library
Fixes: https://pagure.io/pungi/issue/1094
Merges: https://pagure.io/pungi/pull-request/1097
Signed-off-by: Frédéric Pierret (fepitre) <frederic.pierret@qubes-os.org>
2019-01-03 10:37:59 +01:00
Frédéric Pierret (fepitre)
c69bc13068 Fix import of ConfigParser for NoSectionError and NoOptionError
Merges: https://pagure.io/pungi/pull-request/1100
Signed-off-by: Frédéric Pierret (fepitre) <frederic.pierret@qubes-os.org>
2019-01-03 10:18:05 +01:00
Ken Dreyer
27d5073b98 doc: explain product_id_allow_missing results in detail
Add more details to the documentation for the product_id_allow_missing
setting:

- Pungi expects a product cert for each arch in each variant.

- Pungi will exit with an error by default if a variant+arch's cert is
  missing.

- Pungi will log a warning if product_id_allow_missing is True.

Merges: https://pagure.io/pungi/pull-request/1102
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2019-01-03 10:14:53 +01:00
Ken Dreyer
7c45157a92 doc: describe product_id's output and purpose
Document the "productid" files that Pungi creates when the user
configures the "product_id" setting. Describe where to find the
"productid" files, and how they relate to subscription-manager.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2019-01-02 14:22:21 -05:00
Lubomír Sedlář
5f6dcb37f6 ostree: Send correct ref in the message
If configuration requests a different ref than what treefile has, a
wrong message can be sent.

If the treefile is in JSON, the local copy will be updated in-place and
it works. However with YAML the updated version is still written as JSON
and thus we are still sending the original value.

Fixes: https://pagure.io/pungi/issue/1092
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-12-05 11:20:17 +01:00
Lubomír Sedlář
7c48b808f1 4.1.31 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-26 13:14:09 +01:00
Lubomír Sedlář
df35f26910 Add script to merge and dump multiple configuration files
The script can either take config from an existing compose, or load
files by path. By default it will perform full validation (and add
default values and resolved git references). This can be turned off.

The final JSON is printed to stdout.

JIRA: COMPOSE-3066
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-26 07:59:31 +01:00
Lubomír Sedlář
ca7d6256e5 Move resolving git reference to config validation
Instead of multiple places handling the same thing duplicating the
logic, it's better to do it once upfront. This allows easy caching of
the results.

Additional advantage of this approach is that the config dump will
include resolved URLs. The original reference will still be available in
the copy of the original config.

JIRA: COMPOSE-3065
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-21 13:25:35 +01:00
Lubomír Sedlář
444af0396e util: Add a cache for resolved git urls
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-21 10:15:52 +01:00
Mohan Boddu
4864a0f58e Copy config files into logs/global/config-copy/ directory
Fixes: https://pagure.io/pungi/issue/994
Merges: https://pagure.io/pungi/pull-request/1013
Signed-off-by: Mohan Boddu <mboddu@bhujji.com>
2018-11-19 14:42:38 +01:00
Lubomír Sedlář
7e923d3823 Remove timestamp from config dump
This makes it hard to programatically access the dump.

The timestamp was included originally to make sure that restarting a
compose in debug mode will preserve all configs. However we don't really
support that usecase anyway.

JIRA: COMPOSE-3063
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-19 13:53:34 +01:00
Lubomír Sedlář
7c2e701a74 extra_iso: Support extra files in directory
It works, only the config schema does not allow it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-19 13:18:42 +01:00
Lubomír Sedlář
c2a4700446 extra_iso: Include extra_files.json metadata
This should be on top level of the ISO, and list files added
specifically to the ISO. If there's anything inherited from one some
variant, the files will be listed in metadata in the variant directory.

JIRA: COMPOSE-3069
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-19 13:18:42 +01:00
Lubomír Sedlář
605c9ca435 Allow reading configuration from JSON
JIRA: COMPOSE-3064
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-19 12:02:19 +01:00
Lubomír Sedlář
f809cac0b2 Cleanup parsing treefile
...and also add tests for it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-19 11:59:32 +01:00
Mohan Boddu
324b371cff Fix convert rpm_ostree config to YAML
Merges: https://pagure.io/pungi/pull-request/1088
Signed-off-by: Mohan Boddu <mboddu@bhujji.com>
2018-11-19 11:59:32 +01:00
Lubomír Sedlář
07d08627c6 koji_wrapper: Change owner of runroot output
The files created in runroot task are owned by root by default (because
that's who is running the processes to create them). Making the results
world readable allows the compose to work, but it still can be difficult
to clean up old composes if they contain random files owned by root.

Fixes: https://pagure.io/pungi/issue/1039
JIRA: COMPOSE-2906
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-15 12:07:27 +01:00
Lubomír Sedlář
1d020dbedd util: Preserve symlinks when copying
Currently the `copy_all` function resolves all symlinks and crashes on
broken symlinks. Instead it should copy symlinks as symlinks.

Fixes: https://pagure.io/pungi/issue/1084
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-15 09:10:16 +01:00
Patrick Uiterwijk
8e88373a82 Move from yaml.load to yaml.safe_load
yaml.load is equally powerful as python pickles, and we don't
need that level of power for the ostree yaml files.
Better safe than sorry.

Signed-off-by: Patrick Uiterwijk <patrick@puiterwijk.org>
2018-11-13 21:11:02 +01:00
Lubomír Sedlář
dc692bc604 extra_iso: Stop including variant extra files
They are not always wanted, so let's not include them by default.
There's a new option to include the same files that extra files phases
uses, or alternatively they can be configured specifically and put into
the variant subdirectory.

JIRA: COMPOSE-3084
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-09 12:06:25 +01:00