Commit Graph

30 Commits

Author SHA1 Message Date
Lubomír Sedlář
f822ee324a ostree: Run commands in universal_newlines mode
This will mean the output is returned as unicode (decoded as UTF-8).
Thus Kobo will not have to do any decoding. This should work around
possible errors with breaking multibyte unicode character sequences into
different chunks.

Relates: https://pagure.io/releng/failed-composes/issue/237
Relates: https://github.com/release-engineering/kobo/issues/119
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 09:30:53 +02:00
Lubomír Sedlář
f809cac0b2 Cleanup parsing treefile
...and also add tests for it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-19 11:59:32 +01:00
Mohan Boddu
324b371cff Fix convert rpm_ostree config to YAML
Merges: https://pagure.io/pungi/pull-request/1088
Signed-off-by: Mohan Boddu <mboddu@bhujji.com>
2018-11-19 11:59:32 +01:00
Patrick Uiterwijk
8e88373a82 Move from yaml.load to yaml.safe_load
yaml.load is equally powerful as python pickles, and we don't
need that level of power for the ostree yaml files.
Better safe than sorry.

Signed-off-by: Patrick Uiterwijk <patrick@puiterwijk.org>
2018-11-13 21:11:02 +01:00
Lubomír Sedlář
36c347eb79 ostree: Use --touch-if-changed
There are three different cases:

 * we expect commitid and it's there
 * we expect commitid and it's missing
 * we don't expect commitid

This patch helps differentiate between the second two. In former one we
should report an error and mark the phase as failed. The latter is
perfectly fine and no error should be reported

Fixes: https://pagure.io/pungi/issue/1046
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-19 13:07:03 +02:00
Lubomír Sedlář
b85cd7ff9f ostree: Update tests for working with YAML file
Merges: https://pagure.io/pungi/pull-request/1019
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-14 16:13:23 +02:00
Colin Walters
b5d5e8da4a pungi/ostree: Convert rpm-ostree YAML to JSON
We'd like to use YAML for future rpm-ostree work; among
other things it supports comments.

See https://pagure.io/fedora-atomic/pull-request/125
and the original https://github.com/projectatomic/rpm-ostree/pull/1377

Signed-off-by: Colin Walters <walters@verbum.org>
2018-08-14 16:13:23 +02:00
Lubomír Sedlář
071792bdd0 ostree: Always substitute basearch
When ref is not modified via pungi config, we read it from the treefile
and substitute in basearch.

When pungi is configured to replace it, it modifies the treefile and
then used the value from config to avoid parsing the file. This however
did not substitute the basearch value.

We can simply use one code path for getting the value. This will work
for both cases.

Fixes: https://pagure.io/pungi/issue/866
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-03-21 15:37:15 +01:00
Ondrej Nosek
2ae056e021 Minor correction for python backward compatibility
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-01-25 13:28:08 +01:00
Ondrej Nosek
f301158974 ostree/utils: Drop timestamps from generated repo names - tests
Issue: https://pagure.io/pungi/issue/811

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-01-17 16:48:22 +01:00
Colin Walters
3427d6abe2 ostree/utils: Generate a single pungi.repo file, use repo-<num> IDs
Ideally, pungi would generate repository IDs like `fedora-updates`
or so, and we'd have versioning inside the rpm-md.  But for now
let's do this to avoid invalidating rpm-ostree's change detection.

Closes: https://pagure.io/pungi/issue/811
Signed-off-by: Colin Walters <walters@verbum.org>
2018-01-17 16:48:22 +01:00
Colin Walters
b2bdc8a608 ostree/utils: Drop timestamps from generated repo names
Since we drop these files in a separate workdir each time,
there's no need to datestamp them.  Doing so is part of the
cause for invalidating's rpm-ostree input change hashing.

Issue: https://pagure.io/pungi/issue/811
Signed-off-by: Colin Walters <walters@verbum.org>
2018-01-17 16:48:22 +01:00
Ondrej Nosek
51cd359057 ostree: Add force_new_commit option - test added
Relates: https://pagure.io/pungi/issue/811

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-01-11 15:08:07 +01:00
Colin Walters
161b7f974b ostree: Add force_new_commit option
Followup from discussion in: https://pagure.io/pungi/issue/811

It's likely now that for Fedora Atomic Host we'll use this, to work
around other issues, after we fix the FAW change detection.

Signed-off-by: Colin Walters <walters@verbum.org>
2018-01-11 15:03:36 +01:00
Ondrej Nosek
9642c1171c add ability to specify ostree ref in OSTREE phase - update
Additionally ostree_ref (if parameter is given) should be placed into treefile.

Relates: https://pagure.io/pungi/issue/777

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2017-12-04 18:07:36 +01:00
Ondrej Nosek
a6c65e026a add ability to specify ostree ref in OSTREE phase
It allows specify what ref we want this compose to commit to.
New parameter 'ostree_ref' overrides the default value from the treefile json.

Relates: https://pagure.io/pungi/issue/777

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2017-12-04 09:27:19 +01:00
Ondrej Nosek
91ee1fb854 Remove deprecated options
Options that are currently marked as deprecated do not have any effect
anymore (other than printing warning). We should remove them and update
the message so that we can mark options as deprecated even when they
still work.

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2017-11-16 11:12:34 +01:00
Qixiang Wan
0a3e5b27bf remove the dependency of rpmUtils
The rpmUtils module is provided yum-utils package, which is only
available for Python 2. There is no replacement for the functionality in
DNF.

There is a proposal to add this functionality to rpm itself, but it's
not really moving forward very much:
https://bugzilla.redhat.com/show_bug.cgi?id=1072972

As a short term solution let's copy the needed parts of rpmUtils.arch
module directly to pungi code base.

Fixes: https://pagure.io/pungi/issue/533
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-07-26 11:23:16 +02:00
Colin Walters
af631f560e ostree: Don't automatically create a repo
Creating an OSTree repository is a notable event; the general
expectation is that rather than having lots of repositories,
one has branches inside a single repository.

For $reasons, Fedora is not currently doing this, but we will
change it to do so.

The reason I'm making this change is we discovered that
it looked like Fedora had somehow made a repo inside a repo,
presumably due to a configuration error.

https://lists.fedoraproject.org/archives/list/cloud@lists.fedoraproject.org/message/GBFSOLULGGZFGEFCIW6FG23NZZV5VH4K/
Signed-off-by: Colin Walters <walters@verbum.org>
2017-06-26 17:47:35 -04:00
Lubomír Sedlář
a2c42aee3c ostree: Refactor writing repo file
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-05-29 13:26:49 +02:00
Qixiang Wan
2f5d6d7dcd unify repo and repo_from options
Config option 'repo' and 'repo_from' are used in several phases, merge
them with one option 'repo'. 'append' in schema is used for appending
the values from deprecated options to 'repo', so it won't break on any
existing config files that have the old options of 'repo_from' and
'source_repo_from' (which is an alias of 'repo_from').

And 'repo' schema is updated to support repo dict as the value or an
item in the values, a repo dict is just a dict contains repo options,
'baseurl' is required in the dict, like:

{"baseurl": "http://example.com/url/to/repo"}

or:

{"baseurl": "Serer"}

currently this is used in ostree phase to support extra repo options
like:

{"baseurl": "Server", "exclude": "systemd-container"}

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-29 10:12:32 +08:00
Qixiang Wan
2ae8710934 consolidate repo option names
live_images:            additional_repos        -> repo
ostree:                 source_repo_from        -> repo_from
                        extra_source_repos      -> repo
ostree_installer:       source_repo_from        -> repo_from

With the change, the phases have consolidate option names for variant
repos and external repos.

Old option names will continue to work, old names will be converted
to new names after validation automatically if new options are not
specified in config.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-03-06 09:26:42 +08:00
Lubomír Sedlář
9e020c2782 ostree: Silence logger in tests
Instead of creating and configuring the logger at module import time, we
can only get the logger if it's actually needed and configure it from
the main script.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-23 11:08:31 +01:00
Lubomír Sedlář
3162fea60d ostree: Fix crash when extra repos are missing
The default needs to be an empty list, not `None`.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-22 15:48:06 +01:00
Qixiang Wan
a57bc13e30 Add --quiet option to pungi-koji
log will not be printed to screen when quiet mode is enabled.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-02-20 15:00:03 +08:00
Dennis Gilmore
59dd4dbcd8 replace ${basearch} when updating the ref
pungi-make-ostree has to run on the target arch so that rpm
scriptlets can be ran. as a reult we can ask rpm what the
basearch is for the running environment. For notifications
we have to pass in the arch we are running for.

Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2017-01-18 21:21:26 -06:00
Patrick Uiterwijk
4edf567bd4 Add some debugging about ref updating
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2017-01-18 14:09:15 -06:00
Qixiang Wan
2d404c88e6 [pungi-make-ostree] change 'tree' command '--log-dir' arg to be required
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-12-15 18:55:04 +08:00
Qixiang Wan
83428a06bf [ostree-installer] Add 'installer' sub-command to pungi-make-ostree
The new sub-command 'installer' is added to support build OSTree
installer image with pungi-make-ostree. It can take an optional argument
'--extra-config' to read some of configurations from a json file. The
content of the json file can contains the configuration which are
supported in OSTree installer phase, the difference is variant UID is
not supported as a repo url in this case. A valid json file can be like
the following:

{
    "source_repo_from": "http://www.example.com/repo/workstation/os",
    "installpkgs": [
        "fedora-productimg-workstation"
    ],
    "add_template": [
        "/path/to/installer/template/lorax-configure-repo.tmpl"
    ],
    "add_template_var": [
        "ostree_osname=fedora-workstation",
        "ostree_ref=fedora/25/x86_64/workstation"
    ],
    "add_arch_template": [
        "/path/to/installer/template/lorax-embed-repo.tmpl"
    ],
    "add_arch_template_var": [
        "ostree_repo=https://www.example.com/compose/ostree",
        "ostree_osname=fedora-workstation",
        "ostree_ref=fedora/25/x86_64/workstation"
    ]
}

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-12-12 19:30:35 +08:00
Qixiang Wan
e043604822 [ostree] Add 'tree' sub-command to pungi-make-ostree script
Update pungi-make-ostree to supourt sub-command 'tree', which is just
as the original feature of pungi-make-ostree to compose OSTree tree.
With the change we can add other sub commands later to build other
OSTree artifacts, like the installer image.

Inaddtional to the change, now the the 'tree' command can accept an
optional '--extra-config' parameter to update the original tree
configuration with extra configurations specified in a json file
before composing the OSTree tree.

Example:

pungi-make-ostree tree --repo=/ostree --treefile=/path/to/treefile \
--log-dir=/path/to/log --extra-config=/path/to/extra-config.json

The extra-config file can contains the same configuration as OSTree
phase, the difference is it doesn't understand variant UID as source
repo since it's not ran in the chain of phases. A valid configuration
can be like:

{
    "source_repo_from": "http://example.com/repo/x86_64/Server",
    "extra_source_repos": [
        {
            "name": "optional",
            "baseurl": "http://example.com/repo/x86_64/optional",
            "exclude": "systemd-container",
            "gpgcheck": False
        },
        {
            "name": "extra",
            "baseurl": "http://example.com/repo/x86_64/extra",
        }
    ],
    "keep_original_sources": True
}

The OSTree phase is updated to move out the task of updating treefile,
instead of that, it writes the extra configurations to a json file,
then 'pungi-make-ostree tree' will take it by option '--extra-config'.

Signed-off-by: Qixiang Wan <qwan@redhat.com>
2016-12-12 19:30:35 +08:00