diff --git a/docs/Makefile b/docs/Makefile index 1bab5ce5..fe841b99 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -7,8 +7,8 @@ SPHINXBUILD = sphinx-build-3 SPHINXAPIDOC = sphinx-apidoc-3 PAPER = BUILDDIR = . -SOURCEDIR = ../src/pylorax -MODULE_NAMES = pylorax.rst modules.rst +SOURCEDIR = ../src +MODULE_NAMES = pylorax.rst pylorax.api.rst modules.rst composer.rst composer.cli.rst # User-friendly check for sphinx-build ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1) diff --git a/docs/composer.cli.rst b/docs/composer.cli.rst new file mode 100644 index 00000000..f6e56582 --- /dev/null +++ b/docs/composer.cli.rst @@ -0,0 +1,62 @@ +composer.cli package +==================== + +Submodules +---------- + +composer.cli.blueprints module +------------------------------ + +.. automodule:: composer.cli.blueprints + :members: + :undoc-members: + :show-inheritance: + +composer.cli.cmdline module +--------------------------- + +.. automodule:: composer.cli.cmdline + :members: + :undoc-members: + :show-inheritance: + +composer.cli.compose module +--------------------------- + +.. automodule:: composer.cli.compose + :members: + :undoc-members: + :show-inheritance: + +composer.cli.modules module +--------------------------- + +.. automodule:: composer.cli.modules + :members: + :undoc-members: + :show-inheritance: + +composer.cli.projects module +---------------------------- + +.. automodule:: composer.cli.projects + :members: + :undoc-members: + :show-inheritance: + +composer.cli.utilities module +----------------------------- + +.. automodule:: composer.cli.utilities + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: composer.cli + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/composer.rst b/docs/composer.rst new file mode 100644 index 00000000..5314fc0c --- /dev/null +++ b/docs/composer.rst @@ -0,0 +1,37 @@ +composer package +================ + +Subpackages +----------- + +.. toctree:: + + composer.cli + +Submodules +---------- + +composer.http\_client module +---------------------------- + +.. automodule:: composer.http_client + :members: + :undoc-members: + :show-inheritance: + +composer.unix\_socket module +---------------------------- + +.. automodule:: composer.unix_socket + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: composer + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/html/.buildinfo b/docs/html/.buildinfo index 2d8fe27b..74f1e784 100644 --- a/docs/html/.buildinfo +++ b/docs/html/.buildinfo @@ -1,4 +1,4 @@ # Sphinx build info version 1 # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. -config: 6d8b53f57aa5089097c41bc46e52d283 +config: 05d2025242aa5b20580579fa62643e7f tags: 645f666f9bcd5a90fca523b33c5a78b7 diff --git a/docs/html/.doctrees/composer-cli.doctree b/docs/html/.doctrees/composer-cli.doctree new file mode 100644 index 00000000..62bff524 Binary files /dev/null and b/docs/html/.doctrees/composer-cli.doctree differ diff --git a/docs/html/.doctrees/composer.cli.doctree b/docs/html/.doctrees/composer.cli.doctree new file mode 100644 index 00000000..75236635 Binary files /dev/null and b/docs/html/.doctrees/composer.cli.doctree differ diff --git a/docs/html/.doctrees/composer.doctree b/docs/html/.doctrees/composer.doctree new file mode 100644 index 00000000..b1854b41 Binary files /dev/null and b/docs/html/.doctrees/composer.doctree differ diff --git a/docs/html/.doctrees/environment.pickle b/docs/html/.doctrees/environment.pickle index 1f39ec18..0cccac05 100644 Binary files a/docs/html/.doctrees/environment.pickle and b/docs/html/.doctrees/environment.pickle differ diff --git a/docs/html/.doctrees/index.doctree b/docs/html/.doctrees/index.doctree index 3051cc25..801ef94f 100644 Binary files a/docs/html/.doctrees/index.doctree and b/docs/html/.doctrees/index.doctree differ diff --git a/docs/html/.doctrees/intro.doctree b/docs/html/.doctrees/intro.doctree index 78b76b2d..3fed1889 100644 Binary files a/docs/html/.doctrees/intro.doctree and b/docs/html/.doctrees/intro.doctree differ diff --git a/docs/html/.doctrees/livemedia-creator.doctree b/docs/html/.doctrees/livemedia-creator.doctree index 222b1b6c..14aa21e7 100644 Binary files a/docs/html/.doctrees/livemedia-creator.doctree and b/docs/html/.doctrees/livemedia-creator.doctree differ diff --git a/docs/html/.doctrees/lorax-composer.doctree b/docs/html/.doctrees/lorax-composer.doctree new file mode 100644 index 00000000..13292a25 Binary files /dev/null and b/docs/html/.doctrees/lorax-composer.doctree differ diff --git a/docs/html/.doctrees/lorax.doctree b/docs/html/.doctrees/lorax.doctree index b29302ee..7edba50d 100644 Binary files a/docs/html/.doctrees/lorax.doctree and b/docs/html/.doctrees/lorax.doctree differ diff --git a/docs/html/.doctrees/modules.doctree b/docs/html/.doctrees/modules.doctree index 8b55504f..5894f5a4 100644 Binary files a/docs/html/.doctrees/modules.doctree and b/docs/html/.doctrees/modules.doctree differ diff --git a/docs/html/.doctrees/product-images.doctree b/docs/html/.doctrees/product-images.doctree index f54cf679..76279ee0 100644 Binary files a/docs/html/.doctrees/product-images.doctree and b/docs/html/.doctrees/product-images.doctree differ diff --git a/docs/html/.doctrees/pylorax.api.doctree b/docs/html/.doctrees/pylorax.api.doctree new file mode 100644 index 00000000..f28b102b Binary files /dev/null and b/docs/html/.doctrees/pylorax.api.doctree differ diff --git a/docs/html/.doctrees/pylorax.doctree b/docs/html/.doctrees/pylorax.doctree index 6e8ebbae..a0ec42b2 100644 Binary files a/docs/html/.doctrees/pylorax.doctree and b/docs/html/.doctrees/pylorax.doctree differ diff --git a/docs/html/.doctrees/source/index.doctree b/docs/html/.doctrees/source/index.doctree deleted file mode 100644 index a1aebf96..00000000 Binary files a/docs/html/.doctrees/source/index.doctree and /dev/null differ diff --git a/docs/html/_modules/composer/cli.html b/docs/html/_modules/composer/cli.html new file mode 100644 index 00000000..c996041d --- /dev/null +++ b/docs/html/_modules/composer/cli.html @@ -0,0 +1,274 @@ + + + + + + + + + + + composer.cli — Lorax 29.1 documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + +
+ + + + +
+
+
+
+ +

Source code for composer.cli

+#!/usr/bin/python
+#
+# composer-cli
+#
+# Copyright (C) 2018  Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+import logging
+log = logging.getLogger("composer-cli")
+
+from composer.cli.blueprints import blueprints_cmd
+from composer.cli.modules import modules_cmd
+from composer.cli.projects import projects_cmd
+from composer.cli.compose import compose_cmd
+
+command_map = {
+    "blueprints": blueprints_cmd,
+    "modules": modules_cmd,
+    "projects": projects_cmd,
+    "compose": compose_cmd
+    }
+
+
+
[docs]def main(opts):
+    """ Main program execution
+
+    :param opts: Cmdline arguments
+    :type opts: argparse.Namespace
+    """
+    if len(opts.args) == 0:
+        log.error("Missing command")
+        return 1
+    elif opts.args[0] not in command_map:
+        log.error("Unknown command %s", opts.args[0])
+        return 1
+    if len(opts.args) == 1:
+        log.error("Missing %s sub-command", opts.args[0])
+        return 1
+    else:
+        try:
+            return command_map[opts.args[0]](opts)
+        except Exception as e:
+            log.error(str(e))
+            return 1

+
+ +
+ +
+ + +
+
+ +
+ +
+ + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/docs/html/_modules/composer/cli/blueprints.html b/docs/html/_modules/composer/cli/blueprints.html new file mode 100644 index 00000000..7a0f27ec --- /dev/null +++ b/docs/html/_modules/composer/cli/blueprints.html @@ -0,0 +1,740 @@ + + + + + + + + + + + composer.cli.blueprints — Lorax 29.1 documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + +
+ + + + +
+
+
+
+ +

Source code for composer.cli.blueprints

+#
+# Copyright (C) 2018  Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+import logging
+log = logging.getLogger("composer-cli")
+
+import os
+import json
+
+from composer import http_client as client
+from composer.cli.utilities import argify, frozen_toml_filename, toml_filename, handle_api_result
+from composer.cli.utilities import packageNEVRA
+
+
[docs]def blueprints_cmd(opts):
+    """Process blueprints commands
+
+    :param opts: Cmdline arguments
+    :type opts: argparse.Namespace
+    :returns: Value to return from sys.exit()
+    :rtype: int
+
+    This dispatches the blueprints commands to a function
+    """
+    cmd_map = {
+        "list":      blueprints_list,
+        "show":      blueprints_show,
+        "changes":   blueprints_changes,
+        "diff":      blueprints_diff,
+        "save":      blueprints_save,
+        "delete":    blueprints_delete,
+        "depsolve":  blueprints_depsolve,
+        "push":      blueprints_push,
+        "freeze":    blueprints_freeze,
+        "tag":       blueprints_tag,
+        "undo":      blueprints_undo,
+        "workspace": blueprints_workspace
+        }
+    if opts.args[1] not in cmd_map:
+        log.error("Unknown blueprints command: %s", opts.args[1])
+        return 1
+
+    return cmd_map[opts.args[1]](opts.socket, opts.api_version, opts.args[2:], opts.json)

+
+
[docs]def blueprints_list(socket_path, api_version, args, show_json=False):
+    """Output the list of available blueprints
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param api_version: Version of the API to talk to. eg. "0"
+    :type api_version: str
+    :param args: List of remaining arguments from the cmdline
+    :type args: list of str
+    :param show_json: Set to True to show the JSON output instead of the human readable output
+    :type show_json: bool
+
+    blueprints list
+    """
+    api_route = client.api_url(api_version, "/blueprints/list")
+    result = client.get_url_json(socket_path, api_route)
+    if show_json:
+        print(json.dumps(result, indent=4))
+        return 0
+
+    print("blueprints: " + ", ".join([r for r in result["blueprints"]]))
+
+    return 0

+
+
[docs]def blueprints_show(socket_path, api_version, args, show_json=False):
+    """Show the blueprints, in TOML format
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param api_version: Version of the API to talk to. eg. "0"
+    :type api_version: str
+    :param args: List of remaining arguments from the cmdline
+    :type args: list of str
+    :param show_json: Set to True to show the JSON output instead of the human readable output
+    :type show_json: bool
+
+    blueprints show <blueprint,...>        Display the blueprint in TOML format.
+
+    Multiple blueprints will be separated by \n\n
+    """
+    for blueprint in argify(args):
+        api_route = client.api_url(api_version, "/blueprints/info/%s?format=toml" % blueprint)
+        print(client.get_url_raw(socket_path, api_route) + "\n\n")
+
+    return 0

+
+
[docs]def blueprints_changes(socket_path, api_version, args, show_json=False):
+    """Display the changes for each of the blueprints
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param api_version: Version of the API to talk to. eg. "0"
+    :type api_version: str
+    :param args: List of remaining arguments from the cmdline
+    :type args: list of str
+    :param show_json: Set to True to show the JSON output instead of the human readable output
+    :type show_json: bool
+
+    blueprints changes <blueprint,...>     Display the changes for each blueprint.
+    """
+    api_route = client.api_url(api_version, "/blueprints/changes/%s" % (",".join(argify(args))))
+    result = client.get_url_json(socket_path, api_route)
+    if show_json:
+        print(json.dumps(result, indent=4))
+        return 0
+
+    for blueprint in result["blueprints"]:
+        print(blueprint["name"])
+        for change in blueprint["changes"]:
+            prettyCommitDetails(change)
+
+    return 0

+
+
[docs]def prettyCommitDetails(change, indent=4):
+    """Print the blueprint's change in a nice way
+
+    :param change: The individual blueprint change dict
+    :type change: dict
+    :param indent: Number of spaces to indent
+    :type indent: int
+    """
+    def revision():
+        if change["revision"]:
+            return "  revision %d" % change["revision"]
+        else:
+            return ""
+
+    print(" " * indent + change["timestamp"] + "  " + change["commit"] + revision())
+    print(" " * indent + change["message"] + "\n")

+
+
[docs]def blueprints_diff(socket_path, api_version, args, show_json=False):
+    """Display the differences between 2 versions of a blueprint
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param api_version: Version of the API to talk to. eg. "0"
+    :type api_version: str
+    :param args: List of remaining arguments from the cmdline
+    :type args: list of str
+    :param show_json: Set to True to show the JSON output instead of the human readable output
+    :type show_json: bool
+
+    blueprints diff <blueprint-name>       Display the differences between 2 versions of a blueprint.
+                 <from-commit>       Commit hash or NEWEST
+                 <to-commit>         Commit hash, NEWEST, or WORKSPACE
+    """
+    if len(args) == 0:
+        log.error("blueprints diff is missing the blueprint name, from commit, and to commit")
+        return 1
+    elif len(args) == 1:
+        log.error("blueprints diff is missing the from commit, and the to commit")
+        return 1
+    elif len(args) == 2:
+        log.error("blueprints diff is missing the to commit")
+        return 1
+
+    api_route = client.api_url(api_version, "/blueprints/diff/%s/%s/%s" % (args[0], args[1], args[2]))
+    result = client.get_url_json(socket_path, api_route)
+
+    if show_json:
+        print(json.dumps(result, indent=4))
+        return 0
+
+    for err in result.get("errors", []):
+        log.error(err)
+
+    if result.get("errors", False):
+        return 1
+
+    for diff in result["diff"]:
+        print(prettyDiffEntry(diff))
+
+    return 0

+
+
[docs]def prettyDiffEntry(diff):
+    """Generate nice diff entry string.
+
+    :param diff: Difference entry dict
+    :type diff: dict
+    :returns: Nice string
+    """
+    def change(diff):
+        if diff["old"] and diff["new"]:
+            return "Changed"
+        elif diff["new"] and not diff["old"]:
+            return "Added"
+        elif diff["old"] and not diff["new"]:
+            return "Removed"
+        else:
+            return "Unknown"
+
+    def name(diff):
+        if diff["old"]:
+            return list(diff["old"].keys())[0]
+        elif diff["new"]:
+            return list(diff["new"].keys())[0]
+        else:
+            return "Unknown"
+
+    def details(diff):
+        if change(diff) == "Changed":
+            if name(diff) == "Description":
+                return '"%s" -> "%s"' % (diff["old"][name(diff)], diff["new"][name(diff)])
+            elif name(diff) == "Version":
+                return "%s -> %s" % (diff["old"][name(diff)], diff["new"][name(diff)])
+            elif name(diff) in ["Module", "Package"]:
+                return "%s %s -> %s" % (diff["old"][name(diff)]["name"], diff["old"][name(diff)]["version"],
+                                        diff["new"][name(diff)]["version"])
+            else:
+                return "Unknown"
+        elif change(diff) == "Added":
+            if name(diff) in ["Module", "Package"]:
+                return "%s %s" % (diff["new"][name(diff)]["name"], diff["new"][name(diff)]["version"])
+            else:
+                return " ".join([diff["new"][k] for k in diff["new"]])
+        elif change(diff) == "Removed":
+            if name(diff) in ["Module", "Package"]:
+                return "%s %s" % (diff["old"][name(diff)]["name"], diff["old"][name(diff)]["version"])
+            else:
+                return " ".join([diff["old"][k] for k in diff["old"]])
+
+    return change(diff) + " " + name(diff) + " " + details(diff)

+
+
[docs]def blueprints_save(socket_path, api_version, args, show_json=False):
+    """Save the blueprint to a TOML file
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param api_version: Version of the API to talk to. eg. "0"
+    :type api_version: str
+    :param args: List of remaining arguments from the cmdline
+    :type args: list of str
+    :param show_json: Set to True to show the JSON output instead of the human readable output
+    :type show_json: bool
+
+    blueprints save <blueprint,...>        Save the blueprint to a file, <blueprint-name>.toml
+    """
+    for blueprint in argify(args):
+        api_route = client.api_url(api_version, "/blueprints/info/%s?format=toml" % blueprint)
+        blueprint_toml = client.get_url_raw(socket_path, api_route)
+        open(toml_filename(blueprint), "w").write(blueprint_toml)
+
+    return 0

+
+
[docs]def blueprints_delete(socket_path, api_version, args, show_json=False):
+    """Delete a blueprint from the server
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param api_version: Version of the API to talk to. eg. "0"
+    :type api_version: str
+    :param args: List of remaining arguments from the cmdline
+    :type args: list of str
+    :param show_json: Set to True to show the JSON output instead of the human readable output
+    :type show_json: bool
+
+    delete <blueprint>          Delete a blueprint from the server
+    """
+    api_route = client.api_url(api_version, "/blueprints/delete/%s" % args[0])
+    result = client.delete_url_json(socket_path, api_route)
+
+    return handle_api_result(result, show_json)

+
+
[docs]def blueprints_depsolve(socket_path, api_version, args, show_json=False):
+    """Display the packages needed to install the blueprint
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param api_version: Version of the API to talk to. eg. "0"
+    :type api_version: str
+    :param args: List of remaining arguments from the cmdline
+    :type args: list of str
+    :param show_json: Set to True to show the JSON output instead of the human readable output
+    :type show_json: bool
+
+    blueprints depsolve <blueprint,...>    Display the packages needed to install the blueprint.
+    """
+    api_route = client.api_url(api_version, "/blueprints/depsolve/%s" % (",".join(argify(args))))
+    result = client.get_url_json(socket_path, api_route)
+
+    if show_json:
+        print(json.dumps(result, indent=4))
+        return 0
+
+    for blueprint in result["blueprints"]:
+        if blueprint["blueprint"].get("version", ""):
+            print("blueprint: %s v%s" % (blueprint["blueprint"]["name"], blueprint["blueprint"]["version"]))
+        else:
+            print("blueprint: %s" % (blueprint["blueprint"]["name"]))
+        for dep in blueprint["dependencies"]:
+            print("    " + packageNEVRA(dep))
+
+    return 0

+
+
[docs]def blueprints_push(socket_path, api_version, args, show_json=False):
+    """Push a blueprint TOML file to the server, updating the blueprint
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param api_version: Version of the API to talk to. eg. "0"
+    :type api_version: str
+    :param args: List of remaining arguments from the cmdline
+    :type args: list of str
+    :param show_json: Set to True to show the JSON output instead of the human readable output
+    :type show_json: bool
+
+    push <blueprint>            Push a blueprint TOML file to the server.
+    """
+    api_route = client.api_url(api_version, "/blueprints/new")
+    rval = 0
+    for blueprint in argify(args):
+        if not os.path.exists(blueprint):
+            log.error("Missing blueprint file: %s", blueprint)
+            continue
+        blueprint_toml = open(blueprint, "r").read()
+
+        result = client.post_url_toml(socket_path, api_route, blueprint_toml)
+        if handle_api_result(result, show_json):
+            rval = 1
+
+    return rval

+
+
[docs]def blueprints_freeze(socket_path, api_version, args, show_json=False):
+    """Handle the blueprints freeze commands
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param api_version: Version of the API to talk to. eg. "0"
+    :type api_version: str
+    :param args: List of remaining arguments from the cmdline
+    :type args: list of str
+    :param show_json: Set to True to show the JSON output instead of the human readable output
+    :type show_json: bool
+
+    blueprints freeze <blueprint,...>      Display the frozen blueprint's modules and packages.
+    blueprints freeze show <blueprint,...> Display the frozen blueprint in TOML format.
+    blueprints freeze save <blueprint,...> Save the frozen blueprint to a file, <blueprint-name>.frozen.toml.
+    """
+    if args[0] == "show":
+        return blueprints_freeze_show(socket_path, api_version, args[1:], show_json)
+    elif args[0] == "save":
+        return blueprints_freeze_save(socket_path, api_version, args[1:], show_json)
+
+    if len(args) == 0:
+        log.error("freeze is missing the blueprint name")
+        return 1
+
+    api_route = client.api_url(api_version, "/blueprints/freeze/%s" % (",".join(argify(args))))
+    result = client.get_url_json(socket_path, api_route)
+
+    if show_json:
+        print(json.dumps(result, indent=4))
+    else:
+        for entry in result["blueprints"]:
+            blueprint = entry["blueprint"]
+            if blueprint.get("version", ""):
+                print("blueprint: %s v%s" % (blueprint["name"], blueprint["version"]))
+            else:
+                print("blueprint: %s" % (blueprint["name"]))
+
+            for m in blueprint["modules"]:
+                print("    %s-%s" % (m["name"], m["version"]))
+
+            for p in blueprint["packages"]:
+                print("    %s-%s" % (p["name"], p["version"]))
+
+        # Print any errors
+        for err in result.get("errors", []):
+            log.error(err)
+
+    # Return a 1 if there are any errors
+    if result.get("errors", []):
+        return 1
+    else:
+        return 0

+
+
[docs]def blueprints_freeze_show(socket_path, api_version, args, show_json=False):
+    """Show the frozen blueprint in TOML format
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param api_version: Version of the API to talk to. eg. "0"
+    :type api_version: str
+    :param args: List of remaining arguments from the cmdline
+    :type args: list of str
+    :param show_json: Set to True to show the JSON output instead of the human readable output
+    :type show_json: bool
+
+    blueprints freeze show <blueprint,...> Display the frozen blueprint in TOML format.
+    """
+    if len(args) == 0:
+        log.error("freeze show is missing the blueprint name")
+        return 1
+
+    for blueprint in argify(args):
+        api_route = client.api_url(api_version, "/blueprints/freeze/%s?format=toml" % blueprint)
+        print(client.get_url_raw(socket_path, api_route))
+
+    return 0

+
+
[docs]def blueprints_freeze_save(socket_path, api_version, args, show_json=False):
+    """Save the frozen blueprint to a TOML file
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param api_version: Version of the API to talk to. eg. "0"
+    :type api_version: str
+    :param args: List of remaining arguments from the cmdline
+    :type args: list of str
+    :param show_json: Set to True to show the JSON output instead of the human readable output
+    :type show_json: bool
+
+    blueprints freeze save <blueprint,...> Save the frozen blueprint to a file, <blueprint-name>.frozen.toml.
+    """
+    if len(args) == 0:
+        log.error("freeze save is missing the blueprint name")
+        return 1
+
+    for blueprint in argify(args):
+        api_route = client.api_url(api_version, "/blueprints/freeze/%s?format=toml" % blueprint)
+        blueprint_toml = client.get_url_raw(socket_path, api_route)
+        open(frozen_toml_filename(blueprint), "w").write(blueprint_toml)
+
+    return 0

+
+
[docs]def blueprints_tag(socket_path, api_version, args, show_json=False):
+    """Tag the most recent blueprint commit as a release
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param api_version: Version of the API to talk to. eg. "0"
+    :type api_version: str
+    :param args: List of remaining arguments from the cmdline
+    :type args: list of str
+    :param show_json: Set to True to show the JSON output instead of the human readable output
+    :type show_json: bool
+
+    blueprints tag <blueprint>             Tag the most recent blueprint commit as a release.
+    """
+    api_route = client.api_url(api_version, "/blueprints/tag/%s" % args[0])
+    result = client.post_url(socket_path, api_route, "")
+
+    return handle_api_result(result, show_json)

+
+
[docs]def blueprints_undo(socket_path, api_version, args, show_json=False):
+    """Undo changes to a blueprint
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param api_version: Version of the API to talk to. eg. "0"
+    :type api_version: str
+    :param args: List of remaining arguments from the cmdline
+    :type args: list of str
+    :param show_json: Set to True to show the JSON output instead of the human readable output
+    :type show_json: bool
+
+    blueprints undo <blueprint> <commit>   Undo changes to a blueprint by reverting to the selected commit.
+    """
+    if len(args) == 0:
+        log.error("undo is missing the blueprint name and commit hash")
+        return 1
+    elif len(args) == 1:
+        log.error("undo is missing commit hash")
+        return 1
+
+    api_route = client.api_url(api_version, "/blueprints/undo/%s/%s" % (args[0], args[1]))
+    result = client.post_url(socket_path, api_route, "")
+
+    return handle_api_result(result, show_json)

+
+
[docs]def blueprints_workspace(socket_path, api_version, args, show_json=False):
+    """Push the blueprint TOML to the temporary workspace storage
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param api_version: Version of the API to talk to. eg. "0"
+    :type api_version: str
+    :param args: List of remaining arguments from the cmdline
+    :type args: list of str
+    :param show_json: Set to True to show the JSON output instead of the human readable output
+    :type show_json: bool
+
+    blueprints workspace <blueprint>       Push the blueprint TOML to the temporary workspace storage.
+    """
+    api_route = client.api_url(api_version, "/blueprints/workspace")
+    rval = 0
+    for blueprint in argify(args):
+        if not os.path.exists(blueprint):
+            log.error("Missing blueprint file: %s", blueprint)
+            continue
+        blueprint_toml = open(blueprint, "r").read()
+
+        result = client.post_url_toml(socket_path, api_route, blueprint_toml)
+        if show_json:
+            print(json.dumps(result, indent=4))
+
+        for err in result.get("errors", []):
+            log.error(err)
+
+        # Any errors results in returning a 1, but we continue with the rest first
+        if not result.get("status", False):
+            rval = 1
+
+    return rval

+
+ +
+ +
+ + +
+
+ +
+ +
+ + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/docs/html/_modules/composer/cli/cmdline.html b/docs/html/_modules/composer/cli/cmdline.html new file mode 100644 index 00000000..06d08c25 --- /dev/null +++ b/docs/html/_modules/composer/cli/cmdline.html @@ -0,0 +1,359 @@ + + + + + + + + + + + composer.cli.cmdline — Lorax 29.1 documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + +
+ + + + +
+
+
+
+ +

Source code for composer.cli.cmdline

+#
+# Copyright (C) 2018 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+import os
+import sys
+import argparse
+
+from composer import vernum
+
+VERSION = "{0}-{1}".format(os.path.basename(sys.argv[0]), vernum)
+
+# Documentation for the commands
+epilog = """
+compose start <BLUEPRINT> <TYPE>
+    Start a compose using the selected blueprint and output type.
+
+compose types
+    List the supported output types.
+
+compose status
+    List the status of all running and finished composes.
+
+compose log <UUID> [<SIZE>]
+    Show the last SIZE kB of the compose log.
+
+compose cancel <UUID>
+    Cancel a running compose and delete any intermediate results.
+
+compose delete <UUID,...>
+    Delete the listed compose results.
+
+compose info <UUID>
+    Show detailed information on the compose.
+
+compose metadata <UUID>
+    Download the metadata use to create the compose to <uuid>-metadata.tar
+
+compose logs <UUID>
+    Download the compose logs to <uuid>-logs.tar
+
+compose results <UUID>
+    Download all of the compose results; metadata, logs, and image to <uuid>.tar
+
+compose image <UUID>
+    Download the output image from the compose. Filename depends on the type.
+
+blueprints list
+    List the names of the available blueprints.
+
+blueprints show <BLUEPRINT,...>
+    Display the blueprint in TOML format.
+
+blueprints changes <BLUEPRINT,...>
+    Display the changes for each blueprint.
+
+blueprints diff <BLUEPRINT> <FROM-COMMIT> <TO-COMMIT>
+    Display the differences between 2 versions of a blueprint.
+    FROM-COMMIT can be a commit hash or NEWEST
+    TO-COMMIT  can be a commit hash, NEWEST, or WORKSPACE
+
+blueprints save <BLUEPRINT,...>
+    Save the blueprint to a file, <BLUEPRINT>.toml
+
+blueprints delete <BLUEPRINT>
+    Delete a blueprint from the server
+
+blueprints depsolve <BLUEPRINT,...>
+    Display the packages needed to install the blueprint.
+
+blueprints push <BLUEPRINT>
+    Push a blueprint TOML file to the server.
+
+blueprints freeze <BLUEPRINT,...>
+    Display the frozen blueprint's modules and packages.
+
+blueprints freeze show <BLUEPRINT,...>
+    Display the frozen blueprint in TOML format.
+
+blueprints freeze save <BLUEPRINT,...>
+    Save the frozen blueprint to a file, <blueprint-name>.frozen.toml.
+
+blueprints tag <BLUEPRINT>
+    Tag the most recent blueprint commit as a release.
+
+blueprints undo <BLUEPRINT> <COMMIT>
+    Undo changes to a blueprint by reverting to the selected commit.
+
+blueprints workspace <BLUEPRINT>
+    Push the blueprint TOML to the temporary workspace storage.
+
+modules list
+    List the available modules.
+
+projects list
+    List the available projects.
+
+projects info <PROJECT,...>
+    Show details about the listed projects.
+
+"""
+
+
[docs]def composer_cli_parser():
+    """ Return the ArgumentParser for composer-cli"""
+
+    parser = argparse.ArgumentParser(description="Lorax Composer commandline tool",
+                                     epilog=epilog,
+                                     formatter_class=argparse.RawDescriptionHelpFormatter,
+                                     fromfile_prefix_chars="@")
+
+    parser.add_argument("-j", "--json", action="store_true", default=False,
+                        help="Output the raw JSON response instead of the normal output.")
+    parser.add_argument("-s", "--socket", default="/run/weldr/api.socket", metavar="SOCKET",
+                        help="Path to the socket file to listen on")
+    parser.add_argument("--log", dest="logfile", default="./composer-cli.log", metavar="LOG",
+                        help="Path to logfile (./composer-cli.log)")
+    parser.add_argument("-a", "--api", dest="api_version", default="0", metavar="APIVER",
+                        help="API Version to use")
+    parser.add_argument("--test", dest="testmode", default=0, type=int, metavar="TESTMODE",
+                        help="Pass test mode to compose. 1=Mock compose with fail. 2=Mock compose with finished.")
+    parser.add_argument("-V", action="store_true", dest="showver",
+                        help="show program's version number and exit")
+
+    # Commands are implemented by parsing the remaining arguments outside of argparse
+    parser.add_argument('args', nargs=argparse.REMAINDER)
+
+    return parser

+
+ +
+ +
+ + +
+
+ +
+ +
+ + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/docs/html/_modules/composer/cli/compose.html b/docs/html/_modules/composer/cli/compose.html new file mode 100644 index 00000000..d1c788c3 --- /dev/null +++ b/docs/html/_modules/composer/cli/compose.html @@ -0,0 +1,677 @@ + + + + + + + + + + + composer.cli.compose — Lorax 29.1 documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + +
+ + + + +
+
+
+
+ +

Source code for composer.cli.compose

+#
+# Copyright (C) 2018  Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+import logging
+log = logging.getLogger("composer-cli")
+
+import sys
+import json
+
+from composer import http_client as client
+from composer.cli.utilities import argify, handle_api_result, packageNEVRA
+
+
[docs]def compose_cmd(opts):
+    """Process compose commands
+
+    :param opts: Cmdline arguments
+    :type opts: argparse.Namespace
+    :returns: Value to return from sys.exit()
+    :rtype: int
+
+    This dispatches the compose commands to a function
+    """
+    cmd_map = {
+        "status":   compose_status,
+        "types":    compose_types,
+        "start":    compose_start,
+        "log":      compose_log,
+        "cancel":   compose_cancel,
+        "delete":   compose_delete,
+        "info":     compose_info,
+        "metadata": compose_metadata,
+        "results":  compose_results,
+        "logs":     compose_logs,
+        "image":    compose_image,
+        }
+    if opts.args[1] not in cmd_map:
+        log.error("Unknown compose command: %s", opts.args[1])
+        return 1
+
+    return cmd_map[opts.args[1]](opts.socket, opts.api_version, opts.args[2:], opts.json, opts.testmode)

+
+
[docs]def compose_status(socket_path, api_version, args, show_json=False, testmode=0):
+    """Return the status of all known composes
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param api_version: Version of the API to talk to. eg. "0"
+    :type api_version: str
+    :param args: List of remaining arguments from the cmdline
+    :type args: list of str
+    :param show_json: Set to True to show the JSON output instead of the human readable output
+    :type show_json: bool
+    :param testmode: unused in this function
+    :type testmode: int
+
+    This doesn't map directly to an API command, it combines the results from queue, finished,
+    and failed so raw JSON output is not available.
+    """
+    def get_status(compose):
+        return {"id": compose["id"],
+                "blueprint": compose["blueprint"],
+                "version": compose["version"],
+                "compose_type": compose["compose_type"],
+                "image_size": compose["image_size"],
+                "status": compose["queue_status"]}
+
+    # Sort the status in a specific order
+    def sort_status(a):
+        order = ["RUNNING", "WAITING", "FINISHED", "FAILED"]
+        return (order.index(a["status"]), a["blueprint"], a["version"], a["compose_type"])
+
+    status = []
+
+    # Get the composes currently in the queue
+    api_route = client.api_url(api_version, "/compose/queue")
+    result = client.get_url_json(socket_path, api_route)
+    status.extend(list(map(get_status, result["run"] + result["new"])))
+
+    # Get the list of finished composes
+    api_route = client.api_url(api_version, "/compose/finished")
+    result = client.get_url_json(socket_path, api_route)
+    status.extend(list(map(get_status, result["finished"])))
+
+    # Get the list of failed composes
+    api_route = client.api_url(api_version, "/compose/failed")
+    result = client.get_url_json(socket_path, api_route)
+    status.extend(list(map(get_status, result["failed"])))
+
+    # Sort them by status (running, waiting, finished, failed) and then by name and version.
+    status.sort(key=sort_status)
+
+    if show_json:
+        print(json.dumps(status, indent=4))
+        return 0
+
+    # Print them as UUID blueprint STATUS
+    for c in status:
+        if c["image_size"] > 0:
+            image_size = str(c["image_size"])
+        else:
+            image_size = ""
+
+        print("%s %-8s %-15s %s %-16s %s" % (c["id"], c["status"], c["blueprint"], c["version"], c["compose_type"],
+                                             image_size))

+
+
+
[docs]def compose_types(socket_path, api_version, args, show_json=False, testmode=0):
+    """Return information about the supported compose types
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param api_version: Version of the API to talk to. eg. "0"
+    :type api_version: str
+    :param args: List of remaining arguments from the cmdline
+    :type args: list of str
+    :param show_json: Set to True to show the JSON output instead of the human readable output
+    :type show_json: bool
+    :param testmode: unused in this function
+    :type testmode: int
+
+    Add additional details to types that are known to composer-cli. Raw JSON output does not
+    include this extra information.
+    """
+    api_route = client.api_url(api_version, "/compose/types")
+    result = client.get_url_json(socket_path, api_route)
+    if show_json:
+        print(json.dumps(result, indent=4))
+        return 0
+
+    print("Compose Types: " + ", ".join([t["name"] for t in result["types"]]))

+
+
[docs]def compose_start(socket_path, api_version, args, show_json=False, testmode=0):
+    """Start a new compose using the selected blueprint and type
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param api_version: Version of the API to talk to. eg. "0"
+    :type api_version: str
+    :param args: List of remaining arguments from the cmdline
+    :type args: list of str
+    :param show_json: Set to True to show the JSON output instead of the human readable output
+    :type show_json: bool
+    :param testmode: Set to 1 to simulate a failed compose, set to 2 to simulate a finished one.
+    :type testmode: int
+
+    compose start <blueprint-name> <compose-type>
+    """
+    if len(args) == 0:
+        log.error("start is missing the blueprint name and output type")
+        return 1
+    if len(args) == 1:
+        log.error("start is missing the output type")
+        return 1
+
+    config = {
+        "blueprint_name": args[0],
+        "compose_type": args[1],
+        "branch": "master"
+        }
+    if testmode:
+        test_url = "?test=%d" % testmode
+    else:
+        test_url = ""
+    api_route = client.api_url(api_version, "/compose" + test_url)
+    result = client.post_url_json(socket_path, api_route, json.dumps(config))
+
+    if show_json:
+        print(json.dumps(result, indent=4))
+        return 0
+
+    for err in result.get("errors", []):
+        log.error(err)
+
+    if result["status"] == False or result.get("errors", False):
+        return 1
+
+    print("Compose %s added to the queue" % result["build_id"])
+    return 0

+
+
[docs]def compose_log(socket_path, api_version, args, show_json=False, testmode=0):
+    """Show the last part of the compose log
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param api_version: Version of the API to talk to. eg. "0"
+    :type api_version: str
+    :param args: List of remaining arguments from the cmdline
+    :type args: list of str
+    :param show_json: Set to True to show the JSON output instead of the human readable output
+    :type show_json: bool
+    :param testmode: unused in this function
+    :type testmode: int
+
+    compose log <uuid> [<size>kB]
+
+    This will display the last 1kB of the compose's log file. Can be used to follow progress
+    during the build.
+    """
+    if len(args) == 0:
+        log.error("log is missing the compose build id")
+        return 1
+    if len(args) == 2:
+        try:
+            log_size = int(args[1])
+        except ValueError:
+            log.error("Log size must be an integer.")
+            return 1
+    else:
+        log_size = 1024
+
+    api_route = client.api_url(api_version, "/compose/log/%s?size=%d" % (args[0], log_size))
+    try:
+        result = client.get_url_raw(socket_path, api_route)
+    except RuntimeError as e:
+        print(str(e))
+        return 1
+
+    print(result)
+    return 0

+
+
[docs]def compose_cancel(socket_path, api_version, args, show_json=False, testmode=0):
+    """Cancel a running compose
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param api_version: Version of the API to talk to. eg. "0"
+    :type api_version: str
+    :param args: List of remaining arguments from the cmdline
+    :type args: list of str
+    :param show_json: Set to True to show the JSON output instead of the human readable output
+    :type show_json: bool
+    :param testmode: unused in this function
+    :type testmode: int
+
+    compose cancel <uuid>
+
+    This will cancel a running compose. It does nothing if the compose has finished.
+    """
+    if len(args) == 0:
+        log.error("cancel is missing the compose build id")
+        return 1
+
+    api_route = client.api_url(api_version, "/compose/cancel/%s" % args[0])
+    result = client.delete_url_json(socket_path, api_route)
+    return handle_api_result(result, show_json)

+
+
[docs]def compose_delete(socket_path, api_version, args, show_json=False, testmode=0):
+    """Delete a finished compose's results
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param api_version: Version of the API to talk to. eg. "0"
+    :type api_version: str
+    :param args: List of remaining arguments from the cmdline
+    :type args: list of str
+    :param show_json: Set to True to show the JSON output instead of the human readable output
+    :type show_json: bool
+    :param testmode: unused in this function
+    :type testmode: int
+
+    compose delete <uuid,...>
+
+    Delete the listed compose results. It will only delete results for composes that have finished
+    or failed, not a running compose.
+    """
+    if len(args) == 0:
+        log.error("delete is missing the compose build id")
+        return 1
+
+    api_route = client.api_url(api_version, "/compose/delete/%s" % (",".join(argify(args))))
+    result = client.delete_url_json(socket_path, api_route)
+
+    if show_json:
+        print(json.dumps(result, indent=4))
+        return 0
+
+    # Print any errors
+    for err in result.get("errors", []):
+        log.error(err)
+
+    if result.get("errors", []):
+        return 1
+    else:
+        return 0

+
+
[docs]def compose_info(socket_path, api_version, args, show_json=False, testmode=0):
+    """Return detailed information about the compose
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param api_version: Version of the API to talk to. eg. "0"
+    :type api_version: str
+    :param args: List of remaining arguments from the cmdline
+    :type args: list of str
+    :param show_json: Set to True to show the JSON output instead of the human readable output
+    :type show_json: bool
+    :param testmode: unused in this function
+    :type testmode: int
+
+    compose info <uuid>
+
+    This returns information about the compose, including the blueprint and the dependencies.
+    """
+    if len(args) == 0:
+        log.error("info is missing the compose build id")
+        return 1
+
+    api_route = client.api_url(api_version, "/compose/info/%s" % args[0])
+    result = client.get_url_json(socket_path, api_route)
+    if show_json:
+        print(json.dumps(result, indent=4))
+        return 0
+
+    for err in result.get("errors", []):
+        log.error(err)
+
+    if result.get("errors", []):
+        return 1
+
+    if result["image_size"] > 0:
+        image_size = str(result["image_size"])
+    else:
+        image_size = ""
+
+
+    print("%s %-8s %-15s %s %-16s %s" % (result["id"],
+                                         result["queue_status"],
+                                         result["blueprint"]["name"],
+                                         result["blueprint"]["version"],
+                                         result["compose_type"],
+                                         image_size))
+    print("Packages:")
+    for p in result["blueprint"]["packages"]:
+        print("    %s-%s" % (p["name"], p["version"]))
+
+    print("Modules:")
+    for m in result["blueprint"]["modules"]:
+        print("    %s-%s" % (m["name"], m["version"]))
+
+    print("Dependencies:")
+    for d in result["deps"]["packages"]:
+        print("    " + packageNEVRA(d))

+
+
[docs]def compose_metadata(socket_path, api_version, args, show_json=False, testmode=0):
+    """Download a tar file of the compose's metadata
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param api_version: Version of the API to talk to. eg. "0"
+    :type api_version: str
+    :param args: List of remaining arguments from the cmdline
+    :type args: list of str
+    :param show_json: Set to True to show the JSON output instead of the human readable output
+    :type show_json: bool
+    :param testmode: unused in this function
+    :type testmode: int
+
+    compose metadata <uuid>
+
+    Saves the metadata as uuid-metadata.tar
+    """
+    if len(args) == 0:
+        log.error("metadata is missing the compose build id")
+        return 1
+
+    api_route = client.api_url(api_version, "/compose/metadata/%s" % args[0])
+    return client.download_file(socket_path, api_route)

+
+
[docs]def compose_results(socket_path, api_version, args, show_json=False, testmode=0):
+    """Download a tar file of the compose's results
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param api_version: Version of the API to talk to. eg. "0"
+    :type api_version: str
+    :param args: List of remaining arguments from the cmdline
+    :type args: list of str
+    :param show_json: Set to True to show the JSON output instead of the human readable output
+    :type show_json: bool
+    :param testmode: unused in this function
+    :type testmode: int
+
+    compose results <uuid>
+
+    The results includes the metadata, output image, and logs.
+    It is saved as uuid.tar
+    """
+    if len(args) == 0:
+        log.error("results is missing the compose build id")
+        return 1
+
+    api_route = client.api_url(api_version, "/compose/results/%s" % args[0])
+    return client.download_file(socket_path, api_route, sys.stdout.isatty())

+
+
[docs]def compose_logs(socket_path, api_version, args, show_json=False, testmode=0):
+    """Download a tar of the compose's logs
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param api_version: Version of the API to talk to. eg. "0"
+    :type api_version: str
+    :param args: List of remaining arguments from the cmdline
+    :type args: list of str
+    :param show_json: Set to True to show the JSON output instead of the human readable output
+    :type show_json: bool
+    :param testmode: unused in this function
+    :type testmode: int
+
+    compose logs <uuid>
+
+    Saves the logs as uuid-logs.tar
+    """
+    if len(args) == 0:
+        log.error("logs is missing the compose build id")
+        return 1
+
+    api_route = client.api_url(api_version, "/compose/logs/%s" % args[0])
+    return client.download_file(socket_path, api_route, sys.stdout.isatty())

+
+
[docs]def compose_image(socket_path, api_version, args, show_json=False, testmode=0):
+    """Download the compose's output image
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param api_version: Version of the API to talk to. eg. "0"
+    :type api_version: str
+    :param args: List of remaining arguments from the cmdline
+    :type args: list of str
+    :param show_json: Set to True to show the JSON output instead of the human readable output
+    :type show_json: bool
+    :param testmode: unused in this function
+    :type testmode: int
+
+    compose image <uuid>
+
+    This downloads only the result image, saving it as the image name, which depends on the type
+    of compose that was selected.
+    """
+    if len(args) == 0:
+        log.error("logs is missing the compose build id")
+        return 1
+
+    api_route = client.api_url(api_version, "/compose/image/%s" % args[0])
+    return client.download_file(socket_path, api_route, sys.stdout.isatty())

+
+ +
+ +
+ + +
+
+ +
+ +
+ + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/docs/html/_modules/composer/cli/modules.html b/docs/html/_modules/composer/cli/modules.html new file mode 100644 index 00000000..21d8a58b --- /dev/null +++ b/docs/html/_modules/composer/cli/modules.html @@ -0,0 +1,264 @@ + + + + + + + + + + + composer.cli.modules — Lorax 29.1 documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + +
+ + + + +
+
+
+
+ +

Source code for composer.cli.modules

+#
+# Copyright (C) 2018  Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+import logging
+log = logging.getLogger("composer-cli")
+
+import json
+
+from composer import http_client as client
+
+
[docs]def modules_cmd(opts):
+    """Process modules commands
+
+    :param opts: Cmdline arguments
+    :type opts: argparse.Namespace
+    :returns: Value to return from sys.exit()
+    :rtype: int
+    """
+    if opts.args[1] != "list":
+        log.error("Unknown modules command: %s", opts.args[1])
+        return 1
+
+    api_route = client.api_url(opts.api_version, "/modules/list")
+    result = client.get_url_json(opts.socket, api_route)
+    if opts.json:
+        print(json.dumps(result, indent=4))
+        return 0
+
+    print("Modules:\n" + "\n".join(["    "+r["name"] for r in result["modules"]]))
+
+    return 0

+
+ +
+ +
+ + +
+
+ +
+ +
+ + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/docs/html/_modules/composer/cli/projects.html b/docs/html/_modules/composer/cli/projects.html new file mode 100644 index 00000000..1fa8aa56 --- /dev/null +++ b/docs/html/_modules/composer/cli/projects.html @@ -0,0 +1,326 @@ + + + + + + + + + + + composer.cli.projects — Lorax 29.1 documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + +
+ + + + +
+
+
+
+ +

Source code for composer.cli.projects

+#
+# Copyright (C) 2018  Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+import logging
+log = logging.getLogger("composer-cli")
+
+import json
+import textwrap
+
+from composer import http_client as client
+
+
[docs]def projects_cmd(opts):
+    """Process projects commands
+
+    :param opts: Cmdline arguments
+    :type opts: argparse.Namespace
+    :returns: Value to return from sys.exit()
+    :rtype: int
+    """
+    cmd_map = {
+        "list":      projects_list,
+        "info":      projects_info,
+        }
+    if opts.args[1] not in cmd_map:
+        log.error("Unknown projects command: %s", opts.args[1])
+        return 1
+
+    return cmd_map[opts.args[1]](opts.socket, opts.api_version, opts.args[2:], opts.json)

+
+
[docs]def projects_list(socket_path, api_version, args, show_json=False):
+    """Output the list of available projects
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param api_version: Version of the API to talk to. eg. "0"
+    :type api_version: str
+    :param args: List of remaining arguments from the cmdline
+    :type args: list of str
+    :param show_json: Set to True to show the JSON output instead of the human readable output
+    :type show_json: bool
+
+    projects list
+    """
+    api_route = client.api_url(api_version, "/projects/list")
+    result = client.get_url_json(socket_path, api_route)
+    if show_json:
+        print(json.dumps(result, indent=4))
+        return 0
+
+    for proj in result["projects"]:
+        for k in ["name", "summary", "homepage", "description"]:
+            print("%s: %s" % (k.title(), textwrap.fill(proj[k], subsequent_indent=" " * (len(k)+2))))
+        print("\n\n")
+
+    return 0

+
+
[docs]def projects_info(socket_path, api_version, args, show_json=False):
+    """Output info on a list of projects
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param api_version: Version of the API to talk to. eg. "0"
+    :type api_version: str
+    :param args: List of remaining arguments from the cmdline
+    :type args: list of str
+    :param show_json: Set to True to show the JSON output instead of the human readable output
+    :type show_json: bool
+
+    projects info <project,...>
+    """
+    if len(args) == 0:
+        log.error("projects info is missing the packages")
+        return 1
+
+    api_route = client.api_url(api_version, "/projects/info/%s" % ",".join(args))
+    result = client.get_url_json(socket_path, api_route)
+    if show_json:
+        print(json.dumps(result, indent=4))
+        return 0
+
+    for proj in result["projects"]:
+        for k in ["name", "summary", "homepage", "description"]:
+            print("%s: %s" % (k.title(), textwrap.fill(proj[k], subsequent_indent=" " * (len(k)+2))))
+        print("Builds: ")
+        for build in proj["builds"]:
+            print("    %s%s-%s.%s at %s for %s" % ("" if not build["epoch"] else build["epoch"] + ":",
+                                                       build["source"]["version"],
+                                                       build["release"],
+                                                       build["arch"],
+                                                       build["build_time"],
+                                                       build["changelog"]))
+        print("")
+    return 0

+
+ +
+ +
+ + +
+
+ +
+ +
+ + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/docs/html/_modules/composer/cli/utilities.html b/docs/html/_modules/composer/cli/utilities.html new file mode 100644 index 00000000..b8cf5b62 --- /dev/null +++ b/docs/html/_modules/composer/cli/utilities.html @@ -0,0 +1,304 @@ + + + + + + + + + + + composer.cli.utilities — Lorax 29.1 documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + +
+ + + + +
+
+
+
+ +

Source code for composer.cli.utilities

+#
+# Copyright (C) 2018  Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+import logging
+log = logging.getLogger("composer-cli")
+
+import json
+
+
[docs]def argify(args):
+    """Take a list of human args and return a list with each item
+
+    :param args: list of strings with possible commas and spaces
+    :type args: list of str
+    :returns: List of all the items
+    :rtype: list of str
+
+    Examples:
+
+    ["one,two", "three", ",four", ",five,"] returns ["one", "two", "three", "four", "five"]
+    """
+    return [i for i in [arg for entry in args for arg in entry.split(",")] if i]

+
+
[docs]def toml_filename(blueprint_name):
+    """Convert a blueprint name into a filename.toml
+
+    :param blueprint_name: The blueprint's name
+    :type blueprint_name: str
+    :returns: The blueprint name with ' ' converted to - and .toml appended
+    :rtype: str
+    """
+    return blueprint_name.replace(" ", "-") + ".toml"

+
+
[docs]def frozen_toml_filename(blueprint_name):
+    """Convert a blueprint name into a filename.toml
+
+    :param blueprint_name: The blueprint's name
+    :type blueprint_name: str
+    :returns: The blueprint name with ' ' converted to - and .toml appended
+    :rtype: str
+    """
+    return blueprint_name.replace(" ", "-") + ".frozen.toml"

+
+
[docs]def handle_api_result(result, show_json=False):
+    """Log any errors, return the correct value
+
+    :param result: JSON result from the http query
+    :type result: dict
+    """
+    if show_json:
+        print(json.dumps(result, indent=4))
+
+    for err in result.get("errors", []):
+        log.error(err)
+
+    if result["status"] == True:
+        return 0
+    else:
+        return 1

+
+
[docs]def packageNEVRA(pkg):
+    """Return the package info as a NEVRA
+
+    :param pkg: The package details
+    :type pkg: dict
+    :returns: name-[epoch:]version-release-arch
+    :rtype: str
+    """
+    if pkg["epoch"]:
+        return "%s-%s:%s-%s.%s" % (pkg["name"], pkg["epoch"], pkg["version"], pkg["release"], pkg["arch"])
+    else:
+        return "%s-%s-%s.%s" % (pkg["name"], pkg["version"], pkg["release"], pkg["arch"])

+
+ +
+ +
+ + +
+
+ +
+ +
+ + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/docs/html/_modules/composer/http_client.html b/docs/html/_modules/composer/http_client.html new file mode 100644 index 00000000..4995a72e --- /dev/null +++ b/docs/html/_modules/composer/http_client.html @@ -0,0 +1,419 @@ + + + + + + + + + + + composer.http_client — Lorax 29.1 documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + +
+ + + + +
+
+
+
+ +

Source code for composer.http_client

+#
+# Copyright (C) 2018  Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+import logging
+log = logging.getLogger("composer-cli")
+
+import os
+import sys
+import json
+
+from composer.unix_socket import UnixHTTPConnectionPool
+
+
[docs]def api_url(api_version, url):
+    """Return the versioned path to the API route
+
+    :param api_version: The version of the API to talk to. eg. "0"
+    :type api_version: str
+    :param url: The API route to talk to
+    :type url: str
+    :returns: The full url to use for the route and API version
+    :rtype: str
+    """
+    return os.path.normpath("/api/v%s/%s" % (api_version, url))

+
+
[docs]def get_url_raw(socket_path, url):
+    """Return the raw results of a GET request
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param url: URL to request
+    :type url: str
+    :returns: The raw response from the server
+    :rtype: str
+    """
+    http = UnixHTTPConnectionPool(socket_path)
+    r = http.request("GET", url)
+    if r.status == 400:
+        err = json.loads(r.data.decode("utf-8"))
+        if "status" in err and err["status"] == False:
+            raise RuntimeError(", ".join(err["errors"]))
+
+    return r.data.decode('utf-8')

+
+
[docs]def get_url_json(socket_path, url):
+    """Return the JSON results of a GET request
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param url: URL to request
+    :type url: str
+    :returns: The json response from the server
+    :rtype: dict
+    """
+    http = UnixHTTPConnectionPool(socket_path)
+    r = http.request("GET", url)
+    return json.loads(r.data.decode('utf-8'))

+
+
[docs]def delete_url_json(socket_path, url):
+    """Send a DELETE request to the url and return JSON response
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param url: URL to send DELETE to
+    :type url: str
+    :returns: The json response from the server
+    :rtype: dict
+    """
+    http = UnixHTTPConnectionPool(socket_path)
+    r = http.request("DELETE", url)
+    return json.loads(r.data.decode("utf-8"))

+
+
[docs]def post_url(socket_path, url, body):
+    """POST raw data to the URL
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param url: URL to send POST to
+    :type url: str
+    :param body: The data for the body of the POST
+    :type body: str
+    :returns: The json response from the server
+    :rtype: dict
+    """
+    http = UnixHTTPConnectionPool(socket_path)
+    r = http.request("POST", url,
+                     body=body.encode("utf-8"))
+    return json.loads(r.data.decode("utf-8"))

+
+
[docs]def post_url_toml(socket_path, url, body):
+    """POST a TOML string to the URL
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param url: URL to send POST to
+    :type url: str
+    :param body: The data for the body of the POST
+    :type body: str
+    :returns: The json response from the server
+    :rtype: dict
+    """
+    http = UnixHTTPConnectionPool(socket_path)
+    r = http.request("POST", url,
+                     body=body.encode("utf-8"),
+                     headers={"Content-Type": "text/x-toml"})
+    return json.loads(r.data.decode("utf-8"))

+
+
[docs]def post_url_json(socket_path, url, body):
+    """POST some JSON data to the URL
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param url: URL to send POST to
+    :type url: str
+    :param body: The data for the body of the POST
+    :type body: str
+    :returns: The json response from the server
+    :rtype: dict
+    """
+    http = UnixHTTPConnectionPool(socket_path)
+    r = http.request("POST", url,
+                     body=body.encode("utf-8"),
+                     headers={"Content-Type": "application/json"})
+    return json.loads(r.data.decode("utf-8"))

+
+
[docs]def get_filename(headers):
+    """Get the filename from the response header
+
+    :param response: The urllib3 response object
+    :type response: Response
+    :raises: RuntimeError if it cannot find a filename in the header
+    :returns: Filename from content-disposition header
+    :rtype: str
+    """
+    log.debug("Headers = %s", headers)
+    if "content-disposition" not in headers:
+        raise RuntimeError("No Content-Disposition header; cannot get filename")
+
+    try:
+        k, _, v = headers["content-disposition"].split(";")[1].strip().partition("=")
+        if k != "filename":
+            raise RuntimeError("No filename= found in content-disposition header")
+    except RuntimeError:
+        raise
+    except Exception as e:
+        raise RuntimeError("Error parsing filename from content-disposition header: %s" % str(e))
+
+    return os.path.basename(v)

+
+
[docs]def download_file(socket_path, url, progress=True):
+    """Download a file, saving it to the CWD with the included filename
+
+    :param socket_path: Path to the Unix socket to use for API communication
+    :type socket_path: str
+    :param url: URL to send POST to
+    :type url: str
+    """
+    http = UnixHTTPConnectionPool(socket_path)
+    r = http.request("GET", url, preload_content=False)
+    if r.status == 400:
+        err = json.loads(r.data.decode("utf-8"))
+        if not err["status"]:
+            raise RuntimeError(", ".join(err["errors"]))
+
+    filename = get_filename(r.headers)
+    if os.path.exists(filename):
+        msg = "%s exists, skipping download" % filename
+        log.error(msg)
+        raise RuntimeError(msg)
+
+    with open(filename, "wb") as f:
+        while True:
+            data = r.read(10 * 1024**2)
+            if not data:
+                break
+            f.write(data)
+
+            if progress:
+                data_written = f.tell()
+                if data_written > 5 * 1024**2:
+                    sys.stdout.write("%s: %0.2f MB    \r" % (filename, data_written / 1024**2))
+                else:
+                    sys.stdout.write("%s: %0.2f kB\r" % (filename, data_written / 1024))
+                sys.stdout.flush()
+
+    print("")
+    r.release_conn()
+
+    return 0

+
+ +
+ +
+ + +
+
+ +
+ +
+ + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/docs/html/_modules/composer/unix_socket.html b/docs/html/_modules/composer/unix_socket.html new file mode 100644 index 00000000..7e26736e --- /dev/null +++ b/docs/html/_modules/composer/unix_socket.html @@ -0,0 +1,279 @@ + + + + + + + + + + + composer.unix_socket — Lorax 29.1 documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + +
+ + + + +
+
+
+
+ +

Source code for composer.unix_socket

+#
+# Copyright (C) 2018  Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+import http.client
+import socket
+import urllib3
+
+
+# These 2 classes were adapted and simplified for use with just urllib3.
+# Originally from https://github.com/msabramo/requests-unixsocket/blob/master/requests_unixsocket/adapters.py
+
+# The following was adapted from some code from docker-py
+# https://github.com/docker/docker-py/blob/master/docker/transport/unixconn.py
+
[docs]class UnixHTTPConnection(http.client.HTTPConnection, object):
+
+    def __init__(self, socket_path, timeout=60):
+        """Create an HTTP connection to a unix domain socket
+
+        :param socket_path: The path to the Unix domain socket
+        :param timeout: Number of seconds to timeout the connection
+        """
+        super(UnixHTTPConnection, self).__init__('localhost', timeout=timeout)
+        self.socket_path = socket_path
+        self.sock = None
+
+    def __del__(self):  # base class does not have d'tor
+        if self.sock:
+            self.sock.close()
+
+
[docs]    def connect(self):
+        sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+        sock.settimeout(self.timeout)
+        sock.connect(self.socket_path)
+        self.sock = sock

+
+
[docs]class UnixHTTPConnectionPool(urllib3.connectionpool.HTTPConnectionPool):
+
+    def __init__(self, socket_path, timeout=60):
+        """Create a connection pool using a Unix domain socket
+
+        :param socket_path: The path to the Unix domain socket
+        :param timeout: Number of seconds to timeout the connection
+        """
+        super(UnixHTTPConnectionPool, self).__init__('localhost', timeout=timeout)
+        self.socket_path = socket_path
+
+    def _new_conn(self):
+        return UnixHTTPConnection(self.socket_path, self.timeout)

+
+ +
+ +
+ + +
+
+ +
+ +
+ + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/docs/html/_modules/index.html b/docs/html/_modules/index.html index 94d994e8..469df984 100644 --- a/docs/html/_modules/index.html +++ b/docs/html/_modules/index.html @@ -8,7 +8,7 @@ - Overview: module code — Lorax 29.0 documentation + Overview: module code — Lorax 29.1 documentation @@ -25,24 +25,17 @@ - - - - - - - - - + + + - +
@@ -64,7 +57,7 @@
- 29.0 + 29.1
@@ -93,8 +86,10 @@
  • Before Lorax
  • Lorax
  • livemedia-creator
    • +
  • lorax-composer
    • +
  • composer-cli
  • Product and Updates Images
    • -
  • pylorax
    • +
  • src
    • @@ -106,7 +101,7 @@
    -
    - Built with Sphinx using a theme provided by Read the Docs. + Built with Sphinx using a theme provided by Read the Docs. @@ -640,7 +631,8 @@ - - - + \ No newline at end of file diff --git a/docs/html/_modules/pylorax/api/cmdline.html b/docs/html/_modules/pylorax/api/cmdline.html new file mode 100644 index 00000000..7f9cb910 --- /dev/null +++ b/docs/html/_modules/pylorax/api/cmdline.html @@ -0,0 +1,278 @@ + + + + + + + + + + + pylorax.api.cmdline — Lorax 29.1 documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    + + + + +
    + + + + + +
    + +
    + + + + + + + + + + + + + + + + + +
    + + + + +
    +
    +
    +
    + +

    Source code for pylorax.api.cmdline

    +#
+# cmdline.py
+#
+# Copyright (C) 2018 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+import os
+import sys
+import argparse
+
+from pylorax import vernum
+
+version = "{0}-{1}".format(os.path.basename(sys.argv[0]), vernum)
+
+
    [docs]def lorax_composer_parser():
+    """ Return the ArgumentParser for lorax-composer"""
+
+    parser = argparse.ArgumentParser(description="Lorax Composer API Server",
+                                     fromfile_prefix_chars="@")
+
+    parser.add_argument("--socket", default="/run/weldr/api.socket", metavar="SOCKET",
+                        help="Path to the socket file to listen on")
+    parser.add_argument("--user", default="weldr", metavar="USER",
+                        help="User to use for reduced permissions")
+    parser.add_argument("--group", default="weldr", metavar="GROUP",
+                        help="Group to set ownership of the socket to")
+    parser.add_argument("--log", dest="logfile", default="/var/log/lorax-composer/composer.log", metavar="LOG",
+                        help="Path to logfile (/var/log/lorax-composer/composer.log)")
+    parser.add_argument("--mockfiles", default="/var/tmp/bdcs-mockfiles/", metavar="MOCKFILES",
+                        help="Path to JSON files used for /api/mock/ paths (/var/tmp/bdcs-mockfiles/)")
+    parser.add_argument("--sharedir", type=os.path.abspath, metavar="SHAREDIR",
+                        help="Directory containing all the templates. Overrides config file sharedir")
+    parser.add_argument("-V", action="store_true", dest="showver",
+                        help="show program's version number and exit")
+    parser.add_argument("-c", "--config", default="/etc/lorax/composer.conf", metavar="CONFIG",
+                        help="Path to lorax-composer configuration file.")
+    parser.add_argument("--releasever", default=None, metavar="STRING",
+                        help="Release version to use for $releasever in dnf repository urls")
+    parser.add_argument("--tmp", default="/var/tmp",
+                        help="Top level temporary directory")
+    parser.add_argument("--proxy", default=None, metavar="PROXY",
+                        help="Set proxy for DNF, overrides configuration file setting.")
+    parser.add_argument("BLUEPRINTS", metavar="BLUEPRINTS",
+                        help="Path to the blueprints")
+
+    return parser
    
+
    + +
    + +
    + + +
    +
    + +
    + +
    + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/docs/html/_modules/pylorax/api/compose.html b/docs/html/_modules/pylorax/api/compose.html new file mode 100644 index 00000000..5631122f --- /dev/null +++ b/docs/html/_modules/pylorax/api/compose.html @@ -0,0 +1,728 @@ + + + + + + + + + + + pylorax.api.compose — Lorax 29.1 documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    + + + + +
    + + + + + +
    + +
    + + + + + + + + + + + + + + + + + +
    + + + + +
    +
    +
    +
    + +

    Source code for pylorax.api.compose

    +# Copyright (C) 2018 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+""" Setup for composing an image
+
+Adding New Output Types
+-----------------------
+
+The new output type must add a kickstart template to ./share/composer/ where the
+name of the kickstart (without the trailing .ks) matches the entry in compose_args.
+
+The kickstart should not have any url or repo entries, these will be added at build
+time. The %packages section should be the last thing, and while it can contain mandatory
+packages required by the output type, it should not have the trailing %end because the
+package NEVRAs will be appended to it at build time.
+
+compose_args should have a name matching the kickstart, and it should set the novirt_install
+parameters needed to generate the desired output. Other types should be set to False.
+
+"""
+import logging
+log = logging.getLogger("lorax-composer")
+
+import os
+from glob import glob
+from math import ceil
+import pytoml as toml
+import shutil
+from uuid import uuid4
+
+from pyanaconda.simpleconfig import SimpleConfigFile
+
+# Use pykickstart to calculate disk image size
+from pykickstart.parser import KickstartParser
+from pykickstart.version import makeVersion
+
+from pylorax.api.projects import projects_depsolve_with_size, dep_nevra
+from pylorax.api.projects import ProjectsError
+from pylorax.api.recipes import read_recipe_and_id
+from pylorax.imgutils import default_image_name
+from pylorax.sysutils import joinpaths
+
+
+
    [docs]def repo_to_ks(r, url="url"):
+    """ Return a kickstart line with the correct args.
+    :param r: DNF repository information
+    :type r: dnf.Repo
+    :param url: "url" or "baseurl" to use for the baseurl parameter
+    :type url: str
+    :returns: kickstart command arguments for url/repo command
+    :rtype: str
+
+    Set url to "baseurl" if it is a repo, leave it as "url" for the installation url.
+    """
+    cmd = ""
+    # url uses --url not --baseurl
+    if r.baseurl:
+        cmd += '--%s="%s" ' % (url, r.baseurl[0])
+    elif r.metalink:
+        cmd += '--metalink="%s" ' % r.metalink
+    elif r.mirrorlist:
+        cmd += '--mirrorlist="%s" ' % r.mirrorlist
+    else:
+        raise RuntimeError("Repo has no baseurl, metalink, or mirrorlist")
+
+    if r.proxy:
+        cmd += '--proxy="%s" ' % r.proxy
+
+    if not r.sslverify:
+        cmd += '--noverifyssl'
+
+    return cmd
    
+
+
+
    [docs]def write_ks_user(f, user):
+    """ Write kickstart user and sshkey entry
+
+    :param f: kickstart file object
+    :type f: open file object
+    :param user: A blueprint user dictionary
+    :type user: dict
+
+    If the entry contains a ssh key, use sshkey to write it
+    All of the user fields are optional, except name, write out a kickstart user entry
+    with whatever options are relevant.
+    """
+    if "name" not in user:
+        raise RuntimeError("user entry requires a name")
+
+    # ssh key uses the sshkey kickstart command
+    if "key" in user:
+        f.write('sshkey --user %s "%s"\n' % (user["name"], user["key"]))
+
+    # Write out the user kickstart command, much of it is optional
+    f.write("user --name %s" % user["name"])
+    if "home" in user:
+        f.write(" --homedir %s" % user["home"])
+
+    if "password" in user:
+        if any(user["password"].startswith(prefix) for prefix in ["$2b$", "$6$", "$5$"]):
+            log.debug("Detected pre-crypted password")
+            f.write(" --iscrypted")
+        else:
+            log.debug("Detected plaintext password")
+            f.write(" --plaintext")
+
+        f.write(" --password \"%s\"" % user["password"])
+
+    if "shell" in user:
+        f.write(" --shell %s" % user["shell"])
+
+    if "uid" in user:
+        f.write(" --uid %d" % int(user["uid"]))
+
+    if "gid" in user:
+        f.write(" --gid %d" % int(user["gid"]))
+
+    if "description" in user:
+        f.write(" --gecos \"%s\"" % user["description"])
+
+    if "groups" in user:
+        f.write(" --groups %s" % ",".join(user["groups"]))
+
+    f.write("\n")
    
+
+
+
    [docs]def write_ks_group(f, group):
+    """ Write kickstart group entry
+
+    :param f: kickstart file object
+    :type f: open file object
+    :param group: A blueprint group dictionary
+    :type user: dict
+
+    gid is optional
+    """
+    if "name" not in group:
+        raise RuntimeError("group entry requires a name")
+
+    f.write("group --name %s" % group["name"])
+    if "gid" in group:
+        f.write(" --gid %d" % int(group["gid"]))
+
+    f.write("\n")
    
+
+
+
    [docs]def add_customizations(f, recipe):
+    """ Add customizations to the kickstart file
+
+    :param f: kickstart file object
+    :type f: open file object
+    :param recipe:
+    :type recipe: Recipe object
+    :returns: None
+    :raises: RuntimeError if there was a problem writing to the kickstart
+    """
+    if "customizations" not in recipe:
+        return
+    customizations = recipe["customizations"]
+
+    if "hostname" in customizations:
+        f.write("network --hostname=%s\n" % customizations["hostname"])
+
+    # TODO - remove this, should use user section to define this
+    if "sshkey" in customizations:
+        # This is a list of entries
+        for sshkey in customizations["sshkey"]:
+            if "user" not in sshkey or "key" not in sshkey:
+                log.error("%s is incorrect, skipping", sshkey)
+                continue
+            f.write('sshkey --user %s "%s"\n' % (sshkey["user"], sshkey["key"]))
+
+    # Creating a user also creates a group. Make a list of the names for later
+    user_groups = []
+    if "user" in customizations:
+        # only name is required, everything else is optional
+        for user in customizations["user"]:
+            write_ks_user(f, user)
+            user_groups.append(user["name"])
+
+    if "group" in customizations:
+        for group in customizations["group"]:
+            if group["name"] not in user_groups:
+                write_ks_group(f, group)
+            else:
+                log.warning("Skipping group %s, already created by user", group["name"])
    
+
+
    [docs]def start_build(cfg, dnflock, gitlock, branch, recipe_name, compose_type, test_mode=0):
+    """ Start the build
+
+    :param cfg: Configuration object
+    :type cfg: ComposerConfig
+    :param dnflock: Lock and YumBase for depsolving
+    :type dnflock: YumLock
+    :param recipe: The recipe to build
+    :type recipe: str
+    :param compose_type: The type of output to create from the recipe
+    :type compose_type: str
+    :returns: Unique ID for the build that can be used to track its status
+    :rtype: str
+    """
+    share_dir = cfg.get("composer", "share_dir")
+    lib_dir = cfg.get("composer", "lib_dir")
+
+    # Make sure compose_type is valid
+    if compose_type not in compose_types(share_dir):
+        raise RuntimeError("Invalid compose type (%s), must be one of %s" % (compose_type, compose_types(share_dir)))
+
+    with gitlock.lock:
+        (commit_id, recipe) = read_recipe_and_id(gitlock.repo, branch, recipe_name)
+
+    # Combine modules and packages and depsolve the list
+    # TODO include the version/glob in the depsolving
+    module_names = [m["name"] for m in recipe["modules"] or []]
+    package_names = [p["name"] for p in recipe["packages"] or []]
+    projects = sorted(set(module_names+package_names), key=lambda n: n.lower())
+    deps = []
+    try:
+        with dnflock.lock:
+            (installed_size, deps) = projects_depsolve_with_size(dnflock.dbo, projects, with_core=False)
+    except ProjectsError as e:
+        log.error("start_build depsolve: %s", str(e))
+        raise RuntimeError("Problem depsolving %s: %s" % (recipe["name"], str(e)))
+
+    # Read the kickstart template for this type
+    ks_template_path = joinpaths(share_dir, "composer", compose_type) + ".ks"
+    ks_template = open(ks_template_path, "r").read()
+
+    # How much space will the packages in the default template take?
+    ks_version = makeVersion()
+    ks = KickstartParser(ks_version, errorsAreFatal=False, missingIncludeIsFatal=False)
+    ks.readKickstartFromString(ks_template+"\n%end\n")
+    try:
+        with dnflock.lock:
+            (template_size, _) = projects_depsolve_with_size(dnflock.dbo, ks.handler.packages.packageList,
+                                                             with_core=not ks.handler.packages.nocore)
+    except ProjectsError as e:
+        log.error("start_build depsolve: %s", str(e))
+        raise RuntimeError("Problem depsolving %s: %s" % (recipe["name"], str(e)))
+    log.debug("installed_size = %d, template_size=%d", installed_size, template_size)
+
+    # Minimum LMC disk size is 1GiB, and anaconda bumps the estimated size up by 10% (which doesn't always work).
+    # XXX BUT Anaconda has a bug, it won't execute a kickstart on a disk smaller than 3000 MB
+    # XXX There is an upstream patch pending, but until then, use that as the minimum
+    installed_size = max(3e9, int((installed_size+template_size))) * 1.2
+    log.debug("/ partition size = %d", installed_size)
+
+    # Create the results directory
+    build_id = str(uuid4())
+    results_dir = joinpaths(lib_dir, "results", build_id)
+    os.makedirs(results_dir)
+
+    # Write the recipe commit hash
+    commit_path = joinpaths(results_dir, "COMMIT")
+    with open(commit_path, "w") as f:
+        f.write(commit_id)
+
+    # Write the original recipe
+    recipe_path = joinpaths(results_dir, "blueprint.toml")
+    with open(recipe_path, "w") as f:
+        f.write(recipe.toml())
+
+    # Write the frozen recipe
+    frozen_recipe = recipe.freeze(deps)
+    recipe_path = joinpaths(results_dir, "frozen.toml")
+    with open(recipe_path, "w") as f:
+        f.write(frozen_recipe.toml())
+
+    # Write out the dependencies to the results dir
+    deps_path = joinpaths(results_dir, "deps.toml")
+    with open(deps_path, "w") as f:
+        f.write(toml.dumps({"packages":deps}))
+
+    # Save a copy of the original kickstart
+    shutil.copy(ks_template_path, results_dir)
+
+    with dnflock.lock:
+        repos = list(dnflock.dbo.repos.iter_enabled())
+    if not repos:
+        raise RuntimeError("No enabled repos, canceling build.")
+
+    # Create the final kickstart with repos and package list
+    ks_path = joinpaths(results_dir, "final-kickstart.ks")
+    with open(ks_path, "w") as f:
+        ks_url = repo_to_ks(repos[0], "url")
+        log.debug("url = %s", ks_url)
+        f.write('url %s\n' % ks_url)
+        for idx, r in enumerate(repos[1:]):
+            ks_repo = repo_to_ks(r, "baseurl")
+            log.debug("repo composer-%s = %s", idx, ks_repo)
+            f.write('repo --name="composer-%s" %s\n' % (idx, ks_repo))
+
+        # Write the root partition and it's size in MB (rounded up)
+        f.write('part / --fstype="ext4" --size=%d\n' % ceil(installed_size / 1024**2))
+
+        f.write(ks_template)
+
+        for d in deps:
+            f.write(dep_nevra(d)+"\n")
+        f.write("%end\n")
+
+        add_customizations(f, recipe)
+
+    # Setup the config to pass to novirt_install
+    log_dir = joinpaths(results_dir, "logs/")
+    cfg_args = compose_args(compose_type)
+
+    # Get the title, project, and release version from the host
+    if not os.path.exists("/etc/os-release"):
+        log.error("/etc/os-release is missing, cannot determine product or release version")
+    os_release = SimpleConfigFile("/etc/os-release")
+    os_release.read()
+
+    log.debug("os_release = %s", os_release)
+
+    cfg_args["title"] = os_release.get("PRETTY_NAME")
+    cfg_args["project"] = os_release.get("NAME")
+    cfg_args["releasever"] = os_release.get("VERSION_ID")
+    cfg_args["volid"] = ""
+
+    cfg_args.update({
+        "compression":      "xz",
+        "compress_args":    [],
+        "ks":               [ks_path],
+        "logfile":          log_dir,
+        "timeout":          60,                     # 60 minute timeout
+    })
+    with open(joinpaths(results_dir, "config.toml"), "w") as f:
+        f.write(toml.dumps(cfg_args))
+
+    # Set the initial status
+    open(joinpaths(results_dir, "STATUS"), "w").write("WAITING")
+
+    # Set the test mode, if requested
+    if test_mode > 0:
+        open(joinpaths(results_dir, "TEST"), "w").write("%s" % test_mode)
+
+    log.info("Adding %s (%s %s) to compose queue", build_id, recipe["name"], compose_type)
+    os.symlink(results_dir, joinpaths(lib_dir, "queue/new/", build_id))
+
+    return build_id
    
+
+# Supported output types
+
    [docs]def compose_types(share_dir):
+    r""" Returns a list of the supported output types
+
+    The output types come from the kickstart names in /usr/share/lorax/composer/\*ks
+    """
+    return sorted([os.path.basename(ks)[:-3] for ks in glob(joinpaths(share_dir, "composer/*.ks"))])
    
+
+
    [docs]def compose_args(compose_type):
+    """ Returns the settings to pass to novirt_install for the compose type
+
+    :param compose_type: The type of compose to create, from `compose_types()`
+    :type compose_type: str
+
+    This will return a dict of options that match the ArgumentParser options for livemedia-creator.
+    These are the ones the define the type of output, it's filename, etc.
+    Other options will be filled in by `make_compose()`
+    """
+    _MAP = {"tar":              {"make_iso":                False,
+                                 "make_disk":               False,
+                                 "make_fsimage":            False,
+                                 "make_appliance":          False,
+                                 "make_ami":                False,
+                                 "make_tar":                True,
+                                 "make_pxe_live":           False,
+                                 "make_ostree_live":        False,
+                                 "make_oci":                False,
+                                 "make_vagrant":            False,
+                                 "ostree":                  False,
+                                 "live_rootfs_keep_size":   False,
+                                 "live_rootfs_size":        0,
+                                 "image_type":              False,          # False instead of None because of TOML
+                                 "qemu_args":               [],
+                                 "image_name":              default_image_name("xz", "root.tar"),
+                                 "image_only":              True,
+                                 "app_name":                None,
+                                 "app_template":            None,
+                                 "app_file":                None,
+                                },
+            "live-iso":         {"make_iso":                True,
+                                 "make_disk":               False,
+                                 "make_fsimage":            False,
+                                 "make_appliance":          False,
+                                 "make_ami":                False,
+                                 "make_tar":                False,
+                                 "make_pxe_live":           False,
+                                 "make_ostree_live":        False,
+                                 "make_oci":                False,
+                                 "make_vagrant":            False,
+                                 "ostree":                  False,
+                                 "live_rootfs_keep_size":   False,
+                                 "live_rootfs_size":        0,
+                                 "image_type":              False,          # False instead of None because of TOML
+                                 "qemu_args":               [],
+                                 "image_name":              "live.iso",
+                                 "fs_label":                "Anaconda",     # Live booting may expect this to be 'Anaconda'
+                                 "image_only":              False,
+                                 "app_name":                None,
+                                 "app_template":            None,
+                                 "app_file":                None,
+                                 "iso_only":                True,
+                                 "iso_name":                "live.iso",
+                                },
+            "partitioned-disk": {"make_iso":                False,
+                                 "make_disk":               True,
+                                 "make_fsimage":            False,
+                                 "make_appliance":          False,
+                                 "make_ami":                False,
+                                 "make_tar":                False,
+                                 "make_pxe_live":           False,
+                                 "make_ostree_live":        False,
+                                 "make_oci":                False,
+                                 "make_vagrant":            False,
+                                 "ostree":                  False,
+                                 "live_rootfs_keep_size":   False,
+                                 "live_rootfs_size":        0,
+                                 "image_type":              False,          # False instead of None because of TOML
+                                 "qemu_args":               [],
+                                 "image_name":              "disk.img",
+                                 "fs_label":                "",
+                                 "image_only":              True,
+                                 "app_name":                None,
+                                 "app_template":            None,
+                                 "app_file":                None,
+                                },
+            "qcow2":            {"make_iso":                False,
+                                 "make_disk":               True,
+                                 "make_fsimage":            False,
+                                 "make_appliance":          False,
+                                 "make_ami":                False,
+                                 "make_tar":                False,
+                                 "make_pxe_live":           False,
+                                 "make_ostree_live":        False,
+                                 "make_oci":                False,
+                                 "make_vagrant":            False,
+                                 "ostree":                  False,
+                                 "live_rootfs_keep_size":   False,
+                                 "live_rootfs_size":        0,
+                                 "image_type":              "qcow2",
+                                 "qemu_args":               [],
+                                 "image_name":              "disk.qcow2",
+                                 "fs_label":                "",
+                                 "image_only":              True,
+                                 "app_name":                None,
+                                 "app_template":            None,
+                                 "app_file":                None,
+                                },
+            "ext4-filesystem":  {"make_iso":                False,
+                                 "make_disk":               False,
+                                 "make_fsimage":            True,
+                                 "make_appliance":          False,
+                                 "make_ami":                False,
+                                 "make_tar":                False,
+                                 "make_pxe_live":           False,
+                                 "make_ostree_live":        False,
+                                 "make_oci":                False,
+                                 "make_vagrant":            False,
+                                 "ostree":                  False,
+                                 "live_rootfs_keep_size":   False,
+                                 "live_rootfs_size":        0,
+                                 "image_type":              False,          # False instead of None because of TOML
+                                 "qemu_args":               [],
+                                 "image_name":              "filesystem.img",
+                                 "fs_label":                "",
+                                 "image_only":              True,
+                                 "app_name":                None,
+                                 "app_template":            None,
+                                 "app_file":                None,
+                                },
+            }
+    return _MAP[compose_type]
    
+
+
    [docs]def move_compose_results(cfg, results_dir):
+    """Move the final image to the results_dir and cleanup the unneeded compose files
+
+    :param cfg: Build configuration
+    :type cfg: DataHolder
+    :param results_dir: Directory to put the results into
+    :type results_dir: str
+    """
+    if cfg["make_tar"]:
+        shutil.move(joinpaths(cfg["result_dir"], cfg["image_name"]), results_dir)
+    elif cfg["make_iso"]:
+        # Output from live iso is always a boot.iso under images/, move and rename it
+        shutil.move(joinpaths(cfg["result_dir"], cfg["iso_name"]), joinpaths(results_dir, cfg["image_name"]))
+    elif cfg["make_disk"] or cfg["make_fsimage"]:
+        shutil.move(joinpaths(cfg["result_dir"], cfg["image_name"]), joinpaths(results_dir, cfg["image_name"]))
+
+
+    # Cleanup the compose directory, but only if it looks like a compose directory
+    if os.path.basename(cfg["result_dir"]) == "compose":
+        shutil.rmtree(cfg["result_dir"])
+    else:
+        log.error("Incorrect compose directory, not cleaning up")
    
+
    + +
    + +
    + + +
    +
    + +
    + +
    + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/docs/html/_modules/pylorax/api/config.html b/docs/html/_modules/pylorax/api/config.html new file mode 100644 index 00000000..3ffea113 --- /dev/null +++ b/docs/html/_modules/pylorax/api/config.html @@ -0,0 +1,330 @@ + + + + + + + + + + + pylorax.api.config — Lorax 29.1 documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    + + + + +
    + + + + + +
    + +
    + + + + + + + + + + + + + + + + + +
    + + + + +
    +
    +
    +
    + +

    Source code for pylorax.api.config

    +#
+# Copyright (C) 2017  Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+import configparser
+import grp
+import os
+
+from pylorax.sysutils import joinpaths
+
+
    [docs]class ComposerConfig(configparser.SafeConfigParser):
+
    [docs]    def get_default(self, section, option, default):
+        try:
+            return self.get(section, option)
+        except configparser.Error:
+            return default
    
+
+
+
    [docs]def configure(conf_file="/etc/lorax/composer.conf", root_dir="/", test_config=False):
+    """lorax-composer configuration
+
+    :param conf_file: Path to the config file overriding the default settings
+    :type conf_file: str
+    :param root_dir: Directory to prepend to paths, defaults to /
+    :type root_dir: str
+    :param test_config: Set to True to skip reading conf_file
+    :type test_config: bool
+    """
+    conf = ComposerConfig()
+
+    # set defaults
+    conf.add_section("composer")
+    conf.set("composer", "share_dir", os.path.realpath(joinpaths(root_dir, "/usr/share/lorax/")))
+    conf.set("composer", "lib_dir", os.path.realpath(joinpaths(root_dir, "/var/lib/lorax/composer/")))
+    conf.set("composer", "dnf_conf", os.path.realpath(joinpaths(root_dir, "/var/tmp/composer/dnf.conf")))
+    conf.set("composer", "dnf_root", os.path.realpath(joinpaths(root_dir, "/var/tmp/composer/dnf/root/")))
+    conf.set("composer", "repo_dir", os.path.realpath(joinpaths(root_dir, "/var/tmp/composer/repos.d/")))
+    conf.set("composer", "cache_dir", os.path.realpath(joinpaths(root_dir, "/var/tmp/composer/cache/")))
+    conf.set("composer", "tmp", os.path.realpath(joinpaths(root_dir, "/var/tmp/")))
+
+    conf.add_section("users")
+    conf.set("users", "root", "1")
+
+    # Enable all available repo files by default
+    conf.add_section("repos")
+    conf.set("repos", "use_system_repos", "1")
+    conf.set("repos", "enabled", "*")
+
+    conf.add_section("dnf")
+
+    if not test_config:
+        # read the config file
+        if os.path.isfile(conf_file):
+            conf.read(conf_file)
+
+    return conf
    
+
+
    [docs]def make_dnf_dirs(conf):
+    """Make any missing dnf directories
+
+    :param conf: The configuration to use
+    :type conf: ComposerConfig
+    :returns: None
+    """
+    for p in ["dnf_conf", "repo_dir", "cache_dir", "dnf_root"]:
+        p_dir = os.path.dirname(conf.get("composer", p))
+        if not os.path.exists(p_dir):
+            os.makedirs(p_dir)
    
+
+
    [docs]def make_queue_dirs(conf, gid):
+    """Make any missing queue directories
+
+    :param conf: The configuration to use
+    :type conf: ComposerConfig
+    :param gid: Group ID that has access to the queue directories
+    :type gid: int
+    :returns: list of errors
+    :rtype: list of str
+    """
+    errors = []
+    lib_dir = conf.get("composer", "lib_dir")
+    for p in ["queue/run", "queue/new", "results"]:
+        p_dir = joinpaths(lib_dir, p)
+        if not os.path.exists(p_dir):
+            orig_umask = os.umask(0)
+            os.makedirs(p_dir, 0o771)
+            os.chown(p_dir, 0, gid)
+            os.umask(orig_umask)
+        else:
+            p_stat = os.stat(p_dir)
+            if p_stat.st_mode & 0o006 != 0:
+                errors.append("Incorrect permissions on %s, no o+rw permissions are allowed." % p_dir)
+
+            if p_stat.st_gid != gid or p_stat.st_uid != 0:
+                gr_name = grp.getgrgid(gid).gr_name
+                errors.append("%s should be owned by root:%s" % (p_dir, gr_name))
+
+    return errors
    
+
    + +
    + +
    + + +
    +
    + +
    + +
    + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/docs/html/_modules/pylorax/api/crossdomain.html b/docs/html/_modules/pylorax/api/crossdomain.html new file mode 100644 index 00000000..c3b5e4ec --- /dev/null +++ b/docs/html/_modules/pylorax/api/crossdomain.html @@ -0,0 +1,284 @@ + + + + + + + + + + + pylorax.api.crossdomain — Lorax 29.1 documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    + + + + +
    + + + + + +
    + +
    + + + + + + + + + + + + + + + + + +
    + + + + +
    +
    +
    +
    + +

    Source code for pylorax.api.crossdomain

    +#
+# Copyright (C) 2017  Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+# crossdomain decorator from - http://flask.pocoo.org/snippets/56/
+from datetime import timedelta
+from flask import make_response, request, current_app
+from functools import update_wrapper
+
+
+
    [docs]def crossdomain(origin, methods=None, headers=None,
+                max_age=21600, attach_to_all=True,
+                automatic_options=True):
+    if methods is not None:
+        methods = ', '.join(sorted(x.upper() for x in methods))
+    if headers is not None and not isinstance(headers, str):
+        headers = ', '.join(x.upper() for x in headers)
+    if not isinstance(origin, list):
+        origin = [origin]
+    if isinstance(max_age, timedelta):
+        max_age = int(max_age.total_seconds())
+
+    def get_methods():
+        if methods is not None:
+            return methods
+
+        options_resp = current_app.make_default_options_response()
+        return options_resp.headers['allow']
+
+    def decorator(f):
+        def wrapped_function(*args, **kwargs):
+            if automatic_options and request.method == 'OPTIONS':
+                resp = current_app.make_default_options_response()
+            else:
+                resp = make_response(f(*args, **kwargs))
+            if not attach_to_all and request.method != 'OPTIONS':
+                return resp
+
+            h = resp.headers
+
+            h.extend([("Access-Control-Allow-Origin", orig) for orig in origin])
+            h['Access-Control-Allow-Methods'] = get_methods()
+            h['Access-Control-Max-Age'] = str(max_age)
+            if headers is not None:
+                h['Access-Control-Allow-Headers'] = headers
+            return resp
+
+        f.provide_automatic_options = False
+        f.required_methods = ['OPTIONS']
+        return update_wrapper(wrapped_function, f)
+    return decorator
    
+
    + +
    + +
    + + +
    +
    + +
    + +
    + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/docs/html/_modules/pylorax/api/projects.html b/docs/html/_modules/pylorax/api/projects.html new file mode 100644 index 00000000..1c684dbf --- /dev/null +++ b/docs/html/_modules/pylorax/api/projects.html @@ -0,0 +1,524 @@ + + + + + + + + + + + pylorax.api.projects — Lorax 29.1 documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    + + + + +
    + + + + + +
    + +
    + + + + + + + + + + + + + + + + + +
    + + + + +
    +
    +
    +
    + +

    Source code for pylorax.api.projects

    +#
+# Copyright (C) 2017  Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+import logging
+log = logging.getLogger("lorax-composer")
+
+import dnf
+import time
+
+TIME_FORMAT = "%Y-%m-%dT%H:%M:%S"
+
+
+
    [docs]class ProjectsError(Exception):
+    pass
    
+
+
+
    [docs]def api_time(t):
+    """Convert time since epoch to a string
+
+    :param t: Seconds since epoch
+    :type t: int
+    :returns: Time string
+    :rtype: str
+    """
+    return time.strftime(TIME_FORMAT, time.localtime(t))
    
+
+
+
    [docs]def api_changelog(changelog):
+    """Convert the changelog to a string
+
+    :param changelog: A list of time, author, string tuples.
+    :type changelog: tuple
+    :returns: The most recent changelog text or ""
+    :rtype: str
+
+    This returns only the most recent changelog entry.
+    """
+    try:
+        entry = changelog[0][2]
+    except IndexError:
+        entry = ""
+    return entry
    
+
+
+
    [docs]def pkg_to_project(pkg):
+    """Extract the details from a hawkey.Package object
+
+    :param pkgs: hawkey.Package object with package details
+    :type pkgs: hawkey.Package
+    :returns: A dict with the name, summary, description, and url.
+    :rtype: dict
+
+    upstream_vcs is hard-coded to UPSTREAM_VCS
+    """
+    return {"name":         pkg.name,
+            "summary":      pkg.summary,
+            "description":  pkg.description,
+            "homepage":     pkg.url,
+            "upstream_vcs": "UPSTREAM_VCS"}
    
+
+
+
    [docs]def pkg_to_project_info(pkg):
+    """Extract the details from a hawkey.Package object
+
+    :param pkg: hawkey.Package object with package details
+    :type pkg: hawkey.Package
+    :returns: A dict with the project details, as well as epoch, release, arch, build_time, changelog, ...
+    :rtype: dict
+
+    metadata entries are hard-coded to {}
+    """
+    build = {"epoch":      pkg.epoch,
+             "release":    pkg.release,
+             "arch":       pkg.arch,
+             "build_time": api_time(pkg.buildtime),
+             "changelog":  "CHANGELOG_NEEDED",                  # XXX Not in hawkey.Package
+             "build_config_ref": "BUILD_CONFIG_REF",
+             "build_env_ref":    "BUILD_ENV_REF",
+             "metadata":    {},
+             "source":      {"license":    pkg.license,
+                             "version":    pkg.version,
+                             "source_ref": "SOURCE_REF",
+                             "metadata":   {}}}
+
+    return {"name":         pkg.name,
+            "summary":      pkg.summary,
+            "description":  pkg.description,
+            "homepage":     pkg.url,
+            "upstream_vcs": "UPSTREAM_VCS",
+            "builds":       [build]}
    
+
+
+
    [docs]def pkg_to_dep(pkg):
+    """Extract the info from a hawkey.Package object
+
+    :param pkg: A hawkey.Package object
+    :type pkg: hawkey.Package
+    :returns: A dict with name, epoch, version, release, arch
+    :rtype: dict
+    """
+    return {"name":     pkg.name,
+            "epoch":    pkg.epoch,
+            "version":  pkg.version,
+            "release":  pkg.release,
+            "arch":     pkg.arch}
    
+
+
+
    [docs]def proj_to_module(proj):
+    """Extract the name from a project_info dict
+
+    :param pkg: dict with package details
+    :type pkg: dict
+    :returns: A dict with name, and group_type
+    :rtype: dict
+
+    group_type is hard-coded to "rpm"
+    """
+    return {"name": proj["name"],
+            "group_type": "rpm"}
    
+
+
+
    [docs]def dep_evra(dep):
+    """Return the epoch:version-release.arch for the dep
+
+    :param dep: dependency dict
+    :type dep: dict
+    :returns: epoch:version-release.arch
+    :rtype: str
+    """
+    if dep["epoch"] == 0:
+        return dep["version"]+"-"+dep["release"]+"."+dep["arch"]
+    else:
+        return str(dep["epoch"])+":"+dep["version"]+"-"+dep["release"]+"."+dep["arch"]
    
+
+
    [docs]def dep_nevra(dep):
+    """Return the name-epoch:version-release.arch"""
+    return dep["name"]+"-"+dep_evra(dep)
    
+
+
+
    [docs]def projects_list(dbo):
+    """Return a list of projects
+
+    :param dbo: dnf base object
+    :type dbo: dnf.Base
+    :returns: List of project info dicts with name, summary, description, homepage, upstream_vcs
+    :rtype: list of dicts
+    """
+    return projects_info(dbo, None)
    
+
+
+
    [docs]def projects_info(dbo, project_names):
+    """Return details about specific projects
+
+    :param dbo: dnf base object
+    :type dbo: dnf.Base
+    :param project_names: List of names of projects to get info about
+    :type project_names: str
+    :returns: List of project info dicts with pkg_to_project as well as epoch, version, release, etc.
+    :rtype: list of dicts
+
+    If project_names is None it will return the full list of available packages
+    """
+    if project_names:
+        pkgs = dbo.sack.query().available().filter(name__glob=project_names)
+    else:
+        pkgs = dbo.sack.query().available()
+    return sorted(map(pkg_to_project_info, pkgs), key=lambda p: p["name"].lower())
    
+
+
+
    [docs]def projects_depsolve(dbo, project_names):
+    """Return the dependencies for a list of projects
+
+    :param dbo: dnf base object
+    :type dbo: dnf.Base
+    :param project_names: The projects to find the dependencies for
+    :type project_names: List of Strings
+    :returns: NEVRA's of the project and its dependencies
+    :rtype: list of dicts
+    """
+    # This resets the transaction
+    dbo.reset(goal=True)
+    for p in project_names:
+        try:
+            dbo.install(p)
+        except dnf.exceptions.MarkingError:
+            raise ProjectsError("No match for %s" % p)
+
+    try:
+        dbo.resolve()
+    except dnf.exceptions.DepsolveError as e:
+        raise ProjectsError("There was a problem depsolving %s: %s" % (project_names, str(e)))
+
+    if len(dbo.transaction) == 0:
+        return []
+
+    return sorted(map(pkg_to_dep, dbo.transaction.install_set), key=lambda p: p["name"].lower())
    
+
+
+
    [docs]def estimate_size(packages, block_size=6144):
+    """Estimate the installed size of a package list
+
+    :param packages: The packages to be installed
+    :type packages: list of hawkey.Package objects
+    :param block_size: The block size to use for rounding up file sizes.
+    :type block_size: int
+    :returns: The estimated size of installed packages
+    :rtype: int
+
+    Estimating actual requirements is difficult without the actual file sizes, which
+    dnf doesn't provide access to. So use the file count and block size to estimate
+    a minimum size for each package.
+    """
+    installed_size = 0
+    for p in packages:
+        installed_size += len(p.files) * block_size
+        installed_size += p.installsize
+    return installed_size
    
+
+
+
    [docs]def projects_depsolve_with_size(dbo, project_names, with_core=True):
+    """Return the dependencies and installed size for a list of projects
+
+    :param dbo: dnf base object
+    :type dbo: dnf.Base
+    :param project_names: The projects to find the dependencies for
+    :type project_names: List of Strings
+    :returns: installed size and a list of NEVRA's of the project and its dependencies
+    :rtype: tuple of (int, list of dicts)
+    """
+    # This resets the transaction
+    dbo.reset(goal=True)
+    for p in project_names:
+        try:
+            dbo.install(p)
+        except dnf.exceptions.MarkingError:
+            raise ProjectsError("No match for %s" % p)
+
+    if with_core:
+        dbo.group_install("core", ['mandatory', 'default', 'optional'])
+
+    try:
+        dbo.resolve()
+    except dnf.exceptions.DepsolveError as e:
+        raise ProjectsError("There was a problem depsolving %s: %s" % (project_names, str(e)))
+
+    if len(dbo.transaction) == 0:
+        return (0, [])
+
+    installed_size = estimate_size(dbo.transaction.install_set)
+    deps = sorted(map(pkg_to_dep, dbo.transaction.install_set), key=lambda p: p["name"].lower())
+    return (installed_size, deps)
    
+
+
+
    [docs]def modules_list(dbo, module_names):
+    """Return a list of modules
+
+    :param dbo: dnf base object
+    :type dbo: dnf.Base
+    :param offset: Number of modules to skip
+    :type limit: int
+    :param limit: Maximum number of modules to return
+    :type limit: int
+    :returns: List of module information and total count
+    :rtype: tuple of a list of dicts and an Int
+
+    Modules don't exist in RHEL7 so this only returns projects
+    and sets the type to "rpm"
+
+    """
+    # TODO - Figure out what to do with this for Fedora 'modules'
+    projs = projects_info(dbo, module_names)
+    return sorted(map(proj_to_module, projs), key=lambda p: p["name"].lower())
    
+
+
+
    [docs]def modules_info(dbo, module_names):
+    """Return details about a module, including dependencies
+
+    :param dbo: dnf base object
+    :type dbo: dnf.Base
+    :param module_names: Names of the modules to get info about
+    :type module_names: str
+    :returns: List of dicts with module details and dependencies.
+    :rtype: list of dicts
+    """
+    modules = projects_info(dbo, module_names)
+
+    # Add the dependency info to each one
+    for module in modules:
+        module["dependencies"] = projects_depsolve(dbo, [module["name"]])
+
+    return modules
    
+
    + +
    + +
    + + +
    +
    + +
    + +
    + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/docs/html/_modules/pylorax/api/queue.html b/docs/html/_modules/pylorax/api/queue.html new file mode 100644 index 00000000..3eab45e2 --- /dev/null +++ b/docs/html/_modules/pylorax/api/queue.html @@ -0,0 +1,836 @@ + + + + + + + + + + + pylorax.api.queue — Lorax 29.1 documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    + + + + +
    + + + + + +
    + +
    + + + + + + + + + + + + + + + + + +
    + + + + +
    +
    +
    +
    + +

    Source code for pylorax.api.queue

    +# Copyright (C) 2018 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+""" Functions to monitor compose queue and run anaconda"""
+import logging
+log = logging.getLogger("pylorax")
+
+import os
+import grp
+from glob import glob
+import multiprocessing as mp
+import pytoml as toml
+import pwd
+import shutil
+import subprocess
+from subprocess import Popen, PIPE
+import time
+
+from pylorax import find_templates
+from pylorax.api.compose import move_compose_results
+from pylorax.api.recipes import recipe_from_file
+from pylorax.base import DataHolder
+from pylorax.creator import run_creator
+from pylorax.sysutils import joinpaths
+
+
    [docs]def start_queue_monitor(cfg, uid, gid):
+    """Start the queue monitor as a mp process
+
+    :param cfg: Configuration settings
+    :type cfg: ComposerConfig
+    :param uid: User ID that owns the queue
+    :type uid: int
+    :param gid: Group ID that owns the queue
+    :type gid: int
+    :returns: None
+    """
+    lib_dir = cfg.get("composer", "lib_dir")
+    share_dir = cfg.get("composer", "share_dir")
+    tmp = cfg.get("composer", "tmp")
+    monitor_cfg = DataHolder(composer_dir=lib_dir, share_dir=share_dir, uid=uid, gid=gid, tmp=tmp)
+    p = mp.Process(target=monitor, args=(monitor_cfg,))
+    p.daemon = True
+    p.start()
    
+
+
    [docs]def monitor(cfg):
+    """Monitor the queue for new compose requests
+
+    :param cfg: Configuration settings
+    :type cfg: DataHolder
+    :returns: Does not return
+
+    The queue has 2 subdirectories, new and run. When a compose is ready to be run
+    a symlink to the uniquely named results directory should be placed in ./queue/new/
+
+    When the it is ready to be run (it is checked every 30 seconds or after a previous
+    compose is finished) the symlink will be moved into ./queue/run/ and a STATUS file
+    will be created in the results directory.
+
+    STATUS can contain one of: RUNNING, FINISHED, FAILED
+
+    If the system is restarted while a compose is running it will move any old symlinks
+    from ./queue/run/ to ./queue/new/ and rerun them.
+    """
+    def queue_sort(uuid):
+        """Sort the queue entries by their mtime, not their names"""
+        return os.stat(joinpaths(cfg.composer_dir, "queue/new", uuid)).st_mtime
+
+    # Move any symlinks in the run queue back to the new queue
+    for link in os.listdir(joinpaths(cfg.composer_dir, "queue/run")):
+        src = joinpaths(cfg.composer_dir, "queue/run", link)
+        dst = joinpaths(cfg.composer_dir, "queue/new", link)
+        os.rename(src, dst)
+        log.debug("Moved unfinished compose %s back to new state", src)
+
+    while True:
+        uuids = sorted(os.listdir(joinpaths(cfg.composer_dir, "queue/new")), key=queue_sort)
+
+        # Pick the oldest and move it into ./run/
+        if not uuids:
+            # No composes left to process, sleep for a bit
+            time.sleep(30)
+        else:
+            src = joinpaths(cfg.composer_dir, "queue/new", uuids[0])
+            dst = joinpaths(cfg.composer_dir, "queue/run", uuids[0])
+            try:
+                os.rename(src, dst)
+            except OSError:
+                # The symlink may vanish if uuid_cancel() has been called
+                continue
+
+            log.info("Starting new compose: %s", dst)
+            open(joinpaths(dst, "STATUS"), "w").write("RUNNING\n")
+
+            try:
+                make_compose(cfg, os.path.realpath(dst))
+                log.info("Finished building %s, results are in %s", dst, os.path.realpath(dst))
+                open(joinpaths(dst, "STATUS"), "w").write("FINISHED\n")
+            except Exception:
+                import traceback
+                log.error("traceback: %s", traceback.format_exc())
+
+# TODO - Write the error message to an ERROR-LOG file to include with the status
+#                log.error("Error running compose: %s", e)
+                open(joinpaths(dst, "STATUS"), "w").write("FAILED\n")
+
+            os.unlink(dst)
    
+
+
    [docs]def make_compose(cfg, results_dir):
+    """Run anaconda with the final-kickstart.ks from results_dir
+
+    :param cfg: Configuration settings
+    :type cfg: DataHolder
+    :param results_dir: The directory containing the metadata and results for the build
+    :type results_dir: str
+    :returns: Nothing
+    :raises: May raise various exceptions
+
+    This takes the final-kickstart.ks, and the settings in config.toml and runs Anaconda
+    in no-virt mode (directly on the host operating system). Exceptions should be caught
+    at the higer level.
+
+    If there is a failure, the build artifacts will be cleaned up, and any logs will be
+    moved into logs/anaconda/ and their ownership will be set to the user from the cfg
+    object.
+    """
+
+    # Check on the ks's presence
+    ks_path = joinpaths(results_dir, "final-kickstart.ks")
+    if not os.path.exists(ks_path):
+        raise RuntimeError("Missing kickstart file at %s" % ks_path)
+
+    # The anaconda logs are copied into ./anaconda/ in this directory
+    log_dir = joinpaths(results_dir, "logs/")
+    if not os.path.exists(log_dir):
+        os.makedirs(log_dir)
+
+    # Load the compose configuration
+    cfg_path = joinpaths(results_dir, "config.toml")
+    if not os.path.exists(cfg_path):
+        raise RuntimeError("Missing config.toml for %s" % results_dir)
+    cfg_dict = toml.loads(open(cfg_path, "r").read())
+
+    # The keys in cfg_dict correspond to the arguments setup in livemedia-creator
+    # keys that define what to build should be setup in compose_args, and keys with
+    # defaults should be setup here.
+
+    # Make sure that image_name contains no path components
+    cfg_dict["image_name"] = os.path.basename(cfg_dict["image_name"])
+
+    # Only support novirt installation, set some other defaults
+    cfg_dict["no_virt"] = True
+    cfg_dict["disk_image"] = None
+    cfg_dict["fs_image"] = None
+    cfg_dict["keep_image"] = False
+    cfg_dict["domacboot"] = False
+    cfg_dict["anaconda_args"] = ""
+    cfg_dict["proxy"] = ""
+    cfg_dict["armplatform"] = ""
+    cfg_dict["squashfs_args"] = None
+
+    cfg_dict["lorax_templates"] = find_templates(cfg.share_dir)
+    cfg_dict["tmp"] = cfg.tmp
+    cfg_dict["dracut_args"] = None                  # Use default args for dracut
+
+    # TODO How to support other arches?
+    cfg_dict["arch"] = None
+
+    # Compose things in a temporary directory inside the results directory
+    cfg_dict["result_dir"] = joinpaths(results_dir, "compose")
+    os.makedirs(cfg_dict["result_dir"])
+
+    install_cfg = DataHolder(**cfg_dict)
+
+    # Some kludges for the 99-copy-logs %post, failure in it will crash the build
+    for f in ["/tmp/NOSAVE_INPUT_KS", "/tmp/NOSAVE_LOGS"]:
+        open(f, "w")
+
+    # Placing a CANCEL file in the results directory will make execWithRedirect send anaconda a SIGTERM
+    def cancel_build():
+        return os.path.exists(joinpaths(results_dir, "CANCEL"))
+
+    log.debug("cfg  = %s", install_cfg)
+    try:
+        test_path = joinpaths(results_dir, "TEST")
+        if os.path.exists(test_path):
+            # Pretend to run the compose
+            time.sleep(10)
+            try:
+                test_mode = int(open(test_path, "r").read())
+            except Exception:
+                test_mode = 1
+            if test_mode == 1:
+                raise RuntimeError("TESTING FAILED compose")
+            else:
+                open(joinpaths(results_dir, install_cfg.image_name), "w").write("TEST IMAGE")
+        else:
+            run_creator(install_cfg, callback_func=cancel_build)
+
+            # Extract the results of the compose into results_dir and cleanup the compose directory
+            move_compose_results(install_cfg, results_dir)
+    finally:
+        # Make sure any remaining temporary directories are removed (eg. if there was an exception)
+        for d in glob(joinpaths(cfg.tmp, "lmc-*")):
+            if os.path.isdir(d):
+                shutil.rmtree(d)
+            elif os.path.isfile(d):
+                os.unlink(d)
+
+        # Make sure that everything under the results directory is owned by the user
+        user = pwd.getpwuid(cfg.uid).pw_name
+        group = grp.getgrgid(cfg.gid).gr_name
+        log.debug("Install finished, chowning results to %s:%s", user, group)
+        subprocess.call(["chown", "-R", "%s:%s" % (user, group), results_dir])
    
+
+
    [docs]def get_compose_type(results_dir):
+    """Return the type of composition.
+
+    :param results_dir: The directory containing the metadata and results for the build
+    :type results_dir: str
+    :returns: The type of compose (eg. 'tar')
+    :rtype: str
+    :raises: RuntimeError if no kickstart template can be found.
+    """
+    # Should only be 2 kickstarts, the final-kickstart.ks and the template
+    t = [os.path.basename(ks)[:-3] for ks in glob(joinpaths(results_dir, "*.ks"))
+                                   if "final-kickstart" not in ks]
+    if len(t) != 1:
+        raise RuntimeError("Cannot find ks template for build %s" % os.path.basename(results_dir))
+    return t[0]
    
+
+
    [docs]def compose_detail(results_dir):
+    """Return details about the build.
+
+    :param results_dir: The directory containing the metadata and results for the build
+    :type results_dir: str
+    :returns: A dictionary with details about the compose
+    :rtype: dict
+    :raises: IOError if it cannot read the directory, STATUS, or blueprint file.
+
+    The following details are included in the dict:
+
+    * id - The uuid of the comoposition
+    * queue_status - The final status of the composition (FINISHED or FAILED)
+    * timestamp - The time of the last status change
+    * compose_type - The type of output generated (tar, iso, etc.)
+    * blueprint - Blueprint name
+    * version - Blueprint version
+    * image_size - Size of the image, if finished. 0 otherwise.
+    """
+    build_id = os.path.basename(os.path.abspath(results_dir))
+    status = open(joinpaths(results_dir, "STATUS")).read().strip()
+    mtime = os.stat(joinpaths(results_dir, "STATUS")).st_mtime
+    blueprint = recipe_from_file(joinpaths(results_dir, "blueprint.toml"))
+
+    compose_type = get_compose_type(results_dir)
+
+    image_path = get_image_name(results_dir)[1]
+    if status == "FINISHED" and os.path.exists(image_path):
+        image_size = os.stat(image_path).st_size
+    else:
+        image_size = 0
+
+    return {"id":           build_id,
+            "queue_status": status,
+            "timestamp":    mtime,
+            "compose_type": compose_type,
+            "blueprint":    blueprint["name"],
+            "version":      blueprint["version"],
+            "image_size":   image_size
+            }
    
+
+
    [docs]def queue_status(cfg):
+    """Return details about what is in the queue.
+
+    :param cfg: Configuration settings
+    :type cfg: ComposerConfig
+    :returns: A list of the new composes, and a list of the running composes
+    :rtype: dict
+
+    This returns a dict with 2 lists. "new" is the list of uuids that are waiting to be built,
+    and "run" has the uuids that are being built (currently limited to 1 at a time).
+    """
+    queue_dir = joinpaths(cfg.get("composer", "lib_dir"), "queue")
+    new_queue = [os.path.realpath(p) for p in glob(joinpaths(queue_dir, "new/*"))]
+    run_queue = [os.path.realpath(p) for p in glob(joinpaths(queue_dir, "run/*"))]
+
+    new_details = []
+    for n in new_queue:
+        try:
+            d = compose_detail(n)
+        except IOError:
+            continue
+        new_details.append(d)
+
+    run_details = []
+    for r in run_queue:
+        try:
+            d = compose_detail(r)
+        except IOError:
+            continue
+        run_details.append(d)
+
+    return {
+        "new": new_details,
+        "run": run_details
+    }
    
+
+
    [docs]def uuid_status(cfg, uuid):
+    """Return the details of a specific UUID compose
+
+    :param cfg: Configuration settings
+    :type cfg: ComposerConfig
+    :param uuid: The UUID of the build
+    :type uuid: str
+    :returns: Details about the build
+    :rtype: dict or None
+
+    Returns the same dict as `compose_details()`
+    """
+    uuid_dir = joinpaths(cfg.get("composer", "lib_dir"), "results", uuid)
+    try:
+        return compose_detail(uuid_dir)
+    except IOError:
+        return None
    
+
+
    [docs]def build_status(cfg, status_filter=None):
+    """Return the details of finished or failed builds
+
+    :param cfg: Configuration settings
+    :type cfg: ComposerConfig
+    :param status_filter: What builds to return. None == all, "FINISHED", or "FAILED"
+    :type status_filter: str
+    :returns: A list of the build details (from compose_details)
+    :rtype: list of dicts
+
+    This returns a list of build details for each of the matching builds on the
+    system. It does not return the status of builds that have not been finished.
+    Use queue_status() for those.
+    """
+    if status_filter:
+        status_filter = [status_filter]
+    else:
+        status_filter = ["FINISHED", "FAILED"]
+
+    results = []
+    result_dir = joinpaths(cfg.get("composer", "lib_dir"), "results")
+    for build in glob(result_dir + "/*"):
+        log.debug("Checking status of build %s", build)
+
+        try:
+            status = open(joinpaths(build, "STATUS"), "r").read().strip()
+            if status in status_filter:
+                results.append(compose_detail(build))
+        except IOError:
+            pass
+    return results
    
+
+
    [docs]def uuid_cancel(cfg, uuid):
+    """Cancel a build and delete its results
+
+    :param cfg: Configuration settings
+    :type cfg: ComposerConfig
+    :param uuid: The UUID of the build
+    :type uuid: str
+    :returns: True if it was canceled and deleted
+    :rtype: bool
+
+    Only call this if the build status is WAITING or RUNNING
+    """
+    # This status can change (and probably will) while it is in the middle of doing this:
+    # It can move from WAITING -> RUNNING or it can move from RUNNING -> FINISHED|FAILED
+
+    # If it is in WAITING remove the symlink and then check to make sure it didn't show up
+    # in RUNNING
+    queue_dir = joinpaths(cfg.get("composer", "lib_dir"), "queue")
+    uuid_new = joinpaths(queue_dir, "new", uuid)
+    if os.path.exists(uuid_new):
+        try:
+            os.unlink(uuid_new)
+        except OSError:
+            # The symlink may vanish if the queue monitor started the build
+            pass
+        uuid_run = joinpaths(queue_dir, "run", uuid)
+        if not os.path.exists(uuid_run):
+            # Successfully removed it before the build started
+            return uuid_delete(cfg, uuid)
+
+    # Tell the build to stop running
+    cancel_path = joinpaths(cfg.get("composer", "lib_dir"), "results", uuid, "CANCEL")
+    open(cancel_path, "w").write("\n")
+
+    # Wait for status to move to FAILED
+    started = time.time()
+    while True:
+        status = uuid_status(cfg, uuid)
+        if status is None or status["queue_status"] == "FAILED":
+            break
+
+        # Is this taking too long? Exit anyway and try to cleanup.
+        if time.time() > started + (10 * 60):
+            log.error("Failed to cancel the build of %s", uuid)
+            break
+
+        time.sleep(5)
+
+    # Remove the partial results
+    uuid_delete(cfg, uuid)
    
+
+
    [docs]def uuid_delete(cfg, uuid):
+    """Delete all of the results from a compose
+
+    :param cfg: Configuration settings
+    :type cfg: ComposerConfig
+    :param uuid: The UUID of the build
+    :type uuid: str
+    :returns: True if it was deleted
+    :rtype: bool
+    :raises: This will raise an error if the delete failed
+    """
+    uuid_dir = joinpaths(cfg.get("composer", "lib_dir"), "results", uuid)
+    if not uuid_dir or len(uuid_dir) < 10:
+        raise RuntimeError("Directory length is too short: %s" % uuid_dir)
+    shutil.rmtree(uuid_dir)
+    return True
    
+
+
    [docs]def uuid_info(cfg, uuid):
+    """Return information about the composition
+
+    :param cfg: Configuration settings
+    :type cfg: ComposerConfig
+    :param uuid: The UUID of the build
+    :type uuid: str
+    :returns: dictionary of information about the composition
+    :rtype: dict
+    :raises: RuntimeError if there was a problem
+
+    This will return a dict with the following fields populated:
+
+    * id - The uuid of the comoposition
+    * config - containing the configuration settings used to run Anaconda
+    * blueprint - The depsolved blueprint used to generate the kickstart
+    * commit - The (local) git commit hash for the blueprint used
+    * deps - The NEVRA of all of the dependencies used in the composition
+    * compose_type - The type of output generated (tar, iso, etc.)
+    * queue_status - The final status of the composition (FINISHED or FAILED)
+    """
+    uuid_dir = joinpaths(cfg.get("composer", "lib_dir"), "results", uuid)
+    if not os.path.exists(uuid_dir):
+        raise RuntimeError("%s is not a valid build_id" % uuid)
+
+    # Load the compose configuration
+    cfg_path = joinpaths(uuid_dir, "config.toml")
+    if not os.path.exists(cfg_path):
+        raise RuntimeError("Missing config.toml for %s" % uuid)
+    cfg_dict = toml.loads(open(cfg_path, "r").read())
+
+    frozen_path = joinpaths(uuid_dir, "frozen.toml")
+    if not os.path.exists(frozen_path):
+        raise RuntimeError("Missing frozen.toml for %s" % uuid)
+    frozen_dict = toml.loads(open(frozen_path, "r").read())
+
+    deps_path = joinpaths(uuid_dir, "deps.toml")
+    if not os.path.exists(deps_path):
+        raise RuntimeError("Missing deps.toml for %s" % uuid)
+    deps_dict = toml.loads(open(deps_path, "r").read())
+
+    details = compose_detail(uuid_dir)
+
+    commit_path = joinpaths(uuid_dir, "COMMIT")
+    if not os.path.exists(commit_path):
+        raise RuntimeError("Missing commit hash for %s" % uuid)
+    commit_id = open(commit_path, "r").read().strip()
+
+    return {"id":           uuid,
+            "config":       cfg_dict,
+            "blueprint":    frozen_dict,
+            "commit":       commit_id,
+            "deps":         deps_dict,
+            "compose_type": details["compose_type"],
+            "queue_status": details["queue_status"],
+            "image_size":   details["image_size"]
+    }
    
+
+
    [docs]def uuid_tar(cfg, uuid, metadata=False, image=False, logs=False):
+    """Return a tar of the build data
+
+    :param cfg: Configuration settings
+    :type cfg: ComposerConfig
+    :param uuid: The UUID of the build
+    :type uuid: str
+    :param metadata: Set to true to include all the metadata needed to reproduce the build
+    :type metadata: bool
+    :param image: Set to true to include the output image
+    :type image: bool
+    :param logs: Set to true to include the logs from the build
+    :type logs: bool
+    :returns: A stream of bytes from tar
+    :rtype: A generator
+    :raises: RuntimeError if there was a problem (eg. missing config file)
+
+    This yields an uncompressed tar's data to the caller. It includes
+    the selected data to the caller by returning the Popen stdout from the tar process.
+    """
+    uuid_dir = joinpaths(cfg.get("composer", "lib_dir"), "results", uuid)
+    if not os.path.exists(uuid_dir):
+        raise RuntimeError("%s is not a valid build_id" % uuid)
+
+    # Load the compose configuration
+    cfg_path = joinpaths(uuid_dir, "config.toml")
+    if not os.path.exists(cfg_path):
+        raise RuntimeError("Missing config.toml for %s" % uuid)
+    cfg_dict = toml.loads(open(cfg_path, "r").read())
+    image_name = cfg_dict["image_name"]
+
+    def include_file(f):
+        if f.endswith("/logs"):
+            return logs
+        if f.endswith(image_name):
+            return image
+        return metadata
+    filenames = [os.path.basename(f) for f in glob(joinpaths(uuid_dir, "*")) if include_file(f)]
+
+    tar = Popen(["tar", "-C", uuid_dir, "-cf-"] + filenames, stdout=PIPE)
+    return tar.stdout
    
+
+
    [docs]def uuid_image(cfg, uuid):
+    """Return the filename and full path of the build's image file
+
+    :param cfg: Configuration settings
+    :type cfg: ComposerConfig
+    :param uuid: The UUID of the build
+    :type uuid: str
+    :returns: The image filename and full path
+    :rtype: tuple of strings
+    :raises: RuntimeError if there was a problem (eg. invalid uuid, missing config file)
+    """
+    uuid_dir = joinpaths(cfg.get("composer", "lib_dir"), "results", uuid)
+    return get_image_name(uuid_dir)
    
+
+
    [docs]def get_image_name(uuid_dir):
+    """Return the filename and full path of the build's image file
+
+    :param uuid: The UUID of the build
+    :type uuid: str
+    :returns: The image filename and full path
+    :rtype: tuple of strings
+    :raises: RuntimeError if there was a problem (eg. invalid uuid, missing config file)
+    """
+    uuid = os.path.basename(os.path.abspath(uuid_dir))
+    if not os.path.exists(uuid_dir):
+        raise RuntimeError("%s is not a valid build_id" % uuid)
+
+    # Load the compose configuration
+    cfg_path = joinpaths(uuid_dir, "config.toml")
+    if not os.path.exists(cfg_path):
+        raise RuntimeError("Missing config.toml for %s" % uuid)
+    cfg_dict = toml.loads(open(cfg_path, "r").read())
+    image_name = cfg_dict["image_name"]
+
+    return (image_name, joinpaths(uuid_dir, image_name))
    
+
+
    [docs]def uuid_log(cfg, uuid, size=1024):
+    """Return `size` kbytes from the end of the anaconda.log
+
+    :param cfg: Configuration settings
+    :type cfg: ComposerConfig
+    :param uuid: The UUID of the build
+    :type uuid: str
+    :param size: Number of kbytes to read. Default is 1024
+    :type size: int
+    :returns: Up to `size` kbytes from the end of the log
+    :rtype: str
+    :raises: RuntimeError if there was a problem (eg. no log file available)
+
+    This function tries to return lines from the end of the log, it will
+    attempt to start on a line boundry, and may return less than `size` kbytes.
+    """
+    uuid_dir = joinpaths(cfg.get("composer", "lib_dir"), "results", uuid)
+    if not os.path.exists(uuid_dir):
+        raise RuntimeError("%s is not a valid build_id" % uuid)
+
+    # While a build is running the logs will be in /tmp/anaconda.log and when it
+    # has finished they will be in the results directory
+    status = uuid_status(cfg, uuid)
+    if status is None:
+        raise RuntimeError("Status is missing for %s" % uuid)
+
+    if status["queue_status"] == "RUNNING":
+        log_path = "/tmp/anaconda.log"
+    else:
+        log_path = joinpaths(uuid_dir, "logs", "anaconda", "anaconda.log")
+    if not os.path.exists(log_path):
+        raise RuntimeError("No anaconda.log available.")
+
+    with open(log_path, "r") as f:
+        f.seek(0, 2)
+        end = f.tell()
+        if end < 1024 * size:
+            f.seek(0, 0)
+        else:
+            f.seek(end - (1024 * size))
+        # Find the start of the next line and return the rest
+        f.readline()
+        return f.read()
    
+
    + +
    + +
    + + +
    +
    + +
    + +
    + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/docs/html/_modules/pylorax/api/recipes.html b/docs/html/_modules/pylorax/api/recipes.html new file mode 100644 index 00000000..2e0daf0d --- /dev/null +++ b/docs/html/_modules/pylorax/api/recipes.html @@ -0,0 +1,1111 @@ + + + + + + + + + + + pylorax.api.recipes — Lorax 29.1 documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    + + + + +
    + + + + + +
    + +
    + + + + + + + + + + + + + + + + + +
    + + + + +
    +
    +
    +
    + +

    Source code for pylorax.api.recipes

    +#
+# Copyright (C) 2017  Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import gi
+gi.require_version("Ggit", "1.0")
+from gi.repository import Ggit as Git
+from gi.repository import Gio
+from gi.repository import GLib
+
+import os
+import pytoml as toml
+import semantic_version as semver
+
+from pylorax.api.projects import dep_evra
+from pylorax.base import DataHolder
+from pylorax.sysutils import joinpaths
+
+
+
    [docs]class CommitTimeValError(Exception):
+    pass
    
+
+
    [docs]class RecipeFileError(Exception):
+    pass
    
+
+
    [docs]class RecipeError(Exception):
+    pass
    
+
+
+
    [docs]class Recipe(dict):
+    """A Recipe of package and modules
+
+    This is a subclass of dict that enforces the constructor arguments
+    and adds a .filename property to return the recipe's filename,
+    and a .toml() function to return the recipe as a TOML string.
+    """
+    def __init__(self, name, description, version, modules, packages, customizations=None):
+        # Check that version is empty or semver compatible
+        if version:
+            semver.Version(version)
+
+        # Make sure modules and packages are listed by their case-insensitive names
+        if modules is not None:
+            modules = sorted(modules, key=lambda m: m["name"].lower())
+        if packages is not None:
+            packages = sorted(packages, key=lambda p: p["name"].lower())
+        dict.__init__(self, name=name,
+                            description=description,
+                            version=version,
+                            modules=modules,
+                            packages=packages,
+                            customizations=customizations)
+
+        # We don't want customizations=None to show up in the TOML so remove it
+        if customizations is None:
+            del self["customizations"]
+
+    @property
+    def package_names(self):
+        """Return the names of the packages"""
+        return [p["name"] for p in self["packages"] or []]
+
+    @property
+    def module_names(self):
+        """Return the names of the modules"""
+        return [m["name"] for m in self["modules"] or []]
+
+    @property
+    def filename(self):
+        """Return the Recipe's filename
+
+        Replaces spaces in the name with '-' and appends .toml
+        """
+        return recipe_filename(self.get("name"))
+
+
    [docs]    def toml(self):
+        """Return the Recipe in TOML format"""
+        return toml.dumps(self)
    
+
+
    [docs]    def bump_version(self, old_version=None):
+        """semver recipe version number bump
+
+        :param old_version: An optional old version number
+        :type old_version: str
+        :returns: The new version number or None
+        :rtype: str
+        :raises: ValueError
+
+        If neither have a version, 0.0.1 is returned
+        If there is no old version the new version is checked and returned
+        If there is no new version, but there is a old one, bump its patch level
+        If the old and new versions are the same, bump the patch level
+        If they are different, check and return the new version
+        """
+        new_version = self.get("version")
+        if not new_version and not old_version:
+            self["version"] = "0.0.1"
+
+        elif new_version and not old_version:
+            semver.Version(new_version)
+            self["version"] = new_version
+
+        elif not new_version or new_version == old_version:
+            new_version = str(semver.Version(old_version).next_patch())
+            self["version"] = new_version
+
+        else:
+            semver.Version(new_version)
+            self["version"] = new_version
+
+        # Return the new version
+        return str(semver.Version(self["version"]))
    
+
+
    [docs]    def freeze(self, deps):
+        """ Return a new Recipe with full module and package NEVRA
+
+        :param deps: A list of dependency NEVRA to use to fill in the modules and packages
+        :type deps: list(
+        :returns: A new Recipe object
+        :rtype: Recipe
+        """
+        module_names = self.module_names
+        package_names = self.package_names
+
+        new_modules = []
+        new_packages = []
+        for dep in deps:
+            if dep["name"] in package_names:
+                new_packages.append(RecipePackage(dep["name"], dep_evra(dep)))
+            elif dep["name"] in module_names:
+                new_modules.append(RecipeModule(dep["name"], dep_evra(dep)))
+        if "customizations" in self:
+            customizations = self["customizations"]
+        else:
+            customizations = None
+
+        return Recipe(self["name"], self["description"], self["version"],
+                      new_modules, new_packages, customizations)
    
+
+
    [docs]class RecipeModule(dict):
+    def __init__(self, name, version):
+        dict.__init__(self, name=name, version=version)
    
+
+
    [docs]class RecipePackage(RecipeModule):
+    pass
    
+
+
    [docs]def recipe_from_file(recipe_path):
+    """Return a recipe file as a Recipe object
+
+    :param recipe_path: Path to the recipe fila
+    :type recipe_path: str
+    :returns: A Recipe object
+    :rtype: Recipe
+    """
+    with open(recipe_path, 'rb') as f:
+        return recipe_from_toml(f.read())
    
+
+
    [docs]def recipe_from_toml(recipe_str):
+    """Create a Recipe object from a toml string.
+
+    :param recipe_str: The Recipe TOML string
+    :type recipe_str: str
+    :returns: A Recipe object
+    :rtype: Recipe
+    :raises: TomlError
+    """
+    recipe_dict = toml.loads(recipe_str)
+    return recipe_from_dict(recipe_dict)
    
+
+
    [docs]def recipe_from_dict(recipe_dict):
+    """Create a Recipe object from a plain dict.
+
+    :param recipe_dict: A plain dict of the recipe
+    :type recipe_dict: dict
+    :returns: A Recipe object
+    :rtype: Recipe
+    :raises: RecipeError
+    """
+    # Make RecipeModule objects from the toml
+    # The TOML may not have modules or packages in it. Set them to None in this case
+    try:
+        if recipe_dict.get("modules"):
+            modules = [RecipeModule(m.get("name"), m.get("version")) for m in recipe_dict["modules"]]
+        else:
+            modules = []
+        if recipe_dict.get("packages"):
+            packages = [RecipePackage(p.get("name"), p.get("version")) for p in recipe_dict["packages"]]
+        else:
+            packages = []
+        name = recipe_dict["name"]
+        description = recipe_dict["description"]
+        version = recipe_dict.get("version", None)
+        customizations = recipe_dict.get("customizations", None)
+    except KeyError as e:
+        raise RecipeError("There was a problem parsing the recipe: %s" % str(e))
+
+    return Recipe(name, description, version, modules, packages, customizations)
    
+
+
    [docs]def gfile(path):
+    """Convert a string path to GFile for use with Git"""
+    return Gio.file_new_for_path(path)
    
+
+
    [docs]def recipe_filename(name):
+    """Return the toml filename for a recipe
+
+    Replaces spaces with '-' and appends '.toml'
+    """
+    # XXX Raise and error if this is empty?
+    return name.replace(" ", "-") + ".toml"
    
+
+
    [docs]def head_commit(repo, branch):
+    """Get the branch's HEAD Commit Object
+
+    :param repo: Open repository
+    :type repo: Git.Repository
+    :param branch: Branch name
+    :type branch: str
+    :returns: Branch's head commit
+    :rtype: Git.Commit
+    :raises: Can raise errors from Ggit
+    """
+    branch_obj = repo.lookup_branch(branch, Git.BranchType.LOCAL)
+    commit_id = branch_obj.get_target()
+    return repo.lookup(commit_id, Git.Commit)
    
+
+
    [docs]def prepare_commit(repo, branch, builder):
+    """Prepare for a commit
+
+    :param repo: Open repository
+    :type repo: Git.Repository
+    :param branch: Branch name
+    :type branch: str
+    :param builder: instance of TreeBuilder
+    :type builder: TreeBuilder
+    :returns: (Tree, Sig, Ref)
+    :rtype: tuple
+    :raises: Can raise errors from Ggit
+    """
+    tree_id = builder.write()
+    tree = repo.lookup(tree_id, Git.Tree)
+    sig = Git.Signature.new_now("bdcs-api-server", "user-email")
+    ref = "refs/heads/%s" % branch
+    return (tree, sig, ref)
    
+
+
    [docs]def open_or_create_repo(path):
+    """Open an existing repo, or create a new one
+
+    :param path: path to recipe directory
+    :type path: string
+    :returns: A repository object
+    :rtype: Git.Repository
+    :raises: Can raise errors from Ggit
+
+    A bare git repo will be created in the git directory of the specified path.
+    If a repo already exists it will be opened and returned instead of
+    creating a new one.
+    """
+    Git.init()
+    git_path = joinpaths(path, "git")
+    if os.path.exists(joinpaths(git_path, "HEAD")):
+        return Git.Repository.open(gfile(git_path))
+
+    repo = Git.Repository.init_repository(gfile(git_path), True)
+
+    # Make an initial empty commit
+    sig = Git.Signature.new_now("bdcs-api-server", "user-email")
+    tree_id = repo.get_index().write_tree()
+    tree = repo.lookup(tree_id, Git.Tree)
+    repo.create_commit("HEAD", sig, sig, "UTF-8", "Initial Recipe repository commit", tree, [])
+    return repo
    
+
+
    [docs]def write_commit(repo, branch, filename, message, content):
+    """Make a new commit to a repository's branch
+
+    :param repo: Open repository
+    :type repo: Git.Repository
+    :param branch: Branch name
+    :type branch: str
+    :param filename: full path of the file to add
+    :type filename: str
+    :param message: The commit message
+    :type message: str
+    :param content: The data to write
+    :type content: str
+    :returns: OId of the new commit
+    :rtype: Git.OId
+    :raises: Can raise errors from Ggit
+    """
+    try:
+        parent_commit = head_commit(repo, branch)
+    except GLib.GError:
+        # Branch doesn't exist, make a new one based on master
+        master_head = head_commit(repo, "master")
+        repo.create_branch(branch, master_head, 0)
+        parent_commit = head_commit(repo, branch)
+
+    parent_commit = head_commit(repo, branch)
+    blob_id = repo.create_blob_from_buffer(content.encode("UTF-8"))
+
+    # Use treebuilder to make a new entry for this filename and blob
+    parent_tree = parent_commit.get_tree()
+    builder = repo.create_tree_builder_from_tree(parent_tree)
+    builder.insert(filename, blob_id, Git.FileMode.BLOB)
+    (tree, sig, ref) = prepare_commit(repo, branch, builder)
+    return repo.create_commit(ref, sig, sig, "UTF-8", message, tree, [parent_commit])
    
+
+
    [docs]def read_commit_spec(repo, spec):
+    """Return the raw content of the blob specified by the spec
+
+    :param repo: Open repository
+    :type repo: Git.Repository
+    :param spec: Git revparse spec
+    :type spec: str
+    :returns: Contents of the commit
+    :rtype: str
+    :raises: Can raise errors from Ggit
+
+    eg. To read the README file from master the spec is "master:README"
+    """
+    commit_id = repo.revparse(spec).get_id()
+    blob = repo.lookup(commit_id, Git.Blob)
+    return blob.get_raw_content()
    
+
+
    [docs]def read_commit(repo, branch, filename, commit=None):
+    """Return the contents of a file on a specific branch or commit.
+
+    :param repo: Open repository
+    :type repo: Git.Repository
+    :param branch: Branch name
+    :type branch: str
+    :param filename: filename to read
+    :type filename: str
+    :param commit: Optional commit hash
+    :type commit: str
+    :returns: The commit id, and the contents of the commit
+    :rtype: tuple(str, str)
+    :raises: Can raise errors from Ggit
+
+    If no commit is passed the master:filename is returned, otherwise it will be
+    commit:filename
+    """
+    if not commit:
+        # Find the most recent commit for filename on the selected branch
+        commits = list_commits(repo, branch, filename, 1)
+        if not commits:
+            raise RecipeError("No commits for %s on the %s branch." % (filename, branch))
+        commit = commits[0].commit
+    return (commit, read_commit_spec(repo, "%s:%s" % (commit, filename)))
    
+
+
    [docs]def read_recipe_commit(repo, branch, recipe_name, commit=None):
+    """Read a recipe commit from git and return a Recipe object
+
+    :param repo: Open repository
+    :type repo: Git.Repository
+    :param branch: Branch name
+    :type branch: str
+    :param recipe_name: Recipe name to read
+    :type recipe_name: str
+    :param commit: Optional commit hash
+    :type commit: str
+    :returns: A Recipe object
+    :rtype: Recipe
+    :raises: Can raise errors from Ggit
+
+    If no commit is passed the master:filename is returned, otherwise it will be
+    commit:filename
+    """
+    (_, recipe_toml) = read_commit(repo, branch, recipe_filename(recipe_name), commit)
+    return recipe_from_toml(recipe_toml)
    
+
+
    [docs]def read_recipe_and_id(repo, branch, recipe_name, commit=None):
+    """Read a recipe commit and its id from git
+
+    :param repo: Open repository
+    :type repo: Git.Repository
+    :param branch: Branch name
+    :type branch: str
+    :param recipe_name: Recipe name to read
+    :type recipe_name: str
+    :param commit: Optional commit hash
+    :type commit: str
+    :returns: The commit id, and a Recipe object
+    :rtype: tuple(str, Recipe)
+    :raises: Can raise errors from Ggit
+
+    If no commit is passed the master:filename is returned, otherwise it will be
+    commit:filename
+    """
+    (commit_id, recipe_toml) = read_commit(repo, branch, recipe_filename(recipe_name), commit)
+    return (commit_id, recipe_from_toml(recipe_toml))
    
+
+
    [docs]def list_branch_files(repo, branch):
+    """Return a sorted list of the files on the branch HEAD
+
+    :param repo: Open repository
+    :type repo: Git.Repository
+    :param branch: Branch name
+    :type branch: str
+    :returns: A sorted list of the filenames
+    :rtype: list(str)
+    :raises: Can raise errors from Ggit
+    """
+    commit = head_commit(repo, branch).get_id().to_string()
+    return list_commit_files(repo, commit)
    
+
+
    [docs]def list_commit_files(repo, commit):
+    """Return a sorted list of the files on a commit
+
+    :param repo: Open repository
+    :type repo: Git.Repository
+    :param commit: The commit hash to list
+    :type commit: str
+    :returns: A sorted list of the filenames
+    :rtype: list(str)
+    :raises: Can raise errors from Ggit
+    """
+    commit_id = Git.OId.new_from_string(commit)
+    commit_obj = repo.lookup(commit_id, Git.Commit)
+    tree = commit_obj.get_tree()
+    return sorted([tree.get(i).get_name() for i in range(0, tree.size())])
    
+
+
    [docs]def delete_recipe(repo, branch, recipe_name):
+    """Delete a recipe from a branch.
+
+    :param repo: Open repository
+    :type repo: Git.Repository
+    :param branch: Branch name
+    :type branch: str
+    :param recipe_name: Recipe name to delete
+    :type recipe_name: str
+    :returns: OId of the new commit
+    :rtype: Git.OId
+    :raises: Can raise errors from Ggit
+    """
+    return delete_file(repo, branch, recipe_filename(recipe_name))
    
+
+
    [docs]def delete_file(repo, branch, filename):
+    """Delete a file from a branch.
+
+    :param repo: Open repository
+    :type repo: Git.Repository
+    :param branch: Branch name
+    :type branch: str
+    :param filename: filename to delete
+    :type filename: str
+    :returns: OId of the new commit
+    :rtype: Git.OId
+    :raises: Can raise errors from Ggit
+    """
+    parent_commit = head_commit(repo, branch)
+    parent_tree = parent_commit.get_tree()
+    builder = repo.create_tree_builder_from_tree(parent_tree)
+    builder.remove(filename)
+    (tree, sig, ref) = prepare_commit(repo, branch, builder)
+    message = "Recipe %s deleted" % filename
+    return repo.create_commit(ref, sig, sig, "UTF-8", message, tree, [parent_commit])
    
+
+
    [docs]def revert_recipe(repo, branch, recipe_name, commit):
+    """Revert the contents of a recipe to that of a previous commit
+
+    :param repo: Open repository
+    :type repo: Git.Repository
+    :param branch: Branch name
+    :type branch: str
+    :param recipe_name: Recipe name to revert
+    :type recipe_name: str
+    :param commit: Commit hash
+    :type commit: str
+    :returns: OId of the new commit
+    :rtype: Git.OId
+    :raises: Can raise errors from Ggit
+    """
+    return revert_file(repo, branch, recipe_filename(recipe_name), commit)
    
+
+
    [docs]def revert_file(repo, branch, filename, commit):
+    """Revert the contents of a file to that of a previous commit
+
+    :param repo: Open repository
+    :type repo: Git.Repository
+    :param branch: Branch name
+    :type branch: str
+    :param filename: filename to revert
+    :type filename: str
+    :param commit: Commit hash
+    :type commit: str
+    :returns: OId of the new commit
+    :rtype: Git.OId
+    :raises: Can raise errors from Ggit
+    """
+    commit_id = Git.OId.new_from_string(commit)
+    commit_obj = repo.lookup(commit_id, Git.Commit)
+    revert_tree = commit_obj.get_tree()
+    entry = revert_tree.get_by_name(filename)
+    blob_id = entry.get_id()
+    parent_commit = head_commit(repo, branch)
+
+    # Use treebuilder to modify the tree
+    parent_tree = parent_commit.get_tree()
+    builder = repo.create_tree_builder_from_tree(parent_tree)
+    builder.insert(filename, blob_id, Git.FileMode.BLOB)
+    (tree, sig, ref) = prepare_commit(repo, branch, builder)
+    commit_hash = commit_id.to_string()
+    message = "%s reverted to commit %s" % (filename, commit_hash)
+    return repo.create_commit(ref, sig, sig, "UTF-8", message, tree, [parent_commit])
    
+
+
    [docs]def commit_recipe(repo, branch, recipe):
+    """Commit a recipe to a branch
+
+    :param repo: Open repository
+    :type repo: Git.Repository
+    :param branch: Branch name
+    :type branch: str
+    :param recipe: Recipe to commit
+    :type recipe: Recipe
+    :returns: OId of the new commit
+    :rtype: Git.OId
+    :raises: Can raise errors from Ggit
+    """
+    try:
+        old_recipe = read_recipe_commit(repo, branch, recipe["name"])
+        old_version = old_recipe["version"]
+    except Exception:
+        old_version = None
+
+    recipe.bump_version(old_version)
+    recipe_toml = recipe.toml()
+    message = "Recipe %s, version %s saved." % (recipe["name"], recipe["version"])
+    return write_commit(repo, branch, recipe.filename, message, recipe_toml)
    
+
+
    [docs]def commit_recipe_file(repo, branch, filename):
+    """Commit a recipe file to a branch
+
+    :param repo: Open repository
+    :type repo: Git.Repository
+    :param branch: Branch name
+    :type branch: str
+    :param filename: Path to the recipe file to commit
+    :type filename: str
+    :returns: OId of the new commit
+    :rtype: Git.OId
+    :raises: Can raise errors from Ggit or RecipeFileError
+    """
+    try:
+        recipe = recipe_from_file(filename)
+    except IOError:
+        raise RecipeFileError
+
+    return commit_recipe(repo, branch, recipe)
    
+
+
    [docs]def commit_recipe_directory(repo, branch, directory):
+    r"""Commit all \*.toml files from a directory, if they aren't already in git.
+
+    :param repo: Open repository
+    :type repo: Git.Repository
+    :param branch: Branch name
+    :type branch: str
+    :param directory: The directory of \*.toml recipes to commit
+    :type directory: str
+    :returns: None
+    :raises: Can raise errors from Ggit or RecipeFileError
+
+    Files with Toml or RecipeFileErrors will be skipped, and the remainder will
+    be tried.
+    """
+    dir_files = set([e for e in os.listdir(directory) if e.endswith(".toml")])
+    branch_files = set(list_branch_files(repo, branch))
+    new_files = dir_files.difference(branch_files)
+
+    for f in new_files:
+        # Skip files with errors, but try the others
+        try:
+            commit_recipe_file(repo, branch, joinpaths(directory, f))
+        except (RecipeFileError, toml.TomlError):
+            pass
    
+
+
    [docs]def tag_recipe_commit(repo, branch, recipe_name):
+    """Tag a file's most recent commit
+
+    :param repo: Open repository
+    :type repo: Git.Repository
+    :param branch: Branch name
+    :type branch: str
+    :param recipe_name: Recipe name to tag
+    :type recipe_name: str
+    :returns: Tag id or None if it failed.
+    :rtype: Git.OId
+    :raises: Can raise errors from Ggit
+
+    Uses tag_file_commit()
+    """
+    return tag_file_commit(repo, branch, recipe_filename(recipe_name))
    
+
+
    [docs]def tag_file_commit(repo, branch, filename):
+    """Tag a file's most recent commit
+
+    :param repo: Open repository
+    :type repo: Git.Repository
+    :param branch: Branch name
+    :type branch: str
+    :param filename: Filename to tag
+    :type filename: str
+    :returns: Tag id or None if it failed.
+    :rtype: Git.OId
+    :raises: Can raise errors from Ggit
+
+    This uses git tags, of the form `refs/tags/<branch>/<filename>/r<revision>`
+    Only the most recent recipe commit can be tagged to prevent out of order tagging.
+    Revisions start at 1 and increment for each new commit that is tagged.
+    If the commit has already been tagged it will return false.
+    """
+    file_commits = list_commits(repo, branch, filename)
+    if not file_commits:
+        return None
+
+    # Find the most recently tagged version (may not be one) and add 1 to it.
+    for details in file_commits:
+        if details.revision is not None:
+            new_revision = details.revision + 1
+            break
+    else:
+        new_revision = 1
+
+    name = "%s/%s/r%d" % (branch, filename, new_revision)
+    sig = Git.Signature.new_now("bdcs-api-server", "user-email")
+    commit_id = Git.OId.new_from_string(file_commits[0].commit)
+    commit = repo.lookup(commit_id, Git.Commit)
+    return repo.create_tag(name, commit, sig, name, Git.CreateFlags.NONE)
    
+
+
    [docs]def find_commit_tag(repo, branch, filename, commit_id):
+    """Find the tag that matches the commit_id
+
+    :param repo: Open repository
+    :type repo: Git.Repository
+    :param branch: Branch name
+    :type branch: str
+    :param filename: filename to revert
+    :type filename: str
+    :param commit_id: The commit id to check
+    :type commit_id: Git.OId
+    :returns: The tag or None if there isn't one
+    :rtype: str or None
+
+    There should be only 1 tag pointing to a commit, but there may not
+    be a tag at all.
+
+    The tag will look like: 'refs/tags/<branch>/<filename>/r<revision>'
+    """
+    pattern = "%s/%s/r*" % (branch, filename)
+    tags = [t for t in repo.list_tags_match(pattern) if is_commit_tag(repo, commit_id, t)]
+    if len(tags) != 1:
+        return None
+    else:
+        return tags[0]
    
+
+
    [docs]def is_commit_tag(repo, commit_id, tag):
+    """Check to see if a tag points to a specific commit.
+
+    :param repo: Open repository
+    :type repo: Git.Repository
+    :param commit_id: The commit id to check
+    :type commit_id: Git.OId
+    :param tag: The tag to check
+    :type tag: str
+    :returns: True if the tag points to the commit, False otherwise
+    :rtype: bool
+    """
+    ref = repo.lookup_reference("refs/tags/" + tag)
+    tag_id = ref.get_target()
+    tag = repo.lookup(tag_id, Git.Tag)
+    target_id = tag.get_target_id()
+    return commit_id.compare(target_id) == 0
    
+
+
    [docs]def get_revision_from_tag(tag):
+    """Return the revision number from a tag
+
+    :param tag: The tag to exract the revision from
+    :type tag: str
+    :returns: The integer revision or None
+    :rtype: int or None
+
+    The revision is the part after the r in 'branch/filename/rXXX'
+    """
+    if tag is None:
+        return None
+    try:
+        return int(tag.rsplit('r', 2)[-1])
+    except (ValueError, IndexError):
+        return None
    
+
+
    [docs]class CommitDetails(DataHolder):
+    def __init__(self, commit, timestamp, message, revision=None):
+        DataHolder.__init__(self,
+                            commit = commit,
+                            timestamp = timestamp,
+                            message = message,
+                            revision = revision)
    
+
+
    [docs]def list_commits(repo, branch, filename, limit=0):
+    """List the commit history of a file on a branch.
+
+    :param repo: Open repository
+    :type repo: Git.Repository
+    :param branch: Branch name
+    :type branch: str
+    :param filename: filename to revert
+    :type filename: str
+    :param limit: Number of commits to return (0=all)
+    :type limit: int
+    :returns: A list of commit details
+    :rtype: list(CommitDetails)
+    :raises: Can raise errors from Ggit
+    """
+    revwalk = Git.RevisionWalker.new(repo)
+    branch_ref = "refs/heads/%s" % branch
+    revwalk.push_ref(branch_ref)
+
+    commits = []
+    while True:
+        commit_id = revwalk.next()
+        if not commit_id:
+            break
+        commit = repo.lookup(commit_id, Git.Commit)
+
+        parents = commit.get_parents()
+        # No parents? Must be the first commit.
+        if parents.get_size() == 0:
+            continue
+
+        tree = commit.get_tree()
+        # Is the filename in this tree? If not, move on.
+        if not tree.get_by_name(filename):
+            continue
+
+        # Is filename different in all of the parent commits?
+        parent_commits = list(map(parents.get, range(0, parents.get_size())))
+        is_diff = all([is_parent_diff(repo, filename, tree, pc) for pc in parent_commits])
+        # No changes from parents, skip it.
+        if not is_diff:
+            continue
+
+        tag = find_commit_tag(repo, branch, filename, commit.get_id())
+        try:
+            commits.append(get_commit_details(commit, get_revision_from_tag(tag)))
+            if limit and len(commits) > limit:
+                break
+        except CommitTimeValError:
+            # Skip any commits that have trouble converting the time
+            # TODO - log details about this failure
+            pass
+
+    # These will be in reverse time sort order thanks to revwalk
+    return commits
    
+
+
    [docs]def get_commit_details(commit, revision=None):
+    """Return the details about a specific commit.
+
+    :param commit: The commit to get details from
+    :type commit: Git.Commit
+    :param revision: Optional commit revision
+    :type revision: int
+    :returns: Details about the commit
+    :rtype: CommitDetails
+    :raises: CommitTimeValError or Ggit exceptions
+
+    """
+    message = commit.get_message()
+    commit_str = commit.get_id().to_string()
+    sig = commit.get_committer()
+
+    datetime = sig.get_time()
+    # XXX What do we do with timezone?
+    _timezone = sig.get_time_zone()
+    timeval = GLib.TimeVal()
+    ok = datetime.to_timeval(timeval)
+    if not ok:
+        raise CommitTimeValError
+    time_str = timeval.to_iso8601()
+
+    return CommitDetails(commit_str, time_str, message, revision)
    
+
+
    [docs]def is_parent_diff(repo, filename, tree, parent):
+    """Check to see if the commit is different from its parents
+
+    :param repo: Open repository
+    :type repo: Git.Repository
+    :param filename: filename to revert
+    :type filename: str
+    :param tree: The commit's tree
+    :type tree: Git.Tree
+    :param parent: The commit's parent commit
+    :type parent: Git.Commit
+    :retuns: True if filename in the commit is different from its parents
+    :rtype: bool
+    """
+    diff_opts = Git.DiffOptions.new()
+    diff_opts.set_pathspec([filename])
+    diff = Git.Diff.new_tree_to_tree(repo, parent.get_tree(), tree, diff_opts)
+    return diff.get_num_deltas() > 0
    
+
+
    [docs]def find_name(name, lst):
+    """Find the dict matching the name in a list and return it.
+
+    :param name: Name to search for
+    :type name: str
+    :param lst: List of dict's with "name" field
+    :returns: First dict with matching name, or None
+    :rtype: dict or None
+    """
+    for e in lst:
+        if e["name"] == name:
+            return e
+    return None
    
+
+
    [docs]def diff_items(title, old_items, new_items):
+    """Return the differences between two lists of dicts.
+
+    :param title: Title of the entry
+    :type title: str
+    :param old_items: List of item dicts with "name" field
+    :type old_items: list(dict)
+    :param new_items: List of item dicts with "name" field
+    :type new_items: list(dict)
+    :returns: List of diff dicts with old/new entries
+    :rtype: list(dict)
+    """
+    diffs = []
+    old_names = set(m["name"] for m in old_items)
+    new_names = set(m["name"] for m in new_items)
+
+    added_items = new_names.difference(old_names)
+    added_items = sorted(added_items, key=lambda n: n.lower())
+
+    removed_items = old_names.difference(new_names)
+    removed_items = sorted(removed_items, key=lambda n: n.lower())
+
+    same_items = old_names.intersection(new_names)
+    same_items = sorted(same_items, key=lambda n: n.lower())
+
+    for name in added_items:
+        diffs.append({"old":None,
+                      "new":{title:find_name(name, new_items)}})
+
+    for name in removed_items:
+        diffs.append({"old":{title:find_name(name, old_items)},
+                      "new":None})
+
+    for name in same_items:
+        old_item = find_name(name, old_items)
+        new_item = find_name(name, new_items)
+        if old_item != new_item:
+            diffs.append({"old":{title:old_item},
+                          "new":{title:new_item}})
+
+    return diffs
    
+
+
+
    [docs]def recipe_diff(old_recipe, new_recipe):
+    """Diff two versions of a recipe
+
+    :param old_recipe: The old version of the recipe
+    :type old_recipe: Recipe
+    :param new_recipe: The new version of the recipe
+    :type new_recipe: Recipe
+    :returns: A list of diff dict entries with old/new
+    :rtype: list(dict)
+    """
+
+    diffs = []
+    # These cannot be added or removed, just different
+    for element in ["name", "description", "version"]:
+        if old_recipe[element] != new_recipe[element]:
+            diffs.append({"old":{element.title():old_recipe[element]},
+                          "new":{element.title():new_recipe[element]}})
+
+    diffs.extend(diff_items("Module", old_recipe["modules"], new_recipe["modules"]))
+    diffs.extend(diff_items("Package", old_recipe["packages"], new_recipe["packages"]))
+
+    return diffs
    
+
    + +
    + +
    + + +
    +
    + +
    + +
    + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/docs/html/_modules/pylorax/api/server.html b/docs/html/_modules/pylorax/api/server.html new file mode 100644 index 00000000..9b11428e --- /dev/null +++ b/docs/html/_modules/pylorax/api/server.html @@ -0,0 +1,297 @@ + + + + + + + + + + + pylorax.api.server — Lorax 29.1 documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    + + + + +
    + + + + + +
    + +
    + + + + + + + + + + + + + + + + + +
    + + + + +
    +
    +
    +
    + +

    Source code for pylorax.api.server

    +#
+# Copyright (C) 2017  Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+import logging
+log = logging.getLogger("lorax-composer")
+
+from collections import namedtuple
+from flask import Flask, jsonify, redirect, send_from_directory
+from glob import glob
+import os
+
+from pylorax import vernum
+from pylorax.api.crossdomain import crossdomain
+from pylorax.api.v0 import v0_api
+from pylorax.sysutils import joinpaths
+
+GitLock = namedtuple("GitLock", ["repo", "lock", "dir"])
+DNFLock = namedtuple("DNFLock", ["dbo", "lock"])
+
+server = Flask(__name__)
+
+__all__ = ["server", "GitLock"]
+
+@server.route('/')
+def server_root():
+    redirect("/api/docs/")
+
+@server.route("/api/docs/")
+@server.route("/api/docs/<path:path>")
+def api_docs(path=None):
+    # Find the html docs
+    try:
+        # This assumes it is running from the source tree
+        docs_path = os.path.abspath(joinpaths(os.path.dirname(__file__), "../../../docs/html"))
+    except IndexError:
+        docs_path = glob("/usr/share/doc/lorax-*/html/")[0]
+
+    if not path:
+        path="index.html"
+    return send_from_directory(docs_path, path)
+
+@server.route("/api/status")
+@crossdomain(origin="*")
+def v0_status():
+    """
+    `/api/v0/status`
+    ^^^^^^^^^^^^^^^^
+    Return the status of the API Server::
+
+          { "api": "0",
+            "build": "devel",
+            "db_supported": true,
+            "db_version": "0",
+            "schema_version": "0",
+            "backend": "lorax-composer"}
+    """
+    return jsonify(backend="lorax-composer",
+                   build=vernum,
+                   api="0",
+                   db_version="0",
+                   schema_version="0",
+                   db_supported=True)
+
+v0_api(server)
+
    + +
    + +
    + + +
    +
    + +
    + +
    + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/docs/html/_modules/pylorax/api/v0.html b/docs/html/_modules/pylorax/api/v0.html new file mode 100644 index 00000000..a47bf9ec --- /dev/null +++ b/docs/html/_modules/pylorax/api/v0.html @@ -0,0 +1,1783 @@ + + + + + + + + + + + pylorax.api.v0 — Lorax 29.1 documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    + + + + +
    + + + + + +
    + +
    + + + + + + + + + + + + + + + + + +
    + + + + +
    +
    +
    +
    + +

    Source code for pylorax.api.v0

    +#
+# Copyright (C) 2017-2018  Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+""" Setup v0 of the API server
+
+v0_api() must be called to setup the API routes for Flask
+
+Status Responses
+----------------
+
+Some requests only return a status/error response.
+
+  The response will be a status response with `status` set to true, or an
+  error response with it set to false and an error message included.
+
+  Example response::
+
+      {
+        "status": true
+      }
+
+  Error response::
+
+      {
+        "errors": ["ggit-error: Failed to remove entry. File isn't in the tree - jboss.toml (-1)"]
+        "status": false
+      }
+
+API Routes
+----------
+
+All of the blueprints routes support the optional `branch` argument. If it is not
+used then the API will use the `master` branch for blueprints. If you want to create
+a new branch use the `new` or `workspace` routes with ?branch=<branch-name> to
+store the new blueprint on the new branch.
+
+`/api/v0/blueprints/list`
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  List the available blueprints::
+
+      { "limit": 20,
+        "offset": 0,
+        "blueprints": [
+          "atlas",
+          "development",
+          "glusterfs",
+          "http-server",
+          "jboss",
+          "kubernetes" ],
+        "total": 6 }
+
+`/api/v0/blueprints/info/<blueprint_names>`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  Return the JSON representation of the blueprint. This includes 3 top level
+  objects.  `changes` which lists whether or not the workspace is different from
+  the most recent commit. `blueprints` which lists the JSON representation of the
+  blueprint, and `errors` which will list any errors, like non-existant blueprints.
+
+  Example::
+
+      {
+        "changes": [
+          {
+            "changed": false,
+            "name": "glusterfs"
+          }
+        ],
+        "errors": [],
+        "blueprints": [
+          {
+            "description": "An example GlusterFS server with samba",
+            "modules": [
+              {
+                "name": "glusterfs",
+                "version": "3.7.*"
+              },
+              {
+                "name": "glusterfs-cli",
+                "version": "3.7.*"
+              }
+            ],
+            "name": "glusterfs",
+            "packages": [
+              {
+                "name": "2ping",
+                "version": "3.2.1"
+              },
+              {
+                "name": "samba",
+                "version": "4.2.*"
+              }
+            ],
+            "version": "0.0.6"
+          }
+        ]
+      }
+
+  Error example::
+
+      {
+        "changes": [],
+        "errors": ["ggit-error: the path 'missing.toml' does not exist in the given tree (-3)"]
+        "blueprints": []
+      }
+
+`/api/v0/blueprints/changes/<blueprint_names>[?offset=0&limit=20]`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  Return the commits to a blueprint. By default it returns the first 20 commits, this
+  can be changed by passing `offset` and/or `limit`. The response will include the
+  commit hash, summary, timestamp, and optionally the revision number. The commit
+  hash can be passed to `/api/v0/blueprints/diff/` to retrieve the exact changes.
+
+  Example::
+
+      {
+        "errors": [],
+        "limit": 20,
+        "offset": 0,
+        "blueprints": [
+          {
+            "changes": [
+              {
+                "commit": "e083921a7ed1cf2eec91ad12b9ad1e70ef3470be",
+                "message": "blueprint glusterfs, version 0.0.6 saved.",
+                "revision": null,
+                "timestamp": "2017-11-23T00:18:13Z"
+              },
+              {
+                "commit": "cee5f4c20fc33ea4d54bfecf56f4ad41ad15f4f3",
+                "message": "blueprint glusterfs, version 0.0.5 saved.",
+                "revision": null,
+                "timestamp": "2017-11-11T01:00:28Z"
+              },
+              {
+                "commit": "29b492f26ed35d80800b536623bafc51e2f0eff2",
+                "message": "blueprint glusterfs, version 0.0.4 saved.",
+                "revision": null,
+                "timestamp": "2017-11-11T00:28:30Z"
+              },
+              {
+                "commit": "03374adbf080fe34f5c6c29f2e49cc2b86958bf2",
+                "message": "blueprint glusterfs, version 0.0.3 saved.",
+                "revision": null,
+                "timestamp": "2017-11-10T23:15:52Z"
+              },
+              {
+                "commit": "0e08ecbb708675bfabc82952599a1712a843779d",
+                "message": "blueprint glusterfs, version 0.0.2 saved.",
+                "revision": null,
+                "timestamp": "2017-11-10T23:14:56Z"
+              },
+              {
+                "commit": "3e11eb87a63d289662cba4b1804a0947a6843379",
+                "message": "blueprint glusterfs, version 0.0.1 saved.",
+                "revision": null,
+                "timestamp": "2017-11-08T00:02:47Z"
+              }
+            ],
+            "name": "glusterfs",
+            "total": 6
+          }
+        ]
+      }
+
+POST `/api/v0/blueprints/new`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  Create a new blueprint, or update an existing blueprint. This supports both JSON and TOML
+  for the blueprint format. The blueprint should be in the body of the request with the
+  `Content-Type` header set to either `application/json` or `text/x-toml`.
+
+  The response will be a status response with `status` set to true, or an
+  error response with it set to false and an error message included.
+
+DELETE `/api/v0/blueprints/delete/<blueprint_name>`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  Delete a blueprint. The blueprint is deleted from the branch, and will no longer
+  be listed by the `list` route. A blueprint can be undeleted using the `undo` route
+  to revert to a previous commit.
+
+  The response will be a status response with `status` set to true, or an
+  error response with it set to false and an error message included.
+
+POST `/api/v0/blueprints/workspace`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  Write a blueprint to the temporary workspace. This works exactly the same as `new` except
+  that it does not create a commit. JSON and TOML bodies are supported.
+
+  The workspace is meant to be used as a temporary blueprint storage for clients.
+  It will be read by the `info` and `diff` routes if it is different from the
+  most recent commit.
+
+  The response will be a status response with `status` set to true, or an
+  error response with it set to false and an error message included.
+
+DELETE `/api/v0/blueprints/workspace/<blueprint_name>`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  Remove the temporary workspace copy of a blueprint. The `info` route will now
+  return the most recent commit of the blueprint. Any changes that were in the
+  workspace will be lost.
+
+  The response will be a status response with `status` set to true, or an
+  error response with it set to false and an error message included.
+
+POST `/api/v0/blueprints/undo/<blueprint_name>/<commit>`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  This will revert the blueprint to a previous commit. The commit hash from the `changes`
+  route can be used in this request.
+
+  The response will be a status response with `status` set to true, or an
+  error response with it set to false and an error message included.
+
+POST `/api/v0/blueprints/tag/<blueprint_name>`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  Tag a blueprint as a new release. This uses git tags with a special format.
+  `refs/tags/<branch>/<filename>/r<revision>`. Only the most recent blueprint commit
+  can be tagged. Revisions start at 1 and increment for each new tag
+  (per-blueprint). If the commit has already been tagged it will return false.
+
+  The response will be a status response with `status` set to true, or an
+  error response with it set to false and an error message included.
+
+`/api/v0/blueprints/diff/<blueprint_name>/<from_commit>/<to_commit>`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  Return the differences between two commits, or the workspace. The commit hash
+  from the `changes` response can be used here, or several special strings:
+
+  - NEWEST will select the newest git commit. This works for `from_commit` or `to_commit`
+  - WORKSPACE will select the workspace copy. This can only be used in `to_commit`
+
+  eg. `/api/v0/blueprints/diff/glusterfs/NEWEST/WORKSPACE` will return the differences
+  between the most recent git commit and the contents of the workspace.
+
+  Each entry in the response's diff object contains the old blueprint value and the new one.
+  If old is null and new is set, then it was added.
+  If new is null and old is set, then it was removed.
+  If both are set, then it was changed.
+
+  The old/new entries will have the name of the blueprint field that was changed. This
+  can be one of: Name, Description, Version, Module, or Package.
+  The contents for these will be the old/new values for them.
+
+  In the example below the version was changed and the ping package was added.
+
+  Example::
+
+      {
+        "diff": [
+          {
+            "new": {
+              "Version": "0.0.6"
+            },
+            "old": {
+              "Version": "0.0.5"
+            }
+          },
+          {
+            "new": {
+              "Package": {
+                "name": "ping",
+                "version": "3.2.1"
+              }
+            },
+            "old": null
+          }
+        ]
+      }
+
+`/api/v0/blueprints/freeze/<blueprint_names>`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  Return a JSON representation of the blueprint with the package and module versions set
+  to the exact versions chosen by depsolving the blueprint.
+
+  Example::
+
+      {
+        "errors": [],
+        "blueprints": [
+          {
+            "blueprint": {
+              "description": "An example GlusterFS server with samba",
+              "modules": [
+                {
+                  "name": "glusterfs",
+                  "version": "3.8.4-18.4.el7.x86_64"
+                },
+                {
+                  "name": "glusterfs-cli",
+                  "version": "3.8.4-18.4.el7.x86_64"
+                }
+              ],
+              "name": "glusterfs",
+              "packages": [
+                {
+                  "name": "ping",
+                  "version": "2:3.2.1-2.el7.noarch"
+                },
+                {
+                  "name": "samba",
+                  "version": "4.6.2-8.el7.x86_64"
+                }
+              ],
+              "version": "0.0.6"
+            }
+          }
+        ]
+      }
+
+`/api/v0/blueprints/depsolve/<blueprint_names>`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  Depsolve the blueprint using yum, return the blueprint used, and the NEVRAs of the packages
+  chosen to satisfy the blueprint's requirements. The response will include a list of results,
+  with the full dependency list in `dependencies`, the NEVRAs for the blueprint's direct modules
+  and packages in `modules`, and any error will be in `errors`.
+
+  Example::
+
+      {
+        "errors": [],
+        "blueprints": [
+          {
+            "dependencies": [
+              {
+                "arch": "noarch",
+                "epoch": "0",
+                "name": "2ping",
+                "release": "2.el7",
+                "version": "3.2.1"
+              },
+              {
+                "arch": "x86_64",
+                "epoch": "0",
+                "name": "acl",
+                "release": "12.el7",
+                "version": "2.2.51"
+              },
+              {
+                "arch": "x86_64",
+                "epoch": "0",
+                "name": "audit-libs",
+                "release": "3.el7",
+                "version": "2.7.6"
+              },
+              {
+                "arch": "x86_64",
+                "epoch": "0",
+                "name": "avahi-libs",
+                "release": "17.el7",
+                "version": "0.6.31"
+              },
+              ...
+            ],
+            "modules": [
+              {
+                "arch": "noarch",
+                "epoch": "0",
+                "name": "2ping",
+                "release": "2.el7",
+                "version": "3.2.1"
+              },
+              {
+                "arch": "x86_64",
+                "epoch": "0",
+                "name": "glusterfs",
+                "release": "18.4.el7",
+                "version": "3.8.4"
+              },
+              ...
+            ],
+            "blueprint": {
+              "description": "An example GlusterFS server with samba",
+              "modules": [
+                {
+                  "name": "glusterfs",
+                  "version": "3.7.*"
+                },
+             ...
+            }
+          }
+        ]
+      }
+
+`/api/v0/projects/list[?offset=0&limit=20]`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  List all of the available projects. By default this returns the first 20 items,
+  but this can be changed by setting the `offset` and `limit` arguments.
+
+  Example::
+
+      {
+        "limit": 20,
+        "offset": 0,
+        "projects": [
+          {
+            "description": "0 A.D. (pronounced \"zero ey-dee\") is a ...",
+            "homepage": "http://play0ad.com",
+            "name": "0ad",
+            "summary": "Cross-Platform RTS Game of Ancient Warfare",
+            "upstream_vcs": "UPSTREAM_VCS"
+          },
+          ...
+        ],
+        "total": 21770
+      }
+
+`/api/v0/projects/info/<project_names>`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  Return information about the comma-separated list of projects. It includes the description
+  of the package along with the list of available builds.
+
+  Example::
+
+      {
+        "projects": [
+          {
+            "builds": [
+              {
+                "arch": "x86_64",
+                "build_config_ref": "BUILD_CONFIG_REF",
+                "build_env_ref": "BUILD_ENV_REF",
+                "build_time": "2017-03-01T08:39:23",
+                "changelog": "- restore incremental backups correctly, files ...",
+                "epoch": "2",
+                "metadata": {},
+                "release": "32.el7",
+                "source": {
+                  "license": "GPLv3+",
+                  "metadata": {},
+                  "source_ref": "SOURCE_REF",
+                  "version": "1.26"
+                }
+              }
+            ],
+            "description": "The GNU tar program saves many ...",
+            "homepage": "http://www.gnu.org/software/tar/",
+            "name": "tar",
+            "summary": "A GNU file archiving program",
+            "upstream_vcs": "UPSTREAM_VCS"
+          }
+        ]
+      }
+
+`/api/v0/projects/depsolve/<project_names>`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  Depsolve the comma-separated list of projects and return the list of NEVRAs needed
+  to satisfy the request.
+
+  Example::
+
+      {
+        "projects": [
+          {
+            "arch": "noarch",
+            "epoch": "0",
+            "name": "basesystem",
+            "release": "7.el7",
+            "version": "10.0"
+          },
+          {
+            "arch": "x86_64",
+            "epoch": "0",
+            "name": "bash",
+            "release": "28.el7",
+            "version": "4.2.46"
+          },
+          {
+            "arch": "x86_64",
+            "epoch": "0",
+            "name": "filesystem",
+            "release": "21.el7",
+            "version": "3.2"
+          },
+          ...
+        ]
+      }
+
+`/api/v0/modules/list[?offset=0&limit=20]`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  Return a list of all of the available modules. This includes the name and the
+  group_type, which is always "rpm" for lorax-composer. By default this returns
+  the first 20 items. This can be changed by setting the `offset` and `limit`
+  arguments.
+
+  Example::
+
+      {
+        "limit": 20,
+        "modules": [
+          {
+            "group_type": "rpm",
+            "name": "0ad"
+          },
+          {
+            "group_type": "rpm",
+            "name": "0ad-data"
+          },
+          {
+            "group_type": "rpm",
+            "name": "0install"
+          },
+          {
+            "group_type": "rpm",
+            "name": "2048-cli"
+          },
+          ...
+        ]
+        "total": 21770
+      }
+
+`/api/v0/modules/list/<module_names>[?offset=0&limit=20]`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  Return the list of comma-separated modules. Output is the same as `/modules/list`
+
+  Example::
+
+      {
+        "limit": 20,
+        "modules": [
+          {
+            "group_type": "rpm",
+            "name": "tar"
+          }
+        ],
+        "offset": 0,
+        "total": 1
+      }
+
+`/api/v0/modules/info/<module_names>`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  Return the module's dependencies, and the information about the module.
+
+  Example::
+
+      {
+        "modules": [
+          {
+            "dependencies": [
+              {
+                "arch": "noarch",
+                "epoch": "0",
+                "name": "basesystem",
+                "release": "7.el7",
+                "version": "10.0"
+              },
+              {
+                "arch": "x86_64",
+                "epoch": "0",
+                "name": "bash",
+                "release": "28.el7",
+                "version": "4.2.46"
+              },
+              ...
+            ],
+            "description": "The GNU tar program saves ...",
+            "homepage": "http://www.gnu.org/software/tar/",
+            "name": "tar",
+            "summary": "A GNU file archiving program",
+            "upstream_vcs": "UPSTREAM_VCS"
+          }
+        ]
+      }
+
+POST `/api/v0/compose`
+^^^^^^^^^^^^^^^^^^^^^^
+
+  Start a compose. The content type should be 'application/json' and the body of the POST
+  should look like this::
+
+      {
+        "blueprint_name": "http-server",
+        "compose_type": "tar",
+        "branch": "master"
+      }
+
+  Pass it the name of the blueprint, the type of output (from '/api/v0/compose/types'), and the
+  blueprint branch to use. 'branch' is optional and will default to master. It will create a new
+  build and add it to the queue. It returns the build uuid and a status if it succeeds::
+
+      {
+        "build_id": "e6fa6db4-9c81-4b70-870f-a697ca405cdf",
+        "status": true
+      }
+
+`/api/v0/compose/types`
+^^^^^^^^^^^^^^^^^^^^^^^
+
+  Returns the list of supported output types that are valid for use with 'POST /api/v0/compose'
+
+      {
+        "types": [
+          {
+            "enabled": true,
+            "name": "tar"
+          }
+        ]
+      }
+
+`/api/v0/compose/queue`
+^^^^^^^^^^^^^^^^^^^^^^^
+
+  Return the status of the build queue. It includes information about the builds waiting,
+  and the build that is running.
+
+  Example::
+
+      {
+        "new": [
+          {
+            "id": "45502a6d-06e8-48a5-a215-2b4174b3614b",
+            "blueprint": "glusterfs",
+            "queue_status": "WAITING",
+            "timestamp": 1517362647.4570868,
+            "version": "0.0.6"
+          },
+          {
+            "id": "6d292bd0-bec7-4825-8d7d-41ef9c3e4b73",
+            "blueprint": "kubernetes",
+            "queue_status": "WAITING",
+            "timestamp": 1517362659.0034983,
+            "version": "0.0.1"
+          }
+        ],
+        "run": [
+          {
+            "id": "745712b2-96db-44c0-8014-fe925c35e795",
+            "blueprint": "glusterfs",
+            "queue_status": "RUNNING",
+            "timestamp": 1517362633.7965999,
+            "version": "0.0.6"
+          }
+        ]
+      }
+
+`/api/v0/compose/finished`
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  Return the details on all of the finished composes on the system.
+
+  Example::
+
+      {
+        "finished": [
+          {
+            "id": "70b84195-9817-4b8a-af92-45e380f39894",
+            "blueprint": "glusterfs",
+            "queue_status": "FINISHED",
+            "timestamp": 1517351003.8210032,
+            "version": "0.0.6"
+          },
+          {
+            "id": "e695affd-397f-4af9-9022-add2636e7459",
+            "blueprint": "glusterfs",
+            "queue_status": "FINISHED",
+            "timestamp": 1517362289.7193348,
+            "version": "0.0.6"
+          }
+        ]
+      }
+
+`/api/v0/compose/failed`
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  Return the details on all of the failed composes on the system.
+
+  Example::
+
+      {
+        "failed": [
+           {
+            "id": "8c8435ef-d6bd-4c68-9bf1-a2ef832e6b1a",
+            "blueprint": "http-server",
+            "queue_status": "FAILED",
+            "timestamp": 1517523249.9301329,
+            "version": "0.0.2"
+          }
+        ]
+      }
+
+`/api/v0/compose/status/<uuids>`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  Return the details for each of the comma-separated list of uuids.
+
+  Example::
+
+      {
+        "uuids": [
+          {
+            "id": "8c8435ef-d6bd-4c68-9bf1-a2ef832e6b1a",
+            "blueprint": "http-server",
+            "queue_status": "FINISHED",
+            "timestamp": 1517523644.2384307,
+            "version": "0.0.2"
+          },
+          {
+            "id": "45502a6d-06e8-48a5-a215-2b4174b3614b",
+            "blueprint": "glusterfs",
+            "queue_status": "FINISHED",
+            "timestamp": 1517363442.188399,
+            "version": "0.0.6"
+          }
+        ]
+      }
+
+DELETE `/api/v0/blueprints/cancel/<uuid>`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  Cancel the build, if it is not finished, and delete the results. It will return a
+  status of True if it is successful.
+
+  Example::
+
+      {
+        "status": true,
+        "uuid": "03397f8d-acff-4cdb-bd31-f629b7a948f5"
+      }
+
+DELETE `/api/v0/compose/delete/<uuids>`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  Delete the list of comma-separated uuids from the compose results.
+
+  Example::
+
+      {
+        "errors": [],
+        "uuids": [
+          {
+            "status": true,
+            "uuid": "ae1bf7e3-7f16-4c9f-b36e-3726a1093fd0"
+          }
+        ]
+      }
+
+`/api/v0/compose/info/<uuid>`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  Get detailed information about the compose. The returned JSON string will
+  contain the following information:
+
+    * id - The uuid of the comoposition
+    * config - containing the configuration settings used to run Anaconda
+    * blueprint - The depsolved blueprint used to generate the kickstart
+    * commit - The (local) git commit hash for the blueprint used
+    * deps - The NEVRA of all of the dependencies used in the composition
+    * compose_type - The type of output generated (tar, iso, etc.)
+    * queue_status - The final status of the composition (FINISHED or FAILED)
+
+  Example::
+
+      {
+        "commit": "7078e521a54b12eae31c3fd028680da7a0815a4d",
+        "compose_type": "tar",
+        "config": {
+          "anaconda_args": "",
+          "armplatform": "",
+          "compress_args": [],
+          "compression": "xz",
+          "image_name": "root.tar.xz",
+          ...
+        },
+        "deps": {
+          "packages": [
+            {
+              "arch": "x86_64",
+              "epoch": "0",
+              "name": "acl",
+              "release": "14.el7",
+              "version": "2.2.51"
+            }
+          ]
+        },
+        "id": "c30b7d80-523b-4a23-ad52-61b799739ce8",
+        "queue_status": "FINISHED",
+        "blueprint": {
+          "description": "An example kubernetes master",
+          ...
+        }
+      }
+
+`/api/v0/compose/metadata/<uuid>`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  Returns a .tar of the metadata used for the build. This includes all the
+  information needed to reproduce the build, including the final kickstart
+  populated with repository and package NEVRA.
+
+  The mime type is set to 'application/x-tar' and the filename is set to
+  UUID-metadata.tar
+
+  The .tar is uncompressed, but is not large.
+
+`/api/v0/compose/results/<uuid>`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  Returns a .tar of the metadata, logs, and output image of the build. This
+  includes all the information needed to reproduce the build, including the
+  final kickstart populated with repository and package NEVRA. The output image
+  is already in compressed form so the returned tar is not compressed.
+
+  The mime type is set to 'application/x-tar' and the filename is set to
+  UUID.tar
+
+`/api/v0/compose/logs/<uuid>`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  Returns a .tar of the anaconda build logs. The tar is not compressed, but is
+  not large.
+
+  The mime type is set to 'application/x-tar' and the filename is set to
+  UUID-logs.tar
+
+`/api/v0/compose/image/<uuid>`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  Returns the output image from the build. The filename is set to the filename
+  from the build with the UUID as a prefix. eg. UUID-root.tar.xz or UUID-boot.iso.
+
+`/api/v0/compose/log/<uuid>[?size=kbytes]`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  Returns the end of the anaconda.log. The size parameter is optional and defaults to 1Mbytes
+  if it is not included. The returned data is raw text from the end of the logfile, starting on
+  a line boundry.
+
+  Example::
+
+      12:59:24,222 INFO anaconda: Running Thread: AnaConfigurationThread (140629395244800)
+      12:59:24,223 INFO anaconda: Configuring installed system
+      12:59:24,912 INFO anaconda: Configuring installed system
+      12:59:24,912 INFO anaconda: Creating users
+      12:59:24,913 INFO anaconda: Clearing libuser.conf at /tmp/libuser.Dyy8Gj
+      12:59:25,154 INFO anaconda: Creating users
+      12:59:25,155 INFO anaconda: Configuring addons
+      12:59:25,155 INFO anaconda: Configuring addons
+      12:59:25,155 INFO anaconda: Generating initramfs
+      12:59:49,467 INFO anaconda: Generating initramfs
+      12:59:49,467 INFO anaconda: Running post-installation scripts
+      12:59:49,467 INFO anaconda: Running kickstart %%post script(s)
+      12:59:50,782 INFO anaconda: All kickstart %%post script(s) have been run
+      12:59:50,782 INFO anaconda: Running post-installation scripts
+      12:59:50,784 INFO anaconda: Thread Done: AnaConfigurationThread (140629395244800)
+
+"""
+
+import logging
+log = logging.getLogger("lorax-composer")
+
+import os
+from flask import jsonify, request, Response, send_file
+
+from pylorax.api.compose import start_build, compose_types
+from pylorax.api.crossdomain import crossdomain
+from pylorax.api.projects import projects_list, projects_info, projects_depsolve
+from pylorax.api.projects import modules_list, modules_info, ProjectsError
+from pylorax.api.queue import queue_status, build_status, uuid_delete, uuid_status, uuid_info
+from pylorax.api.queue import uuid_tar, uuid_image, uuid_cancel, uuid_log
+from pylorax.api.recipes import list_branch_files, read_recipe_commit, recipe_filename, list_commits
+from pylorax.api.recipes import recipe_from_dict, recipe_from_toml, commit_recipe, delete_recipe, revert_recipe
+from pylorax.api.recipes import tag_recipe_commit, recipe_diff
+from pylorax.api.workspace import workspace_read, workspace_write, workspace_delete
+
+# The API functions don't actually get called by any code here
+# pylint: disable=unused-variable
+
+
    [docs]def take_limits(iterable, offset, limit):
+    """ Apply offset and limit to an iterable object
+
+    :param iterable: The object to limit
+    :type iterable: iter
+    :param offset: The number of items to skip
+    :type offset: int
+    :param limit: The total number of items to return
+    :type limit: int
+    :returns: A subset of the iterable
+    """
+    return iterable[offset:][:limit]
    
+
+
    [docs]def v0_api(api):
+    # Note that Sphinx will not generate documentations for any of these.
+    @api.route("/api/v0/blueprints/list")
+    @crossdomain(origin="*")
+    def v0_blueprints_list():
+        """List the available blueprints on a branch."""
+        branch = request.args.get("branch", "master")
+        try:
+            limit = int(request.args.get("limit", "20"))
+            offset = int(request.args.get("offset", "0"))
+        except ValueError as e:
+            return jsonify(status=False, errors=[str(e)]), 400
+
+        with api.config["GITLOCK"].lock:
+            blueprints = take_limits([f[:-5] for f in list_branch_files(api.config["GITLOCK"].repo, branch)], offset, limit)
+        return jsonify(blueprints=blueprints, limit=limit, offset=offset, total=len(blueprints))
+
+    @api.route("/api/v0/blueprints/info/<blueprint_names>")
+    @crossdomain(origin="*")
+    def v0_blueprints_info(blueprint_names):
+        """Return the contents of the blueprint, or a list of blueprints"""
+        branch = request.args.get("branch", "master")
+        out_fmt = request.args.get("format", "json")
+        blueprints = []
+        changes = []
+        errors = []
+        for blueprint_name in [n.strip() for n in blueprint_names.split(",")]:
+            exceptions = []
+            # Get the workspace version (if it exists)
+            try:
+                with api.config["GITLOCK"].lock:
+                    ws_blueprint = workspace_read(api.config["GITLOCK"].repo, branch, blueprint_name)
+            except Exception as e:
+                ws_blueprint = None
+                exceptions.append(str(e))
+                log.error("(v0_blueprints_info) %s", str(e))
+
+            # Get the git version (if it exists)
+            try:
+                with api.config["GITLOCK"].lock:
+                    git_blueprint = read_recipe_commit(api.config["GITLOCK"].repo, branch, blueprint_name)
+            except Exception as e:
+                git_blueprint = None
+                exceptions.append(str(e))
+                log.error("(v0_blueprints_info) %s", str(e))
+
+            if not ws_blueprint and not git_blueprint:
+                # Neither blueprint, return an error
+                errors.append("%s: %s" % (blueprint_name, ", ".join(exceptions)))
+            elif ws_blueprint and not git_blueprint:
+                # No git blueprint, return the workspace blueprint
+                changes.append({"name":blueprint_name, "changed":True})
+                blueprints.append(ws_blueprint)
+            elif not ws_blueprint and git_blueprint:
+                # No workspace blueprint, no change, return the git blueprint
+                changes.append({"name":blueprint_name, "changed":False})
+                blueprints.append(git_blueprint)
+            else:
+                # Both exist, maybe changed, return the workspace blueprint
+                changes.append({"name":blueprint_name, "changed":ws_blueprint != git_blueprint})
+                blueprints.append(ws_blueprint)
+
+        # Sort all the results by case-insensitive blueprint name
+        changes = sorted(changes, key=lambda c: c["name"].lower())
+        blueprints = sorted(blueprints, key=lambda r: r["name"].lower())
+        errors = sorted(errors, key=lambda e: e.lower())
+
+        if out_fmt == "toml":
+            # With TOML output we just want to dump the raw blueprint, skipping the rest.
+            return "\n\n".join([r.toml() for r in blueprints])
+        else:
+            return jsonify(changes=changes, blueprints=blueprints, errors=errors)
+
+    @api.route("/api/v0/blueprints/changes/<blueprint_names>")
+    @crossdomain(origin="*")
+    def v0_blueprints_changes(blueprint_names):
+        """Return the changes to a blueprint or list of blueprints"""
+        branch = request.args.get("branch", "master")
+        try:
+            limit = int(request.args.get("limit", "20"))
+            offset = int(request.args.get("offset", "0"))
+        except ValueError as e:
+            return jsonify(status=False, errors=[str(e)]), 400
+
+        blueprints = []
+        errors = []
+        for blueprint_name in [n.strip() for n in blueprint_names.split(",")]:
+            filename = recipe_filename(blueprint_name)
+            try:
+                with api.config["GITLOCK"].lock:
+                    commits = take_limits(list_commits(api.config["GITLOCK"].repo, branch, filename), offset, limit)
+            except Exception as e:
+                errors.append("%s: %s" % (blueprint_name, str(e)))
+                log.error("(v0_blueprints_changes) %s", str(e))
+            else:
+                blueprints.append({"name":blueprint_name, "changes":commits, "total":len(commits)})
+
+        blueprints = sorted(blueprints, key=lambda r: r["name"].lower())
+        errors = sorted(errors, key=lambda e: e.lower())
+
+        return jsonify(blueprints=blueprints, errors=errors, offset=offset, limit=limit)
+
+    @api.route("/api/v0/blueprints/new", methods=["POST"])
+    @crossdomain(origin="*")
+    def v0_blueprints_new():
+        """Commit a new blueprint"""
+        branch = request.args.get("branch", "master")
+        try:
+            if request.headers['Content-Type'] == "text/x-toml":
+                blueprint = recipe_from_toml(request.data)
+            else:
+                blueprint = recipe_from_dict(request.get_json(cache=False))
+
+            with api.config["GITLOCK"].lock:
+                commit_recipe(api.config["GITLOCK"].repo, branch, blueprint)
+
+                # Read the blueprint with new version and write it to the workspace
+                blueprint = read_recipe_commit(api.config["GITLOCK"].repo, branch, blueprint["name"])
+                workspace_write(api.config["GITLOCK"].repo, branch, blueprint)
+        except Exception as e:
+            log.error("(v0_blueprints_new) %s", str(e))
+            return jsonify(status=False, errors=[str(e)]), 400
+        else:
+            return jsonify(status=True)
+
+    @api.route("/api/v0/blueprints/delete/<blueprint_name>", methods=["DELETE"])
+    @crossdomain(origin="*")
+    def v0_blueprints_delete(blueprint_name):
+        """Delete a blueprint from git"""
+        branch = request.args.get("branch", "master")
+        try:
+            with api.config["GITLOCK"].lock:
+                delete_recipe(api.config["GITLOCK"].repo, branch, blueprint_name)
+        except Exception as e:
+            log.error("(v0_blueprints_delete) %s", str(e))
+            return jsonify(status=False, errors=[str(e)]), 400
+        else:
+            return jsonify(status=True)
+
+    @api.route("/api/v0/blueprints/workspace", methods=["POST"])
+    @crossdomain(origin="*")
+    def v0_blueprints_workspace():
+        """Write a blueprint to the workspace"""
+        branch = request.args.get("branch", "master")
+        try:
+            if request.headers['Content-Type'] == "text/x-toml":
+                blueprint = recipe_from_toml(request.data)
+            else:
+                blueprint = recipe_from_dict(request.get_json(cache=False))
+
+            with api.config["GITLOCK"].lock:
+                workspace_write(api.config["GITLOCK"].repo, branch, blueprint)
+        except Exception as e:
+            log.error("(v0_blueprints_workspace) %s", str(e))
+            return jsonify(status=False, errors=[str(e)]), 400
+        else:
+            return jsonify(status=True)
+
+    @api.route("/api/v0/blueprints/workspace/<blueprint_name>", methods=["DELETE"])
+    @crossdomain(origin="*")
+    def v0_blueprints_delete_workspace(blueprint_name):
+        """Delete a blueprint from the workspace"""
+        branch = request.args.get("branch", "master")
+        try:
+            with api.config["GITLOCK"].lock:
+                workspace_delete(api.config["GITLOCK"].repo, branch, blueprint_name)
+        except Exception as e:
+            log.error("(v0_blueprints_delete_workspace) %s", str(e))
+            return jsonify(status=False, error=[str(e)]), 400
+        else:
+            return jsonify(status=True)
+
+    @api.route("/api/v0/blueprints/undo/<blueprint_name>/<commit>", methods=["POST"])
+    @crossdomain(origin="*")
+    def v0_blueprints_undo(blueprint_name, commit):
+        """Undo changes to a blueprint by reverting to a previous commit."""
+        branch = request.args.get("branch", "master")
+        try:
+            with api.config["GITLOCK"].lock:
+                revert_recipe(api.config["GITLOCK"].repo, branch, blueprint_name, commit)
+
+                # Read the new recipe and write it to the workspace
+                blueprint = read_recipe_commit(api.config["GITLOCK"].repo, branch, blueprint_name)
+                workspace_write(api.config["GITLOCK"].repo, branch, blueprint)
+        except Exception as e:
+            log.error("(v0_blueprints_undo) %s", str(e))
+            return jsonify(status=False, errors=[str(e)]), 400
+        else:
+            return jsonify(status=True)
+
+    @api.route("/api/v0/blueprints/tag/<blueprint_name>", methods=["POST"])
+    @crossdomain(origin="*")
+    def v0_blueprints_tag(blueprint_name):
+        """Tag a blueprint's latest blueprint commit as a 'revision'"""
+        branch = request.args.get("branch", "master")
+        try:
+            with api.config["GITLOCK"].lock:
+                tag_recipe_commit(api.config["GITLOCK"].repo, branch, blueprint_name)
+        except Exception as e:
+            log.error("(v0_blueprints_tag) %s", str(e))
+            return jsonify(status=False, errors=[str(e)]), 400
+        else:
+            return jsonify(status=True)
+
+    @api.route("/api/v0/blueprints/diff/<blueprint_name>/<from_commit>/<to_commit>")
+    @crossdomain(origin="*")
+    def v0_blueprints_diff(blueprint_name, from_commit, to_commit):
+        """Return the differences between two commits of a blueprint"""
+        branch = request.args.get("branch", "master")
+        try:
+            if from_commit == "NEWEST":
+                with api.config["GITLOCK"].lock:
+                    old_blueprint = read_recipe_commit(api.config["GITLOCK"].repo, branch, blueprint_name)
+            else:
+                with api.config["GITLOCK"].lock:
+                    old_blueprint = read_recipe_commit(api.config["GITLOCK"].repo, branch, blueprint_name, from_commit)
+        except Exception as e:
+            log.error("(v0_blueprints_diff) %s", str(e))
+            return jsonify(status=False, errors=[str(e)]), 400
+
+        try:
+            if to_commit == "WORKSPACE":
+                with api.config["GITLOCK"].lock:
+                    new_blueprint = workspace_read(api.config["GITLOCK"].repo, branch, blueprint_name)
+                # If there is no workspace, use the newest commit instead
+                if not new_blueprint:
+                    with api.config["GITLOCK"].lock:
+                        new_blueprint = read_recipe_commit(api.config["GITLOCK"].repo, branch, blueprint_name)
+            elif to_commit == "NEWEST":
+                with api.config["GITLOCK"].lock:
+                    new_blueprint = read_recipe_commit(api.config["GITLOCK"].repo, branch, blueprint_name)
+            else:
+                with api.config["GITLOCK"].lock:
+                    new_blueprint = read_recipe_commit(api.config["GITLOCK"].repo, branch, blueprint_name, to_commit)
+        except Exception as e:
+            log.error("(v0_blueprints_diff) %s", str(e))
+            return jsonify(status=False, errors=[str(e)]), 400
+
+        diff = recipe_diff(old_blueprint, new_blueprint)
+        return jsonify(diff=diff)
+
+    @api.route("/api/v0/blueprints/freeze/<blueprint_names>")
+    @crossdomain(origin="*")
+    def v0_blueprints_freeze(blueprint_names):
+        """Return the blueprint with the exact modules and packages selected by depsolve"""
+        branch = request.args.get("branch", "master")
+        out_fmt = request.args.get("format", "json")
+        blueprints = []
+        errors = []
+        for blueprint_name in [n.strip() for n in sorted(blueprint_names.split(","), key=lambda n: n.lower())]:
+            # get the blueprint
+            # Get the workspace version (if it exists)
+            blueprint = None
+            try:
+                with api.config["GITLOCK"].lock:
+                    blueprint = workspace_read(api.config["GITLOCK"].repo, branch, blueprint_name)
+            except Exception:
+                pass
+
+            if not blueprint:
+                # No workspace version, get the git version (if it exists)
+                try:
+                    with api.config["GITLOCK"].lock:
+                        blueprint = read_recipe_commit(api.config["GITLOCK"].repo, branch, blueprint_name)
+                except Exception as e:
+                    errors.append("%s: %s" % (blueprint_name, str(e)))
+                    log.error("(v0_blueprints_freeze) %s", str(e))
+
+            # No blueprint found, skip it.
+            if not blueprint:
+                errors.append("%s: blueprint_not_found" % (blueprint_name))
+                continue
+
+            # Combine modules and packages and depsolve the list
+            # TODO include the version/glob in the depsolving
+            module_names = blueprint.module_names
+            package_names = blueprint.package_names
+            projects = sorted(set(module_names+package_names), key=lambda n: n.lower())
+            deps = []
+            try:
+                with api.config["DNFLOCK"].lock:
+                    deps = projects_depsolve(api.config["DNFLOCK"].dbo, projects)
+            except ProjectsError as e:
+                errors.append("%s: %s" % (blueprint_name, str(e)))
+                log.error("(v0_blueprints_freeze) %s", str(e))
+
+            blueprints.append({"blueprint": blueprint.freeze(deps)})
+
+        if out_fmt == "toml":
+            # With TOML output we just want to dump the raw blueprint, skipping the rest.
+            return "\n\n".join([e["blueprint"].toml() for e in blueprints])
+        else:
+            return jsonify(blueprints=blueprints, errors=errors)
+
+    @api.route("/api/v0/blueprints/depsolve/<blueprint_names>")
+    @crossdomain(origin="*")
+    def v0_blueprints_depsolve(blueprint_names):
+        """Return the dependencies for a blueprint"""
+        branch = request.args.get("branch", "master")
+        blueprints = []
+        errors = []
+        for blueprint_name in [n.strip() for n in sorted(blueprint_names.split(","), key=lambda n: n.lower())]:
+            # get the blueprint
+            # Get the workspace version (if it exists)
+            blueprint = None
+            try:
+                with api.config["GITLOCK"].lock:
+                    blueprint = workspace_read(api.config["GITLOCK"].repo, branch, blueprint_name)
+            except Exception:
+                pass
+
+            if not blueprint:
+                # No workspace version, get the git version (if it exists)
+                try:
+                    with api.config["GITLOCK"].lock:
+                        blueprint = read_recipe_commit(api.config["GITLOCK"].repo, branch, blueprint_name)
+                except Exception as e:
+                    errors.append("%s: %s" % (blueprint_name, str(e)))
+                    log.error("(v0_blueprints_depsolve) %s", str(e))
+
+            # No blueprint found, skip it.
+            if not blueprint:
+                errors.append("%s: blueprint not found" % blueprint_name)
+                continue
+
+            # Combine modules and packages and depsolve the list
+            # TODO include the version/glob in the depsolving
+            module_names = [m["name"] for m in blueprint["modules"] or []]
+            package_names = [p["name"] for p in blueprint["packages"] or []]
+            projects = sorted(set(module_names+package_names), key=lambda n: n.lower())
+            deps = []
+            try:
+                with api.config["DNFLOCK"].lock:
+                    deps = projects_depsolve(api.config["DNFLOCK"].dbo, projects)
+            except ProjectsError as e:
+                errors.append("%s: %s" % (blueprint_name, str(e)))
+                log.error("(v0_blueprints_depsolve) %s", str(e))
+
+            # Get the NEVRA's of the modules and projects, add as "modules"
+            modules = []
+            for dep in deps:
+                if dep["name"] in projects:
+                    modules.append(dep)
+            modules = sorted(modules, key=lambda m: m["name"].lower())
+
+            blueprints.append({"blueprint":blueprint, "dependencies":deps, "modules":modules})
+
+        return jsonify(blueprints=blueprints, errors=errors)
+
+    @api.route("/api/v0/projects/list")
+    @crossdomain(origin="*")
+    def v0_projects_list():
+        """List all of the available projects/packages"""
+        try:
+            limit = int(request.args.get("limit", "20"))
+            offset = int(request.args.get("offset", "0"))
+        except ValueError as e:
+            return jsonify(status=False, errors=[str(e)]), 400
+
+        try:
+            with api.config["DNFLOCK"].lock:
+                available = projects_list(api.config["DNFLOCK"].dbo)
+        except ProjectsError as e:
+            log.error("(v0_projects_list) %s", str(e))
+            return jsonify(status=False, errors=[str(e)]), 400
+
+        projects = take_limits(available, offset, limit)
+        return jsonify(projects=projects, offset=offset, limit=limit, total=len(available))
+
+    @api.route("/api/v0/projects/info/<project_names>")
+    @crossdomain(origin="*")
+    def v0_projects_info(project_names):
+        """Return detailed information about the listed projects"""
+        try:
+            with api.config["DNFLOCK"].lock:
+                projects = projects_info(api.config["DNFLOCK"].dbo, project_names.split(","))
+        except ProjectsError as e:
+            log.error("(v0_projects_info) %s", str(e))
+            return jsonify(status=False, errors=[str(e)]), 400
+
+        return jsonify(projects=projects)
+
+    @api.route("/api/v0/projects/depsolve/<project_names>")
+    @crossdomain(origin="*")
+    def v0_projects_depsolve(project_names):
+        """Return detailed information about the listed projects"""
+        try:
+            with api.config["DNFLOCK"].lock:
+                deps = projects_depsolve(api.config["DNFLOCK"].dbo, project_names.split(","))
+        except ProjectsError as e:
+            log.error("(v0_projects_depsolve) %s", str(e))
+            return jsonify(status=False, errors=[str(e)]), 400
+
+        return jsonify(projects=deps)
+
+    @api.route("/api/v0/modules/list")
+    @api.route("/api/v0/modules/list/<module_names>")
+    @crossdomain(origin="*")
+    def v0_modules_list(module_names=None):
+        """List available modules, filtering by module_names"""
+        try:
+            limit = int(request.args.get("limit", "20"))
+            offset = int(request.args.get("offset", "0"))
+        except ValueError as e:
+            return jsonify(status=False, errors=[str(e)]), 400
+
+        if module_names:
+            module_names = module_names.split(",")
+
+        try:
+            with api.config["DNFLOCK"].lock:
+                available = modules_list(api.config["DNFLOCK"].dbo, module_names)
+        except ProjectsError as e:
+            log.error("(v0_modules_list) %s", str(e))
+            return jsonify(status=False, errors=[str(e)]), 400
+
+        modules = take_limits(available, offset, limit)
+        return jsonify(modules=modules, offset=offset, limit=limit, total=len(available))
+
+    @api.route("/api/v0/modules/info/<module_names>")
+    @crossdomain(origin="*")
+    def v0_modules_info(module_names):
+        """Return detailed information about the listed modules"""
+        try:
+            with api.config["DNFLOCK"].lock:
+                modules = modules_info(api.config["DNFLOCK"].dbo, module_names.split(","))
+        except ProjectsError as e:
+            log.error("(v0_modules_info) %s", str(e))
+            return jsonify(status=False, errors=[str(e)]), 400
+
+        return jsonify(modules=modules)
+
+    @api.route("/api/v0/compose", methods=["POST"])
+    @crossdomain(origin="*")
+    def v0_compose_start():
+        """Start a compose
+
+        The body of the post should have these fields:
+          blueprint_name - The blueprint name from /blueprints/list/
+          compose_type   - The type of output to create, from /compose/types
+          branch         - Optional, defaults to master, selects the git branch to use for the blueprint.
+        """
+        # Passing ?test=1 will generate a fake FAILED compose.
+        # Passing ?test=2 will generate a fake FINISHED compose.
+        try:
+            test_mode = int(request.args.get("test", "0"))
+        except ValueError:
+            test_mode = 0
+
+        compose = request.get_json(cache=False)
+
+        errors = []
+        if not compose:
+            return jsonify(status=False, errors=["Missing POST body"]), 400
+
+        if "blueprint_name" not in compose:
+            errors.append("No 'blueprint_name' in the JSON request")
+        else:
+            blueprint_name = compose["blueprint_name"]
+
+        if "branch" not in compose or not compose["branch"]:
+            branch = "master"
+        else:
+            branch = compose["branch"]
+
+        if "compose_type" not in compose:
+            errors.append("No 'compose_type' in the JSON request")
+        else:
+            compose_type = compose["compose_type"]
+
+        if errors:
+            return jsonify(status=False, errors=errors), 400
+
+        try:
+            build_id = start_build(api.config["COMPOSER_CFG"], api.config["DNFLOCK"], api.config["GITLOCK"],
+                                   branch, blueprint_name, compose_type, test_mode)
+        except Exception as e:
+            return jsonify(status=False, errors=[str(e)]), 400
+
+        return jsonify(status=True, build_id=build_id)
+
+    @api.route("/api/v0/compose/types")
+    @crossdomain(origin="*")
+    def v0_compose_types():
+        """Return the list of enabled output types
+
+        (only enabled types are returned)
+        """
+        share_dir = api.config["COMPOSER_CFG"].get("composer", "share_dir")
+        return jsonify(types=[{"name": k, "enabled": True} for k in compose_types(share_dir)])
+
+    @api.route("/api/v0/compose/queue")
+    @crossdomain(origin="*")
+    def v0_compose_queue():
+        """Return the status of the new and running queues"""
+        return jsonify(queue_status(api.config["COMPOSER_CFG"]))
+
+    @api.route("/api/v0/compose/finished")
+    @crossdomain(origin="*")
+    def v0_compose_finished():
+        """Return the list of finished composes"""
+        return jsonify(finished=build_status(api.config["COMPOSER_CFG"], "FINISHED"))
+
+    @api.route("/api/v0/compose/failed")
+    @crossdomain(origin="*")
+    def v0_compose_failed():
+        """Return the list of failed composes"""
+        return jsonify(failed=build_status(api.config["COMPOSER_CFG"], "FAILED"))
+
+    @api.route("/api/v0/compose/status/<uuids>")
+    @crossdomain(origin="*")
+    def v0_compose_status(uuids):
+        """Return the status of the listed uuids"""
+        results = []
+        for uuid in [n.strip().lower() for n in uuids.split(",")]:
+            details = uuid_status(api.config["COMPOSER_CFG"], uuid)
+            if details is not None:
+                results.append(details)
+
+        return jsonify(uuids=results)
+
+    @api.route("/api/v0/compose/cancel/<uuid>", methods=["DELETE"])
+    @crossdomain(origin="*")
+    def v0_compose_cancel(uuid):
+        """Cancel a running compose and delete its results directory"""
+        status = uuid_status(api.config["COMPOSER_CFG"], uuid)
+        if status is None:
+            return jsonify(status=False, errors=["%s is not a valid build uuid" % uuid]), 400
+
+        if status["queue_status"] not in ["WAITING", "RUNNING"]:
+            return jsonify(status=False, errors=["Build %s is not in WAITING or RUNNING." % uuid])
+
+        try:
+            uuid_cancel(api.config["COMPOSER_CFG"], uuid)
+        except Exception as e:
+            return jsonify(status=False, errors=["%s: %s" % (uuid, str(e))]), 400
+        else:
+            return jsonify(status=True, uuid=uuid)
+
+    @api.route("/api/v0/compose/delete/<uuids>", methods=["DELETE"])
+    @crossdomain(origin="*")
+    def v0_compose_delete(uuids):
+        """Delete the compose results for the listed uuids"""
+        results = []
+        errors = []
+        for uuid in [n.strip().lower() for n in uuids.split(",")]:
+            status = uuid_status(api.config["COMPOSER_CFG"], uuid)
+            if status is None:
+                errors.append("%s is not a valid build uuid" % uuid)
+            elif status["queue_status"] not in ["FINISHED", "FAILED"]:
+                errors.append("Build %s is not in FINISHED or FAILED." % uuid)
+            else:
+                try:
+                    uuid_delete(api.config["COMPOSER_CFG"], uuid)
+                except Exception as e:
+                    errors.append("%s: %s" % (uuid, str(e)))
+                else:
+                    results.append({"uuid":uuid, "status":True})
+        return jsonify(uuids=results, errors=errors)
+
+    @api.route("/api/v0/compose/info/<uuid>")
+    @crossdomain(origin="*")
+    def v0_compose_info(uuid):
+        """Return detailed info about a compose"""
+        try:
+            info = uuid_info(api.config["COMPOSER_CFG"], uuid)
+        except Exception as e:
+            return jsonify(status=False, errors=[str(e)]), 400
+
+        return jsonify(**info)
+
+    @api.route("/api/v0/compose/metadata/<uuid>")
+    @crossdomain(origin="*")
+    def v0_compose_metadata(uuid):
+        """Return a tar of the metadata for the build"""
+        status = uuid_status(api.config["COMPOSER_CFG"], uuid)
+        if status is None:
+            return jsonify(status=False, errors=["%s is not a valid build uuid" % uuid]), 400
+        if status["queue_status"] not in ["FINISHED", "FAILED"]:
+            return jsonify(status=False, errors=["Build %s not in FINISHED or FAILED state." % uuid]), 400
+        else:
+            return Response(uuid_tar(api.config["COMPOSER_CFG"], uuid, metadata=True, image=False, logs=False),
+                            mimetype="application/x-tar",
+                            headers=[("Content-Disposition", "attachment; filename=%s-metadata.tar;" % uuid)],
+                            direct_passthrough=True)
+
+    @api.route("/api/v0/compose/results/<uuid>")
+    @crossdomain(origin="*")
+    def v0_compose_results(uuid):
+        """Return a tar of the metadata and the results for the build"""
+        status = uuid_status(api.config["COMPOSER_CFG"], uuid)
+        if status is None:
+            return jsonify(status=False, errors=["%s is not a valid build uuid" % uuid]), 400
+        elif status["queue_status"] not in ["FINISHED", "FAILED"]:
+            return jsonify(status=False, errors=["Build %s not in FINISHED or FAILED state." % uuid]), 400
+        else:
+            return Response(uuid_tar(api.config["COMPOSER_CFG"], uuid, metadata=True, image=True, logs=True),
+                            mimetype="application/x-tar",
+                            headers=[("Content-Disposition", "attachment; filename=%s.tar;" % uuid)],
+                            direct_passthrough=True)
+
+    @api.route("/api/v0/compose/logs/<uuid>")
+    @crossdomain(origin="*")
+    def v0_compose_logs(uuid):
+        """Return a tar of the metadata for the build"""
+        status = uuid_status(api.config["COMPOSER_CFG"], uuid)
+        if status is None:
+            return jsonify(status=False, errors=["%s is not a valid build uuid" % uuid]), 400
+        elif status["queue_status"] not in ["FINISHED", "FAILED"]:
+            return jsonify(status=False, errors=["Build %s not in FINISHED or FAILED state." % uuid]), 400
+        else:
+            return Response(uuid_tar(api.config["COMPOSER_CFG"], uuid, metadata=False, image=False, logs=True),
+                            mimetype="application/x-tar",
+                            headers=[("Content-Disposition", "attachment; filename=%s-logs.tar;" % uuid)],
+                            direct_passthrough=True)
+
+    @api.route("/api/v0/compose/image/<uuid>")
+    @crossdomain(origin="*")
+    def v0_compose_image(uuid):
+        """Return the output image for the build"""
+        status = uuid_status(api.config["COMPOSER_CFG"], uuid)
+        if status is None:
+            return jsonify(status=False, errors=["%s is not a valid build uuid" % uuid]), 400
+        elif status["queue_status"] not in ["FINISHED", "FAILED"]:
+            return jsonify(status=False, errors=["Build %s not in FINISHED or FAILED state." % uuid]), 400
+        else:
+            image_name, image_path = uuid_image(api.config["COMPOSER_CFG"], uuid)
+
+            # Make sure it really exists
+            if not os.path.exists(image_path):
+                return jsonify(status=False, errors=["Build %s is missing image file %s" % (uuid, image_name)]), 400
+
+            # Make the image name unique
+            image_name = uuid + "-" + image_name
+            # XXX - Will mime type guessing work for all our output?
+            return send_file(image_path, as_attachment=True, attachment_filename=image_name, add_etags=False)
+
+    @api.route("/api/v0/compose/log/<uuid>")
+    @crossdomain(origin="*")
+    def v0_compose_log_tail(uuid):
+        """Return the end of the main anaconda.log, defaults to 1Mbytes"""
+        try:
+            size = int(request.args.get("size", "1024"))
+        except ValueError as e:
+            return jsonify(status=False, errors=[str(e)]), 400
+
+        status = uuid_status(api.config["COMPOSER_CFG"], uuid)
+        if status is None:
+            return jsonify(status=False, errors=["%s is not a valid build uuid" % uuid]), 400
+        elif status["queue_status"] == "WAITING":
+            return jsonify(status=False, errors=["Build %s has not started yet. No logs to view" % uuid])
+        try:
+            return Response(uuid_log(api.config["COMPOSER_CFG"], uuid, size), direct_passthrough=True)
+        except RuntimeError as e:
+            return jsonify(status=False, errors=[str(e)]), 400
    
+
    + +
    + +
    + + +
    +
    + +
    + +
    + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/docs/html/_modules/pylorax/api/workspace.html b/docs/html/_modules/pylorax/api/workspace.html new file mode 100644 index 00000000..b2aa0d0a --- /dev/null +++ b/docs/html/_modules/pylorax/api/workspace.html @@ -0,0 +1,319 @@ + + + + + + + + + + + pylorax.api.workspace — Lorax 29.1 documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    + + + + +
    + + + + + +
    + +
    + + + + + + + + + + + + + + + + + +
    + + + + +
    +
    +
    +
    + +

    Source code for pylorax.api.workspace

    +#
+# Copyright (C) 2017  Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+import os
+
+from pylorax.api.recipes import recipe_filename, recipe_from_toml, RecipeFileError
+from pylorax.sysutils import joinpaths
+
+
+
    [docs]def workspace_dir(repo, branch):
+    """Create the workspace's path from a Repository and branch
+
+    :param repo: Open repository
+    :type repo: Git.Repository
+    :param branch: Branch name
+    :type branch: str
+    :returns: The path to the branch's workspace directory
+    :rtype: str
+
+    """
+    repo_path = repo.get_location().get_path()
+    return joinpaths(repo_path, "workspace", branch)
    
+
+
+
    [docs]def workspace_read(repo, branch, recipe_name):
+    """Read a Recipe from the branch's workspace
+
+    :param repo: Open repository
+    :type repo: Git.Repository
+    :param branch: Branch name
+    :type branch: str
+    :param recipe_name: The name of the recipe
+    :type recipe_name: str
+    :returns: The workspace copy of the recipe, or None if it doesn't exist
+    :rtype: Recipe or None
+    :raises: RecipeFileError
+    """
+    ws_dir = workspace_dir(repo, branch)
+    if not os.path.isdir(ws_dir):
+        os.makedirs(ws_dir)
+    filename = joinpaths(ws_dir, recipe_filename(recipe_name))
+    if not os.path.exists(filename):
+        return None
+    try:
+        f = open(filename, 'rb')
+        recipe = recipe_from_toml(f.read().decode("UTF-8"))
+    except IOError:
+        raise RecipeFileError
+    return recipe
    
+
+
+
    [docs]def workspace_write(repo, branch, recipe):
+    """Write a recipe to the workspace
+
+    :param repo: Open repository
+    :type repo: Git.Repository
+    :param branch: Branch name
+    :type branch: str
+    :param recipe: The recipe to write to the workspace
+    :type recipe: Recipe
+    :returns: None
+    :raises: IO related errors
+    """
+    ws_dir = workspace_dir(repo, branch)
+    if not os.path.isdir(ws_dir):
+        os.makedirs(ws_dir)
+    filename = joinpaths(ws_dir, recipe.filename)
+    open(filename, 'wb').write(recipe.toml().encode("UTF-8"))
    
+
+
+
    [docs]def workspace_delete(repo, branch, recipe_name):
+    """Delete the recipe from the workspace
+
+    :param repo: Open repository
+    :type repo: Git.Repository
+    :param branch: Branch name
+    :type branch: str
+    :param recipe_name: The name of the recipe
+    :type recipe_name: str
+    :returns: None
+    :raises: IO related errors
+    """
+    ws_dir = workspace_dir(repo, branch)
+    filename = joinpaths(ws_dir, recipe_filename(recipe_name))
+    if os.path.exists(filename):
+        os.unlink(filename)
    
+
    + +
    + +
    + + +
    +
    + +
    + +
    + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/docs/html/_modules/pylorax/base.html b/docs/html/_modules/pylorax/base.html index a886ebec..8649c3b4 100644 --- a/docs/html/_modules/pylorax/base.html +++ b/docs/html/_modules/pylorax/base.html @@ -8,7 +8,7 @@ - pylorax.base — Lorax 29.0 documentation + pylorax.base — Lorax 29.1 documentation @@ -25,25 +25,17 @@ - - - - - - - - - - + + + - +
    @@ -65,7 +57,7 @@
    - 29.0 + 29.1
    @@ -94,8 +86,10 @@
  • Before Lorax
  • Lorax
  • livemedia-creator
    • +
  • lorax-composer
    • +
  • composer-cli
  • Product and Updates Images
    • -
  • pylorax
    • +
  • src
    • @@ -107,7 +101,7 @@
    -
    @@ -265,7 +256,8 @@ - - - + \ No newline at end of file diff --git a/docs/html/_modules/pylorax/buildstamp.html b/docs/html/_modules/pylorax/buildstamp.html index 3e6f7504..3a73a231 100644 --- a/docs/html/_modules/pylorax/buildstamp.html +++ b/docs/html/_modules/pylorax/buildstamp.html @@ -8,7 +8,7 @@ - pylorax.buildstamp — Lorax 29.0 documentation + pylorax.buildstamp — Lorax 29.1 documentation @@ -25,25 +25,17 @@ - - - - - - - - - - + + + - +
    @@ -65,7 +57,7 @@
    - 29.0 + 29.1
    @@ -94,8 +86,10 @@
  • Before Lorax
  • Lorax
  • livemedia-creator
    • +
  • lorax-composer
    • +
  • composer-cli
  • Product and Updates Images
    • -
  • pylorax
    • +
  • src
    • @@ -107,7 +101,7 @@
    -
    @@ -259,7 +250,8 @@ - - - + \ No newline at end of file diff --git a/docs/html/_modules/pylorax/cmdline.html b/docs/html/_modules/pylorax/cmdline.html index e1318ad9..ae241131 100644 --- a/docs/html/_modules/pylorax/cmdline.html +++ b/docs/html/_modules/pylorax/cmdline.html @@ -8,7 +8,7 @@ - pylorax.cmdline — Lorax 29.0 documentation + pylorax.cmdline — Lorax 29.1 documentation @@ -25,25 +25,17 @@ - - - - - - - - - - + + + - +
    @@ -65,7 +57,7 @@
    - 29.0 + 29.1
    @@ -94,8 +86,10 @@
  • Before Lorax
  • Lorax
  • livemedia-creator
    • +
  • lorax-composer
    • +
  • composer-cli
  • Product and Updates Images
    • -
  • pylorax
    • +
  • src
    • @@ -107,7 +101,7 @@
    -
    @@ -492,7 +483,8 @@ - - - + \ No newline at end of file diff --git a/docs/html/_modules/pylorax/creator.html b/docs/html/_modules/pylorax/creator.html new file mode 100644 index 00000000..b041d2da --- /dev/null +++ b/docs/html/_modules/pylorax/creator.html @@ -0,0 +1,934 @@ + + + + + + + + + + + pylorax.creator — Lorax 29.1 documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    + + + + +
    + + + + + +
    + +
    + + + + + + + + + + + + + + + + + +
    + + + + +
    +
    +
    +
    + +

    Source code for pylorax.creator

    +#
+# Copyright (C) 2011-2018  Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+import logging
+log = logging.getLogger("pylorax")
+
+import os
+import tempfile
+import subprocess
+import shutil
+import hashlib
+import glob
+
+# Use Mako templates for appliance builder descriptions
+from mako.template import Template
+from mako.exceptions import text_error_template
+
+# Use pykickstart to calculate disk image size
+from pykickstart.parser import KickstartParser
+from pykickstart.constants import KS_SHUTDOWN
+from pykickstart.version import makeVersion
+
+# Use the Lorax treebuilder branch for iso creation
+from pylorax import ArchData
+from pylorax.base import DataHolder
+from pylorax.executils import execWithRedirect, runcmd
+from pylorax.imgutils import PartitionMount
+from pylorax.imgutils import mount, umount, Mount
+from pylorax.imgutils import mksquashfs, mkrootfsimg
+from pylorax.imgutils import copytree
+from pylorax.installer import novirt_install, virt_install, InstallError
+from pylorax.treebuilder import TreeBuilder, RuntimeBuilder
+from pylorax.treebuilder import findkernels
+from pylorax.sysutils import joinpaths, remove
+
+
+# Default parameters for rebuilding initramfs, override with --dracut-args
+DRACUT_DEFAULT = ["--xz", "--add", "livenet dmsquash-live convertfs pollcdrom qemu qemu-net",
+                  "--omit", "plymouth", "--no-hostonly", "--debug", "--no-early-microcode"]
+
+RUNTIME = "images/install.img"
+
+
    [docs]class FakeDNF(object):
+    """
+    A minimal DNF object suitable for passing to RuntimeBuilder
+
+    lmc uses RuntimeBuilder to run the arch specific iso creation
+    templates, so the the installroot config value is the important part of
+    this. Everything else should be a nop.
+    """
+    def __init__(self, conf):
+        self.conf = conf
+
+
    [docs]    def reset(self):
+        pass
    
+
+
    [docs]def is_image_mounted(disk_img):
+    """
+    Check to see if the disk_img is mounted
+
+    :returns: True if disk_img is in /proc/mounts
+    :rtype: bool
+    """
+    with open("/proc/mounts") as mounts:
+        for mnt in mounts:
+            fields = mnt.split()
+            if len(fields) > 2 and fields[1] == disk_img:
+                return True
+    return False
    
+
+
    [docs]def find_ostree_root(phys_root):
+    """
+    Find root of ostree deployment
+
+    :param str phys_root: Path to physical root
+    :returns: Relative path of ostree deployment root
+    :rtype: str
+    :raise Exception: More than one deployment roots were found
+    """
+    ostree_root = ""
+    ostree_sysroots = glob.glob(joinpaths(phys_root, "ostree/boot.?/*/*/0"))
+    log.debug("ostree_sysroots = %s", ostree_sysroots)
+    if ostree_sysroots:
+        if len(ostree_sysroots) > 1:
+            raise Exception("Too many deployment roots found: %s" % ostree_sysroots)
+        ostree_root = os.path.relpath(ostree_sysroots[0], phys_root)
+    return ostree_root
    
+
+
    [docs]def get_arch(mount_dir):
+    """
+    Get the kernel arch
+
+    :returns: Arch of first kernel found at mount_dir/boot/ or i386
+    :rtype: str
+    """
+    kernels = findkernels(mount_dir)
+    if not kernels:
+        return "i386"
+    return kernels[0].arch
    
+
+
    [docs]def squashfs_args(opts):
+    """ Returns the compression type and args to use when making squashfs
+
+    :param opts: ArgumentParser object with compression and compressopts
+    :returns: tuple of compression type and args
+    :rtype: tuple
+    """
+    compression = opts.compression or "xz"
+    arch = ArchData(opts.arch or os.uname().machine)
+    if compression == "xz" and arch.bcj:
+        compressargs = ["-Xbcj", arch.bcj]
+    else:
+        compressargs = []
+    return (compression, compressargs)
    
+
+
+
    [docs]def make_appliance(disk_img, name, template, outfile, networks=None, ram=1024,
+                   vcpus=1, arch=None, title="Linux", project="Linux",
+                   releasever="29"):
+    """
+    Generate an appliance description file
+
+    :param str disk_img: Full path of the disk image
+    :param str name: Name of the appliance, passed to the template
+    :param str template: Full path of Mako template
+    :param str outfile: Full path of file to write, using template
+    :param list networks: List of networks(str) from the kickstart
+    :param int ram: Ram, in MiB, passed to template. Default is 1024
+    :param int vcpus: CPUs, passed to template. Default is 1
+    :param str arch: CPU architecture. Default is 'x86_64'
+    :param str title: Title, passed to template. Default is 'Linux'
+    :param str project: Project, passed to template. Default is 'Linux'
+    :param str releasever: Release version, passed to template. Default is 29
+    """
+    if not (disk_img and template and outfile):
+        return None
+
+    log.info("Creating appliance definition using %s", template)
+
+    if not arch:
+        arch = "x86_64"
+
+    log.info("Calculating SHA256 checksum of %s", disk_img)
+    sha256 = hashlib.sha256()
+    with open(disk_img) as f:
+        while True:
+            data = f.read(1024**2)
+            if not data:
+                break
+            sha256.update(data)
+    log.info("SHA256 of %s is %s", disk_img, sha256.hexdigest())
+    disk_info = DataHolder(name=os.path.basename(disk_img), format="raw",
+                           checksum_type="sha256", checksum=sha256.hexdigest())
+    try:
+        result = Template(filename=template).render(disks=[disk_info], name=name,
+                          arch=arch, memory=ram, vcpus=vcpus, networks=networks,
+                          title=title, project=project, releasever=releasever)
+    except Exception:
+        log.error(text_error_template().render())
+        raise
+
+    with open(outfile, "w") as f:
+        f.write(result)
    
+
+
+
    [docs]def make_runtime(opts, mount_dir, work_dir, size=None):
+    """
+    Make the squashfs image from a directory
+
+    :param opts: options passed to livemedia-creator
+    :type opts: argparse options
+    :param str mount_dir: Directory tree to compress
+    :param str work_dir: Output compressed image to work_dir+images/install.img
+    :param int size: Size of disk image, in GiB
+    """
+    kernel_arch = get_arch(mount_dir)
+
+    # Fake dnf  object
+    fake_dbo = FakeDNF(conf=DataHolder(installroot=mount_dir))
+    # Fake arch with only basearch set
+    arch = ArchData(kernel_arch)
+    # TODO: Need to get release info from someplace...
+    product = DataHolder(name=opts.project, version=opts.releasever, release="",
+                            variant="", bugurl="", isfinal=False)
+
+    # This is a mounted image partition, cannot hardlink to it, so just use it
+    # symlink mount_dir/images to work_dir/images so we don't run out of space
+    os.makedirs(joinpaths(work_dir, "images"))
+
+    rb = RuntimeBuilder(product, arch, fake_dbo)
+    compression, compressargs = squashfs_args(opts)
+    log.info("Creating runtime")
+    rb.create_runtime(joinpaths(work_dir, RUNTIME), size=size,
+                      compression=compression, compressargs=compressargs)
    
+
+
+
    [docs]def rebuild_initrds_for_live(opts, sys_root_dir, results_dir):
+    """
+    Rebuild intrds for pxe live image (root=live:http://)
+
+    :param opts: options passed to livemedia-creator
+    :type opts: argparse options
+    :param str sys_root_dir: Path to root of the system
+    :param str results_dir: Path of directory for storing results
+    """
+    if not opts.dracut_args:
+        dracut_args = DRACUT_DEFAULT
+    else:
+        dracut_args = []
+        for arg in opts.dracut_args:
+            dracut_args += arg.split(" ", 1)
+    log.info("dracut args = %s", dracut_args)
+
+    dracut = ["dracut", "--nomdadmconf", "--nolvmconf"] + dracut_args
+
+    kdir = "boot"
+    if opts.ostree:
+        kernels_dir = glob.glob(joinpaths(sys_root_dir, "boot/ostree/*"))
+        if kernels_dir:
+            kdir = os.path.relpath(kernels_dir[0], sys_root_dir)
+
+    kernels = [kernel for kernel in findkernels(sys_root_dir, kdir)]
+    if not kernels:
+        raise Exception("No initrds found, cannot rebuild_initrds")
+
+    # Hush some dracut warnings. TODO: bind-mount proc in place?
+    open(joinpaths(sys_root_dir,"/proc/modules"),"w")
+
+    if opts.ostree:
+        # Dracut assumes to have some dirs in disk image
+        # /var/tmp for temp files
+        vartmp_dir = joinpaths(sys_root_dir, "var/tmp")
+        if not os.path.isdir(vartmp_dir):
+            os.mkdir(vartmp_dir)
+        # /root (maybe not fatal)
+        root_dir = joinpaths(sys_root_dir, "var/roothome")
+        if not os.path.isdir(root_dir):
+            os.mkdir(root_dir)
+        # /tmp (maybe not fatal)
+        tmp_dir = joinpaths(sys_root_dir, "sysroot/tmp")
+        if not os.path.isdir(tmp_dir):
+            os.mkdir(tmp_dir)
+
+    # Write the new initramfs directly to the results directory
+    os.mkdir(joinpaths(sys_root_dir, "results"))
+    mount(results_dir, opts="bind", mnt=joinpaths(sys_root_dir, "results"))
+    # Dracut runs out of space inside the minimal rootfs image
+    mount("/var/tmp", opts="bind", mnt=joinpaths(sys_root_dir, "var/tmp"))
+    for kernel in kernels:
+        if hasattr(kernel, "initrd"):
+            outfile = os.path.basename(kernel.initrd.path)
+        else:
+            # Construct an initrd from the kernel name
+            outfile = os.path.basename(kernel.path.replace("vmlinuz-", "initrd-") + ".img")
+        log.info("rebuilding %s", outfile)
+
+        kver = kernel.version
+
+        cmd = dracut + ["/results/"+outfile, kver]
+        runcmd(cmd, root=sys_root_dir)
+
+        shutil.copy2(joinpaths(sys_root_dir, kernel.path), results_dir)
+    umount(joinpaths(sys_root_dir, "var/tmp"), delete=False)
+    umount(joinpaths(sys_root_dir, "results"), delete=False)
+    os.unlink(joinpaths(sys_root_dir,"/proc/modules"))
    
+
+
    [docs]def create_pxe_config(template, images_dir, live_image_name, add_args = None):
+    """
+    Create template for pxe to live configuration
+
+    :param str images_dir: Path of directory with images to be used
+    :param str live_image_name: Name of live rootfs image file
+    :param list add_args: Arguments to be added to initrd= pxe config
+    """
+
+    add_args = add_args or []
+
+    kernels = [kernel for kernel in findkernels(images_dir, kdir="")
+               if hasattr(kernel, "initrd")]
+    if not kernels:
+        return
+
+    kernel = kernels[0]
+
+    add_args_str = " ".join(add_args)
+
+
+    try:
+        result = Template(filename=template).render(kernel=kernel.path,
+                          initrd=kernel.initrd.path, liveimg=live_image_name,
+                          addargs=add_args_str)
+    except Exception:
+        log.error(text_error_template().render())
+        raise
+
+    with open (joinpaths(images_dir, "PXE_CONFIG"), "w") as f:
+        f.write(result)
    
+
+
+
    [docs]def make_livecd(opts, mount_dir, work_dir):
+    """
+    Take the content from the disk image and make a livecd out of it
+
+    :param opts: options passed to livemedia-creator
+    :type opts: argparse options
+    :param str mount_dir: Directory tree to compress
+    :param str work_dir: Output compressed image to work_dir+images/install.img
+
+    This uses wwood's squashfs live initramfs method:
+     * put the real / into LiveOS/rootfs.img
+     * make a squashfs of the LiveOS/rootfs.img tree
+     * This is loaded by dracut when the cmdline is passed to the kernel:
+       root=live:CDLABEL=<volid> rd.live.image
+    """
+    kernel_arch = get_arch(mount_dir)
+
+    arch = ArchData(kernel_arch)
+    # TODO: Need to get release info from someplace...
+    product = DataHolder(name=opts.project, version=opts.releasever, release="",
+                            variant="", bugurl="", isfinal=False)
+
+    # Link /images to work_dir/images to make the templates happy
+    if os.path.islink(joinpaths(mount_dir, "images")):
+        os.unlink(joinpaths(mount_dir, "images"))
+    execWithRedirect("/bin/ln", ["-s", joinpaths(work_dir, "images"),
+                                 joinpaths(mount_dir, "images")])
+
+    # The templates expect the config files to be in /tmp/config_files
+    # I think these should be release specific, not from lorax, but for now
+    configdir = joinpaths(opts.lorax_templates,"live/config_files/")
+    configdir_path = "tmp/config_files"
+    fullpath = joinpaths(mount_dir, configdir_path)
+    if os.path.exists(fullpath):
+        remove(fullpath)
+    copytree(configdir, fullpath)
+
+    isolabel = opts.volid or "{0.name}-{0.version}-{1.basearch}".format(product, arch)
+    if len(isolabel) > 32:
+        isolabel = isolabel[:32]
+        log.warning("Truncating isolabel to 32 chars: %s", isolabel)
+
+    tb = TreeBuilder(product=product, arch=arch, domacboot=opts.domacboot,
+                     inroot=mount_dir, outroot=work_dir,
+                     runtime=RUNTIME, isolabel=isolabel,
+                     templatedir=joinpaths(opts.lorax_templates,"live/"))
+    log.info("Rebuilding initrds")
+    if not opts.dracut_args:
+        dracut_args = DRACUT_DEFAULT
+    else:
+        dracut_args = []
+        for arg in opts.dracut_args:
+            dracut_args += arg.split(" ", 1)
+    log.info("dracut args = %s", dracut_args)
+    tb.rebuild_initrds(add_args=dracut_args)
+    log.info("Building boot.iso")
+    tb.build()
+
+    return work_dir
    
+
+
    [docs]def mount_boot_part_over_root(img_mount):
+    """
+    Mount boot partition to /boot of root fs mounted in img_mount
+
+    Used for OSTree so it finds deployment configurations on live rootfs
+
+    param img_mount: object with mounted disk image root partition
+    type img_mount: imgutils.PartitionMount
+    """
+    root_dir = img_mount.mount_dir
+    is_boot_part = lambda dir: os.path.exists(dir+"/loader.0")
+    tmp_mount_dir = tempfile.mkdtemp(prefix="lmc-tmpdir-")
+    sysroot_boot_dir = None
+    for dev, _size in img_mount.loop_devices:
+        if dev is img_mount.mount_dev:
+            continue
+        try:
+            mount("/dev/mapper/"+dev, mnt=tmp_mount_dir)
+            if is_boot_part(tmp_mount_dir):
+                umount(tmp_mount_dir)
+                sysroot_boot_dir = joinpaths(root_dir, "boot")
+                mount("/dev/mapper/"+dev, mnt=sysroot_boot_dir)
+                break
+            else:
+                umount(tmp_mount_dir)
+        except subprocess.CalledProcessError as e:
+            log.debug("Looking for boot partition error: %s", e)
+    remove(tmp_mount_dir)
+    return sysroot_boot_dir
    
+
+
    [docs]def make_squashfs(opts, disk_img, work_dir):
+    """
+    Create a squashfs image of an unpartitioned filesystem disk image
+
+    :param str disk_img: Path to the unpartitioned filesystem disk image
+    :param str work_dir: Output compressed image to work_dir+images/install.img
+    :param str compression: Compression type to use
+    :returns: True if squashfs creation was successful. False if there was an error.
+    :rtype: bool
+
+    Take disk_img and put it into LiveOS/rootfs.img and squashfs this
+    tree into work_dir+images/install.img
+
+    fsck.ext4 is run on the disk image to make sure there are no errors and to zero
+    out any deleted blocks to make it compress better. If this fails for any reason
+    it will return False and log the error.
+    """
+    # Make sure free blocks are actually zeroed so it will compress
+    rc = execWithRedirect("/usr/sbin/fsck.ext4", ["-y", "-f", "-E", "discard", disk_img])
+    if rc != 0:
+        log.error("Problem zeroing free blocks of %s", disk_img)
+        return False
+
+    liveos_dir = joinpaths(work_dir, "runtime/LiveOS")
+    os.makedirs(liveos_dir)
+    os.makedirs(os.path.dirname(joinpaths(work_dir, RUNTIME)))
+
+    rc = execWithRedirect("/bin/ln", [disk_img, joinpaths(liveos_dir, "rootfs.img")])
+    if rc != 0:
+        shutil.copy2(disk_img, joinpaths(liveos_dir, "rootfs.img"))
+
+    compression, compressargs = squashfs_args(opts)
+    mksquashfs(joinpaths(work_dir, "runtime"),
+               joinpaths(work_dir, RUNTIME), compression, compressargs)
+    remove(joinpaths(work_dir, "runtime"))
+    return True
    
+
+
    [docs]def calculate_disk_size(opts, ks):
+    """ Calculate the disk size from the kickstart
+
+    :param opts: options passed to livemedia-creator
+    :type opts: argparse options
+    :param str ks: Path to the kickstart to use for the installation
+    :returns: Disk size in MiB
+    :rtype: int
+    """
+    # Disk size for a filesystem image should only be the size of /
+    # to prevent surprises when using the same kickstart for different installations.
+    unique_partitions = dict((p.mountpoint, p) for p in ks.handler.partition.partitions)
+    if opts.no_virt and (opts.make_iso or opts.make_fsimage):
+        disk_size = 2 + sum(p.size for p in unique_partitions.values() if p.mountpoint == "/")
+    else:
+        disk_size = 2 + sum(p.size for p in unique_partitions.values())
+    log.info("Using disk size of %sMiB", disk_size)
+    return disk_size
    
+
+
    [docs]def make_image(opts, ks):
+    """
+    Install to a disk image
+
+    :param opts: options passed to livemedia-creator
+    :type opts: argparse options
+    :param str ks: Path to the kickstart to use for the installation
+    :returns: Path of the image created
+    :rtype: str
+
+    Use qemu+boot.iso or anaconda to install to a disk image.
+    """
+    if opts.image_name:
+        disk_img = joinpaths(opts.result_dir, opts.image_name)
+    else:
+        disk_img = tempfile.mktemp(prefix="lmc-disk-", suffix=".img", dir=opts.result_dir)
+    log.info("disk_img = %s", disk_img)
+    disk_size = calculate_disk_size(opts, ks)
+    try:
+        if opts.no_virt:
+            novirt_install(opts, disk_img, disk_size)
+        else:
+            install_log = os.path.abspath(os.path.dirname(opts.logfile))+"/virt-install.log"
+            log.info("install_log = %s", install_log)
+
+            virt_install(opts, install_log, disk_img, disk_size)
+    except InstallError as e:
+        log.error("Install failed: %s", e)
+        if not opts.keep_image and os.path.exists(disk_img):
+            log.info("Removing bad disk image")
+            os.unlink(disk_img)
+        raise
+
+    log.info("Disk Image install successful")
+    return disk_img
    
+
+
+
    [docs]def make_live_images(opts, work_dir, disk_img):
+    """
+    Create live images from direcory or rootfs image
+
+    :param opts: options passed to livemedia-creator
+    :type opts: argparse options
+    :param str work_dir: Directory for storing results
+    :param str disk_img: Path to disk image (fsimage or partitioned)
+    :returns: Path of directory with created images or None
+    :rtype: str
+
+    fsck.ext4 is run on the rootfs_image to make sure there are no errors and to zero
+    out any deleted blocks to make it compress better. If this fails for any reason
+    it will return None and log the error.
+    """
+    sys_root = ""
+
+    squashfs_root_dir = joinpaths(work_dir, "squashfs_root")
+    liveos_dir = joinpaths(squashfs_root_dir, "LiveOS")
+    os.makedirs(liveos_dir)
+    rootfs_img = joinpaths(liveos_dir, "rootfs.img")
+
+    if opts.fs_image or opts.no_virt:
+        # Find the ostree root in the fsimage
+        if opts.ostree:
+            with Mount(disk_img, opts="loop") as mnt_dir:
+                sys_root = find_ostree_root(mnt_dir)
+
+        # Try to hardlink the image, if that fails, copy it
+        rc = execWithRedirect("/bin/ln", [disk_img, rootfs_img])
+        if rc != 0:
+            shutil.copy2(disk_img, rootfs_img)
+    else:
+        is_root_part = None
+        if opts.ostree:
+            is_root_part = lambda dir: os.path.exists(dir+"/ostree/deploy")
+        with PartitionMount(disk_img, mount_ok=is_root_part) as img_mount:
+            if img_mount and img_mount.mount_dir:
+                try:
+                    mounted_sysroot_boot_dir = None
+                    if opts.ostree:
+                        sys_root = find_ostree_root(img_mount.mount_dir)
+                        mounted_sysroot_boot_dir = mount_boot_part_over_root(img_mount)
+                    if opts.live_rootfs_keep_size:
+                        size = img_mount.mount_size / 1024**3
+                    else:
+                        size = opts.live_rootfs_size or None
+                    log.info("Creating live rootfs image")
+                    mkrootfsimg(img_mount.mount_dir, rootfs_img, "LiveOS", size=size, sysroot=sys_root)
+                finally:
+                    if mounted_sysroot_boot_dir:
+                        umount(mounted_sysroot_boot_dir)
+    log.debug("sys_root = %s", sys_root)
+
+    # Make sure free blocks are actually zeroed so it will compress
+    rc = execWithRedirect("/usr/sbin/fsck.ext4", ["-y", "-f", "-E", "discard", rootfs_img])
+    if rc != 0:
+        log.error("Problem zeroing free blocks of %s", disk_img)
+        return None
+
+    log.info("Packing live rootfs image")
+    add_pxe_args = []
+    live_image_name = "live-rootfs.squashfs.img"
+    compression, compressargs = squashfs_args(opts)
+    mksquashfs(squashfs_root_dir, joinpaths(work_dir, live_image_name), compression, compressargs)
+
+    log.info("Rebuilding initramfs for live")
+    with Mount(rootfs_img, opts="loop") as mnt_dir:
+        try:
+            mount(joinpaths(mnt_dir, "boot"), opts="bind", mnt=joinpaths(mnt_dir, sys_root, "boot"))
+            rebuild_initrds_for_live(opts, joinpaths(mnt_dir, sys_root), work_dir)
+        finally:
+            umount(joinpaths(mnt_dir, sys_root, "boot"), delete=False)
+
+    remove(squashfs_root_dir)
+
+    if opts.ostree:
+        add_pxe_args.append("ostree=/%s" % sys_root)
+    template = joinpaths(opts.lorax_templates, "pxe-live/pxe-config.tmpl")
+    create_pxe_config(template, work_dir, live_image_name, add_pxe_args)
+
+    return work_dir
    
+
+
    [docs]def run_creator(opts, callback_func=None):
+    """Run the image creator process
+
+    :param opts: Commandline options to control the process
+    :type opts: Either a DataHolder or ArgumentParser
+    :returns: The result directory and the disk image path.
+    :rtype: Tuple of str
+
+    This function takes the opts arguments and creates the selected output image.
+    See the cmdline --help for livemedia-creator for the possible options
+
+    (Yes, this is not ideal, but we can fix that later)
+    """
+    result_dir = None
+
+    # Parse the kickstart
+    if opts.ks:
+        ks_version = makeVersion()
+        ks = KickstartParser(ks_version, errorsAreFatal=False, missingIncludeIsFatal=False)
+        ks.readKickstart(opts.ks[0])
+
+    # live iso usually needs dracut-live so warn the user if it is missing
+    if opts.ks and opts.make_iso:
+        if "dracut-live" not in ks.handler.packages.packageList:
+            log.error("dracut-live package is missing from the kickstart.")
+            raise RuntimeError("dracut-live package is missing from the kickstart.")
+
+    # Make the disk or filesystem image
+    if not opts.disk_image and not opts.fs_image:
+        if not opts.ks:
+            raise RuntimeError("Image creation requires a kickstart file")
+
+        errors = []
+        if opts.no_virt and ks.handler.method.method not in ("url", "nfs") \
+           and not ks.handler.ostreesetup.seen:
+            errors.append("Only url, nfs and ostreesetup install methods are currently supported."
+                          "Please fix your kickstart file." )
+
+        if ks.handler.method.method in ("url", "nfs") and not ks.handler.network.seen:
+            errors.append("The kickstart must activate networking if "
+                          "the url or nfs install method is used.")
+
+        if ks.handler.displaymode.displayMode is not None:
+            errors.append("The kickstart must not set a display mode (text, cmdline, "
+                          "graphical), this will interfere with livemedia-creator.")
+
+        if opts.make_fsimage or (opts.make_pxe_live and opts.no_virt):
+            # Make sure the kickstart isn't using autopart and only has a / mountpoint
+            part_ok = not any(p for p in ks.handler.partition.partitions
+                                 if p.mountpoint not in ["/", "swap"])
+            if not part_ok or ks.handler.autopart.seen:
+                errors.append("Filesystem images must use a single / part, not autopart or "
+                              "multiple partitions. swap is allowed but not used.")
+
+        if not opts.no_virt and ks.handler.reboot.action != KS_SHUTDOWN:
+            errors.append("The kickstart must include shutdown when using virt installation.")
+
+        if errors:
+            list(log.error(e) for e in errors)
+            raise RuntimeError("\n".join(errors))
+
+        # Make the image. Output of this is either a partitioned disk image or a fsimage
+        try:
+            disk_img = make_image(opts, ks)
+        except InstallError as e:
+            log.error("ERROR: Image creation failed: %s", e)
+            raise RuntimeError("Image creation failed: %s" % e)
+
+    if opts.image_only:
+        return (result_dir, disk_img)
+
+    if opts.make_iso:
+        work_dir = tempfile.mkdtemp(prefix="lmc-work-")
+        log.info("working dir is %s", work_dir)
+
+        if (opts.fs_image or opts.no_virt) and not opts.disk_image:
+            # Create iso from a filesystem image
+            disk_img = opts.fs_image or disk_img
+
+            if not make_squashfs(opts, disk_img, work_dir):
+                log.error("squashfs.img creation failed")
+                raise RuntimeError("squashfs.img creation failed")
+
+            with Mount(disk_img, opts="loop") as mount_dir:
+                result_dir = make_livecd(opts, mount_dir, work_dir)
+        else:
+            # Create iso from a partitioned disk image
+            disk_img = opts.disk_image or disk_img
+            with PartitionMount(disk_img) as img_mount:
+                if img_mount and img_mount.mount_dir:
+                    make_runtime(opts, img_mount.mount_dir, work_dir, calculate_disk_size(opts, ks)/1024.0)
+                    result_dir = make_livecd(opts, img_mount.mount_dir, work_dir)
+
+        # --iso-only removes the extra build artifacts, keeping only the boot.iso
+        if opts.iso_only and result_dir:
+            boot_iso = joinpaths(result_dir, "images/boot.iso")
+            if not os.path.exists(boot_iso):
+                log.error("%s is missing, skipping --iso-only.", boot_iso)
+            else:
+                iso_dir = tempfile.mkdtemp(prefix="lmc-result-")
+                dest_file = joinpaths(iso_dir, opts.iso_name or "boot.iso")
+                shutil.move(boot_iso, dest_file)
+                shutil.rmtree(result_dir)
+                result_dir = iso_dir
+
+        # cleanup the mess
+        # cleanup work_dir?
+        if disk_img and not (opts.keep_image or opts.disk_image or opts.fs_image):
+            os.unlink(disk_img)
+            log.info("Disk image erased")
+            disk_img = None
+    elif opts.make_appliance:
+        if not opts.ks:
+            networks = []
+        else:
+            networks = ks.handler.network.network
+        make_appliance(opts.disk_image or disk_img, opts.app_name,
+                       opts.app_template, opts.app_file, networks, opts.ram,
+                       opts.vcpus or 1, opts.arch, opts.title, opts.project, opts.releasever)
+    elif opts.make_pxe_live:
+        work_dir = tempfile.mkdtemp(prefix="lmc-work-")
+        log.info("working dir is %s", work_dir)
+        disk_img = opts.fs_image or opts.disk_image or disk_img
+        log.debug("disk image is %s", disk_img)
+
+        result_dir = make_live_images(opts, work_dir, disk_img)
+        if result_dir is None:
+            log.error("Creating PXE live image failed.")
+            raise RuntimeError("Creating PXE live image failed.")
+
+    if opts.result_dir != opts.tmp and result_dir:
+        copytree(result_dir, opts.result_dir, preserve=False)
+        shutil.rmtree(result_dir)
+        result_dir = None
+
+    return (result_dir, disk_img)
    
+
    + +
    + +
    + + +
    +
    + +
    + +
    + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/docs/html/_modules/pylorax/decorators.html b/docs/html/_modules/pylorax/decorators.html index ce118962..17b21a52 100644 --- a/docs/html/_modules/pylorax/decorators.html +++ b/docs/html/_modules/pylorax/decorators.html @@ -8,7 +8,7 @@ - pylorax.decorators — Lorax 29.0 documentation + pylorax.decorators — Lorax 29.1 documentation @@ -25,25 +25,17 @@ - - - - - - - - - - + + + - +
    @@ -65,7 +57,7 @@
    - 29.0 + 29.1
    @@ -94,8 +86,10 @@
  • Before Lorax
  • Lorax
  • livemedia-creator
    • +
  • lorax-composer
    • +
  • composer-cli
  • Product and Updates Images
    • -
  • pylorax
    • +
  • src
    • @@ -107,7 +101,7 @@
    -
    @@ -228,7 +219,8 @@ - - - + \ No newline at end of file diff --git a/docs/html/_modules/pylorax/discinfo.html b/docs/html/_modules/pylorax/discinfo.html index be3cc712..e06f3507 100644 --- a/docs/html/_modules/pylorax/discinfo.html +++ b/docs/html/_modules/pylorax/discinfo.html @@ -8,7 +8,7 @@ - pylorax.discinfo — Lorax 29.0 documentation + pylorax.discinfo — Lorax 29.1 documentation @@ -25,25 +25,17 @@ - - - - - - - - - - + + + - +
    @@ -65,7 +57,7 @@
    - 29.0 + 29.1
    @@ -94,8 +86,10 @@
  • Before Lorax
  • Lorax
  • livemedia-creator
    • +
  • lorax-composer
    • +
  • composer-cli
  • Product and Updates Images
    • -
  • pylorax
    • +
  • src
    • @@ -107,7 +101,7 @@
    -
    @@ -237,7 +228,8 @@ - - - + \ No newline at end of file diff --git a/docs/html/_modules/pylorax/dnfhelper.html b/docs/html/_modules/pylorax/dnfhelper.html index 20adf22b..4e7c1811 100644 --- a/docs/html/_modules/pylorax/dnfhelper.html +++ b/docs/html/_modules/pylorax/dnfhelper.html @@ -8,7 +8,7 @@ - pylorax.dnfhelper — Lorax 29.0 documentation + pylorax.dnfhelper — Lorax 29.1 documentation @@ -25,25 +25,17 @@ - - - - - - - - - - + + + - +
    @@ -65,7 +57,7 @@
    - 29.0 + 29.1
    @@ -94,8 +86,10 @@
  • Before Lorax
  • Lorax
  • livemedia-creator
    • +
  • lorax-composer
    • +
  • composer-cli
  • Product and Updates Images
    • -
  • pylorax
    • +
  • src
    • @@ -107,7 +101,7 @@
    -
    @@ -307,7 +298,8 @@ - - - + \ No newline at end of file diff --git a/docs/html/_modules/pylorax/executils.html b/docs/html/_modules/pylorax/executils.html index 2d21ed64..e95f845f 100644 --- a/docs/html/_modules/pylorax/executils.html +++ b/docs/html/_modules/pylorax/executils.html @@ -8,7 +8,7 @@ - pylorax.executils — Lorax 29.0 documentation + pylorax.executils — Lorax 29.1 documentation @@ -25,25 +25,17 @@ - - - - - - - - - - + + + - +
    @@ -65,7 +57,7 @@
    - 29.0 + 29.1
    @@ -94,8 +86,10 @@
  • Before Lorax
  • Lorax
  • livemedia-creator
    • +
  • lorax-composer
    • +
  • composer-cli
  • Product and Updates Images
    • -
  • pylorax
    • +
  • src
    • @@ -107,7 +101,7 @@
    -
    @@ -545,7 +536,8 @@ - - - + \ No newline at end of file diff --git a/docs/html/_modules/pylorax/imgutils.html b/docs/html/_modules/pylorax/imgutils.html index dd122b41..aa692136 100644 --- a/docs/html/_modules/pylorax/imgutils.html +++ b/docs/html/_modules/pylorax/imgutils.html @@ -8,7 +8,7 @@ - pylorax.imgutils — Lorax 29.0 documentation + pylorax.imgutils — Lorax 29.1 documentation @@ -25,25 +25,17 @@ - - - - - - - - - - + + + - +
    @@ -65,7 +57,7 @@
    - 29.0 + 29.1
    @@ -94,8 +86,10 @@
  • Before Lorax
  • Lorax
  • livemedia-creator
    • +
  • lorax-composer
    • +
  • composer-cli
  • Product and Updates Images
    • -
  • pylorax
    • +
  • src
    • @@ -107,7 +101,7 @@
    -
    -
    - -
    + @@ -683,7 +708,8 @@ - - - + \ No newline at end of file diff --git a/docs/html/_modules/pylorax/installer.html b/docs/html/_modules/pylorax/installer.html new file mode 100644 index 00000000..efb5a816 --- /dev/null +++ b/docs/html/_modules/pylorax/installer.html @@ -0,0 +1,823 @@ + + + + + + + + + + + pylorax.installer — Lorax 29.1 documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    + + + + +
    + + + + + +
    + +
    + + + + + + + + + + + + + + + + + +
    + + + + +
    +
    +
    +
    + +

    Source code for pylorax.installer

    +#
+# Copyright (C) 2011-2018  Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+import logging
+log = logging.getLogger("pylorax")
+
+import glob
+import json
+from math import ceil
+import os
+import subprocess
+import shutil
+import socket
+import tempfile
+
+# Use the Lorax treebuilder branch for iso creation
+from pylorax.executils import execWithRedirect, execReadlines
+from pylorax.imgutils import PartitionMount, mksparse, mkext4img, loop_detach
+from pylorax.imgutils import get_loop_name, dm_detach, mount, umount
+from pylorax.imgutils import mkqemu_img, mktar, mkcpio, mkfsimage_from_disk
+from pylorax.monitor import LogMonitor
+from pylorax.mount import IsoMountpoint
+from pylorax.sysutils import joinpaths
+from pylorax.treebuilder import udev_escape
+
+
+ROOT_PATH = "/mnt/sysimage/"
+
+
    [docs]class InstallError(Exception):
+    pass
    
+
+
+
    [docs]def create_vagrant_metadata(path, size=0):
+    """ Create a default Vagrant metadata.json file
+
+    :param str path: Path to metadata.json file
+    :param int size: Disk size in MiB
+    """
+    metadata = { "provider":"libvirt", "format":"qcow2", "virtual_size": ceil(size / 1024) }
+    with open(path, "wt") as f:
+        json.dump(metadata, f, indent=4)
    
+
+
+
    [docs]def update_vagrant_metadata(path, size):
+    """ Update the Vagrant metadata.json file
+
+    :param str path: Path to metadata.json file
+    :param int size: Disk size in MiB
+
+    This function makes sure that the provider, format and virtual size of the
+    metadata file are set correctly. All other values are left untouched.
+    """
+    with open(path, "rt") as f:
+        try:
+            metadata = json.load(f)
+        except ValueError as e:
+            log.error("Problem reading metadata file %s: %s", path, e)
+            return
+
+    metadata["provider"] = "libvirt"
+    metadata["format"] = "qcow2"
+    metadata["virtual_size"] = ceil(size / 1024)
+    with open(path, "wt") as f:
+        json.dump(metadata, f, indent=4)
    
+
+
+
    [docs]def find_free_port(start=5900, end=5999, host="127.0.0.1"):
+    """ Return first free port in range.
+
+    :param int start: Starting port number
+    :param int end: Ending port number
+    :param str host: Host IP to search
+    :returns: First free port or -1 if none found
+    :rtype: int
+    """
+    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+    for port in range(start, end+1):
+        try:
+            s.bind((host, port))
+            s.close()
+            return port
+        except OSError:
+            pass
+
+    return -1
    
+
+
    [docs]def append_initrd(initrd, files):
+    """ Append files to an initrd.
+
+    :param str initrd: Path to initrd
+    :param list files: list of file paths to add
+    :returns: Path to a new initrd
+    :rtype: str
+
+    The files are added to the initrd by creating a cpio image
+    of the files (stored at /) and writing the cpio to the end of a
+    copy of the initrd.
+
+    The initrd is not changed, a copy is made before appending the
+    cpio archive.
+    """
+    qemu_initrd = tempfile.mktemp(prefix="lmc-initrd-", suffix=".img")
+    shutil.copy2(initrd, qemu_initrd)
+    ks_dir = tempfile.mkdtemp(prefix="lmc-ksdir-")
+    for ks in files:
+        shutil.copy2(ks, ks_dir)
+    ks_initrd = tempfile.mktemp(prefix="lmc-ks-", suffix=".img")
+    mkcpio(ks_dir, ks_initrd)
+    shutil.rmtree(ks_dir)
+    with open(qemu_initrd, "ab") as initrd_fp:
+        with open(ks_initrd, "rb") as ks_fp:
+            while True:
+                data = ks_fp.read(1024**2)
+                if not data:
+                    break
+                initrd_fp.write(data)
+    os.unlink(ks_initrd)
+
+    return qemu_initrd
    
+
+
    [docs]class QEMUInstall(object):
+    """
+    Run qemu using an iso and a kickstart
+    """
+    # Mapping of arch to qemu command
+    QEMU_CMDS = {"x86_64":  "qemu-system-x86_64",
+                 "i386":    "qemu-system-i386",
+                 "arm":     "qemu-system-arm",
+                 "aarch64": "qemu-system-aarch64",
+                 "ppc":     "qemu-system-ppc",
+                 "ppc64":   "qemu-system-ppc64"
+                }
+
+    def __init__(self, opts, iso, ks_paths, disk_img, img_size=2048,
+                 kernel_args=None, memory=1024, vcpus=None, vnc=None, arch=None,
+                 log_check=None, virtio_host="127.0.0.1", virtio_port=6080,
+                 image_type=None, boot_uefi=False, ovmf_path=None):
+        """
+        Start the installation
+
+        :param iso: Information about the iso to use for the installation
+        :type iso: IsoMountpoint
+        :param list ks_paths: Paths to kickstart files. All are injected, the
+           first one is the one executed.
+        :param str disk_img: Path to a disk image, created it it doesn't exist
+        :param int img_size: The image size, in MiB, to create if it doesn't exist
+        :param str kernel_args: Extra kernel arguments to pass on the kernel cmdline
+        :param int memory: Amount of RAM to assign to the virt, in MiB
+        :param int vcpus: Number of virtual cpus
+        :param str vnc: Arguments to pass to qemu -display
+        :param str arch: Optional architecture to use in the virt
+        :param log_check: Method that returns True if the installation fails
+        :type log_check: method
+        :param str virtio_host: Hostname to connect virtio log to
+        :param int virtio_port: Port to connect virtio log to
+        :param str image_type: Type of qemu-img disk to create, or None.
+        :param bool boot_uefi: Use OVMF to boot the VM in UEFI mode
+        :param str ovmf_path: Path to the OVMF firmware
+        """
+        # Lookup qemu-system- for arch if passed, or try to guess using host arch
+        qemu_cmd = [self.QEMU_CMDS.get(arch or os.uname().machine, "qemu-system-"+os.uname().machine)]
+        if not os.path.exists("/usr/bin/"+qemu_cmd[0]):
+            raise InstallError("%s does not exist, cannot run qemu" % qemu_cmd[0])
+
+        qemu_cmd += ["-nodefconfig"]
+        qemu_cmd += ["-m", str(memory)]
+        if vcpus:
+            qemu_cmd += ["-smp", str(vcpus)]
+
+        if not opts.no_kvm and os.path.exists("/dev/kvm"):
+            qemu_cmd += ["--machine", "accel=kvm"]
+
+        # Copy the initrd from the iso, create a cpio archive of the kickstart files
+        # and append it to the temporary initrd.
+        qemu_initrd = append_initrd(iso.initrd, ks_paths)
+        qemu_cmd += ["-kernel", iso.kernel]
+        qemu_cmd += ["-initrd", qemu_initrd]
+
+        # Add the disk and cdrom
+        if not os.path.isfile(disk_img):
+            mksparse(disk_img, img_size * 1024**2)
+        drive_args = "file=%s" % disk_img
+        drive_args += ",cache=unsafe,discard=unmap"
+        if image_type:
+            drive_args += ",format=%s" % image_type
+        else:
+            drive_args += ",format=raw"
+        qemu_cmd += ["-drive", drive_args]
+
+        drive_args = "file=%s,media=cdrom,readonly=on" % iso.iso_path
+        qemu_cmd += ["-drive", drive_args]
+
+        # Setup the cmdline args
+        # ======================
+        cmdline_args = "ks=file:/%s" % os.path.basename(ks_paths[0])
+        cmdline_args += " inst.stage2=hd:LABEL=%s" % udev_escape(iso.label)
+        if opts.proxy:
+            cmdline_args += " inst.proxy=%s" % opts.proxy
+        if kernel_args:
+            cmdline_args += " "+kernel_args
+        cmdline_args += " inst.text inst.cmdline"
+
+        qemu_cmd += ["-append", cmdline_args]
+
+        if not opts.vnc:
+            vnc_port = find_free_port()
+            if vnc_port == -1:
+                raise InstallError("No free VNC ports")
+            display_args = "vnc=127.0.0.1:%d" % (vnc_port - 5900)
+        else:
+            display_args = opts.vnc
+        log.info("qemu %s", display_args)
+        qemu_cmd += ["-nographic", "-display", display_args ]
+
+        # Setup the virtio log port
+        qemu_cmd += ["-device", "virtio-serial-pci,id=virtio-serial0"]
+        qemu_cmd += ["-device", "virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0"
+                                ",id=channel0,name=org.fedoraproject.anaconda.log.0"]
+        qemu_cmd += ["-chardev", "socket,id=charchannel0,host=%s,port=%s" % (virtio_host, virtio_port)]
+
+        # PAss through rng from host
+        if opts.with_rng != "none":
+            qemu_cmd += ["-object", "rng-random,id=virtio-rng0,filename=%s" % opts.with_rng]
+            qemu_cmd += ["-device", "virtio-rng-pci,rng=virtio-rng0,id=rng0,bus=pci.0,addr=0x9"]
+
+        if boot_uefi and ovmf_path:
+            qemu_cmd += ["-drive", "file=%s/OVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on" % ovmf_path]
+
+            # Make a copy of the OVMF_VARS.fd for this run
+            ovmf_vars = tempfile.mktemp(prefix="lmc-OVMF_VARS-", suffix=".fd")
+            shutil.copy2(joinpaths(ovmf_path, "/OVMF_VARS.fd"), ovmf_vars)
+
+            qemu_cmd += ["-drive", "file=%s,if=pflash,format=raw,unit=1" % ovmf_vars]
+
+        log.info("Running qemu")
+        log.debug(qemu_cmd)
+        try:
+            execWithRedirect(qemu_cmd[0], qemu_cmd[1:], reset_lang=False, raise_err=True,
+                             callback=lambda p: not log_check())
+        except subprocess.CalledProcessError as e:
+            log.error("Running qemu failed:")
+            log.error("cmd: %s", " ".join(e.cmd))
+            log.error("output: %s", e.output or "")
+            raise InstallError("QEMUInstall failed")
+        except (OSError, KeyboardInterrupt) as e:
+            log.error("Running qemu failed: %s", str(e))
+            raise InstallError("QEMUInstall failed")
+        finally:
+            os.unlink(qemu_initrd)
+            if boot_uefi and ovmf_path:
+                os.unlink(ovmf_vars)
+
+        if log_check():
+            log.error("Installation error detected. See logfile for details.")
+            raise InstallError("QEMUInstall failed")
+        else:
+            log.info("Installation finished without errors.")
    
+
+
+
    [docs]def novirt_log_check(log_check, proc):
+    """
+    Check to see if there has been an error in the logs
+
+    :param log_check: method to call to check for an error in the logs
+    :param proc: Popen object for the anaconda process
+    :returns: True if the process has been terminated
+
+    The log_check method should return a True if an error has been detected.
+    When an error is detected the process is terminated and this returns True
+    """
+    if log_check():
+        proc.terminate()
+        return True
+    return False
    
+
+
+
    [docs]def anaconda_cleanup(dirinstall_path):
+    """
+    Cleanup any leftover mounts from anaconda
+
+    :param str dirinstall_path: Path where anaconda mounts things
+    :returns: True if cleanups were successful. False if any of them failed.
+
+    If anaconda crashes it may leave things mounted under this path. It will
+    typically be set to /mnt/sysimage/
+
+    Attempts to cleanup may also fail. Catch these and continue trying the
+    other mountpoints.
+    """
+    rc = True
+    dirinstall_path = os.path.abspath(dirinstall_path)
+    # unmount filesystems
+    for mounted in reversed(open("/proc/mounts").readlines()):
+        (_device, mountpoint, _rest) = mounted.split(" ", 2)
+        if mountpoint.startswith(dirinstall_path) and os.path.ismount(mountpoint):
+            try:
+                umount(mountpoint)
+            except subprocess.CalledProcessError:
+                log.error("Cleanup of %s failed. See program.log for details", mountpoint)
+                rc = False
+    return rc
    
+
+
+
    [docs]def novirt_install(opts, disk_img, disk_size):
+    """
+    Use Anaconda to install to a disk image
+
+    :param opts: options passed to livemedia-creator
+    :type opts: argparse options
+    :param str disk_img: The full path to the disk image to be created
+    :param int disk_size: The size of the disk_img in MiB
+
+    This method runs anaconda to create the image and then based on the opts
+    passed creates a qemu disk image or tarfile.
+    """
+    dirinstall_path = ROOT_PATH
+
+    # Clean up /tmp/ from previous runs to prevent stale info from being used
+    for path in ["/tmp/yum.repos.d/", "/tmp/yum.cache/"]:
+        if os.path.isdir(path):
+            shutil.rmtree(path)
+
+    args = ["--kickstart", opts.ks[0], "--cmdline", "--loglevel", "debug"]
+    if opts.anaconda_args:
+        for arg in opts.anaconda_args:
+            args += arg.split(" ", 1)
+    if opts.proxy:
+        args += ["--proxy", opts.proxy]
+    if opts.armplatform:
+        args += ["--armplatform", opts.armplatform]
+
+    if opts.make_iso or opts.make_fsimage or opts.make_pxe_live:
+        # Make a blank fs image
+        args += ["--dirinstall"]
+
+        mkext4img(None, disk_img, label=opts.fs_label, size=disk_size * 1024**2)
+        if not os.path.isdir(dirinstall_path):
+            os.mkdir(dirinstall_path)
+        mount(disk_img, opts="loop", mnt=dirinstall_path)
+    elif opts.make_tar or opts.make_oci:
+        # Install under dirinstall_path, make sure it starts clean
+        if os.path.exists(dirinstall_path):
+            shutil.rmtree(dirinstall_path)
+
+        if opts.make_oci:
+            # OCI installs under /rootfs/
+            dirinstall_path = joinpaths(dirinstall_path, "rootfs")
+            args += ["--dirinstall", dirinstall_path]
+        else:
+            args += ["--dirinstall"]
+
+        os.makedirs(dirinstall_path)
+    else:
+        args += ["--image", disk_img]
+
+        # Create the sparse image
+        mksparse(disk_img, disk_size * 1024**2)
+
+    log_monitor = LogMonitor(timeout=opts.timeout)
+    args += ["--remotelog", "%s:%s" % (log_monitor.host, log_monitor.port)]
+
+    # Make sure anaconda has the right product and release
+    log.info("Running anaconda.")
+    try:
+        for line in execReadlines("anaconda", args, reset_lang=False,
+                                  env_add={"ANACONDA_PRODUCTNAME": opts.project,
+                                           "ANACONDA_PRODUCTVERSION": opts.releasever},
+                                  callback=lambda p: not novirt_log_check(log_monitor.server.log_check, p)):
+            log.info(line)
+
+        # Make sure the new filesystem is correctly labeled
+        setfiles_args = ["-e", "/proc", "-e", "/sys", "-e", "/dev",
+                         "/etc/selinux/targeted/contexts/files/file_contexts", "/"]
+
+        # setfiles may not be available, warn instead of fail
+        try:
+            if "--dirinstall" in args:
+                execWithRedirect("setfiles", setfiles_args, root=dirinstall_path)
+            else:
+                with PartitionMount(disk_img) as img_mount:
+                    if img_mount and img_mount.mount_dir:
+                        execWithRedirect("setfiles", setfiles_args, root=img_mount.mount_dir)
+        except (subprocess.CalledProcessError, OSError) as e:
+            log.warning("Running setfiles on install tree failed: %s", str(e))
+
+    except (subprocess.CalledProcessError, OSError) as e:
+        log.error("Running anaconda failed: %s", e)
+        raise InstallError("novirt_install failed")
+    finally:
+        log_monitor.shutdown()
+
+        # Move the anaconda logs over to a log directory
+        log_dir = os.path.abspath(os.path.dirname(opts.logfile))
+        log_anaconda = joinpaths(log_dir, "anaconda")
+        if not os.path.isdir(log_anaconda):
+            os.mkdir(log_anaconda)
+        for l in glob.glob("/tmp/*log")+glob.glob("/tmp/anaconda-tb-*"):
+            shutil.copy2(l, log_anaconda)
+            os.unlink(l)
+
+        # Make sure any leftover anaconda mounts have been cleaned up
+        if not anaconda_cleanup(dirinstall_path):
+            raise InstallError("novirt_install cleanup of anaconda mounts failed.")
+
+        if not opts.make_iso and not opts.make_fsimage and not opts.make_pxe_live:
+            dm_name = os.path.splitext(os.path.basename(disk_img))[0]
+            dm_path = "/dev/mapper/"+dm_name
+            if os.path.exists(dm_path):
+                dm_detach(dm_path)
+                loop_detach(get_loop_name(disk_img))
+
+    # qemu disk image is used by bare qcow2 images and by Vagrant
+    if opts.image_type:
+        log.info("Converting %s to %s", disk_img, opts.image_type)
+        qemu_args = []
+        for arg in opts.qemu_args:
+            qemu_args += arg.split(" ", 1)
+
+        # convert the image to the selected format
+        if "-O" not in qemu_args:
+            qemu_args.extend(["-O", opts.image_type])
+        qemu_img = tempfile.mktemp(prefix="lmc-disk-", suffix=".img")
+        execWithRedirect("qemu-img", ["convert"] + qemu_args + [disk_img, qemu_img], raise_err=True)
+        if not opts.make_vagrant:
+            execWithRedirect("mv", ["-f", qemu_img, disk_img], raise_err=True)
+        else:
+            # Take the new qcow2 image and package it up for Vagrant
+            compress_args = []
+            for arg in opts.compress_args:
+                compress_args += arg.split(" ", 1)
+
+            vagrant_dir = tempfile.mkdtemp(prefix="lmc-tmpdir-")
+            metadata_path = joinpaths(vagrant_dir, "metadata.json")
+            execWithRedirect("mv", ["-f", qemu_img, joinpaths(vagrant_dir, "box.img")], raise_err=True)
+            if opts.vagrant_metadata:
+                shutil.copy2(opts.vagrant_metadata, metadata_path)
+            else:
+                create_vagrant_metadata(metadata_path)
+            update_vagrant_metadata(metadata_path, disk_size)
+            if opts.vagrantfile:
+                shutil.copy2(opts.vagrantfile, joinpaths(vagrant_dir, "vagrantfile"))
+
+            log.info("Creating Vagrant image")
+            rc = mktar(vagrant_dir, disk_img, opts.compression, compress_args, selinux=False)
+            if rc:
+                raise InstallError("novirt_install mktar failed: rc=%s" % rc)
+            shutil.rmtree(vagrant_dir)
+    elif opts.make_tar:
+        compress_args = []
+        for arg in opts.compress_args:
+            compress_args += arg.split(" ", 1)
+
+        rc = mktar(dirinstall_path, disk_img, opts.compression, compress_args)
+        shutil.rmtree(dirinstall_path)
+
+        if rc:
+            raise InstallError("novirt_install mktar failed: rc=%s" % rc)
+    elif opts.make_oci:
+        # An OCI image places the filesystem under /rootfs/ and adds the json files at the top
+        # And then creates a tar of the whole thing.
+        compress_args = []
+        for arg in opts.compress_args:
+            compress_args += arg.split(" ", 1)
+
+        shutil.copy2(opts.oci_config, ROOT_PATH)
+        shutil.copy2(opts.oci_runtime, ROOT_PATH)
+        rc = mktar(ROOT_PATH, disk_img, opts.compression, compress_args)
+
+        if rc:
+            raise InstallError("novirt_install mktar failed: rc=%s" % rc)
    
+
+
+
    [docs]def virt_install(opts, install_log, disk_img, disk_size):
+    """
+    Use qemu to install to a disk image
+
+    :param opts: options passed to livemedia-creator
+    :type opts: argparse options
+    :param str install_log: The path to write the log from qemu
+    :param str disk_img: The full path to the disk image to be created
+    :param int disk_size: The size of the disk_img in MiB
+
+    This uses qemu with a boot.iso and a kickstart to create a disk
+    image and then optionally, based on the opts passed, creates tarfile.
+    """
+    iso_mount = IsoMountpoint(opts.iso, opts.location)
+    if not iso_mount.stage2:
+        iso_mount.umount()
+        raise InstallError("ISO is missing stage2, cannot continue")
+
+    log_monitor = LogMonitor(install_log, timeout=opts.timeout)
+
+    kernel_args = ""
+    if opts.kernel_args:
+        kernel_args += opts.kernel_args
+    if opts.proxy:
+        kernel_args += " proxy="+opts.proxy
+
+    if opts.image_type and not opts.make_fsimage:
+        qemu_args = []
+        for arg in opts.qemu_args:
+            qemu_args += arg.split(" ", 1)
+        if "-f" not in qemu_args:
+            qemu_args += ["-f", opts.image_type]
+
+        mkqemu_img(disk_img, disk_size*1024**2, qemu_args)
+
+    if opts.make_fsimage or opts.make_tar or opts.make_oci:
+        diskimg_path = tempfile.mktemp(prefix="lmc-disk-", suffix=".img")
+    else:
+        diskimg_path = disk_img
+
+    try:
+        QEMUInstall(opts, iso_mount, opts.ks, diskimg_path, disk_size,
+                    kernel_args, opts.ram, opts.vcpus, opts.vnc, opts.arch,
+                    log_check = log_monitor.server.log_check,
+                    virtio_host = log_monitor.host,
+                    virtio_port = log_monitor.port,
+                    image_type=opts.image_type, boot_uefi=opts.virt_uefi,
+                    ovmf_path=opts.ovmf_path)
+        log_monitor.shutdown()
+    except InstallError as e:
+        log.error("VirtualInstall failed: %s", e)
+        raise
+    finally:
+        log.info("unmounting the iso")
+        iso_mount.umount()
+
+    if log_monitor.server.log_check():
+        if not log_monitor.server.error_line and opts.timeout:
+            msg = "virt_install failed due to timeout"
+        else:
+            msg = "virt_install failed on line: %s" % log_monitor.server.error_line
+        raise InstallError(msg)
+
+    if opts.make_fsimage:
+        mkfsimage_from_disk(diskimg_path, disk_img, disk_size, label=opts.fs_label)
+        os.unlink(diskimg_path)
+    elif opts.make_tar:
+        compress_args = []
+        for arg in opts.compress_args:
+            compress_args += arg.split(" ", 1)
+
+        with PartitionMount(diskimg_path) as img_mount:
+            if img_mount and img_mount.mount_dir:
+                rc = mktar(img_mount.mount_dir, disk_img, opts.compression, compress_args)
+            else:
+                rc = 1
+        os.unlink(diskimg_path)
+
+        if rc:
+            raise InstallError("virt_install failed")
+    elif opts.make_oci:
+        # An OCI image places the filesystem under /rootfs/ and adds the json files at the top
+        # And then creates a tar of the whole thing.
+        compress_args = []
+        for arg in opts.compress_args:
+            compress_args += arg.split(" ", 1)
+
+        with PartitionMount(diskimg_path, submount="rootfs") as img_mount:
+            if img_mount and img_mount.temp_dir:
+                shutil.copy2(opts.oci_config, img_mount.temp_dir)
+                shutil.copy2(opts.oci_runtime, img_mount.temp_dir)
+                rc = mktar(img_mount.temp_dir, disk_img, opts.compression, compress_args)
+            else:
+                rc = 1
+        os.unlink(diskimg_path)
+
+        if rc:
+            raise InstallError("virt_install failed")
+    elif opts.make_vagrant:
+        compress_args = []
+        for arg in opts.compress_args:
+            compress_args += arg.split(" ", 1)
+
+        vagrant_dir = tempfile.mkdtemp(prefix="lmc-tmpdir-")
+        metadata_path = joinpaths(vagrant_dir, "metadata.json")
+        execWithRedirect("mv", ["-f", disk_img, joinpaths(vagrant_dir, "box.img")], raise_err=True)
+        if opts.vagrant_metadata:
+            shutil.copy2(opts.vagrant_metadata, metadata_path)
+        else:
+            create_vagrant_metadata(metadata_path)
+        update_vagrant_metadata(metadata_path, disk_size)
+        if opts.vagrantfile:
+            shutil.copy2(opts.vagrantfile, joinpaths(vagrant_dir, "vagrantfile"))
+
+        rc = mktar(vagrant_dir, disk_img, opts.compression, compress_args, selinux=False)
+        if rc:
+            raise InstallError("virt_install failed")
+        shutil.rmtree(vagrant_dir)
    
+
    + +
    + +
    + + +
    +
    + +
    + +
    + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/docs/html/_modules/pylorax/ltmpl.html b/docs/html/_modules/pylorax/ltmpl.html index 73518e85..5e723493 100644 --- a/docs/html/_modules/pylorax/ltmpl.html +++ b/docs/html/_modules/pylorax/ltmpl.html @@ -8,7 +8,7 @@ - pylorax.ltmpl — Lorax 29.0 documentation + pylorax.ltmpl — Lorax 29.1 documentation @@ -25,25 +25,17 @@ - - - - - - - - - - + + + - +
    @@ -65,7 +57,7 @@
    - 29.0 + 29.1
    @@ -94,8 +86,10 @@
  • Before Lorax
  • Lorax
  • livemedia-creator
    • +
  • lorax-composer
    • +
  • composer-cli
  • Product and Updates Images
    • -
  • pylorax
    • +
  • src
    • @@ -107,7 +101,7 @@
    -
    @@ -985,7 +976,8 @@ - - - + \ No newline at end of file diff --git a/docs/html/_modules/pylorax/monitor.html b/docs/html/_modules/pylorax/monitor.html index bef68c56..59d8ead8 100644 --- a/docs/html/_modules/pylorax/monitor.html +++ b/docs/html/_modules/pylorax/monitor.html @@ -8,7 +8,7 @@ - pylorax.monitor — Lorax 29.0 documentation + pylorax.monitor — Lorax 29.1 documentation @@ -25,25 +25,17 @@ - - - - - - - - - - + + + - +
    @@ -65,7 +57,7 @@
    - 29.0 + 29.1
    @@ -94,8 +86,10 @@
  • Before Lorax
  • Lorax
  • livemedia-creator
    • +
  • lorax-composer
    • +
  • composer-cli
  • Product and Updates Images
    • -
  • pylorax
    • +
  • src
    • @@ -107,7 +101,7 @@
    -
    @@ -393,7 +384,8 @@ - - - + \ No newline at end of file diff --git a/docs/html/_modules/pylorax/mount.html b/docs/html/_modules/pylorax/mount.html index be6f244e..5bb9e992 100644 --- a/docs/html/_modules/pylorax/mount.html +++ b/docs/html/_modules/pylorax/mount.html @@ -8,7 +8,7 @@ - pylorax.mount — Lorax 29.0 documentation + pylorax.mount — Lorax 29.1 documentation @@ -25,25 +25,17 @@ - - - - - - - - - - + + + - +
    @@ -65,7 +57,7 @@
    - 29.0 + 29.1
    @@ -94,8 +86,10 @@
  • Before Lorax
  • Lorax
  • livemedia-creator
    • +
  • lorax-composer
    • +
  • composer-cli
  • Product and Updates Images
    • -
  • pylorax
    • +
  • src
    • @@ -107,7 +101,7 @@
    -
    @@ -300,7 +291,8 @@ - - - + \ No newline at end of file diff --git a/docs/html/_modules/pylorax/sysutils.html b/docs/html/_modules/pylorax/sysutils.html index 062ec7bb..c879cb2f 100644 --- a/docs/html/_modules/pylorax/sysutils.html +++ b/docs/html/_modules/pylorax/sysutils.html @@ -8,7 +8,7 @@ - pylorax.sysutils — Lorax 29.0 documentation + pylorax.sysutils — Lorax 29.1 documentation @@ -25,25 +25,17 @@ - - - - - - - - - - + + + - +
    @@ -65,7 +57,7 @@
    - 29.0 + 29.1
    @@ -94,8 +86,10 @@
  • Before Lorax
  • Lorax
  • livemedia-creator
    • +
  • lorax-composer
    • +
  • composer-cli
  • Product and Updates Images
    • -
  • pylorax
    • +
  • src
    • @@ -107,7 +101,7 @@
    -
    @@ -306,7 +297,8 @@ - - - + \ No newline at end of file diff --git a/docs/html/_modules/pylorax/treebuilder.html b/docs/html/_modules/pylorax/treebuilder.html index 17592827..60dc46ff 100644 --- a/docs/html/_modules/pylorax/treebuilder.html +++ b/docs/html/_modules/pylorax/treebuilder.html @@ -8,7 +8,7 @@ - pylorax.treebuilder — Lorax 29.0 documentation + pylorax.treebuilder — Lorax 29.1 documentation @@ -25,25 +25,17 @@ - - - - - - - - - - + + + - +
    @@ -65,7 +57,7 @@
    - 29.0 + 29.1
    @@ -94,8 +86,10 @@
  • Before Lorax
  • Lorax
  • livemedia-creator
    • +
  • lorax-composer
    • +
  • composer-cli
  • Product and Updates Images
    • -
  • pylorax
    • +
  • src
    • @@ -107,7 +101,7 @@
    -
    @@ -598,7 +589,8 @@ - - - + \ No newline at end of file diff --git a/docs/html/_modules/pylorax/treeinfo.html b/docs/html/_modules/pylorax/treeinfo.html index 9c78f0fb..a55f8ed8 100644 --- a/docs/html/_modules/pylorax/treeinfo.html +++ b/docs/html/_modules/pylorax/treeinfo.html @@ -8,7 +8,7 @@ - pylorax.treeinfo — Lorax 29.0 documentation + pylorax.treeinfo — Lorax 29.1 documentation @@ -25,25 +25,17 @@ - - - - - - - - - - + + + - +
    @@ -65,7 +57,7 @@
    - 29.0 + 29.1
    @@ -94,8 +86,10 @@
  • Before Lorax
  • Lorax
  • livemedia-creator
    • +
  • lorax-composer
    • +
  • composer-cli
  • Product and Updates Images
    • -
  • pylorax
    • +
  • src
    • @@ -107,7 +101,7 @@
    -
    @@ -255,7 +246,8 @@ - - - + \ No newline at end of file diff --git a/docs/html/_sources/composer-cli.rst.txt b/docs/html/_sources/composer-cli.rst.txt new file mode 100644 index 00000000..7cde6feb --- /dev/null +++ b/docs/html/_sources/composer-cli.rst.txt @@ -0,0 +1,62 @@ +composer-cli +============ + +:Authors: + Brian C. Lane + +``composer-cli`` is used to interact with the ``lorax-composer`` API server, managing blueprints, exploring available packages, and building new images. + +It requires `lorax-composer `_ to be installed on the +local system, and the user running it needs to be a member of the ``weldr`` +group. They do not need to be root, but all of the `security precautions +`_ apply. + +composer-cli cmdline arguments +------------------------------ + +.. argparse:: + :ref: composer.cli.cmdline.composer_cli_parser + :prog: composer-cli + +Edit a Blueprint +---------------- + +Start out by listing the available blueprints using ``composer-cli blueprints +list``, pick one and save it to the local directory by running ``composer-cli +blueprints save http-server``. If there are no blueprints available you can +copy one of the examples `from the test suite +`_. + +Edit the file (it will be saved with a .toml extension) and chance the +description, add a package or module to it. Send it back to the server by +running ``composer-cli blueprints push http-server.toml``. You can verify that it was +saved by viewing the changelog - ``composer-cli blueprints changes http-server``. + +Build an image +---------------- + +Build a ``qcow2`` disk image from this blueprint by running ``composer-cli +compose start http-server qcow2``. It will print a UUID that you can use to +keep track of the build. You can also cancel the build if needed. + +The available types of images is displayed by ``composer-cli compose types``. +Currently this consists of: ext4-filesystem, live-iso, partitioned-disk, qcow2, +tar + +Monitor the build status +------------------------ + +Monitor it using ``composer-cli compose status``, which will show the status of +all the builds on the system. You can view the end of the anaconda build logs +once it is in the ``RUNNING`` state using ``composer-cli compose log UUID`` +where UUID is the UUID returned by the start command. + +Once the build is in the ``FINISHED`` state you can download the image. + +Download the image +------------------ + +Downloading the final image is done with ``composer-cli compose image UUID`` and it will +save the qcow2 image as ``UUID-disk.qcow2`` which you can then use to boot a VM like this:: + + qemu-kvm --name test-image -m 1024 -hda ./UUID-disk.qcow2 diff --git a/docs/html/_sources/composer.cli.rst.txt b/docs/html/_sources/composer.cli.rst.txt new file mode 100644 index 00000000..f6e56582 --- /dev/null +++ b/docs/html/_sources/composer.cli.rst.txt @@ -0,0 +1,62 @@ +composer.cli package +==================== + +Submodules +---------- + +composer.cli.blueprints module +------------------------------ + +.. automodule:: composer.cli.blueprints + :members: + :undoc-members: + :show-inheritance: + +composer.cli.cmdline module +--------------------------- + +.. automodule:: composer.cli.cmdline + :members: + :undoc-members: + :show-inheritance: + +composer.cli.compose module +--------------------------- + +.. automodule:: composer.cli.compose + :members: + :undoc-members: + :show-inheritance: + +composer.cli.modules module +--------------------------- + +.. automodule:: composer.cli.modules + :members: + :undoc-members: + :show-inheritance: + +composer.cli.projects module +---------------------------- + +.. automodule:: composer.cli.projects + :members: + :undoc-members: + :show-inheritance: + +composer.cli.utilities module +----------------------------- + +.. automodule:: composer.cli.utilities + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: composer.cli + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/html/_sources/composer.rst.txt b/docs/html/_sources/composer.rst.txt new file mode 100644 index 00000000..5314fc0c --- /dev/null +++ b/docs/html/_sources/composer.rst.txt @@ -0,0 +1,37 @@ +composer package +================ + +Subpackages +----------- + +.. toctree:: + + composer.cli + +Submodules +---------- + +composer.http\_client module +---------------------------- + +.. automodule:: composer.http_client + :members: + :undoc-members: + :show-inheritance: + +composer.unix\_socket module +---------------------------- + +.. automodule:: composer.unix_socket + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: composer + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/html/_sources/index.rst.txt b/docs/html/_sources/index.rst.txt index 1b7ac330..7a708ea8 100644 --- a/docs/html/_sources/index.rst.txt +++ b/docs/html/_sources/index.rst.txt @@ -14,6 +14,8 @@ Contents: intro lorax livemedia-creator + lorax-composer + composer-cli product-images modules diff --git a/docs/html/_sources/index.txt b/docs/html/_sources/index.txt deleted file mode 100644 index c9b962d8..00000000 --- a/docs/html/_sources/index.txt +++ /dev/null @@ -1,27 +0,0 @@ -.. Lorax documentation master file, created by - sphinx-quickstart on Wed Apr 8 13:46:00 2015. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. - -Welcome to Lorax's documentation! -================================= - -Contents: - -.. toctree:: - :maxdepth: 1 - - intro - lorax - livemedia-creator - product-images - modules - - -Indices and tables -================== - -* :ref:`genindex` -* :ref:`modindex` -* :ref:`search` - diff --git a/docs/html/_sources/intro.txt b/docs/html/_sources/intro.txt deleted file mode 100644 index 01857ee9..00000000 --- a/docs/html/_sources/intro.txt +++ /dev/null @@ -1,67 +0,0 @@ -Introduction to Lorax -===================== - -I am the Lorax. I speak for the trees [and images]. - -Lorax is used to build the Anaconda Installer boot.iso, it consists of a -library, pylorax, a set of templates, and the lorax script. Its operation -is driven by a customized set of Mako templates that lists the packages -to be installed, steps to execute to remove unneeded files, and creation -of the iso for all of the supported architectures. - - - - - - -Before Lorax -============ - -Tree building tools such as pungi and revisor rely on 'buildinstall' in -anaconda/scripts/ to produce the boot images and other such control files -in the final tree. The existing buildinstall scripts written in a mix of -bash and Python are unmaintainable. Lorax is an attempt to replace them -with something more flexible. - - -EXISTING WORKFLOW: - -pungi and other tools call scripts/buildinstall, which in turn call other -scripts to do the image building and data generation. Here's how it -currently looks: - - -> buildinstall - * process command line options - * write temporary yum.conf to point to correct repo - * find anaconda release RPM - * unpack RPM, pull in those versions of upd-instroot, mk-images, - maketreeinfo.py, makestamp.py, and buildinstall - - -> call upd-instroot - - -> call maketreeinfo.py - - -> call mk-images (which figures out which mk-images.ARCH to call) - - -> call makestamp.py - - * clean up - - -PROBLEMS: - -The existing workflow presents some problems with maintaining the scripts. -First, almost all knowledge of what goes in to the stage 1 and stage 2 -images lives in upd-instroot. The mk-images* scripts copy things from the -root created by upd-instroot in order to build the stage 1 image, though -it's not completely clear from reading the scripts. - - -NEW IDEAS: - -Create a new central driver with all information living in Python modules. -Configuration files will provide the knowledge previously contained in the -upd-instroot and mk-images* scripts. - - - diff --git a/docs/html/_sources/livemedia-creator.txt b/docs/html/_sources/livemedia-creator.txt deleted file mode 100644 index b8825248..00000000 --- a/docs/html/_sources/livemedia-creator.txt +++ /dev/null @@ -1,638 +0,0 @@ -livemedia-creator -================= - -:Authors: - Brian C. Lane - -livemedia-creator uses `Anaconda `_, -`kickstart `_ and `Lorax -`_ to create bootable media that use the -same install path as a normal system installation. It can be used to make live -isos, bootable (partitioned) disk images, tarfiles, and filesystem images for -use with virtualization and container solutions like libvirt, docker, and -OpenStack. - -The general idea is to use qemu with kickstart and an Anaconda boot.iso to -install into a disk image and then use the disk image to create the bootable -media. - -livemedia-creator --help will describe all of the options available. At the -minimum you need: - -``--make-iso`` to create a final bootable .iso or one of the other ``--make-*`` options. - -``--iso`` to specify the Anaconda install media to use with qemu. - -``--ks`` to select the kickstart file describing what to install. - -To use livemedia-creator with virtualization you will need to have qemu installed. - -If you are going to be using Anaconda directly, with ``--no-virt`` mode, make sure -you have the anaconda-tui package installed. - -Conventions used in this document: - -``lmc`` is an abbreviation for livemedia-creator. - -``builder`` is the system where livemedia-creator is being run - -``image`` is the disk image being created by running livemedia-creator - - -livemedia-creator cmdline arguments ------------------------------------ - -.. argparse:: - :ref: pylorax.cmdline.lmc_parser - :prog: livemedia-creator - - -Quickstart ----------- - -Run this to create a bootable live iso:: - - sudo livemedia-creator --make-iso \ - --iso=/extra/iso/boot.iso --ks=./docs/fedora-livemedia.ks - -You can run it directly from the lorax git repo like this:: - - sudo PATH=./src/sbin/:$PATH PYTHONPATH=./src/ ./src/sbin/livemedia-creator \ - --make-iso --iso=/extra/iso/boot.iso \ - --ks=./docs/fedora-livemedia.ks --lorax-templates=./share/ - -You can observe the installation using vnc. The logs will show what port was -chosen, or you can use a specific port by passing it. eg. ``--vnc vnc:127.0.0.1:5`` - -This is usually a good idea when testing changes to the kickstart. lmc tries -to monitor the logs for fatal errors, but may not catch everything. - - -How ISO creation works ----------------------- - -There are 2 stages, the install stage which produces a disk or filesystem image -as its output, and the boot media creation which uses the image as its input. -Normally you would run both stages, but it is possible to stop after the -install stage, by using ``--image-only``, or to skip the install stage and use -a previously created disk image by passing ``--disk-image`` or ``--fs-image`` - -When creating an iso qemu boots using the passed Anaconda installer iso -and installs the system based on the kickstart. The ``%post`` section of the -kickstart is used to customize the installed system in the same way that -current spin-kickstarts do. - -livemedia-creator monitors the install process for problems by watching the -install logs. They are written to the current directory or to the base -directory specified by the --logfile command. You can also monitor the install -by using a vnc client. This is recommended when first modifying a kickstart, -since there are still places where Anaconda may get stuck without the log -monitor catching it. - -The output from this process is a partitioned disk image. kpartx can be used -to mount and examine it when there is a problem with the install. It can also -be booted using kvm. - -When creating an iso the disk image's / partition is copied into a formatted -filesystem image which is then used as the input to lorax for creation of the -final media. - -The final image is created by lorax, using the templates in /usr/share/lorax/live/ -or the live directory below the directory specified by ``--lorax-templates``. The -templates are written using the Mako template system with some extra commands -added by lorax. - -.. note:: - The output from --make-iso includes the artifacts used to create the boot.iso; - the kernel, initrd, the squashfs filesystem, etc. If you only want the - boot.iso you can pass ``--iso-only`` and the other files will be removed. You - can also name the iso by using ``--iso-name my-live.iso``. - - -Kickstarts ----------- - -The docs/ directory includes several example kickstarts, one to create a live -desktop iso using GNOME, and another to create a minimal disk image. When -creating your own kickstarts you should start with the minimal example, it -includes several needed packages that are not always included by dependencies. - -Or you can use existing spin kickstarts to create live media with a few -changes. Here are the steps I used to convert the Fedora XFCE spin. - -1. Flatten the xfce kickstart using ksflatten -2. Add zerombr so you don't get the disk init dialog -3. Add clearpart --all -4. Add swap partition -5. bootloader target -6. Add shutdown to the kickstart -7. Add network --bootproto=dhcp --activate to activate the network - This works for F16 builds but for F15 and before you need to pass - something on the cmdline that activate the network, like sshd: - - ``livemedia-creator --kernel-args="sshd"`` - -8. Add a root password:: - - rootpw rootme - network --bootproto=dhcp --activate - zerombr - clearpart --all - bootloader --location=mbr - part swap --size=512 - shutdown - -9. In the livesys script section of the %post remove the root password. This - really depends on how the spin wants to work. You could add the live user - that you create to the %wheel group so that sudo works if you wanted to. - - ``passwd -d root > /dev/null`` - -10. Remove /etc/fstab in %post, dracut handles mounting the rootfs - - ``cat /dev/null > /dev/fstab`` - - Do this only for live iso's, the filesystem will be mounted read only if - there is no /etc/fstab - -11. Don't delete initramfs files from /boot in %post -12. When creating live iso's you need to have, at least, these packages in the %package section:: - dracut-config-generic - dracut-live - -dracut-config-rescue - grub-efi - memtest86+ - syslinux - -One drawback to using qemu is that it pulls the packages from the repo each -time you run it. To speed things up you either need a local mirror of the -packages, or you can use a caching proxy. When using a proxy you pass it to -livemedia-creator like this: - - ``--proxy=http://proxy.yourdomain.com:3128`` - -You also need to use a specific mirror instead of mirrormanager so that the -packages will get cached, so your kickstart url would look like: - - ``url --url="http://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/x86_64/os/"`` - -You can also add an update repo, but don't name it updates. Add --proxy to it -as well. - - -Anaconda image install (no-virt) --------------------------------- - -You can create images without using qemu by passing ``--no-virt`` on the -cmdline. This will use Anaconda's directory install feature to handle the -install. There are a couple of things to keep in mind when doing this: - -1. It will be most reliable when building images for the same release that the - host is running. Because Anaconda has expectations about the system it is - running under you may encounter strange bugs if you try to build newer or - older releases. - -2. Make sure selinux is set to permissive or disabled. It won't install - correctly with selinux set to enforcing yet. - -3. It may totally trash your host. So far I haven't had this happen, but the - possibility exists that a bug in Anaconda could result in it operating on - real devices. I recommend running it in a virt or on a system that you can - afford to lose all data from. - -The logs from anaconda will be placed in an ./anaconda/ directory in either -the current directory or in the directory used for --logfile - -Example cmdline: - -``sudo livemedia-creator --make-iso --no-virt --ks=./fedora-livemedia.ks`` - -.. note:: - Using no-virt to create a partitioned disk image (eg. --make-disk or - --make-vagrant) will only create disks usable on the host platform (BIOS - or UEFI). You can create BIOS partitioned disk images on UEFI by using - virt. - - -AMI Images ----------- - -Amazon EC2 images can be created by using the --make-ami switch and an appropriate -kickstart file. All of the work to customize the image is handled by the kickstart. -The example currently included was modified from the cloud-kickstarts version so -that it would work with livemedia-creator. - -Example cmdline: - -``sudo livemedia-creator --make-ami --iso=/path/to/boot.iso --ks=./docs/fedora-livemedia-ec2.ks`` - -This will produce an ami-root.img file in the working directory. - -At this time I have not tested the image with EC2. Feedback would be welcome. - - -Appliance Creation ------------------- - -livemedia-creator can now replace appliance-tools by using the --make-appliance -switch. This will create the partitioned disk image and an XML file that can be -used with virt-image to setup a virtual system. - -The XML is generated using the Mako template from -/usr/share/lorax/appliance/libvirt.xml You can use a different template by -passing ``--app-template