phases: implement image-builder

Implement a phase for the `imageBuilderBuild` task that is provided by
the `koji-image-builder` plugin, which schedules tasks to run with
`image-builder`.

This change is part of an accepted change proposal [1] for Fedora to use
`koji-image-builder` to build (some of) its variants.

[1]: https://fedoraproject.org/wiki/Changes/KojiLocalImageBuilder

Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
(cherry picked from commit 69d87c27ff29b128aa8ff1e8aebd278a00d9fed8)

doc/configuration.rst

@@ -1726,6 +1726,102 @@ OSBuild Composer for building images
    arch.
 
 
+Image Builder Settings
+======================
+
+**imagebuilder**
+    (*dict*) -- configuration for building images with the ``koji-image-builder``
+    Koji plugin. Pungi will trigger a Koji task which will build the image with
+    the given configuration using the ``image-builder`` executable in the build
+    root.
+
+    Format: ``{variant_uid_regex: [{...}]}``.
+
+    Required keys in the configuration dict:
+
+    * ``name`` -- name of the Koji package
+    * ``types`` -- a list with a single image type string representing
+      the image type to build (e.g. ``qcow2``). Only a single image type
+      can be provided as an argument.
+
+    Optional keys:
+
+    * ``target`` -- which build target to use for the task. Either this option,
+      the global ``imagebuilder_target``, or ``global_target`` is required.
+    * ``version`` -- version for the final build (as a string). This option is
+      required if the global ``imagebuilder_version`` or its ``global_version``
+      equivalent are not specified.
+    * ``release`` -- release part of the final NVR. If neither this option nor
+      the global ``imagebuilder_release`` nor its ``global_release`` equivalent
+      are set, Koji will automatically generate a value.
+    * ``repos`` -- a list of repositories from which to consume packages for
+      building the image. By default only the variant repository is used.
+      The list items use the following formats:
+
+      * String with just the repository URL.
+      * Variant ID in the current compose.
+
+    * ``arches`` -- list of architectures for which to build the image. By
+      default, the variant arches are used. This option can only restrict it,
+      not add a new one.
+
+    * ``seed`` -- An integer that can be used to make builds more reproducible.
+      When ``image-builder`` builds images various bits and bobs are generated
+      with a PRNG (partition uuids, etc). Pinning the seed with this argument
+      or ``imagebuilder_seed`` to do so globally will make builds use the same
+      random values each time. Note that using ``seed`` requires the Koji side
+      to have at least ``koji-image-builder >= 7`` deployed.
+
+    * ``scratch`` -- A boolean to instruct ``koji-image-builder`` to perform scratch
+      builds. This might have implications on garbage collection within the ``koji``
+      instance you're targeting. Can also be set globally through
+      ``imagebuilder_scratch``.
+
+    * ``ostree`` -- A dictionary describing where to get ``ostree`` content when
+      applicable. The dictionary contains the following keys:
+
+        * ``url`` -- URL of the repository that's used to fetch the parent
+          commit from.
+        * ``ref`` -- Name of an ostree branch or tag
+
+    * ``blueprint`` -- A dictionary with a blueprint to use for the
+      image build. Blueprints can customize images beyond their initial definition.
+      For the list of supported customizations, see external
+      `Documentation <https://osbuild.org/docs/user-guide/blueprint-reference/>`__
+
+.. note::
+   There is initial support for having this task as failable without aborting
+   the whole compose. This can be enabled by setting ``"failable": ["*"]`` in
+   the config for the image. It is an on/off switch without granularity per
+   arch.
+
+
+Example Config
+--------------
+::
+
+    imagebuilder_target = 'f43-image-builder'
+    imagebuilder_seed = 43
+    imagebuilder_scratch = True
+
+    imagebuilder = {
+        "^IoT$": [
+          {
+            "name": "%s-raw" % release_name,
+            "types": ["iot-raw-xz"],
+            "arches": ["x86_64"], #, "aarch64"],
+            "repos": ["https://kojipkgs.fedoraproject.org/compose/rawhide/latest-Fedora-Rawhide/compose/Everything/$arch/os/"],
+            "ostree": {
+               "url": "https://kojipkgs.fedoraproject.org/compose/iot/repo/",
+               "ref": "fedora/rawhide/$arch/iot",
+            },
+            "subvariant": "IoT",
+            "failable": ["*"],
+          },
+        ]
+    }
+
+
 Image container
 ===============
 

doc/phases.rst

@@ -124,6 +124,12 @@ OSBuild
 Similarly to image build, this phases creates a koji `osbuild` task. In the
 background it uses OSBuild Composer to create images.
 
+ImageBuilder
+------------
+
+Similarly to image build, this phases creates a koji `imageBuilderBuild`
+task. In the background it uses `image-builder` to create images.
+
 OSBS
 ----
 

pungi/checks.py

@@ -1428,6 +1428,57 @@ def make_schema():
                     },
                 },
             },
+            "imagebuilder": {
+                "type": "object",
+                "patternProperties": {
+                    # Warning: this pattern is a variant uid regex, but the
+                    # format does not let us validate it as there is no regular
+                    # expression to describe all regular expressions.
+                    ".+": {
+                        "type": "array",
+                        "items": {
+                            "type": "object",
+                            "properties": {
+                                "name": {"type": "string"},
+                                "target": {"type": "string"},
+                                "arches": {"$ref": "#/definitions/list_of_strings"},
+                                "types": {"$ref": "#/definitions/list_of_strings"},
+                                "version": {"type": "string"},
+                                "repos": {"$ref": "#/definitions/list_of_strings"},
+                                "release": {"type": "string"},
+                                "distro": {"type": "string"},
+                                "scratch": {"type": "boolean"},
+                                "ostree": {
+                                    "type": "object",
+                                    "properties": {
+                                        "parent": {"type": "string"},
+                                        "ref": {"type": "string"},
+                                        "url": {"type": "string"},
+                                    },
+                                },
+                                "failable": {"$ref": "#/definitions/list_of_strings"},
+                                "subvariant": {"type": "string"},
+                                "blueprint": {
+                                    "type": "object",
+                                    "additionalProperties": True,
+                                },
+                                "seed": {"type": "integer"},
+                            },
+                            "required": [
+                                "name",
+                                "types",
+                            ],
+                            "additionalProperties": False,
+                        },
+                    }
+                },
+                "additionalProperties": False,
+            },
+            "imagebuilder_target": {"type": "string"},
+            "imagebuilder_release": {"$ref": "#/definitions/optional_string"},
+            "imagebuilder_version": {"type": "string"},
+            "imagebuilder_seed": {"type": "integer"},
+            "imagebuilder_scratch": {"type": "boolean"},
             "lorax_options": _variant_arch_mapping(
                 {
                     "type": "object",

pungi/phases/__init__.py

@@ -29,6 +29,7 @@ from .image_build import ImageBuildPhase  # noqa
 from .image_container import ImageContainerPhase  # noqa
 from .kiwibuild import KiwiBuildPhase  # noqa
 from .osbuild import OSBuildPhase  # noqa
+from .imagebuilder import ImageBuilderPhase  # noqa
 from .repoclosure import RepoclosurePhase  # noqa
 from .test import TestPhase  # noqa
 from .image_checksum import ImageChecksumPhase  # noqa

pungi/phases/imagebuilder.py

@@ -0,0 +1,263 @@
+# -*- coding: utf-8 -*-
+
+import os
+from kobo.threads import ThreadPool
+from kobo import shortcuts
+from productmd.images import Image
+
+from . import base
+from .. import util
+from ..linker import Linker
+from ..wrappers import kojiwrapper
+from .image_build import EXTENSIONS
+from ..threading import TelemetryWorkerThread as WorkerThread
+
+
+IMAGEBUILDEREXTENSIONS = [
+    ("vagrant-libvirt", ["vagrant.libvirt.box"], "vagrant-libvirt.box"),
+    (
+        "vagrant-virtualbox",
+        ["vagrant.virtualbox.box"],
+        "vagrant-virtualbox.box",
+    ),
+    ("container", ["oci.tar.xz"], "tar.xz"),
+    ("wsl2", ["wsl"], "wsl"),
+    # .iso images can be of many types - boot, cd, dvd, live... -
+    # so 'boot' is just a default guess. 'iso' is not a valid
+    # productmd image type
+    ("boot", [".iso"], "iso"),
+]
+
+
+class ImageBuilderPhase(
+    base.PhaseLoggerMixin, base.ImageConfigMixin, base.ConfigGuardedPhase
+):
+    name = "imagebuilder"
+
+    def __init__(self, compose):
+        super(ImageBuilderPhase, self).__init__(compose)
+        self.pool = ThreadPool(logger=self.logger)
+
+    def _get_arches(self, image_conf, arches):
+        """Get an intersection of arches in the config dict and the given ones."""
+        if "arches" in image_conf:
+            arches = set(image_conf["arches"]) & arches
+        return sorted(arches)
+
+    @staticmethod
+    def _get_repo_urls(compose, repos, arch="$basearch"):
+        """
+        Get list of repos with resolved repo URLs. Preserve repos defined
+        as dicts.
+        """
+        resolved_repos = []
+
+        for repo in repos:
+            repo = util.get_repo_url(compose, repo, arch=arch)
+            if repo is None:
+                raise RuntimeError("Failed to resolve repo URL for %s" % repo)
+            resolved_repos.append(repo)
+
+        return resolved_repos
+
+    def _get_repo(self, image_conf, variant):
+        """
+        Get a list of repos. First included are those explicitly listed in
+        config, followed by by repo for current variant if it's not included in
+        the list already.
+        """
+        repos = shortcuts.force_list(image_conf.get("repos", []))
+
+        if not variant.is_empty and variant.uid not in repos:
+            repos.append(variant.uid)
+
+        return ImageBuilderPhase._get_repo_urls(self.compose, repos, arch="$arch")
+
+    def run(self):
+        for variant in self.compose.get_variants():
+            arches = set([x for x in variant.arches if x != "src"])
+
+            for image_conf in self.get_config_block(variant):
+                build_arches = self._get_arches(image_conf, arches)
+                if not build_arches:
+                    self.log_debug("skip: no arches")
+                    continue
+
+                # these properties can be set per-image *or* as e.g.
+                # imagebuilder_release or global_release in the config
+                generics = {
+                    "release": self.get_release(image_conf),
+                    "target": self.get_config(image_conf, "target"),
+                    "types": self.get_config(image_conf, "types"),
+                    "seed": self.get_config(image_conf, "seed"),
+                    "scratch": self.get_config(image_conf, "scratch"),
+                    "version": self.get_version(image_conf),
+                }
+
+                repo = self._get_repo(image_conf, variant)
+
+                failable_arches = image_conf.pop("failable", [])
+                if failable_arches == ["*"]:
+                    failable_arches = image_conf["arches"]
+
+                self.pool.add(RunImageBuilderThread(self.pool))
+                self.pool.queue_put(
+                    (
+                        self.compose,
+                        variant,
+                        image_conf,
+                        build_arches,
+                        generics,
+                        repo,
+                        failable_arches,
+                    )
+                )
+
+        self.pool.start()
+
+
+class RunImageBuilderThread(WorkerThread):
+    def process(self, item, num):
+        (compose, variant, config, arches, generics, repo, failable_arches) = item
+        self.failable_arches = []
+        # the Koji task as a whole can only fail if *all* arches are failable
+        can_task_fail = set(self.failable_arches).issuperset(set(arches))
+        self.num = num
+        with util.failable(
+            compose,
+            can_task_fail,
+            variant,
+            "*",
+            "imageBuilderBuild",
+            logger=self.pool._logger,
+        ):
+            self.worker(compose, variant, config, arches, generics, repo)
+
+    def worker(self, compose, variant, config, arches, generics, repo):
+        msg = "imageBuilderBuild task for variant %s" % variant.uid
+        self.pool.log_info("[BEGIN] %s" % msg)
+        koji = kojiwrapper.KojiWrapper(compose)
+        koji.login()
+
+        opts = {}
+        opts["repos"] = repo
+
+        if generics.get("release"):
+            opts["release"] = generics["release"]
+
+        if generics.get("seed"):
+            opts["seed"] = generics["seed"]
+
+        if generics.get("scratch"):
+            opts["scratch"] = generics["scratch"]
+
+        if config.get("ostree"):
+            opts["ostree"] = config["ostree"]
+
+        if config.get("blueprint"):
+            opts["blueprint"] = config["blueprint"]
+
+        task_id = koji.koji_proxy.imageBuilderBuild(
+            generics["target"],
+            arches,
+            types=generics["types"],
+            name=config["name"],
+            version=generics["version"],
+            opts=opts,
+        )
+
+        koji.save_task_id(task_id)
+
+        # Wait for it to finish and capture the output into log file.
+        log_dir = os.path.join(compose.paths.log.topdir(), "imageBuilderBuild")
+        util.makedirs(log_dir)
+        log_file = os.path.join(
+            log_dir, "%s-%s-watch-task.log" % (variant.uid, self.num)
+        )
+        if koji.watch_task(task_id, log_file) != 0:
+            raise RuntimeError(
+                "imageBuilderBuild task failed: %s. See %s for details"
+                % (task_id, log_file)
+            )
+
+        # Refresh koji session which may have timed out while the task was
+        # running. Watching is done via a subprocess, so the session is
+        # inactive.
+        koji = kojiwrapper.KojiWrapper(compose)
+
+        linker = Linker(logger=self.pool._logger)
+
+        # Process all images in the build. There should be one for each
+        # architecture, but we don't verify that.
+        paths = koji.get_image_paths(task_id)
+
+        for arch, paths in paths.items():
+            for path in paths:
+                type_, format_ = _find_type_and_format(path)
+                if not format_:
+                    # Path doesn't match any known type.
+                    continue
+
+                # image_dir is absolute path to which the image should be copied.
+                # We also need the same path as relative to compose directory for
+                # including in the metadata.
+                if format_ == "iso":
+                    # If the produced image is actually an ISO, it should go to
+                    # iso/ subdirectory.
+                    image_dir = compose.paths.compose.iso_dir(arch, variant)
+                    rel_image_dir = compose.paths.compose.iso_dir(
+                        arch, variant, relative=True
+                    )
+                else:
+                    image_dir = compose.paths.compose.image_dir(variant) % {
+                        "arch": arch
+                    }
+                    rel_image_dir = compose.paths.compose.image_dir(
+                        variant, relative=True
+                    ) % {"arch": arch}
+                util.makedirs(image_dir)
+
+                filename = os.path.basename(path)
+
+                image_dest = os.path.join(image_dir, filename)
+
+                src_file = compose.koji_downloader.get_file(path)
+
+                linker.link(src_file, image_dest, link_type=compose.conf["link_type"])
+
+                # Update image manifest
+                img = Image(compose.im)
+
+                # If user configured exact type, use it, otherwise try to
+                # figure it out based on the koji output.
+                img.type = config.get("manifest_type", type_)
+                img.format = format_
+                img.path = os.path.join(rel_image_dir, filename)
+                img.mtime = util.get_mtime(image_dest)
+                img.size = util.get_file_size(image_dest)
+                img.arch = arch
+                img.disc_number = 1  # We don't expect multiple disks
+                img.disc_count = 1
+
+                img.bootable = format_ == "iso"
+                img.subvariant = config.get("subvariant", variant.uid)
+                setattr(img, "can_fail", arch in self.failable_arches)
+                setattr(img, "deliverable", "imageBuilderBuild")
+                compose.im.add(variant=variant.uid, arch=arch, image=img)
+
+        self.pool.log_info("[DONE ] %s (task id: %s)" % (msg, task_id))
+
+
+def _find_type_and_format(path):
+    # these are our image-builder-exclusive mappings for images whose extensions
+    # aren't quite the same as imagefactory. they come first as we
+    # want our oci.tar.xz mapping to win over the tar.xz one in
+    # EXTENSIONS
+    for type_, suffixes, format_ in IMAGEBUILDEREXTENSIONS:
+        if any(path.endswith(suffix) for suffix in suffixes):
+            return type_, format_
+    for type_, suffixes in EXTENSIONS.items():
+        for suffix in suffixes:
+            if path.endswith(suffix):
+                return type_, suffix
+    return None, None

pungi/scripts/pungi_koji.py

@@ -430,6 +430,7 @@ def run_compose(
     image_build_phase = pungi.phases.ImageBuildPhase(compose, buildinstall_phase)
     kiwibuild_phase = pungi.phases.KiwiBuildPhase(compose)
     osbuild_phase = pungi.phases.OSBuildPhase(compose)
+    imagebuilder_phase = pungi.phases.ImageBuilderPhase(compose)
     osbs_phase = pungi.phases.OSBSPhase(compose, pkgset_phase, buildinstall_phase)
     image_container_phase = pungi.phases.ImageContainerPhase(compose)
     image_checksum_phase = pungi.phases.ImageChecksumPhase(compose)
@@ -457,6 +458,7 @@ def run_compose(
         osbuild_phase,
         image_container_phase,
         kiwibuild_phase,
+        imagebuilder_phase,
     ):
         if phase.skip():
             continue
@@ -513,6 +515,7 @@ def run_compose(
             livemedia_phase,
             osbuild_phase,
             kiwibuild_phase,
+            imagebuilder_phase,
         )
         compose_images_phase = pungi.phases.WeaverPhase(compose, compose_images_schema)
         extra_phase_schema = (
@@ -543,6 +546,7 @@ def run_compose(
         and livemedia_phase.skip()
         and image_build_phase.skip()
         and kiwibuild_phase.skip()
+        and imagebuilder_phase.skip()
         and osbuild_phase.skip()
         and ostree_container_phase.skip()
     ):

pungi/wrappers/kojiwrapper.py

@@ -531,6 +531,7 @@ class KojiWrapper(object):
                 "createLiveMedia",
                 "createAppliance",
                 "createKiwiImage",
+                "imageBuilderBuildArch",
             ]:
                 continue
 

tests/test_imagebuilderphase.py

@@ -0,0 +1,455 @@
+import os
+from unittest import mock
+
+from pungi.phases.imagebuilder import ImageBuilderPhase, RunImageBuilderThread
+from tests.helpers import DummyCompose, PungiTestCase
+
+
+MINIMAL_CONF = {
+    "types": ["minimal-raw-xz"],
+    "name": "Test",
+}
+
+
+def _merge(a, b):
+    """This would be a | b on 3.9 and later, or {**a, **b} or 3.5 and later."""
+    c = a.copy()
+    c.update(b)
+    return c
+
+
+@mock.patch("pungi.phases.imagebuilder.ThreadPool")
+class TestImageBuilderPhase(PungiTestCase):
+    def test_minimal(self, ThreadPool):
+        cfg = _merge({"target": "f40"}, MINIMAL_CONF)
+        compose = DummyCompose(self.topdir, {"imagebuilder": {"^Server$": [cfg]}})
+
+        self.assertValidConfig(compose.conf)
+
+        phase = ImageBuilderPhase(compose)
+
+        phase.run()
+        phase.pool.add.assert_called()
+        assert phase.pool.queue_put.call_args_list == [
+            mock.call(
+                (
+                    compose,
+                    compose.variants["Server"],
+                    cfg,
+                    ["amd64", "x86_64"],
+                    {
+                        "release": None,
+                        "target": "f40",
+                        "types": ["minimal-raw-xz"],
+                        "seed": None,
+                        "scratch": None,
+                        "version": compose.image_version,
+                    },
+                    [self.topdir + "/compose/Server/$arch/os"],
+                    [],
+                )
+            )
+        ]
+
+    def test_full(self, ThreadPool):
+        cfg = _merge(
+            MINIMAL_CONF,
+            {
+                "target": "f40",
+                "release": "1234",
+                "arches": ["x86_64"],
+                "repos": ["https://example.com/repo/", "Client"],
+                "types": ["custom"],
+                "version": "Rawhide",
+            },
+        )
+        compose = DummyCompose(self.topdir, {"imagebuilder": {"^Server$": [cfg]}})
+
+        self.assertValidConfig(compose.conf)
+
+        phase = ImageBuilderPhase(compose)
+
+        phase.run()
+        phase.pool.add.assert_called()
+
+        assert phase.pool.queue_put.call_args_list == [
+            mock.call(
+                (
+                    compose,
+                    compose.variants["Server"],
+                    cfg,
+                    ["x86_64"],
+                    {
+                        "release": "1234",
+                        "target": "f40",
+                        "types": ["custom"],
+                        "seed": None,
+                        "scratch": None,
+                        "version": "Rawhide",
+                    },
+                    [
+                        "https://example.com/repo/",
+                        self.topdir + "/compose/Client/$arch/os",
+                        self.topdir + "/compose/Server/$arch/os",
+                    ],
+                    [],
+                )
+            )
+        ]
+
+    def test_failable(self, ThreadPool):
+        cfg = _merge({"target": "f40", "failable": ["x86_64"]}, MINIMAL_CONF)
+        compose = DummyCompose(self.topdir, {"imagebuilder": {"^Server$": [cfg]}})
+
+        self.assertValidConfig(compose.conf)
+
+        phase = ImageBuilderPhase(compose)
+
+        phase.run()
+        phase.pool.add.assert_called()
+        assert phase.pool.queue_put.call_args_list == [
+            mock.call(
+                (
+                    compose,
+                    compose.variants["Server"],
+                    cfg,
+                    ["amd64", "x86_64"],
+                    {
+                        "release": None,
+                        "target": "f40",
+                        "types": ["minimal-raw-xz"],
+                        "seed": None,
+                        "scratch": None,
+                        "version": compose.image_version,
+                    },
+                    [self.topdir + "/compose/Server/$arch/os"],
+                    ["x86_64"],
+                )
+            )
+        ]
+
+    def test_with_phase_opts(self, ThreadPool):
+        cfg = {"name": "Test", "types": ["minimal-raw-xz"]}
+        compose = DummyCompose(
+            self.topdir,
+            {
+                "imagebuilder": {"^Server$": [cfg]},
+                "imagebuilder_target": "f40",
+                "imagebuilder_release": "1234",
+                "imagebuilder_version": "Rawhide",
+            },
+        )
+
+        self.assertValidConfig(compose.conf)
+
+        phase = ImageBuilderPhase(compose)
+
+        phase.run()
+        phase.pool.add.assert_called()
+        assert phase.pool.queue_put.call_args_list == [
+            mock.call(
+                (
+                    compose,
+                    compose.variants["Server"],
+                    cfg,
+                    ["amd64", "x86_64"],
+                    {
+                        "release": "1234",
+                        "target": "f40",
+                        "types": ["minimal-raw-xz"],
+                        "seed": None,
+                        "scratch": None,
+                        "version": "Rawhide",
+                    },
+                    [self.topdir + "/compose/Server/$arch/os"],
+                    [],
+                )
+            )
+        ]
+
+    def test_with_global_opts(self, ThreadPool):
+        cfg = MINIMAL_CONF
+        compose = DummyCompose(
+            self.topdir,
+            {
+                "imagebuilder": {"^Server$": [cfg]},
+                "global_target": "f40",
+                "global_release": "1234",
+                "global_version": "41",
+            },
+        )
+
+        self.assertValidConfig(compose.conf)
+
+        phase = ImageBuilderPhase(compose)
+
+        phase.run()
+        phase.pool.add.assert_called()
+        assert phase.pool.queue_put.call_args_list == [
+            mock.call(
+                (
+                    compose,
+                    compose.variants["Server"],
+                    cfg,
+                    ["amd64", "x86_64"],
+                    {
+                        "release": "1234",
+                        "target": "f40",
+                        "types": ["minimal-raw-xz"],
+                        "seed": None,
+                        "scratch": None,
+                        "version": "41",
+                    },
+                    [self.topdir + "/compose/Server/$arch/os"],
+                    [],
+                )
+            )
+        ]
+
+
+@mock.patch("pungi.phases.imagebuilder.Linker")
+@mock.patch("pungi.util.get_mtime")
+@mock.patch("pungi.util.get_file_size")
+@mock.patch("pungi.wrappers.kojiwrapper.KojiWrapper")
+class TestImageBuilderThread(PungiTestCase):
+    def _img_path(self, arch, filename=None, dir=None):
+        dir = dir or "images"
+        path = self.topdir + "/compose/Server/%s/%s" % (arch, dir)
+        if filename:
+            path += "/" + filename
+        return path
+
+    def test_process_vagrant_box(self, KojiWrapper, get_file_size, get_mtime, Linker):
+        img_name = "FCBG.{arch}-Rawhide-1.6.vagrant.libvirt.box"
+        self.repo = self.topdir + "/compose/Server/$arch/os"
+        compose = DummyCompose(
+            self.topdir,
+            {
+                "koji_profile": "koji",
+            },
+        )
+        config = _merge({"subvariant": "Test"}, MINIMAL_CONF)
+        pool = mock.Mock()
+
+        get_image_paths = KojiWrapper.return_value.get_image_paths
+        get_image_paths.return_value = {
+            "x86_64": [
+                "/koji/task/1234/FCBG.x86_64-Rawhide-1.6.packages",
+                "/koji/task/1234/%s" % img_name.format(arch="x86_64"),
+            ],
+            "amd64": [
+                "/koji/task/1234/FCBG.amd64-Rawhide-1.6.packages",
+                "/koji/task/1234/%s" % img_name.format(arch="amd64"),
+            ],
+        }
+
+        KojiWrapper.return_value.koji_proxy.imageBuilderBuild.return_value = 1234
+        KojiWrapper.return_value.watch_task.return_value = 0
+
+        t = RunImageBuilderThread(pool)
+        get_file_size.return_value = 1024
+        get_mtime.return_value = 13579
+        t.process(
+            (
+                compose,
+                compose.variants["Server"],
+                config,
+                ["amd64", "x86_64"],
+                {
+                    "release": "1.6",
+                    "target": "f40",
+                    "types": ["t"],
+                    "version": "v",
+                },
+                [self.repo],
+                [],
+            ),
+            1,
+        )
+
+        assert KojiWrapper.return_value.koji_proxy.imageBuilderBuild.mock_calls == [
+            mock.call(
+                "f40",
+                ["amd64", "x86_64"],
+                types=["t"],
+                name="Test",
+                version="v",
+                opts={
+                    "repos": [self.repo],
+                    "release": "1.6",
+                },
+            )
+        ]
+
+        assert get_image_paths.mock_calls == [mock.call(1234)]
+        assert os.path.isdir(self._img_path("x86_64"))
+        assert os.path.isdir(self._img_path("amd64"))
+        Linker.return_value.link.assert_has_calls(
+            [
+                mock.call(
+                    "/koji/task/1234/FCBG.amd64-Rawhide-1.6.vagrant.libvirt.box",
+                    self._img_path("amd64", img_name.format(arch="amd64")),
+                    link_type="hardlink-or-copy",
+                ),
+                mock.call(
+                    "/koji/task/1234/FCBG.x86_64-Rawhide-1.6.vagrant.libvirt.box",
+                    self._img_path("x86_64", img_name.format(arch="x86_64")),
+                    link_type="hardlink-or-copy",
+                ),
+            ],
+            any_order=True,
+        )
+
+        assert len(compose.im.add.call_args_list) == 2
+        for call in compose.im.add.call_args_list:
+            _, kwargs = call
+            image = kwargs["image"]
+            expected_path = "Server/{0.arch}/images/{1}".format(
+                image, img_name.format(arch=image.arch)
+            )
+            assert kwargs["variant"] == "Server"
+            assert kwargs["arch"] in ("amd64", "x86_64")
+            assert kwargs["arch"] == image.arch
+            assert image.path == expected_path
+            assert "vagrant-libvirt.box" == image.format
+            assert "vagrant-libvirt" == image.type
+            assert "Test" == image.subvariant
+            assert not image.bootable
+
+    def test_process_iso(self, KojiWrapper, get_file_size, get_mtime, Linker):
+        img_name = "FCBG.{arch}-Rawhide-1.6.iso"
+        self.repo = self.topdir + "/compose/Server/$arch/os"
+        compose = DummyCompose(
+            self.topdir,
+            {
+                "koji_profile": "koji",
+            },
+        )
+        config = _merge({"subvariant": "Test", "name": "Test"}, MINIMAL_CONF)
+        pool = mock.Mock()
+
+        get_image_paths = KojiWrapper.return_value.get_image_paths
+        get_image_paths.return_value = {
+            "x86_64": [
+                "/koji/task/1234/FCBG.x86_64-Rawhide-1.6.packages",
+                "/koji/task/1234/%s" % img_name.format(arch="x86_64"),
+            ],
+            "amd64": [
+                "/koji/task/1234/FCBG.amd64-Rawhide-1.6.packages",
+                "/koji/task/1234/%s" % img_name.format(arch="amd64"),
+            ],
+        }
+
+        KojiWrapper.return_value.koji_proxy.imageBuilderBuild.return_value = 1234
+        KojiWrapper.return_value.watch_task.return_value = 0
+
+        t = RunImageBuilderThread(pool)
+        get_file_size.return_value = 1024
+        get_mtime.return_value = 13579
+        t.process(
+            (
+                compose,
+                compose.variants["Server"],
+                config,
+                ["amd64", "x86_64"],
+                {
+                    "release": "1.6",
+                    "target": "f40",
+                    "types": ["t"],
+                    "version": "v",
+                },
+                [self.repo],
+                [],
+            ),
+            1,
+        )
+
+        assert KojiWrapper.return_value.koji_proxy.imageBuilderBuild.mock_calls == [
+            mock.call(
+                "f40",
+                ["amd64", "x86_64"],
+                types=["t"],
+                name="Test",
+                version="v",
+                opts={
+                    "repos": [self.repo],
+                    "release": "1.6",
+                },
+            )
+        ]
+
+        assert get_image_paths.mock_calls == [mock.call(1234)]
+        assert os.path.isdir(self._img_path("x86_64", dir="iso"))
+        assert os.path.isdir(self._img_path("amd64", dir="iso"))
+        Linker.return_value.link.assert_has_calls(
+            [
+                mock.call(
+                    "/koji/task/1234/FCBG.amd64-Rawhide-1.6.iso",
+                    self._img_path("amd64", img_name.format(arch="amd64"), dir="iso"),
+                    link_type="hardlink-or-copy",
+                ),
+                mock.call(
+                    "/koji/task/1234/FCBG.x86_64-Rawhide-1.6.iso",
+                    self._img_path("x86_64", img_name.format(arch="x86_64"), dir="iso"),
+                    link_type="hardlink-or-copy",
+                ),
+            ],
+            any_order=True,
+        )
+
+        assert len(compose.im.add.call_args_list) == 2
+        for call in compose.im.add.call_args_list:
+            _, kwargs = call
+            image = kwargs["image"]
+            expected_path = "Server/{0.arch}/iso/{1}".format(
+                image, img_name.format(arch=image.arch)
+            )
+            assert kwargs["variant"] == "Server"
+            assert kwargs["arch"] in ("amd64", "x86_64")
+            assert kwargs["arch"] == image.arch
+            assert image.path == expected_path
+            assert "iso" == image.format
+            assert "boot" == image.type
+            assert image.bootable
+            assert "Test" == image.subvariant
+
+    def test_handle_koji_fail(self, KojiWrapper, get_file_size, get_mtime, Linker):
+        self.repo = self.topdir + "/compose/Server/$arch/os"
+        compose = DummyCompose(self.topdir, {"koji_profile": "koji"})
+        config = MINIMAL_CONF
+        pool = mock.Mock()
+
+        get_image_paths = KojiWrapper.return_value.get_image_paths
+
+        KojiWrapper.return_value.koji_proxy.imageBuilderBuild.return_value = 1234
+        KojiWrapper.return_value.watch_task.return_value = 1
+
+        t = RunImageBuilderThread(pool)
+        try:
+            t.process(
+                (
+                    compose,
+                    compose.variants["Server"],
+                    config,
+                    ["amd64", "x86_64"],
+                    {
+                        "release": "1.6",
+                        "target": "f40",
+                        "types": ["minimal-raw-xz"],
+                        "version": None,
+                    },
+                    [self.repo],
+                    [],
+                ),
+                1,
+            )
+            assert False, "Exception should have been raised"
+        except RuntimeError:
+            pass
+
+        assert (
+            len(KojiWrapper.return_value.koji_proxy.imageBuilderBuild.mock_calls) == 1
+        )
+        assert get_image_paths.mock_calls == []
+        assert Linker.return_value.link.mock_calls == []
+        assert len(compose.im.add.call_args_list) == 0