lorax-composer: Update documentation

This updates the documentation to include composer-cli instead of curl,
warnings about SELinux and needing the RHEL 7 Optional package repo, as
well as other documentation improvements from the master branch.

docs/composer-cli.rst

@@ -0,0 +1,62 @@
+composer-cli
+============
+
+:Authors:
+    Brian C. Lane <bcl@redhat.com>
+
+``composer-cli`` is used to interact with the ``lorax-composer`` API server, managing blueprints, exploring available packages, and building new images.
+
+It requires `lorax-composer <lorax-composer.html>`_ to be installed on the
+local system, and the user running it needs to be a member of the ``weldr``
+group. They do not need to be root, but all of the `security precautions
+<lorax-composer.html#security>`_ apply.
+
+composer-cli cmdline arguments
+------------------------------
+
+.. argparse::
+   :ref: composer.cli.cmdline.composer_cli_parser
+   :prog: composer-cli
+
+Edit a Blueprint
+----------------
+
+Start out by listing the available blueprints using ``composer-cli blueprints
+list``, pick one and save it to the local directory by running ``composer-cli
+blueprints save http-server``. If there are no blueprints available you can
+copy one of the examples `from the test suite
+<https://github.com/weldr/lorax/tree/master/tests/pylorax/blueprints/>`_.
+
+Edit the file (it will be saved with a .toml extension) and change the
+description, add a package or module to it. Send it back to the server by
+running ``composer-cli blueprints push http-server.toml``. You can verify that it was
+saved by viewing the changelog - ``composer-cli blueprints changes http-server``.
+
+Build an image
+----------------
+
+Build a ``qcow2`` disk image from this blueprint by running ``composer-cli
+compose start http-server qcow2``. It will print a UUID that you can use to
+keep track of the build. You can also cancel the build if needed.
+
+The available types of images is displayed by ``composer-cli compose types``.
+Currently this consists of: ext4-filesystem, live-iso, partitioned-disk, qcow2,
+tar
+
+Monitor the build status
+------------------------
+
+Monitor it using ``composer-cli compose status``, which will show the status of
+all the builds on the system. You can view the end of the anaconda build logs
+once it is in the ``RUNNING`` state using ``composer-cli compose log UUID``
+where UUID is the UUID returned by the start command.
+
+Once the build is in the ``FINISHED`` state you can download the image.
+
+Download the image
+------------------
+
+Downloading the final image is done with ``composer-cli compose image UUID`` and it will
+save the qcow2 image as ``UUID-disk.qcow2`` which you can then use to boot a VM like this::
+
+    qemu-kvm --name test-image -m 1024 -hda ./UUID-disk.qcow2

docs/index.rst

@@ -15,6 +15,7 @@ Contents:
    lorax
    livemedia-creator
    lorax-composer
+   composer-cli
    product-images
    modules
 

docs/lorax-composer.rst

@@ -4,29 +4,37 @@ lorax-composer
 :Authors:
     Brian C. Lane <bcl@redhat.com>
 
-lorax-composer is an API server that is compatible with the Weldr project's
+``lorax-composer`` is an API server that allows you to build disk images using
-bdcs-api REST protocol. More information on Weldr can be found `on the Weldr
+`Blueprints`_ to describe the package versions to be installed into the image.
-blog <http://www.weldr.io>`_.
+It is compatible with the Weldr project's bdcs-api REST protocol. More
+information on Weldr can be found `on the Weldr blog <http://www.weldr.io>`_.
 
-The server runs as root, and communication with it is via a unix domain socket
+Behind the scenes it uses `livemedia-creator <livemedia-creator.html>`_ and
-(``/run/weldr/api.socket`` by default). The directory and socket are owned by
+`Anaconda <https://anaconda-installer.readthedocs.io/en/latest/>`_ to handle the
-root:weldr so that any user in the weldr group can use the API to control
+installation and configuration of the images.
-lorax-composer.
 
-When starting the server it will check for the correct permissions and
+Important Things To Note
-ownership of a pre-existing directory, or it will create a new one if it
+------------------------
-doesn't exist.  The socket path and group owner's name can be changed from the
-cmdline by passing it the ``--socket`` and ``--group`` arguments.
 
-As of version 19.7.7 it will drop root privileges for the API thread. The queue
+* SELinux must be in Permissive mode. Anaconda requires SELinux be in permissive mode
-and compose thread still runs as root because it needs to be able to
+  for image creation to work correctly. You can either edit the setting in the
-mount/umount files and run Anaconda.
+  ``/etc/sysconfig/selinux`` file, or run ``setenforce 0`` before starting lorax-composer.
 
-Logs
+* Some output types require packages from the RHEL 7 Optional repository. See the
-----
+  `Red Hat Enterprise Linux 7 documentation <https://access.redhat.com/solutions/392003>`_
+  for information on how to enable it. Otherwise you will see image creation fail to
+  depsolve even if the blueprint itself is correct.
 
-Logs are stored under ``/var/log/lorax-composer/`` and include all console
+Installation
-messages as well as extra debugging info and API requests.
+------------
+
+The best way to install ``lorax-composer`` is to use ``sudo dnf install
+lorax-composer composer-cli``, this will setup the weldr user and install the
+systemd socket activation service. You will then need to enable it with ``sudo
+systemctl enable lorax-composer.socket && sudo systemctl start
+lorax-composer.socket``. This will leave the server off until the first request
+is made. Systemd will then launch the server and it will remain running until
+the system is rebooted.
 
 Quickstart
 ----------
@@ -34,31 +42,162 @@ Quickstart
 1. Create a ``weldr`` user and group by running ``useradd weldr``
 2. Remove any pre-existing socket directory with ``rm -rf /run/weldr/``
    A new directory with correct permissions will be created the first time the server runs.
-3. Either start it via systemd with ``systemctl start lorax-composer`` or
+3. Enable the socket activation with ``systemctl enable lorax-composer.socket
-   run it directly with ``lorax-composer /path/to/blueprints/``
+   && sudo systemctl start lorax-composer.socket`` or run it directly with
+   ``lorax-composer /path/to/blueprints/``
 
-The ``/path/to/blueprints/`` is where the blueprint's git repo will be created, and
+The ``/path/to/blueprints/`` directory is where the blueprints' git repo will
-all the blueprints created with the ``/api/v0/blueprints/new`` route will be stored.
+be created, and all the blueprints created with the ``/api/v0/blueprints/new``
-If there are blueprint ``.toml`` files in the top level of the directory they will
+route will be stored.  If there are blueprint ``.toml`` files in the top level
-be imported into the blueprint git storage.
+of the directory they will be imported into the blueprint git storage when
+``lorax-composer`` starts.
+
+Logs
+----
+
+Logs are stored under ``/var/log/lorax-composer/`` and include all console
+messages as well as extra debugging info and API requests.
+
+Security
+--------
+
+Some security related issues that you should be aware of before running ``lorax-composer``:
+
+* One of the API server threads needs to retain root privileges in order to run Anaconda.
+* SELinux must be set to Permissive or disabled to allow ``livemedia-creator`` to run Anaconda.
+* Only allow authorized users access to the ``weldr`` group and socket.
+
+Since Anaconda kickstarts are used there is the possibility that a user could
+inject commands into a blueprint that would result in the kickstart executing
+arbitrary code on the host.  Only authorized users should be allowed to build
+images using ``lorax-composer``.
+
+How it Works
+------------
+
+The server runs as root, and as ``weldr``. Communication with it is via a unix
+domain socket (``/run/weldr/api.socket`` by default). The directory and socket
+are owned by ``root:weldr`` so that any user in the ``weldr`` group can use the API
+to control ``lorax-composer``.
+
+At startup the server will check for the correct permissions and
+ownership of a pre-existing directory, or it will create a new one if it
+doesn't exist.  The socket path and group owner's name can be changed from the
+cmdline by passing it the ``--socket`` and ``--group`` arguments.
+
+It will then drop root privileges for the API thread and run as the ``weldr``
+user. The queue and compose thread still runs as root because it needs to be
+able to mount/umount files and run Anaconda.
 
 Composing Images
 ----------------
 
-As of version 19.7.7 lorax-composer can create ``tar`` output images. You can use curl to start
+The `welder-web <https://github.com/weldr/welder-web/>`_ GUI project can be used to construct
-a compose like this::
+blueprints and create composes using a web browser.
 
-    curl --unix-socket /run/weldr/api.socket -X POST -H "Content-Type: application/json" -d '{"blueprint_name": "http-server", "compose_type": "tar", "branch": "master"}' http:///api/v0/compose
+Or use the command line with `composer-cli <composer-cli.html>`_.
 
-And then monitor it by passing the returned build UUID to ``/compose/status/<uuid>``.
+Blueprints
+----------
+
+Blueprints are simple text files in `TOML <https://github.com/toml-lang/toml>`_ format that describe
+which packages, and what versions, to install into the image. They can also define a limited set
+of customizations to make to the final image.
+
+Example blueprints can be found in the ``lorax-composer`` `test suite
+<https://github.com/weldr/lorax/tree/master/tests/pylorax/blueprints/>`_, with a simple one
+looking like this::
+
+    name = "base"
+    description = "A base system with bash"
+    version = "0.0.1"
+
+    [[packages]]
+    name = "bash"
+    version = "4.4.*"
+
+The ``name`` field is the name of the blueprint. It can contain spaces, but they will be converted to ``-``
+when it is written to disk. It should be short and descriptive.
+
+``description`` can be a longer description of the blueprint, it is only used for display purposes.
+
+``version`` is a `semver compatible <https://semver.org/>`_ version number. If
+a new blueprint is uploaded with the same ``version`` the server will
+automatically bump the PATCH level of the ``version``. If the ``version``
+doesn't match it will be used as is. eg. Uploading a blueprint with ``version``
+set to ``0.1.0`` when the existing blueprint ``version`` is ``0.0.1`` will
+result in the new blueprint being stored as ``version 0.1.0``.
+
+[[packages]] and [[modules]]
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+These entries describe the package names and matching version glob to be installed into the image.
+
+The names must match the names exactly, and the versions can be an exact match
+or a filesystem-like glob of the version using ``*`` wildcards and ``?``
+character matching.
+
+NOTE: As of lorax-composer-29.2-1 the versions are not used for depsolving,
+that is planned for a future release. And currently there are no differences
+between ``packages`` and ``modules`` in ``lorax-composer``.
+
+Customizations
+~~~~~~~~~~~~~~
+
+The ``[[customizations]]`` section can be used to configure the hostname of the final image. eg.::
+
+    [[customizations]]
+    hostname = "baseimage"
+
+
+[[customizations.sshkey]]
+*************************
+
+Set an existing user's ssh key in the final image::
+
+    [[customizations.sshkey]]
+    user = "root"
+    key = "PUBLIC SSH KEY"
+
+The key will be added to the user's authorized_keys file.
+
+
+[[customizations.user]]
+***********************
+
+Add a user to the image, and/or set their ssh key.
+All fields for this section are optional except for the ``name``, here is a complete example::
+
+    [[customizations.user]]
+    name = "admin"
+    description = "Administrator account"
+    password = "$6$CHO2$3rN8eviE2t50lmVyBYihTgVRHcaecmeCk31L..."
+    key = "PUBLIC SSH KEY"
+    home = "/srv/widget/"
+    shell = "/usr/bin/bash"
+    groups = ["widget", "users", "wheel"]
+    uid = 1200
+    gid = 1200
+
+If the password starts with ``$6$``, ``$5$``, or ``$2b$`` it will be stored as
+an encrypted password. Otherwise it will be treated as a plain text password.
+
+
+[[customizations.group]]
+************************
+
+Add a group to the image. ``name`` is required and ``gid`` is optional::
+
+    [[customizations.group]]
+    name = "widget"
+    gid = 1130
 
-Version 19.7.10 adds support for ``live-iso`` and ``partitioned-disk``
 
 Adding Output Types
 -------------------
 
-livemedia-creator supports a large number of output types, and only some of
+``livemedia-creator`` supports a large number of output types, and only some of
-these are currently available via lorax-composer. To add a new output type to
+these are currently available via ``lorax-composer``. To add a new output type to
 lorax-composer a kickstart file needs to be added to ``./share/composer/``. The
 name of the kickstart is what will be used by the ``/compose/types`` route, and the
 ``compose_type`` field of the POST to start a compose. It also needs to have