Commit Graph

981 Commits

Author SHA1 Message Date
Brian C. Lane 1b84f90963 lifted directories should be under share_dir and lib_dir
Otherwise passing --sharedir pointed to some other path will not use the
correct providers.
2019-10-16 09:17:53 -07:00
Brian C. Lane 3a453eaad7 Make sure V0 API doesn't return uploads information
uploads should only be included in the V1 API routes, add `api`
selection to the relevant helper functions and calls to them from v0.py

Add new V1 routes with api=1 to include the uploads information in the
results.

Also add tests to ensure that V0 requests do not include uploads.
2019-10-16 09:17:53 -07:00
Evan Goode 2692e8138c Automatically upload composed images to the cloud
Currently, Azure, vSphere, and OpenStack are supported. See
https://github.com/weldr/lorax/pull/826 for more details about this new
feature.

I've called the upload library "lifted" as a reference to Seuss'
The Lorax -- in the book, the Lorax lifts himself up by the seat of his
pants through a hole in the smog clouds, and they start calling him the
"Lifted Lorax."

This adds new features to the /compose route under API v1
2019-10-16 09:17:53 -07:00
Evan Goode 584a2d2076 Add load and dump to pylorax.api.toml 2019-10-16 09:17:53 -07:00
Brian C. Lane 697233c14a lorax-composer: Handle RecipeError in commit_recipe_directory
A recipe that is valid TOML can still be an invalid recipe (eg. missing
the 'name' field) so this should also catch RecipeError.

Also added tests for this, as well as making sure commit_recipe_file()
raises the correct errors.

Resolves: rhbz#1755068
2019-10-04 08:07:36 -07:00
Greg Turner 66cd1fdb52 creator.py: include dmsquash-live-ntfs by default
Add dmsquash-live-ntfs to the default dracut modules in livecd builds.

The omission of this is probably why:

  https://bugzilla.redhat.com/show_bug.cgi?id=1449410

didn't work out as intended (I suspect it was closed unfixed).

Without this, initramfs winds up with the ntfs-3g tools lying
around, but no hooks cause them to actually get used.

AFAICT, this is a legit bugfix, not a "proposed enhancement"
so to speak.  To test that this actually fixed the problem (well,
/a/ problem, at least), I used some scripts:

  https://github.com/gmt/test-respin-fedora-ntfsable-iso

Signed-off-by: Greg Turner <gmt@be-evil.net>
Signed-off-by: Brian C. Lane <bcl@redhat.com>
2019-09-04 11:06:26 -07:00
Brian C. Lane 6f686ff9d6 lorax-composer: Add support for dnf variables to repo sources
This loads the system dnf vars from /etc/dnf/vars at startup if
system repos are enabled, and it substitutes the values in the sources
when loaded, and when a new source is added.

Also includes tests.
2019-08-20 16:26:03 -07:00
Brian C. Lane 1e88a99443 lorax-composer: Add v1 API for projects/source/
This changes the source 'name' field to match the DNF usage of it as a
descriptive string. 'id' is now used as the short name to refer to the
source. The v0 API remains unchanged.

Tests for v1 behavior have been added.

Now that the v1 API is in use the status message will return api: 1
2019-08-07 13:26:35 -07:00
Brian C. Lane 278214ff8e Add /api/v1/ handler with no routes 2019-08-06 16:08:10 -07:00
Brian C. Lane f5bb2dca8c Move common functions into pylorax.api.utils
This is so that they can be used from multiple versions of the API. eg.
v0.py, v1.py, etc.
2019-08-06 15:44:36 -07:00
Brian C. Lane b9d18216ef lorax-composer: Add liveimg-tar image type
This creates a tar suitable for use with the anaconda kickstart liveimg
command. It adds the kernel, grub2, and grub2-tools packages to the tar
template.
2019-08-05 14:50:32 -07:00
Brian C. Lane 361c6de2de livemedia-creator: Use --compress-arg in mksquashfs
Previously the --compress-arg option was only used for tar, this adds
support for it to the squashfs.img creation used with live isos.
2019-08-05 14:49:25 -07:00
Brian C. Lane babf823b1b livemedia-creator: Remove unused --squashfs_args option 2019-08-05 14:49:25 -07:00
Brian C. Lane f4a81d6c26 lorax-composer: Add squashfs_only False to all image types
New livemedia-creator flag needs to be set to the default.
2019-07-29 13:21:25 -07:00
Brian C. Lane 1796489839 livemedia-creator: Use make_runtime for all runtime creation 2019-07-29 13:21:25 -07:00
Brian C. Lane cb91fa3c78 livemedia-creator: Add support for a squashfs only runtime image
Normally the runtime image is an ext4 filesystem image that is
compressed with squashfs. dracut now supports setting up an overlayfs
when it detects a bare filesystem tree inside the squashfs.

This commit adds support for a --squashfs-only option which places the
root tree directly in the squashfs.img
2019-07-29 13:21:25 -07:00
Brian C. Lane 0708302464 Use binary mode to tail the file
In python 3 f.seek() on text doesn't work like it does in py2/C because
text is now unicode. So change read_tail to use byte mode and take
unicode into account. Also add tests for it.
2019-07-11 15:16:41 -07:00
Evan Goode 90626f97b6 Return most relevant log file from /compose/log
Return anaconda.log if anaconda is running, composer otherwise

Return packaging.log if it's at least 15s newer than anaconda.log
2019-07-11 15:16:41 -07:00
Pat Riehecky 2c141950b0 lorax: Add debug log of command line options 2019-07-09 17:22:27 -07:00
Pat Riehecky 1f811523b5 lorax: provide runtime lorax config in debug log 2019-07-09 17:20:39 -07:00
Jacob Kozol 6b55c8f8f0 Remove whitespace in v0_blueprints_new 2019-07-09 22:53:51 +02:00
Jacob Kozol 4174186c14 Add seperate validation for blueprint names
The VALID_API_STRING function allows for characters that should not be
allowed in blueprint names. VALID_BLUEPRINT_NAME allows us to
specifically check if a blueprint contains a valid name.
2019-07-09 22:53:51 +02:00
Lars Karlitski fd57e1106e tests: set skip_if_unavailable in test repos
dnf seems to have changed the default for skip_if_unavailable. Some
mock repositories are still around in later tests, which then fail
because metadata cannot be synced.

Also expose skip_if_unavailable in dnf_repo_to_file_repo(), so that
tests checking for equality of repo files continue to pass.
2019-07-09 00:23:33 +02:00
Bastien Nocera eb7f528071 Also search for pxeboot kernel and initrd pairs
As used in the Fedora 30 aarch64 Server DVDs.

Closes: #786
2019-06-28 10:38:38 -07:00
Evan Goode fef76930bc More descriptive error for a bad ref in repos.git
Fixes #771
2019-06-28 08:55:36 -07:00
Brian C. Lane 13a7dcf2b1 lorax-composer: Add basic case check to check_recipe_dict
This will detect if an expected field is not all lower case as is
required.
2019-06-25 11:44:58 +02:00
Brian C. Lane 61059a2699 lorax-composer: Add basic recipe checker function
This makes sure that required fields are included, and that sections are
not empty. It does not check for all optional fields.

If there are errors it will gather up all of them and then raise a
RecipeError with a string of all the errors.
2019-06-25 11:44:58 +02:00
Lars Karlitski 61fc4d2b4e test: Fix test_blueprint_sanity
The new toml library, introduced with abe7df34f, outputs different
whitespace from the old one. Fix the test expectation and strip()
results from toml.dumps(), because it contains superfluous newlines at
the end.
2019-06-24 23:43:08 +02:00
Brian C. Lane abe7df34fc Switch to new toml library
The previous library, pytoml from https://github.com/avakar/pytoml is no
longer supported. So this adds a compatibility layer on top of the
suggested replacement, toml from https://github.com/uiri/toml
2019-06-18 14:01:15 -07:00
Brian C. Lane 932ff5812c composer-cli: Update diff support for customizations and repos.git
composer-cli will now output information about changes to customizations
entries and the repos.git entries.
2019-06-13 14:16:48 -07:00
Brian C. Lane 9011a564e8 Add support for customizations and repos.git to /blueprints/diff/
This also includes extensive tests for each of the currently supported
customizations. It should be generic enough to continue working as long
as the list of dicts includes a 'name' or 'user' field in the dict.
Otherwise support for a new dict key will need to be added to the
customizations_diff function.
2019-06-13 14:16:48 -07:00
Brian C. Lane ffc3432cf2 Move the v0 API documentation into the functions 2019-06-13 11:28:57 -07:00
Brian C. Lane 6d50a5874e Update the /api/v0/ route handling to use the flask_blueprints Blueprint class
Instead of setting up the routes inside a function we can now use a
BlueprintSkip class, which allows us to register them at different
routes (eg. /api/v0/ and /api/v1/) and override any routes that will be
replaced by the new API version.
2019-06-13 11:28:57 -07:00
Brian C. Lane 7071e62985 Extend Flask Blueprint class to allow skipping routes
When adding a new API we want to use the old code for any routes that
aren't being overridden.

This modifies the Flask Blueprint class so that a skip_rules list can be
passed to server.register_blueprint()
2019-06-13 11:28:57 -07:00
Lars Karlitski 4bd03cb8f6 Don't send CORS headers
These are meant for web applications that are accessed by browsers, not
REST APIs.
2019-06-04 20:10:03 +02:00
Lars Karlitski 8ed910b29a composer: Set up a custom HTTP error handler
Override flask's default error handler, because that return html. Return
JSON instead with the usual { "status": false, "errors": [ ... ] }
pattern.
2019-06-04 11:50:25 +03:00
Brian C. Lane d4d050496b Replace isoinfo with pycdlib
isoinfo is part of genisoimage, which we no longer use, switch to using
a python library to read the label from the iso.
2019-05-07 11:22:02 -07:00
Brian C. Lane 3e08389a0f Change customizations.firewall to append items instead of replace
To maintain consistency with the other options this changes firewall to
combine the existing settings from the image template with the settings
from the blueprint.

Also updated the docs, added a new test for it, and sorted the output
for consistency.
2019-05-02 16:21:36 -07:00
Brian C. Lane 1111aee92d lorax-composer: Add services support to blueprints
Add support for enabling and disabling systemd services in the
blueprint. It works like this:

    [customizations.services]
    enabled = ["sshd", "cockpit.socket", "httpd"]
    disabled = ["postfix", "telnetd"]

They are *added* to any existing settings in the kickstart templates.
2019-05-02 16:21:36 -07:00
Brian C. Lane 4d35668ab5 lorax-composer: Add firewall support to blueprints
You can now open ports in the firewall, using port numbers or service
names:

    [customizations.firewall]
    ports = ["22:tcp", "80:tcp", "imap:tcp", "53:tcp", "53:udp"]

Or enable/disable services registered with firewalld:

     [customizations.firewall.services]
     enabled = ["ftp", "ntp", "dhcp"]
     disabled = ["telnet"]

If the template contains firewall --disabled it cannot be overridden,
under the assumption that it is required for the image to boot in the
selected environment.
2019-05-02 16:21:36 -07:00
Brian C. Lane e5a8700bdf lorax-composer: Add locale support to blueprints
You can now set the keyboard layout and language. Eg.

[customizations.locale]
languages = ["en_CA.utf8", "en_HK.utf8"]
keyboard = "de (dvorak)"

Existing entries in the kickstart templates are replaced with the new
ones. If there are no entries then it will default to 'keyboard us' and
'lang en_US.UTF-8'

Includes tests, and leaves the existing keyboard and lang entries in the
templates with a note that they can be replaced by the blueprint.
2019-05-02 16:21:36 -07:00
Brian C. Lane 67007dfa60 lorax-composer: Fix customizations when creating a recipe
This fixes the customizations list problem earlier than in
add_customizations.

In the recipe it should be [customizations] not [[customizations]]
which creates a list. If it was used that way grab the first element and
replace the list with it.
2019-05-02 16:21:36 -07:00
Brian C. Lane 9bdbb29662 lorax-composer: Add timezone support to blueprint
For example:

[customizations.timezone]
timezone = "US/Samoa"
ntpservers = ["0.pool.ntp.org"]

Also includes tests.

This removes the timezone kickstart command from all of the templates
except for google.ks which needs to set it's own ntp servers and timezone.

If timezone isn't included in the blueprint, and it is not already in a
template, it will be set to 'timezone UTC' by default.

If timezone is set in a template it is left as-is, under the assumption
that the image type requires it to boot correctly.
2019-05-02 16:21:36 -07:00
David Shea 44e14176bb Add a compose type for alibaba.
This compose type creates a partitioned disk as a qcow2 file, but with
only one partition instead of using a separate /boot.
2019-04-03 13:05:31 -04:00
David Shea 434fe58c28 Add a new compose type for Hyper-V
This is based on the VHD compose type, with the following differences:

  * Use the vhdx format instead of vhd
  * No WALinuxAgent
  * Install hyperv-daemons

The hyperv-daemons are activated through udev rules, so there is no need
to add them to the services line.
2019-04-03 13:00:37 -04:00
David Shea c941b82b0c Add a new output type, tar-disk.
This option will create an optionally compressed tarball containing a
disk image. This format is used by Google's Compute Engine.

This also adds a new option, tar_disk_name, to set the name of the disk
image that will be wrapped in the final tarball. opts.image_name
continues to be the final output file name.
2019-04-02 10:15:42 -04:00
David Shea c585b91422 Support compressing single files.
Modify imgutils.compress to allow the "rootdir" argument to be either a
directory or a single file to add to an archive.
2019-04-02 10:15:42 -04:00
David Shea 8ef4f431d8 Add an option to align the image size to a multiplier.
If provided, round the disk image size up to a multiple of the value.
This allows for image formats with specific size-alignment requirements
(e.g., disk size must be in GiB).
2019-04-02 10:15:42 -04:00
Brian C. Lane 049f68cb60 lorax-composer: Fix installing files from [[repos.git]] to /
rpmfluff was including / in the rpm, which conflicts with
filesystem.rpm

The rpm globs are pretty limited, and we don't actually know the file
paths until later, so we have to use a glob or a directory.

So when the destination is / it now uses /* to select all the files and
sub-directories in the archive. The limitation of this is that it cannot
support dotfiles directly under /, they will cause a rpmbuild error.

For destinations other than / it uses the name of the directory, so
dotfiles are fine in that situation.
2019-03-29 08:45:39 -07:00
Brian C. Lane d178189ef3 qemu wasn't restoring the terminal if it was terminated early
You would need to run reset to regain control of your terminal after
this happened, so this turns off the monitor and serial port mux to
stdout.
2019-03-26 13:33:47 -07:00