Commit Graph

796 Commits

Author SHA1 Message Date
Brian C. Lane
74f8cd4f34 lorax-composer: Check for valid characters in the undo commit
Return an error 400 with INVALID_CHARS if the commit characters are not
in the allowed list.
2020-05-28 09:39:23 -07:00
Brian C. Lane
f83ae2fed3 mksquashfs: Catch errors with mksquashfs and report them
The host system can run out of space while running mksquashfs, and while
this is logged to program.log it isn't detected by lorax or
livemedia-creator so it will continue running, possibly reporting
unrelated errors and causing confusion.

This adds checks for the return status when calling mksquashfs, logs it
to the log, and either exits or raises an error immediately.
2020-05-07 14:35:55 -07:00
Brian C. Lane
c9c1283c56 lorax: Add --skip-branding cmdline argument
Also document how branding currently works. See docs/lorax.rst

Resolves: rhbz#1826479
2020-04-21 15:57:54 -07:00
Brian C. Lane
91d71b09a4 lorax: Update how the release package is chosen
Previously the release package was chosen by picking the first package
that provides 'system-release' that isn't named generic. This can cause
branding issues with repos containing multiple system-release packages.

This patch changes _install_branding so that it will give preference to
a system-release package that ends with lowercase --variant name. If
there isn't one it will fall back to the previous behavior.

Resolves: rhbz#1472622
2020-03-20 08:28:11 -07:00
Brian C. Lane
d6ce9c907d ltmpl: Fix package logging format
epoch wasn't included in the lorax-packages.log, but arch is. And for
the debuginfo the epoch, including 0, *is* included. This should match
the previous output.

Resolves: rhbz#1815000
2020-03-19 14:03:08 -07:00
Brian C. Lane
e2e8deb906 lorax: Write package lists in run_transaction
A change in glibc now requires /proc be mounted in order to run mknod
which is needed in order to run rpm from runtime-postinstall.

This drops that code from the template and moves writing the package
list into run_transaction, which already has all of the needed
information to generate the list.

Resolves: rhbz#1812895
2020-03-16 11:51:11 -07:00
Brian C. Lane
fe45fa3610 lorax: Catch rootfs out of space failures
It isn't always obvious what happened when the rootfs runs out of space,
especially when using lorax via pungi. So this checks for the out of
space error string when building the runtime image and logs it to the
primary logfile and console as an error with the rootfs size.

eg.
2020-01-20 18:52:58,920: The rootfs ran out of space with size=1
2020-02-05 15:27:41 -08:00
Brian C. Lane
210fe3d8ef lorax: Log dnf solver debug data in ./debugdata/ 2020-01-20 14:04:00 -08:00
Brian C. Lane
a0fce98109 Add --dracut-conf cmdline argument to lorax and livemedia-creator
This adds the ability to use a dracut.conf file instead of passing
--dracut-arg on the cmdline multiple times.
2020-01-16 09:26:27 -08:00
Brian C. Lane
5561bbb3bc executils: Drop bufsize=1 from execReadlines
startProgram is running in binary mode, so bufsize=1 is invalid. The
ExecLineReader class already breaks it up into lines using readline() so
there is no need to change the default buffering.
2020-01-15 10:40:17 -08:00
Brian C. Lane
9bb1a0916e Update to_timeval usage to use format_iso8601
GLib.DateTime.to_timeval is deprecated.
2020-01-15 10:40:17 -08:00
Brian C. Lane
0d3cff0dde ltmpl: Update to use collections.abc 2020-01-15 10:40:17 -08:00
Brian C. Lane
b69e6d725d lorax-composer: Enable ami on aarch64 2020-01-13 14:55:41 -08:00
Dan Horák
c038bb53c6 livemedia-creator: workaround glibc limitation when starting anaconda
On some platforms (aarch64, ppc64le) toolchain limitations/optimizations
can break anaconda startup, as discussed in rhbz#1722181. The workaround
is to preload libgomp.so before starting anaconda.
2020-01-10 11:10:48 +01:00
Brian C. Lane
cd65758ebe anaconda_cleanup: Remove anaconda.pid if it is left behind
When a novirt-install fails or is canceled anaconda may leave its pid
file behind. Add it to the cleanup function.

Resolves: rhbz#1788501
2020-01-07 15:42:16 -08:00
Ondřej Budai
045bddb37a fix typo in api docstring 2019-11-27 12:24:03 +02:00
Brian C. Lane
e365e476af Disable some compose types on other architectures
The 'enabled' field in the /compose/types output now reflects whether or
not the type is supported on the current architecture. Disabled types
are not allowed to be built, and will raise an error like:

Compose type 'alibaba' is disabled on this architecture
2019-10-28 16:42:36 -07:00
Brian C. Lane
af3494fc46 lorax: Drop unused --title option
It has never been hooked up to anything, let alone substitution @TITLE@
(which also hasn't ever been used in the config files).

Closes #863
2019-10-25 11:35:21 -07:00
Brian C. Lane
009c051003 Bump default platform and releasever to 32 2019-10-16 14:22:58 -07:00
Brian C. Lane
3fdc783850 docs: Fix Sphinx errors in docstrings 2019-10-16 13:52:54 -07:00
Brian C. Lane
146560b959 Add upload status examples to compose route docstrings 2019-10-16 09:17:53 -07:00
Brian C. Lane
b23c73e52c Add docstrings to the new upload functions in pylorax.api.queue 2019-10-16 09:17:53 -07:00
Brian C. Lane
d16e8f5bfc Change /compose/uploads/delete to /upload/delete
Remove the requirement to pass the compose uuid and the upload uuid --
only the upload uuid is needed.
2019-10-16 09:17:53 -07:00
Brian C. Lane
0eda252d61 Add profile support to /uploads/schedule/
Pass in the name of a profile in the `profile` field, or pass in
one-time use settings in the `settings` object.
2019-10-16 09:17:53 -07:00
Brian C. Lane
7396c272b2 Use consistent naming for upload uuids
Since we have both compose uuids and upload uuids they need to be
clearly named. This updates the upload naming to use 'upload_uuid' in
the inputs, and 'upload_id' in the output (_id instead of _uuid for
consistency with build_id naming in the status responses).

This also adds 'upload_id' to the /upload/log response.
2019-10-16 09:17:53 -07:00
Brian C. Lane
f8f06ccd23 Fix some docstrings in the v1 API 2019-10-16 09:17:53 -07:00
Brian C. Lane
c384b687ba Add /upload/providers/delete route
This allows deleting a provider's profile. Pass the provider and profile
name like:

/api/v1/upload/providers/delete/azure/test-settings

A standard json response will be returned.
2019-10-16 09:17:53 -07:00
Brian C. Lane
5d7aa95f2f Add support for starting a compose upload with the profile
This extends the /compose/ route to support uploading with either an
existing profile, or with one-time use settings passed in the POST.

To select a profile include the provider and profile, as returned by
`/uploads/providers`:

    "upload": {
      "image_name": "My Image",
      "provider": "azure",
      "profile": "production-azure-settings"
    }
2019-10-16 09:17:53 -07:00
Brian C. Lane
1b84f90963 lifted directories should be under share_dir and lib_dir
Otherwise passing --sharedir pointed to some other path will not use the
correct providers.
2019-10-16 09:17:53 -07:00
Brian C. Lane
3a453eaad7 Make sure V0 API doesn't return uploads information
uploads should only be included in the V1 API routes, add `api`
selection to the relevant helper functions and calls to them from v0.py

Add new V1 routes with api=1 to include the uploads information in the
results.

Also add tests to ensure that V0 requests do not include uploads.
2019-10-16 09:17:53 -07:00
Evan Goode
2692e8138c Automatically upload composed images to the cloud
Currently, Azure, vSphere, and OpenStack are supported. See
https://github.com/weldr/lorax/pull/826 for more details about this new
feature.

I've called the upload library "lifted" as a reference to Seuss'
The Lorax -- in the book, the Lorax lifts himself up by the seat of his
pants through a hole in the smog clouds, and they start calling him the
"Lifted Lorax."

This adds new features to the /compose route under API v1
2019-10-16 09:17:53 -07:00
Evan Goode
584a2d2076 Add load and dump to pylorax.api.toml 2019-10-16 09:17:53 -07:00
Brian C. Lane
697233c14a lorax-composer: Handle RecipeError in commit_recipe_directory
A recipe that is valid TOML can still be an invalid recipe (eg. missing
the 'name' field) so this should also catch RecipeError.

Also added tests for this, as well as making sure commit_recipe_file()
raises the correct errors.

Resolves: rhbz#1755068
2019-10-04 08:07:36 -07:00
Greg Turner
66cd1fdb52 creator.py: include dmsquash-live-ntfs by default
Add dmsquash-live-ntfs to the default dracut modules in livecd builds.

The omission of this is probably why:

  https://bugzilla.redhat.com/show_bug.cgi?id=1449410

didn't work out as intended (I suspect it was closed unfixed).

Without this, initramfs winds up with the ntfs-3g tools lying
around, but no hooks cause them to actually get used.

AFAICT, this is a legit bugfix, not a "proposed enhancement"
so to speak.  To test that this actually fixed the problem (well,
/a/ problem, at least), I used some scripts:

  https://github.com/gmt/test-respin-fedora-ntfsable-iso

Signed-off-by: Greg Turner <gmt@be-evil.net>
Signed-off-by: Brian C. Lane <bcl@redhat.com>
2019-09-04 11:06:26 -07:00
Brian C. Lane
6f686ff9d6 lorax-composer: Add support for dnf variables to repo sources
This loads the system dnf vars from /etc/dnf/vars at startup if
system repos are enabled, and it substitutes the values in the sources
when loaded, and when a new source is added.

Also includes tests.
2019-08-20 16:26:03 -07:00
Brian C. Lane
1e88a99443 lorax-composer: Add v1 API for projects/source/
This changes the source 'name' field to match the DNF usage of it as a
descriptive string. 'id' is now used as the short name to refer to the
source. The v0 API remains unchanged.

Tests for v1 behavior have been added.

Now that the v1 API is in use the status message will return api: 1
2019-08-07 13:26:35 -07:00
Brian C. Lane
278214ff8e Add /api/v1/ handler with no routes 2019-08-06 16:08:10 -07:00
Brian C. Lane
f5bb2dca8c Move common functions into pylorax.api.utils
This is so that they can be used from multiple versions of the API. eg.
v0.py, v1.py, etc.
2019-08-06 15:44:36 -07:00
Brian C. Lane
b9d18216ef lorax-composer: Add liveimg-tar image type
This creates a tar suitable for use with the anaconda kickstart liveimg
command. It adds the kernel, grub2, and grub2-tools packages to the tar
template.
2019-08-05 14:50:32 -07:00
Brian C. Lane
361c6de2de livemedia-creator: Use --compress-arg in mksquashfs
Previously the --compress-arg option was only used for tar, this adds
support for it to the squashfs.img creation used with live isos.
2019-08-05 14:49:25 -07:00
Brian C. Lane
babf823b1b livemedia-creator: Remove unused --squashfs_args option 2019-08-05 14:49:25 -07:00
Brian C. Lane
f4a81d6c26 lorax-composer: Add squashfs_only False to all image types
New livemedia-creator flag needs to be set to the default.
2019-07-29 13:21:25 -07:00
Brian C. Lane
1796489839 livemedia-creator: Use make_runtime for all runtime creation 2019-07-29 13:21:25 -07:00
Brian C. Lane
cb91fa3c78 livemedia-creator: Add support for a squashfs only runtime image
Normally the runtime image is an ext4 filesystem image that is
compressed with squashfs. dracut now supports setting up an overlayfs
when it detects a bare filesystem tree inside the squashfs.

This commit adds support for a --squashfs-only option which places the
root tree directly in the squashfs.img
2019-07-29 13:21:25 -07:00
Brian C. Lane
0708302464 Use binary mode to tail the file
In python 3 f.seek() on text doesn't work like it does in py2/C because
text is now unicode. So change read_tail to use byte mode and take
unicode into account. Also add tests for it.
2019-07-11 15:16:41 -07:00
Evan Goode
90626f97b6 Return most relevant log file from /compose/log
Return anaconda.log if anaconda is running, composer otherwise

Return packaging.log if it's at least 15s newer than anaconda.log
2019-07-11 15:16:41 -07:00
Jacob Kozol
6b55c8f8f0 Remove whitespace in v0_blueprints_new 2019-07-09 22:53:51 +02:00
Jacob Kozol
4174186c14 Add seperate validation for blueprint names
The VALID_API_STRING function allows for characters that should not be
allowed in blueprint names. VALID_BLUEPRINT_NAME allows us to
specifically check if a blueprint contains a valid name.
2019-07-09 22:53:51 +02:00
Lars Karlitski
fd57e1106e tests: set skip_if_unavailable in test repos
dnf seems to have changed the default for skip_if_unavailable. Some
mock repositories are still around in later tests, which then fail
because metadata cannot be synced.

Also expose skip_if_unavailable in dnf_repo_to_file_repo(), so that
tests checking for equality of repo files continue to pass.
2019-07-09 00:23:33 +02:00
Bastien Nocera
eb7f528071 Also search for pxeboot kernel and initrd pairs
As used in the Fedora 30 aarch64 Server DVDs.

Closes: #786
2019-06-28 10:38:38 -07:00