Commit Graph

936 Commits

Author SHA1 Message Date
Brian C. Lane
3613c12464 lorax: Update how the release package is chosen
Previously the release package was chosen by picking the first package
that provides 'system-release' that isn't named generic. This can cause
branding issues with repos containing multiple system-release packages.

This patch changes _install_branding so that it will give preference to
a system-release package that ends with lowercase --variant name. If
there isn't one it will fall back to the previous behavior.

Resolves: rhbz#1826479
2020-04-27 16:02:15 -07:00
Brian C. Lane
b310f3f788 Move get_dnf_base_object into a module
This allows it to be imported by tests.

Related: rhbz#1826479
2020-04-27 16:01:56 -07:00
Brian C. Lane
db8f198304 lorax: Add --skip-branding cmdline argument
Also document how branding currently works. See docs/lorax.rst

Resolves: rhbz#1826479
2020-04-27 16:01:35 -07:00
Brian C. Lane
a08bd603d0 tests: Disable pylint preexec-fn warning
Related: rhbz#1785154
2020-02-05 14:35:16 -08:00
Brian C. Lane
17ce6ee4e4 lorax-composer: Enable ami on aarch64
Resolves: rhbz#1789308
2020-01-13 15:10:16 -08:00
Brian C. Lane
9636a9b1db composer-cli: Increase DELETE timeout to 120s
When the timeout is too short the http library sends the request again.
We return the last response to the user so even if the cancel works they
get an error about the UUID not being valid.

Resolves: rhbz#1788461
2020-01-07 14:12:27 -08:00
Brian C. Lane
15fbef8f97 composer-cli: Remove unneeded % from API WARNING message
Related: rhbz#1779301
2019-12-12 09:32:40 -08:00
Brian C. Lane
054669d027 composer-cli: Only display the available compose types
The enabled bool is now being used so the cli should only show the types
actually available on the architecture.

Also modifies the test in test_compose_sanity.sh

Related: rhbz#1751998
2019-12-10 15:52:39 -08:00
Brian C. Lane
b6b842943d composer-cli: Return int from handle_api_result not bool
The callers, and the documentation, all expect int 0/1 to use as the
exit status for the program. Not True/False, even though that works most
of the time.

(cherry picked from commit fad9b324f7)
Related: rhbz#1779301
2019-12-03 15:13:05 -08:00
Brian C. Lane
32d1451a22 Add support for API v1 commands upload, and providers
This adds the ability to execute API v1 commands when the backend server
supports them. It detects the API version supported by the server and
adjusts the available commands accordingly.

Resolves: rhbz#1779301
2019-12-03 15:13:05 -08:00
Brian C. Lane
7270511379 lorax-composer: Add basic case check to check_recipe_dict
This will detect if an expected field is not all lower case as is
required.

(cherry picked from commit 13a7dcf2b1)

Related: rhbz#1716596
2019-10-31 10:11:13 -07:00
Brian C. Lane
229253b0cb lorax-composer: Add basic recipe checker function
This makes sure that required fields are included, and that sections are
not empty. It does not check for all optional fields.

If there are errors it will gather up all of them and then raise a
RecipeError with a string of all the errors.

(cherry picked from commit 61059a2699)

Resolves: rhbz#1716596
2019-10-31 10:11:13 -07:00
Brian C. Lane
140f40f6be Disable some compose types on other architectures
The 'enabled' field in the /compose/types output now reflects whether or
not the type is supported on the current architecture. Disabled types
are not allowed to be built, and will raise an error like:

Compose type 'alibaba' is disabled on this architecture

Resolves: rhbz#1751998
2019-10-28 17:03:18 -07:00
Evan Goode
12bdd8ba95 More descriptive error for a bad ref in repos.git
Fixes #771

Related: rhbz#1709594

Signed-off-by: Brian C. Lane <bcl@redhat.com>
2019-06-27 08:53:44 -07:00
Brian C. Lane
ff9421c848 lorax-composer: Fix installing files from [[repos.git]] to /
rpmfluff was including / in the rpm, which conflicts with
filesystem.rpm

The rpm globs are pretty limited, and we don't actually know the file
paths until later, so we have to use a glob or a directory.

So when the destination is / it now uses /* to select all the files and
sub-directories in the archive. The limitation of this is that it cannot
support dotfiles directly under /, they will cause a rpmbuild error.

For destinations other than / it uses the name of the directory, so
dotfiles are fine in that situation.

(cherry picked from commit 049f68cb60)

Related: rhbz#1709594
2019-06-26 16:14:40 -07:00
Brian C. Lane
adede63a95 rpmgit: catch potential errors while running git
Log them and report them as RuntimeError. Also add a couple tests for
them.

(cherry picked from commit 61efa91a03)

Related: rhbz#1709594
2019-06-26 16:14:40 -07:00
Brian C. Lane
20389c912f Add repos.git support to lorax-composer builds
This hooks up creation of the rpm to the build, adds it to the
kickstart, and passes the url to Anaconda. The dnf repo with the rpms is
created under the results directory so it will be included when
downloading the build's results.

(cherry picked from commit cd8c884adb)

Related: rhbz#1709594
2019-06-26 16:14:40 -07:00
Brian C. Lane
3b288f0779 Add pylorax.api.gitrpm module and tests
This handles creating the rpm from the dictionary describing the
repository and rpm. Also adds tests for archive and rpm creation.

(cherry picked from commit f6f2308765)

Related: rhbz#1709594
2019-06-26 16:14:40 -07:00
Brian C. Lane
f5113542b1 Add support for [[repos.git]] section to blueprints
This adds support, documentation, and testing for a [[repos.git]]
blueprint section that can be used to install files from a git
repository. It will create an rpm that will be added to the build,
and included in the metadata that can be downloaded. This allows you to
accurately keep track of the source of configuration files and extra
metadata that is added to the build.

The source repo and reference will be listed in the rpm's summary making
it easy to discover on the installed system.

(cherry picked from commit d7b96c8f0f)

Resolves: rhbz#1709594
2019-06-26 16:14:40 -07:00
Brian C. Lane
724ca8b56d composer-cli: Update diff support for customizations and repos.git
composer-cli will now output information about changes to customizations
entries and the repos.git entries.

(cherry picked from commit 156ef0acfd)

Related: rhbz#1709595
2019-05-29 16:35:42 -07:00
Brian C. Lane
eb59a1a26e Add support for customizations and repos.git to /blueprints/diff/
This also includes extensive tests for each of the currently supported
customizations. It should be generic enough to continue working as long
as the list of dicts includes a 'name' or 'user' field in the dict.
Otherwise support for a new dict key will need to be added to the
customizations_diff function.

(cherry picked from commit 850c490b6e)

Related: rhbz#1709595
2019-05-29 16:35:07 -07:00
Brian C. Lane
9f0dd17e2f Change customizations.firewall to append items instead of replace
To maintain consistency with the other options this changes firewall to
combine the existing settings from the image template with the settings
from the blueprint.

Also updated the docs, added a new test for it, and sorted the output
for consistency.

(cherry picked from commit 3e08389a0f)

Resolves: rhbz#1709595
2019-05-29 16:29:17 -07:00
Brian C. Lane
cdf0cbbc5e lorax-composer: Add services support to blueprints
Add support for enabling and disabling systemd services in the
blueprint. It works like this:

    [customizations.services]
    enabled = ["sshd", "cockpit.socket", "httpd"]
    disabled = ["postfix", "telnetd"]

They are *added* to any existing settings in the kickstart templates.

(cherry picked from commit 1111aee92d)

Related: rhbz#1709595
2019-05-29 15:22:30 -07:00
Brian C. Lane
59bad712ad lorax-composer: Add firewall support to blueprints
You can now open ports in the firewall, using port numbers or service
names:

    [customizations.firewall]
    ports = ["22:tcp", "80:tcp", "imap:tcp", "53:tcp", "53:udp"]

Or enable/disable services registered with firewalld:

     [customizations.firewall.services]
     enabled = ["ftp", "ntp", "dhcp"]
     disabled = ["telnet"]

If the template contains firewall --disabled it cannot be overridden,
under the assumption that it is required for the image to boot in the
selected environment.

(cherry picked from commit 4d35668ab5)

Related: rhbz#1709595
2019-05-29 15:21:34 -07:00
Brian C. Lane
ad6fe6cffd lorax-composer: Add locale support to blueprints
You can now set the keyboard layout and language. Eg.

[customizations.locale]
languages = ["en_CA.utf8", "en_HK.utf8"]
keyboard = "de (dvorak)"

Existing entries in the kickstart templates are replaced with the new
ones. If there are no entries then it will default to 'keyboard us' and
'lang en_US.UTF-8'

Includes tests, and leaves the existing keyboard and lang entries in the
templates with a note that they can be replaced by the blueprint.

(cherry picked from commit e5a8700bdf)

Related: rhbz#1709595
2019-05-29 15:20:28 -07:00
Brian C. Lane
86b595e13e lorax-composer: Fix customizations when creating a recipe
This fixes the customizations list problem earlier than in
add_customizations.

In the recipe it should be [customizations] not [[customizations]]
which creates a list. If it was used that way grab the first element and
replace the list with it.

(cherry picked from commit 67007dfa60)

Related: rhbz#1709595
2019-05-29 15:18:41 -07:00
Brian C. Lane
10e46c9178 lorax-composer: Add timezone support to blueprint
For example:

[customizations.timezone]
timezone = "US/Samoa"
ntpservers = ["0.pool.ntp.org"]

Also includes tests.

This removes the timezone kickstart command from all of the templates
except for google.ks which needs to set it's own ntp servers and timezone.

If timezone isn't included in the blueprint, and it is not already in a
template, it will be set to 'timezone UTC' by default.

If timezone is set in a template it is left as-is, under the assumption
that the image type requires it to boot correctly.

(cherry picked from commit 9bdbb29662)

Related: rhbz#1709595
2019-05-29 15:14:58 -07:00
Brian C. Lane
d5fb43844f Switch the --virt-uefi method to use SecureBoot
This updates the qemu arguments so that it will actually work, and
switches to using SecureBoot OVMF firmware.

Resolves: rhbz#1691661
2019-05-16 09:30:53 -07:00
Brian C. Lane
6529b3e5e8 qemu wasn't restoring the terminal if it was terminated early
You would need to run reset to regain control of your terminal after
this happened, so this turns off the monitor and serial port mux to
stdout.

Resolves: rhbz#1691632
2019-05-16 09:23:42 -07:00
Brian C. Lane
59f0870940 Revert "lorax-composer: Add CDN repo checks to startup and compose start."
This reverts commit 6a2f574ed7.

Related: rhbz#1691969
2019-05-13 12:05:33 -07:00
Brian C. Lane
dbee219e13 Revert "lorax-composer: Check for CDN only repos"
This reverts commit f1af108e5b.

Related: rhbz#1691969
2019-05-13 12:03:39 -07:00
Brian C. Lane
e85174abce Use strict=False when reading repo files
In some cases they may contain duplicate sections. With it set to False
the last one wins instead of causing a traceback.

Related: rhbz#1678937
2019-04-05 11:05:58 -07:00
David Shea
0467377810 Add a compose type for alibaba.
This compose type creates a partitioned disk as a qcow2 file, but with
only one partition instead of using a separate /boot.

(cherry picked from commit 44e14176bb)

Resolves: rhbz#1689140
2019-04-05 11:38:01 -04:00
David Shea
d51a79d4fc Add a new output type, tar-disk.
This option will create an optionally compressed tarball containing a
disk image. This format is used by Google's Compute Engine.

This also adds a new option, tar_disk_name, to set the name of the disk
image that will be wrapped in the final tarball. opts.image_name
continues to be the final output file name.

(cherry picked from commit c941b82b0c)

Related: rhbz#1689140
2019-04-05 11:31:39 -04:00
David Shea
96c4f08358 Support compressing single files.
Modify imgutils.compress to allow the "rootdir" argument to be either a
directory or a single file to add to an archive.

(cherry picked from commit c585b91422)

Related: rhbz#1689140
2019-04-05 11:31:09 -04:00
David Shea
a9bb68b26f Add an option to align the image size to a multiplier.
If provided, round the disk image size up to a multiple of the value.
This allows for image formats with specific size-alignment requirements
(e.g., disk size must be in GiB).

(cherry picked from commit 8ef4f431d8)

Related: rhbz#1689140
2019-04-05 11:30:19 -04:00
Lars Karlitski
1511c0f9a2 Pass ssl certificate options to anaconda
If a repository has `sslcacert`, `sslclientcert`, or `ssclientkey` set,
pass them to anaconda through the kickstart file. This is mostly the
case when using RHEL repositories that are accessed through a
subscription.

(cherry picked from commit e194b5926c)

Resolves: rhbz#1663950
2019-04-04 08:46:44 -07:00
Brian C. Lane
d7f5f3064b lorax-composer: pass customization.kernel append to extra_boot_args
This allows iso builds to include the extra kernel boot parameters by
passing them to the arch-specific live/*tmpl template.

Also adds tests to make sure it is written to config.toml in the build
metadata.

Resolves: rhbz#1687743
2019-03-27 11:32:19 -07:00
Brian C. Lane
f2ec60b9e4 Improve logging for template syntax errors
The shlex splitting can fail, resulting in error messages like:

ERROR livemedia-creator: No closing quotation

without any context in the log files. This logs the line that failed to
be split and expanded.

Related: rhbz#1687743
2019-03-27 11:32:19 -07:00
Brian C. Lane
a710d17294 Add extra boot args to the livemedia-creator iso templates
This adds a new livemedia-creator argument, --extra-boot-args, which can
be used to add arguments to the kernel command-line in the templates.

Related: rhbz#1687743
2019-03-27 11:32:19 -07:00
Brian C. Lane
010031a46c lorax-composer: Add the ability to append to the kernel command-line
Sometimes it is necessary to modify the kernel command-line of the
image, this adds support for a [customizations.kernel] section to the
blueprint:

[customizations.kernel]
append = "nosmt=force"

This will be appended to the kickstart's bootloader --append argument.

Includes tests for modifying the bootloader line, the kickstart
template, and examining the final-kickstart.ks created for a compose.

Related: rhbz#1687743
2019-03-27 11:32:19 -07:00
Brian C. Lane
b399076cb0 lorax-composer: Return UnknownBlueprint errors when using deleted blueprints
Reading a blueprint wasn't checking to see if it had been deleted so it
was returning the most recent commit before it had been deleted. This
allowed things like starting a compose with a blueprint that technically
doesn't exist.

One exception to this is the /changes/ route, it must be available so
that you can use the commit hash to undo a delete.

This also adds tests for the various operations.

(cherry picked from commit d32f477e0b)

Resolves: rhbz#1683441
2019-03-27 10:54:47 -07:00
Brian C. Lane
da6e228753 lorax-composer: Delete workspace copy when deleting blueprint
Also extends the blueprint delete test to also check the workspace.

(cherry picked from commit 26bd2c1378)

Related: rhbz#1683441
2019-03-27 10:54:13 -07:00
Brian C. Lane
865fedde4e Remove 3G minimum from lorax-composer
The reason for the 3G minimum was because anaconda had a bug with how it
calculated minimum disk size when using kickstart. The fix for this has
been in Anaconda since 29.16, so we can now remove our limit and
create somewhat smaller disk images.

Resolves: rhbz#1677741
2019-03-27 10:39:58 -07:00
Brian C. Lane
e90e7653df Move the package requirements for live-iso setup out of the template
In order to support iso creation on multiple arches with the templates
we need to be able to select different packages based on arch.
lorax-composer uses the arch-specific Lorax templates in order to
generate the output iso so this patch:

1. Creates a new template and type to parse it, live-install.tmpl
   which contains only installpkg commands and #if clauses for arch
2. Removes bootloader related packages from the live-iso.ks
3. Remove dracut-config-rescue exclusion because it can cause problems
   with some blueprints.
4. Switch logo requirement to system-logos which is satisfied by
   generic-logos or fedora-logos. This prevents conflicts when a blueprint
   installs fedora-release-workstation.

So in the future, if x86.tmpl, etc. need a new package to support
creating the iso it should be added to the correct section in
./share/live/live-install.tmpl

Resolves: rhbz#1673744
2019-03-21 14:48:27 -07:00
Brian C. Lane
4e862e538b Add LiveTemplateRunner to parse per-arch live-iso package requirements
This template runner only uses the installpkg command to gather a list
of package NEVRA to be installed to support live iso creation.

Related: rhbz#1673744
2019-03-21 14:48:27 -07:00
Brian C. Lane
51180ad407 Move the run part of LoraxTemplateRunner into new TemplateRunner class
This will make it easier to add a new subclass that only handles
installpkg for use with livemedia-creator and lorax-composer iso
creation.

Related: rhbz#1673744
2019-03-21 14:48:27 -07:00
Brian C. Lane
776bb18cee livemedia-creator: Add support for reqpart kickstart command
reqpart can be used to make kickstarts more platform agnostic, creating
needed partitions without lmc having to keep track of the arch-specific
needs. eg. ppc64 needs prepboot and /boot

This increases the size of the disk based on whether reqpart or
reqpart --add-boot is in the kickstart.

Note that this is only valid for partitioned disk output types, not
for filesystem images or live iso output.

Related: rhbz#1673744
2019-03-21 14:48:27 -07:00
Brian C. Lane
aa240fb591 Fix make_appliance and the libvirt.tmpl
Apparently nobody has used these since the switch to py3, xrange is now
range and it needs to read the file in binary mode when generating the
sha256.

Related: rhbz#1673744
2019-03-21 14:48:27 -07:00
Brian C. Lane
1f716641cf Clarify the ks repo only error message
This also moves the run_creator kickstart checks into check_kickstart
so that tests may be added.

Related: rhbz#1673744
2019-03-21 14:48:27 -07:00