Commit Graph

945 Commits

Author SHA1 Message Date
Brian C. Lane
088111f087 composer-cli: Make start-ostree parent and ref optional
This changes the start-ostree command to:

compose start-ostree [--size XXXX] [--parent PARENT] [--ref REF] <BLUEPRINT> <TYPE> [<IMAGE-NAME> <PROFILE.TOML>]

Both of them are optional, and if missing a "" is passed to
osbuild-composer. Also adds more tests for all the various possible
options and removes the provider and profile arguments.

(cherry picked from commit 5b0487f47c)

Resolves: rhbz#1859680
2020-07-23 10:35:28 -07:00
Brian C. Lane
dcb2f36c2d composer-cli: Add a get_arg function
This is in preperation for adding more optional arguments. Adds a
generic get_arg function, tests for it, and converts get_size to use it.

(cherry picked from commit 4a4128af23)

Related: rhbz#1859680
2020-07-23 10:35:28 -07:00
Brian C. Lane
952b34dbca composer-cli: Disable retry counter on connection timeout
urllib3 has a retry system that send the request again when the
connection fails, times out, is redircted, etc. This turns off retries,
which aren't very useful anyway.

Related: rhbz#1844649
2020-06-17 10:21:00 -07:00
Brian C. Lane
800fc3b18b composer-cli: Change timeout to 5 minutes
Related: rhbz#1844649

In some cases osbuild-composer with an empty cache can take longer than
a minute to depsolve and start a compose.
2020-06-16 17:04:47 -07:00
Brian C. Lane
c300ac6000 composer-cli: Remove the upload and providers commands
These have never been supported on RHEL8 so remove them. Uploads are
supported by osbuild-composer using the 'compose' command.

Related: rhbz#1844649
2020-06-11 09:14:11 -07:00
Brian C. Lane
ba5a36c982 composer-cli: Return a better error with no value
And add tests for the get_size function.

Related: rhbz#1844649
2020-06-11 09:14:11 -07:00
Brian C. Lane
7da012a92b composer-cli: Add new start-ostree command
This is used to start an ostree build, it is only supported on
ostree-composer, and requires the ostree ref and parent. It may also
include --size and optionally be uploaded.

Resolves: rhbz#1844649
2020-06-09 13:52:44 -07:00
Brian C. Lane
7c705c17d1 composer-cli: Add support for --size to compose start
osbuild-composer can support user specified image sizes, this adds an
optional argument, after start, to specify the size in MiB. eg.

    composer-cli compose start --size 2048 example-http-server qcow2

This only works when the backend is not 'lorax-composer', when it is the
user will get a warning that it will be ignored.

Related: rhbz#1844649
2020-06-09 13:47:40 -07:00
Brian C. Lane
b75a4e2e3c composer-cli: pytoml load() needs an open file not a string
Resolves: rhbz#1843704
2020-06-03 15:16:37 -07:00
Brian C. Lane
3613c12464 lorax: Update how the release package is chosen
Previously the release package was chosen by picking the first package
that provides 'system-release' that isn't named generic. This can cause
branding issues with repos containing multiple system-release packages.

This patch changes _install_branding so that it will give preference to
a system-release package that ends with lowercase --variant name. If
there isn't one it will fall back to the previous behavior.

Resolves: rhbz#1826479
2020-04-27 16:02:15 -07:00
Brian C. Lane
b310f3f788 Move get_dnf_base_object into a module
This allows it to be imported by tests.

Related: rhbz#1826479
2020-04-27 16:01:56 -07:00
Brian C. Lane
db8f198304 lorax: Add --skip-branding cmdline argument
Also document how branding currently works. See docs/lorax.rst

Resolves: rhbz#1826479
2020-04-27 16:01:35 -07:00
Brian C. Lane
a08bd603d0 tests: Disable pylint preexec-fn warning
Related: rhbz#1785154
2020-02-05 14:35:16 -08:00
Brian C. Lane
17ce6ee4e4 lorax-composer: Enable ami on aarch64
Resolves: rhbz#1789308
2020-01-13 15:10:16 -08:00
Brian C. Lane
9636a9b1db composer-cli: Increase DELETE timeout to 120s
When the timeout is too short the http library sends the request again.
We return the last response to the user so even if the cancel works they
get an error about the UUID not being valid.

Resolves: rhbz#1788461
2020-01-07 14:12:27 -08:00
Brian C. Lane
15fbef8f97 composer-cli: Remove unneeded % from API WARNING message
Related: rhbz#1779301
2019-12-12 09:32:40 -08:00
Brian C. Lane
054669d027 composer-cli: Only display the available compose types
The enabled bool is now being used so the cli should only show the types
actually available on the architecture.

Also modifies the test in test_compose_sanity.sh

Related: rhbz#1751998
2019-12-10 15:52:39 -08:00
Brian C. Lane
b6b842943d composer-cli: Return int from handle_api_result not bool
The callers, and the documentation, all expect int 0/1 to use as the
exit status for the program. Not True/False, even though that works most
of the time.

(cherry picked from commit fad9b324f7)
Related: rhbz#1779301
2019-12-03 15:13:05 -08:00
Brian C. Lane
32d1451a22 Add support for API v1 commands upload, and providers
This adds the ability to execute API v1 commands when the backend server
supports them. It detects the API version supported by the server and
adjusts the available commands accordingly.

Resolves: rhbz#1779301
2019-12-03 15:13:05 -08:00
Brian C. Lane
7270511379 lorax-composer: Add basic case check to check_recipe_dict
This will detect if an expected field is not all lower case as is
required.

(cherry picked from commit 13a7dcf2b1)

Related: rhbz#1716596
2019-10-31 10:11:13 -07:00
Brian C. Lane
229253b0cb lorax-composer: Add basic recipe checker function
This makes sure that required fields are included, and that sections are
not empty. It does not check for all optional fields.

If there are errors it will gather up all of them and then raise a
RecipeError with a string of all the errors.

(cherry picked from commit 61059a2699)

Resolves: rhbz#1716596
2019-10-31 10:11:13 -07:00
Brian C. Lane
140f40f6be Disable some compose types on other architectures
The 'enabled' field in the /compose/types output now reflects whether or
not the type is supported on the current architecture. Disabled types
are not allowed to be built, and will raise an error like:

Compose type 'alibaba' is disabled on this architecture

Resolves: rhbz#1751998
2019-10-28 17:03:18 -07:00
Evan Goode
12bdd8ba95 More descriptive error for a bad ref in repos.git
Fixes #771

Related: rhbz#1709594

Signed-off-by: Brian C. Lane <bcl@redhat.com>
2019-06-27 08:53:44 -07:00
Brian C. Lane
ff9421c848 lorax-composer: Fix installing files from [[repos.git]] to /
rpmfluff was including / in the rpm, which conflicts with
filesystem.rpm

The rpm globs are pretty limited, and we don't actually know the file
paths until later, so we have to use a glob or a directory.

So when the destination is / it now uses /* to select all the files and
sub-directories in the archive. The limitation of this is that it cannot
support dotfiles directly under /, they will cause a rpmbuild error.

For destinations other than / it uses the name of the directory, so
dotfiles are fine in that situation.

(cherry picked from commit 049f68cb60)

Related: rhbz#1709594
2019-06-26 16:14:40 -07:00
Brian C. Lane
adede63a95 rpmgit: catch potential errors while running git
Log them and report them as RuntimeError. Also add a couple tests for
them.

(cherry picked from commit 61efa91a03)

Related: rhbz#1709594
2019-06-26 16:14:40 -07:00
Brian C. Lane
20389c912f Add repos.git support to lorax-composer builds
This hooks up creation of the rpm to the build, adds it to the
kickstart, and passes the url to Anaconda. The dnf repo with the rpms is
created under the results directory so it will be included when
downloading the build's results.

(cherry picked from commit cd8c884adb)

Related: rhbz#1709594
2019-06-26 16:14:40 -07:00
Brian C. Lane
3b288f0779 Add pylorax.api.gitrpm module and tests
This handles creating the rpm from the dictionary describing the
repository and rpm. Also adds tests for archive and rpm creation.

(cherry picked from commit f6f2308765)

Related: rhbz#1709594
2019-06-26 16:14:40 -07:00
Brian C. Lane
f5113542b1 Add support for [[repos.git]] section to blueprints
This adds support, documentation, and testing for a [[repos.git]]
blueprint section that can be used to install files from a git
repository. It will create an rpm that will be added to the build,
and included in the metadata that can be downloaded. This allows you to
accurately keep track of the source of configuration files and extra
metadata that is added to the build.

The source repo and reference will be listed in the rpm's summary making
it easy to discover on the installed system.

(cherry picked from commit d7b96c8f0f)

Resolves: rhbz#1709594
2019-06-26 16:14:40 -07:00
Brian C. Lane
724ca8b56d composer-cli: Update diff support for customizations and repos.git
composer-cli will now output information about changes to customizations
entries and the repos.git entries.

(cherry picked from commit 156ef0acfd)

Related: rhbz#1709595
2019-05-29 16:35:42 -07:00
Brian C. Lane
eb59a1a26e Add support for customizations and repos.git to /blueprints/diff/
This also includes extensive tests for each of the currently supported
customizations. It should be generic enough to continue working as long
as the list of dicts includes a 'name' or 'user' field in the dict.
Otherwise support for a new dict key will need to be added to the
customizations_diff function.

(cherry picked from commit 850c490b6e)

Related: rhbz#1709595
2019-05-29 16:35:07 -07:00
Brian C. Lane
9f0dd17e2f Change customizations.firewall to append items instead of replace
To maintain consistency with the other options this changes firewall to
combine the existing settings from the image template with the settings
from the blueprint.

Also updated the docs, added a new test for it, and sorted the output
for consistency.

(cherry picked from commit 3e08389a0f)

Resolves: rhbz#1709595
2019-05-29 16:29:17 -07:00
Brian C. Lane
cdf0cbbc5e lorax-composer: Add services support to blueprints
Add support for enabling and disabling systemd services in the
blueprint. It works like this:

    [customizations.services]
    enabled = ["sshd", "cockpit.socket", "httpd"]
    disabled = ["postfix", "telnetd"]

They are *added* to any existing settings in the kickstart templates.

(cherry picked from commit 1111aee92d)

Related: rhbz#1709595
2019-05-29 15:22:30 -07:00
Brian C. Lane
59bad712ad lorax-composer: Add firewall support to blueprints
You can now open ports in the firewall, using port numbers or service
names:

    [customizations.firewall]
    ports = ["22:tcp", "80:tcp", "imap:tcp", "53:tcp", "53:udp"]

Or enable/disable services registered with firewalld:

     [customizations.firewall.services]
     enabled = ["ftp", "ntp", "dhcp"]
     disabled = ["telnet"]

If the template contains firewall --disabled it cannot be overridden,
under the assumption that it is required for the image to boot in the
selected environment.

(cherry picked from commit 4d35668ab5)

Related: rhbz#1709595
2019-05-29 15:21:34 -07:00
Brian C. Lane
ad6fe6cffd lorax-composer: Add locale support to blueprints
You can now set the keyboard layout and language. Eg.

[customizations.locale]
languages = ["en_CA.utf8", "en_HK.utf8"]
keyboard = "de (dvorak)"

Existing entries in the kickstart templates are replaced with the new
ones. If there are no entries then it will default to 'keyboard us' and
'lang en_US.UTF-8'

Includes tests, and leaves the existing keyboard and lang entries in the
templates with a note that they can be replaced by the blueprint.

(cherry picked from commit e5a8700bdf)

Related: rhbz#1709595
2019-05-29 15:20:28 -07:00
Brian C. Lane
86b595e13e lorax-composer: Fix customizations when creating a recipe
This fixes the customizations list problem earlier than in
add_customizations.

In the recipe it should be [customizations] not [[customizations]]
which creates a list. If it was used that way grab the first element and
replace the list with it.

(cherry picked from commit 67007dfa60)

Related: rhbz#1709595
2019-05-29 15:18:41 -07:00
Brian C. Lane
10e46c9178 lorax-composer: Add timezone support to blueprint
For example:

[customizations.timezone]
timezone = "US/Samoa"
ntpservers = ["0.pool.ntp.org"]

Also includes tests.

This removes the timezone kickstart command from all of the templates
except for google.ks which needs to set it's own ntp servers and timezone.

If timezone isn't included in the blueprint, and it is not already in a
template, it will be set to 'timezone UTC' by default.

If timezone is set in a template it is left as-is, under the assumption
that the image type requires it to boot correctly.

(cherry picked from commit 9bdbb29662)

Related: rhbz#1709595
2019-05-29 15:14:58 -07:00
Brian C. Lane
d5fb43844f Switch the --virt-uefi method to use SecureBoot
This updates the qemu arguments so that it will actually work, and
switches to using SecureBoot OVMF firmware.

Resolves: rhbz#1691661
2019-05-16 09:30:53 -07:00
Brian C. Lane
6529b3e5e8 qemu wasn't restoring the terminal if it was terminated early
You would need to run reset to regain control of your terminal after
this happened, so this turns off the monitor and serial port mux to
stdout.

Resolves: rhbz#1691632
2019-05-16 09:23:42 -07:00
Brian C. Lane
59f0870940 Revert "lorax-composer: Add CDN repo checks to startup and compose start."
This reverts commit 6a2f574ed7.

Related: rhbz#1691969
2019-05-13 12:05:33 -07:00
Brian C. Lane
dbee219e13 Revert "lorax-composer: Check for CDN only repos"
This reverts commit f1af108e5b.

Related: rhbz#1691969
2019-05-13 12:03:39 -07:00
Brian C. Lane
e85174abce Use strict=False when reading repo files
In some cases they may contain duplicate sections. With it set to False
the last one wins instead of causing a traceback.

Related: rhbz#1678937
2019-04-05 11:05:58 -07:00
David Shea
0467377810 Add a compose type for alibaba.
This compose type creates a partitioned disk as a qcow2 file, but with
only one partition instead of using a separate /boot.

(cherry picked from commit 44e14176bb)

Resolves: rhbz#1689140
2019-04-05 11:38:01 -04:00
David Shea
d51a79d4fc Add a new output type, tar-disk.
This option will create an optionally compressed tarball containing a
disk image. This format is used by Google's Compute Engine.

This also adds a new option, tar_disk_name, to set the name of the disk
image that will be wrapped in the final tarball. opts.image_name
continues to be the final output file name.

(cherry picked from commit c941b82b0c)

Related: rhbz#1689140
2019-04-05 11:31:39 -04:00
David Shea
96c4f08358 Support compressing single files.
Modify imgutils.compress to allow the "rootdir" argument to be either a
directory or a single file to add to an archive.

(cherry picked from commit c585b91422)

Related: rhbz#1689140
2019-04-05 11:31:09 -04:00
David Shea
a9bb68b26f Add an option to align the image size to a multiplier.
If provided, round the disk image size up to a multiple of the value.
This allows for image formats with specific size-alignment requirements
(e.g., disk size must be in GiB).

(cherry picked from commit 8ef4f431d8)

Related: rhbz#1689140
2019-04-05 11:30:19 -04:00
Lars Karlitski
1511c0f9a2 Pass ssl certificate options to anaconda
If a repository has `sslcacert`, `sslclientcert`, or `ssclientkey` set,
pass them to anaconda through the kickstart file. This is mostly the
case when using RHEL repositories that are accessed through a
subscription.

(cherry picked from commit e194b5926c)

Resolves: rhbz#1663950
2019-04-04 08:46:44 -07:00
Brian C. Lane
d7f5f3064b lorax-composer: pass customization.kernel append to extra_boot_args
This allows iso builds to include the extra kernel boot parameters by
passing them to the arch-specific live/*tmpl template.

Also adds tests to make sure it is written to config.toml in the build
metadata.

Resolves: rhbz#1687743
2019-03-27 11:32:19 -07:00
Brian C. Lane
f2ec60b9e4 Improve logging for template syntax errors
The shlex splitting can fail, resulting in error messages like:

ERROR livemedia-creator: No closing quotation

without any context in the log files. This logs the line that failed to
be split and expanded.

Related: rhbz#1687743
2019-03-27 11:32:19 -07:00
Brian C. Lane
a710d17294 Add extra boot args to the livemedia-creator iso templates
This adds a new livemedia-creator argument, --extra-boot-args, which can
be used to add arguments to the kernel command-line in the templates.

Related: rhbz#1687743
2019-03-27 11:32:19 -07:00
Brian C. Lane
010031a46c lorax-composer: Add the ability to append to the kernel command-line
Sometimes it is necessary to modify the kernel command-line of the
image, this adds support for a [customizations.kernel] section to the
blueprint:

[customizations.kernel]
append = "nosmt=force"

This will be appended to the kickstart's bootloader --append argument.

Includes tests for modifying the bootloader line, the kickstart
template, and examining the final-kickstart.ks created for a compose.

Related: rhbz#1687743
2019-03-27 11:32:19 -07:00