Commit Graph

901 Commits

Author SHA1 Message Date
David Shea
a9bb68b26f Add an option to align the image size to a multiplier.
If provided, round the disk image size up to a multiple of the value.
This allows for image formats with specific size-alignment requirements
(e.g., disk size must be in GiB).

(cherry picked from commit 8ef4f431d8)

Related: rhbz#1689140
2019-04-05 11:30:19 -04:00
Lars Karlitski
1511c0f9a2 Pass ssl certificate options to anaconda
If a repository has `sslcacert`, `sslclientcert`, or `ssclientkey` set,
pass them to anaconda through the kickstart file. This is mostly the
case when using RHEL repositories that are accessed through a
subscription.

(cherry picked from commit e194b5926c)

Resolves: rhbz#1663950
2019-04-04 08:46:44 -07:00
Brian C. Lane
d7f5f3064b lorax-composer: pass customization.kernel append to extra_boot_args
This allows iso builds to include the extra kernel boot parameters by
passing them to the arch-specific live/*tmpl template.

Also adds tests to make sure it is written to config.toml in the build
metadata.

Resolves: rhbz#1687743
2019-03-27 11:32:19 -07:00
Brian C. Lane
f2ec60b9e4 Improve logging for template syntax errors
The shlex splitting can fail, resulting in error messages like:

ERROR livemedia-creator: No closing quotation

without any context in the log files. This logs the line that failed to
be split and expanded.

Related: rhbz#1687743
2019-03-27 11:32:19 -07:00
Brian C. Lane
a710d17294 Add extra boot args to the livemedia-creator iso templates
This adds a new livemedia-creator argument, --extra-boot-args, which can
be used to add arguments to the kernel command-line in the templates.

Related: rhbz#1687743
2019-03-27 11:32:19 -07:00
Brian C. Lane
010031a46c lorax-composer: Add the ability to append to the kernel command-line
Sometimes it is necessary to modify the kernel command-line of the
image, this adds support for a [customizations.kernel] section to the
blueprint:

[customizations.kernel]
append = "nosmt=force"

This will be appended to the kickstart's bootloader --append argument.

Includes tests for modifying the bootloader line, the kickstart
template, and examining the final-kickstart.ks created for a compose.

Related: rhbz#1687743
2019-03-27 11:32:19 -07:00
Brian C. Lane
b399076cb0 lorax-composer: Return UnknownBlueprint errors when using deleted blueprints
Reading a blueprint wasn't checking to see if it had been deleted so it
was returning the most recent commit before it had been deleted. This
allowed things like starting a compose with a blueprint that technically
doesn't exist.

One exception to this is the /changes/ route, it must be available so
that you can use the commit hash to undo a delete.

This also adds tests for the various operations.

(cherry picked from commit d32f477e0b)

Resolves: rhbz#1683441
2019-03-27 10:54:47 -07:00
Brian C. Lane
da6e228753 lorax-composer: Delete workspace copy when deleting blueprint
Also extends the blueprint delete test to also check the workspace.

(cherry picked from commit 26bd2c1378)

Related: rhbz#1683441
2019-03-27 10:54:13 -07:00
Brian C. Lane
865fedde4e Remove 3G minimum from lorax-composer
The reason for the 3G minimum was because anaconda had a bug with how it
calculated minimum disk size when using kickstart. The fix for this has
been in Anaconda since 29.16, so we can now remove our limit and
create somewhat smaller disk images.

Resolves: rhbz#1677741
2019-03-27 10:39:58 -07:00
Brian C. Lane
e90e7653df Move the package requirements for live-iso setup out of the template
In order to support iso creation on multiple arches with the templates
we need to be able to select different packages based on arch.
lorax-composer uses the arch-specific Lorax templates in order to
generate the output iso so this patch:

1. Creates a new template and type to parse it, live-install.tmpl
   which contains only installpkg commands and #if clauses for arch
2. Removes bootloader related packages from the live-iso.ks
3. Remove dracut-config-rescue exclusion because it can cause problems
   with some blueprints.
4. Switch logo requirement to system-logos which is satisfied by
   generic-logos or fedora-logos. This prevents conflicts when a blueprint
   installs fedora-release-workstation.

So in the future, if x86.tmpl, etc. need a new package to support
creating the iso it should be added to the correct section in
./share/live/live-install.tmpl

Resolves: rhbz#1673744
2019-03-21 14:48:27 -07:00
Brian C. Lane
4e862e538b Add LiveTemplateRunner to parse per-arch live-iso package requirements
This template runner only uses the installpkg command to gather a list
of package NEVRA to be installed to support live iso creation.

Related: rhbz#1673744
2019-03-21 14:48:27 -07:00
Brian C. Lane
51180ad407 Move the run part of LoraxTemplateRunner into new TemplateRunner class
This will make it easier to add a new subclass that only handles
installpkg for use with livemedia-creator and lorax-composer iso
creation.

Related: rhbz#1673744
2019-03-21 14:48:27 -07:00
Brian C. Lane
776bb18cee livemedia-creator: Add support for reqpart kickstart command
reqpart can be used to make kickstarts more platform agnostic, creating
needed partitions without lmc having to keep track of the arch-specific
needs. eg. ppc64 needs prepboot and /boot

This increases the size of the disk based on whether reqpart or
reqpart --add-boot is in the kickstart.

Note that this is only valid for partitioned disk output types, not
for filesystem images or live iso output.

Related: rhbz#1673744
2019-03-21 14:48:27 -07:00
Brian C. Lane
aa240fb591 Fix make_appliance and the libvirt.tmpl
Apparently nobody has used these since the switch to py3, xrange is now
range and it needs to read the file in binary mode when generating the
sha256.

Related: rhbz#1673744
2019-03-21 14:48:27 -07:00
Brian C. Lane
1f716641cf Clarify the ks repo only error message
This also moves the run_creator kickstart checks into check_kickstart
so that tests may be added.

Related: rhbz#1673744
2019-03-21 14:48:27 -07:00
Brian C. Lane
2f8fb6fa8e Add some extra cancel_func protection to QEMUInstall
In livemedia-creator's usage of this it can never pass in None, but if
someone were to import the library and use it, it would crash with
NoneType. So add the extra checks to make sure cancel_func isn't None,
just in case.

Related: rhbz#1684316
2019-03-21 14:25:06 -07:00
Yuval Turgeman
3d376764d4 installer: make sure cancel_func has a value (#612)
When using LMC to virt-install a system to an image, cancel_func is not
provided in run_creator, causing a TypeError (NoneType object is not
callable).

Signed-off-by: Yuval Turgeman <yturgema@redhat.com>

Resolves: rhbz#1684316
2019-03-21 14:25:06 -07:00
Brian C. Lane
6a55184c66 lorax: Move default tmp dir to /var/tmp/lorax
If systemd's tmpfiles.d timer is executed while lorax is running it will
remove any files and directories older than 30 days. This is what has
been causing the occasional error where /proc/ would seem to vanish
during the install.

Upstream has proposed this solution, https://github.com/systemd/systemd/pull/11482
but until that is released we need a work-around to protect the lorax
files.

This commit does several things:

* Move the default tmpdir from /var/tmp/ to /var/tmp/lorax/
* Add a lorax.conf tmpfiles.d file that prevents systemd-tmpfiles from
  removing anything under /var/tmp/lorax/
* Add an exit handler to lorax so that temporary directories are removed on
  exit or on a python traceback.
* Use flock to lock access to the tempdir while lorax is running.
* Remove any unlocked tempdirs named /var/tmp/lorax/lorax.* at startup

Note that the exit handler will not remove the tempdir if lorax is
killed with a signal -- those are being caught by dnf and prevent the
exit handler from running.

systemd-tmpfiles cannot clean up the tempdirs at boot time because they
contain files labeled as shadow_t, so we have to remove those when lorax
runs. It uses the flock to prevent removing any directories created by
parallel instances of lorax and only removes ones that are unlocked.
Worst case they will be around until the first run of lorax after a
reboot.

If you want to keep the working directory around for debugging purposes
use --workdir /var/tmp/lorax/my-workdir and it won't be removed by
lorax.

Resolves: rhbz#1668408
2019-01-30 09:19:24 -08:00
Brian C. Lane
c0c664bc5a Report an error if the blueprint doesn't exist
composer-cli uses TOML for 'blueprints save' which was returning an
empty 200 response if the blueprint didn't exist. Change this to return
a standard 400 error response if the blueprint doesn't exist.

composer-cli is already setup to handle receiving json when an error is
returned so just the toml API response for `blueprints/save` needed to
be changed.

Related: rhbz#1653934
2019-01-11 14:37:01 -08:00
Alexander Todorov
d72ee7bd89 Disable pylint no-member errors for 2 dnf constants
Related: rhbz#1653934
2019-01-11 14:37:01 -08:00
David Shea
59e2da0a96 Allow customizations to be specified as a toml list
Support both

  [customizations]
  hostname = "whatever"

and

  [[customizations]]
  hostname = "whatever"

in the blueprint data. The [[ syntax matches the other customization
directives (user, group, sshkey), and as such it's easy to accidentally
use it for the hostname without even realizing it's specifying something
different.

Add some tests for converting customizations to kickstarts.

(cherry picked from commit 35ab6a1336)

Resolves: rhbz#1664485
2019-01-09 16:22:21 -08:00
Brian C. Lane
f3638762fe Change default releasever to 8
Related: rhbz#1650295
2019-01-09 09:03:27 -08:00
Brian C. Lane
bdc09df37b Revert "lorax-composer: Cancel running Anaconda process"
Drop running pkill. This causes problems if more than one is running on
a system (eg. in parallel using mock). It can kill off other processes
unrelated to this instance of anaconda.

This reverts commit 0ce6d3882b.

Related: rhbz#1656691
2019-01-08 14:47:25 -08:00
Brian C. Lane
0ed7167a03 Make sure cancel_func is not None
Related: rhbz#1656691

(cherry picked from commit ca2c3d9e77)
2019-01-07 14:10:11 -08:00
Brian C. Lane
47eecdfc85 Increase boot.iso rootfs to 3GiB
PPC64 runs out of space when installing into a 2GiB rootfs. Other arches
may also be getting close to running out of space.

Resolves: rhbz#1661169
2019-01-07 08:29:31 -08:00
Brian C. Lane
efc57eae4c lorax: Save information about rootfs filesystem size and usage
Run df on the filesystem image after it has been created.
Output will be in program.log, eg:

Running... df /var/tmp/lorax.imgutils.wm04pg_v
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/loop0       1998672 1619508    362780  82% /var/tmp/lorax.imgutils.wm04pg_v
Return code: 0

Resolves: rhbz#1659625
2018-12-18 08:49:39 -08:00
Brian C. Lane
0ce6d3882b lorax-composer: Cancel running Anaconda process
It ends up that this isn't as easy as you'd think. Anaconda sets up some
signal handlers to handle cleanly exiting, but they are not being run
when sent a TERM after package installation has started. I think DNF
resets them causing it to get ignored.

When the cancel is sent it can take several minutes for it to have an
effect. In my testing it usually takes around 2 minutes for anaconda to
notice and exit.

This sends a TERM to the process and then waits for it to exit. When it
returns it then removed any device-mapper devices that were setup for
image installations, removes any hanging loop devices.

It then kills off any process with pyanaconda. in the cmdline, and
anaconda-bus.conf (because anaconda starts a bunch of helpers and if it
doesn't shut down cleanly they remain running).

Resolves: rhbz#1656691
2018-12-12 16:44:05 -08:00
Brian C. Lane
4b84475612 Add cancel_func to virt and novirt_install functions
In addition to monitoring the logs for errors, call a function (or
functions) that tell it to cancel the anaconda process and cleanup.

Also check for a cancel after creating the squashfs image for live-iso
since that's a long running process.

This required adding a new argument to a number of existing functions,
passing it down to QEMUInstall and novirt_install where the function is
called.

Resolves: rhbz#1656691
2018-12-11 11:37:21 -08:00
Brian C. Lane
a8f616c6da lorax-composer: Check for STATUS before deleting
When there is no run or new symlink do one last check to make sure no
STATUS file was written. If it is missing, go ahead and remove the
results directory.

Related: rhbz#1656691
2018-12-10 10:35:00 -08:00
Brian C. Lane
962d10b3b7 Check for existing CANCEL request, and exit on FINISHED
If another CANCEL request has already been made just exit from
uuid_cancel. If the build is FINISHED before it times out just exit,
don't remove the finished results.

Related: rhbz#1656691
2018-12-07 16:47:19 -08:00
Brian C. Lane
663a0dcd73 lorax-composer: Handle packages with multiple builds
When the repository has multiple arches, eg. i686 and x86_64, it should
add a new entry to the project's builds list, not create a new project
in the list.

This handles that by adding a modified insort_left function and
examining the packages returned from dnf to make sure they aren't
already listed in the results. It also handles adding them in sorted
order so that no further sorting needs to be done on the results.

Resolves: rhbz#1656642
2018-12-07 11:54:02 -08:00
Brian C. Lane
2ad4b20a91 dnf changed the type of gpgkey to a tuple
And in an intermediate version it returns a VectorString object which
isn't serializable by the json or toml modules.

So convert it to a list so that the type is consistent in the sources
code.

(cherry picked from commit e9e5139750)

Resolves: rhbz#1655876
2018-12-07 11:52:22 -08:00
Brian C. Lane
6a2f574ed7 lorax-composer: Add CDN repo checks to startup and compose start.
This will block starting a compose if the sources only reference
cdn.redhat.com urls.

Resolves: rhbz#1655623
2018-12-07 11:50:52 -08:00
Brian C. Lane
f1af108e5b lorax-composer: Check for CDN only repos
Anaconda is currently not able to handle cdn repo urls in the kickstart.
Add a new function that checks for extra repos and returns True.

Related: rhbz#1655623
2018-12-07 11:50:52 -08:00
Brian C. Lane
f0bac40d7f lorax-composer: Check the queue and results at startup
If the system ran out of space, or was rebooted unexpectedly, the state
of the queue symlinks, or the results STATUS files may be inconsistent.
This checks them and:
 * Removes broken symlinks from queue/new and queue/run
 * Removes symlinks from run and sets the build to FAILED
 * Sets builds w/o a STATUS to FAILED
 * Sets builds with STATUS of RUNNING to FAILED
 * Creates missing queue/new symlinks to results with STATUS of WAITING

So, any builds that were running during the reboot will be FAILED, and
any that were waiting to be started will be started upon rebooting.

Resolves: rhbz#1647985
(cherry picked from commit 4dd9004d13)
2018-12-06 15:27:48 -08:00
Brian C. Lane
22061ed4ab Remove setfiles from mkrootfsimage
SELinux applies the correct labels, setfiles is no longer needed.
This allows lorax to run with SELinux in Enforcing mode.

Related: rhbz#1645189
2018-11-29 13:05:01 -08:00
Brian C. Lane
14fb4a9156 Remove SELinux Permissive checks
Anaconda, Lorax, lorax-composer, and livemedia-creator can all now run
with SELinux in Enforcing mode. It does not need to be disabled and if
there are denials they should be reported as a bug.

Log the current state of SELinux when starting, update the
documentation.

Resolves: rhbz#1645189
2018-11-29 13:05:01 -08:00
Brian C. Lane
6de44cd241 Add --no-system-repos to lorax-composer
Running lorax-composer --no-system-repos will prevent it from copying
the dnf repositories from /etc/yum.repos.d/ into the lorax-composer repo
directory. It will *only* use repositories setup using the sources api
or written to /var/lib/lorax/composer/repos.d/

If lorax-composer has previously been run without this switch the system
repos will need to be removed from the composer/repos.d/ directory. It
would also be a good idea to remove the cached metadata in
/var/tmp/composer/

Resolves: rhbz#1650363
2018-11-15 16:19:24 -08:00
Brian C. Lane
5b8341884d Update depsolving with suggestions from dnf (#1636239)
The previous method worked, but wasn't exactly idiomatic. This is more
correct, and appears to work the same (templates depsolve, version globs
work, multiple repos work).

Note that this does use a private dnf attribute ._goal, but the word is
that this is going to become a public api soon, so yes it is there on
purpose.

Resolves: rhbz#1638683
2018-10-12 10:58:04 -07:00
Brian C. Lane
58236a6b61 Add an openstack image type
This is a qcow2 image with cloud-init in the template.

(cherry picked from commit 98f8b23129)

Resolves: rhbz#1628645
2018-10-09 11:55:30 -07:00
Brian C. Lane
077582ee49 Revert "Rename composer-cli to composer"
This reverts commit 2c2e3156d0.

It conflicts with the PHP dependency manager project named 'composer'

Related: rhbz#1635763
2018-10-08 16:22:15 -07:00
Lars Karlitski
2c2e3156d0 Rename composer-cli to composer
Resolves: rhbz#1635763
2018-10-05 08:30:34 -07:00
David Shea
c13aa84f02 Make no-virt generated images sparser
At the end of disk image installs, use fstrim on the generated filesystem to
discard any blocks that were allocated during the install and are now unused.
This will allow tools such as qemu-img to create images that do not include
deleted data.

For raw disk images that do not go through qemu-img, use fallocate --dig-holes
to create sparse holes in place of the unused blocks.

(cherry picked from commit 9717b3fd98)

Related: rhbz#1628645
Related: rhbz#1628646
Related: rhbz#1628647
Related: rhbz#1628648
2018-10-05 11:27:04 -04:00
Brian C. Lane
212e0bcb3e Write a rootpw line if no root customizations in the blueprint
Anaconda requires the root password to be set or locked, so if there
isn't anything setting it we write out 'rootpw --lock'

Also adds tests for this.

Resolves: rhbz#1626122
2018-10-03 08:29:33 -07:00
David Shea
1a23dc0f2b Add a vmdk compose type.
This is similar to the AMI type, but also adds open-vm-tools and does not do
anything special to the partitioning

(cherry picked from commit 1056bfc25b)

Resolves: rhbz#1628646
2018-10-02 13:06:56 -04:00
David Shea
1c1f97ad70 Add a vhd compose type for Azure images
This does pretty much the same things as the AMI compose type, but also
replaces NetworkManager with the Azure linux agent.

(cherry picked from commit e0c236ff36)

Resolves: rhbz#1628648
2018-10-02 13:06:10 -04:00
David Shea
253689ff49 Add an ami compose type for AWS images
This differs from lmc's --make-ami in that creates a full disk image instead of
an fsimage. Create a raw disk image with a / and /boot partitions, and enable
sshd, chronyd, and cockpit by default.

(cherry picked from commit 18188bf6cf)

Resolves: rhbz#1628647
2018-10-02 13:05:40 -04:00
David Shea
b8a7774629 Remove --fstype from the generated part line
Instead of specifying the fstype, just let anaconda use the default.

(cherry picked from commit 847fff4e11)

Related: rhbz#1628647
Related: rhbz#1628648
2018-10-02 12:57:38 -04:00
Brian C. Lane
ca2eb38d92 Always update repo metadata when building an image
When the kickstart is handed off to Anaconda for building it will
download its own copy of the metadata and re-run the depsolve. So if the
dnf cache isn't current there will be a mismatch and the build will
fail to find some of the versions in final-kickstart.ks

This adds a new context to DNFLock, .lock_check, that will force a check
of the metadata. It also implements its own timeout and forces a
refresh of the metadata when that expires because the dnf expiration
doesn't always work as expected.

Resolves: rhbz#1631561
2018-10-01 14:50:11 -07:00
Brian C. Lane
1c99408542 Use rootpw for setting the root password instead of user
Ends up you cannot use the kickstart user command on root, since it
already exists, so we have to translate that into a rootpw command.

So [[customizations.user]] with name = "root" only support key, which
will set the ssh key, and password which will use rootpw to set the
password. plain text or encrypted are supported.

Related: rhbz#1626122
2018-09-28 15:38:02 -07:00