Commit Graph

922 Commits

Author SHA1 Message Date
Brian C. Lane
d32f477e0b lorax-composer: Return UnknownBlueprint errors when using deleted blueprints
Reading a blueprint wasn't checking to see if it had been deleted so it
was returning the most recent commit before it had been deleted. This
allowed things like starting a compose with a blueprint that technically
doesn't exist.

One exception to this is the /changes/ route, it must be available so
that you can use the commit hash to undo a delete.

This also adds tests for the various operations.

Resolves: rhbz#1682113
2019-03-01 14:39:21 -08:00
Brian C. Lane
26bd2c1378 lorax-composer: Delete workspace copy when deleting blueprint
Also extends the blueprint delete test to also check the workspace.
2019-03-01 14:39:21 -08:00
Brian C. Lane
3b8de2a233 Move the package requirements for live-iso setup out of the template
In order to support iso creation on multiple arches with the templates
we need to be able to select different packages based on arch.
lorax-composer uses the arch-specific Lorax templates in order to
generate the output iso so this patch:

1. Creates a new template and type to parse it, live-install.tmpl
   which contains only installpkg commands and #if clauses for arch
2. Removes bootloader related packages from the live-iso.ks
3. Remove dracut-config-rescue exclusion because it can cause problems
   with some blueprints.
4. Switch logo requirement to system-logos which is satisfied by
   generic-logos or fedora-logos. This prevents conflicts when a blueprint
   installs fedora-release-workstation.

So in the future, if x86.tmpl, etc. need a new package to support
creating the iso it should be added to the correct section in
./share/live/live-install.tmpl
2019-02-25 13:49:12 -08:00
Brian C. Lane
6c5e89ed14 Add LiveTemplateRunner to parse per-arch live-iso package requirements
This template runner only uses the installpkg command to gather a list
of package NEVRA to be installed to support live iso creation.
2019-02-25 13:49:12 -08:00
Brian C. Lane
8eaad3bc5e Move the run part of LoraxTemplateRunner into new TemplateRunner class
This will make it easier to add a new subclass that only handles
installpkg for use with livemedia-creator and lorax-composer iso
creation.
2019-02-25 13:49:12 -08:00
Brian C. Lane
b47554d716 livemedia-creator: Add support for reqpart kickstart command
reqpart can be used to make kickstarts more platform agnostic, creating
needed partitions without lmc having to keep track of the arch-specific
needs. eg. ppc64 needs prepboot and /boot

This increases the size of the disk based on whether reqpart or
reqpart --add-boot is in the kickstart.

Note that this is only valid for partitioned disk output types, not
for filesystem images or live iso output.
2019-02-25 13:49:12 -08:00
Brian C. Lane
6443f34337 Drop _unique_dicts function
It is not actually needed. projects_info deduplicates the package list,
placing other builds into the builds list instead of making a new
package entry. So it returns a sorted and deduped list of packages, as
expected.
2019-02-21 08:59:29 -08:00
Brian C. Lane
9041174142 Add some extra cancel_func protection to QEMUInstall
In livemedia-creator's usage of this it can never pass in None, but if
someone were to import the library and use it, it would crash with
NoneType. So add the extra checks to make sure cancel_func isn't None,
just in case.
2019-02-21 08:54:46 -08:00
Yuval Turgeman
1c731b5618 installer: make sure cancel_func has a value (#612)
When using LMC to virt-install a system to an image, cancel_func is not
provided in run_creator, causing a TypeError (NoneType object is not
callable).

Signed-off-by: Yuval Turgeman <yturgema@redhat.com>
2019-02-19 08:09:36 -08:00
Brian C. Lane
7e78dc368f Remove 3G minimum from lorax-composer
The reason for the 3G minimum was because anaconda had a bug with how it
calculated minimum disk size when using kickstart. The gix for this has
been in Anaconda since 29.19-1, so we can now remove our limit and
create somewhat smaller disk images.
2019-02-14 15:20:50 -08:00
Lars Karlitski
6961dce08b installer: Run anaconda in a mount and pid namespace
Anaconda can leave child processes and mounts around when it crashes or
is canceled before finishing. It also sometimes unmounts unrelated file
systems (https://github.com/rhinstaller/anaconda/issues/1791).

Run it in a mount and pid namespace to clean up after it.
2019-02-12 10:30:58 +01:00
Chris Lumens
022e9eba3e Run as root/weldr by default.
We need to be root to read the certificates that give access to the
package repos.  Right now, the alternative seems to be changing
permissions on the certs themselves, which seems less good.  We're
running anaconda as root anyway.
2019-02-11 16:20:39 -08:00
Lars Karlitski
e194b5926c Pass ssl certificate options to anaconda
If a repository has `sslcacert`, `sslclientcert`, or `ssclientkey` set,
pass them to anaconda through the kickstart file. This is mostly the
case when using RHEL repositories that are accessed through a
subscription.
2019-02-11 16:20:39 -08:00
Brian C. Lane
98482e444d Remove duplicate repositories from the sources list
In some cases when the host has, for whatever reason, multiple copies of
the same repo listed the build may fail with an error about running out
of space.

So this commit removes duplicate entries after the host's repos have been
loaded. It also adjusts some of the test repos to use different
temporary repo names for the tests.
2019-01-30 08:30:40 -08:00
Brian C. Lane
3676cb65bb Clarify the ks repo only error message
This also moves the run_creator kickstart checks into check_kickstart
so that tests may be added.

This will close #164
2019-01-29 14:03:11 -08:00
Brian C. Lane
e4fe1aab32 lorax: Move default tmp dir to /var/tmp/lorax
If systemd's tmpfiles.d timer is executed while lorax is running it will
remove any files and directories older than 30 days. This is what has
been causing the occasional error where /proc/ would seem to vanish
during the install.

Upstream has proposed this solution, https://github.com/systemd/systemd/pull/11482
but until that is released we need a work-around to protect the lorax
files.

This commit does several things:

* Move the default tmpdir from /var/tmp/ to /var/tmp/lorax/
* Add a lorax.conf tmpfiles.d file that prevents systemd-tmpfiles from
  removing anything under /var/tmp/lorax/
* Add an exit handler to lorax so that temporary directories are removed on
  exit or on a python traceback.
* Use flock to lock access to the tempdir while lorax is running.
* Remove any unlocked tempdirs named /var/tmp/lorax/lorax.* at startup

Note that the exit handler will not remove the tempdir if lorax is
killed with a signal -- those are being caught by dnf and prevent the
exit handler from running.

systemd-tmpfiles cannot clean up the tempdirs at boot time because they
contain files labeled as shadow_t, so we have to remove those when lorax
runs. It uses the flock to prevent removing any directories created by
parallel instances of lorax and only removes ones that are unlocked.
Worst case they will be around until the first run of lorax after a
reboot.

If you want to keep the working directory around for debugging purposes
use --workdir /var/tmp/lorax/my-workdir and it won't be removed by
lorax.
2019-01-29 13:57:12 -08:00
Adam Williamson
2d3f266373 Don't exclude /dev from the setfiles in novirt_install
After a novirt disk image install, we run `setfiles` in the
install root to ensure some SELinux contexts are correct. /dev
is currently excluded from this run. However, as reported and
discussed in https://bugzilla.redhat.com/show_bug.cgi?id=1663040
it seems that with a recent systemd change, startup of many
services will fail if /dev itself is incorrectly labelled, and
in current Rawhide live images, it *is* incorrectly labelled.
Including `/dev` in this setfiles command appears to resolve the
problem in my testing.

Resolves: rhbz#1663040

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-18 09:13:52 -08:00
Brian C. Lane
4fe21135e3 Turn off pylint warning about docstring with backslash 2019-01-08 11:37:56 -08:00
Anthony F McInerney
f66bff5aa7 fixes #543 qemu -nodefconfig deprecated 2019-01-08 10:19:00 -08:00
Anthony F McInerney
6bb64f94ff fix spinx build warnings 2019-01-08 10:17:44 -08:00
Brian C. Lane
7ee3b3d661 Revert "lorax-composer: Cancel running Anaconda process"
Drop running pkill. This causes problems if more than one is running on
a system (eg. in parallel using mock). It can kill off other processes
unrelated to this instance of anaconda.

This reverts commit 6b5c4df8b5.
2019-01-08 10:14:29 -08:00
David Shea
35ab6a1336 Allow customizations to be specified as a toml list
Support both

  [customizations]
  hostname = "whatever"

and

  [[customizations]]
  hostname = "whatever"

in the blueprint data. The [[ syntax matches the other customization
directives (user, group, sshkey), and as such it's easy to accidentally
use it for the hostname without even realizing it's specifying something
different.

Add some tests for converting customizations to kickstarts.
2019-01-08 10:21:07 -05:00
Brian C. Lane
ca2c3d9e77 Make sure cancel_func is not None 2019-01-07 13:53:44 -08:00
Dan Horák
a477db33d6 drop big endian ppc/ppc64 support 2019-01-07 13:52:46 -08:00
Dan Horák
34ab13b68a add qemu command mapping for ppc64le 2019-01-07 13:52:46 -08:00
Dan Horák
d34acfc14c don't reduce initrd size on ppc64/ppc64le 2019-01-07 13:52:46 -08:00
Brian C. Lane
fd173f7265 Add timestamps to program.log and dnf.log
In lorax, lorax-composer, and livemedia-creator.
2018-12-19 12:13:24 -08:00
Brian C. Lane
6f141658d4 lorax: Save information about rootfs filesystem size and usage
Run df on the filesystem image after it has been created.
Output will be in program.log, eg:

Running... df /var/tmp/lorax.imgutils.wm04pg_v
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/loop0       1998672 1619508    362780  82% /var/tmp/lorax.imgutils.wm04pg_v
Return code: 0
2018-12-17 16:24:23 -08:00
Brian C. Lane
6b5c4df8b5 lorax-composer: Cancel running Anaconda process
It ends up that this isn't as easy as you'd think. Anaconda sets up some
signal handlers to handle cleanly exiting, but they are not being run
when sent a TERM after package installation has started. I think DNF
resets them causing it to get ignored.

When the cancel is sent it can take several minutes for it to have an
effect. In my testing it usually takes around 2 minutes for anaconda to
notice and exit.

This sends a TERM to the process and then waits for it to exit. When it
returns it then removed any device-mapper devices that were setup for
image installations, removes any hanging loop devices.

It then kills off any process with pyanaconda. in the cmdline, and
anaconda-bus.conf (because anaconda starts a bunch of helpers and if it
doesn't shut down cleanly they remain running).

Resolves: rhbz#1656691
2018-12-17 16:10:04 -08:00
Brian C. Lane
26a684932b Add cancel_func to virt and novirt_install functions
In addition to monitoring the logs for errors, call a function (or
functions) that tell it to cancel the anaconda process and cleanup.

Also check for a cancel after creating the squashfs image for live-iso
since that's a long running process.

This required adding a new argument to a number of existing functions,
passing it down to QEMUInstall and novirt_install where the function is
called.

Resolves: rhbz#1656691
2018-12-17 16:10:04 -08:00
Brian C. Lane
3762264f56 lorax-composer: Check for STATUS before deleting
When there is no run or new symlink do one last check to make sure no
STATUS file was written. If it is missing, go ahead and remove the
results directory.

Related: rhbz#1656691
2018-12-17 16:10:04 -08:00
Brian C. Lane
fa461dc837 Check for existing CANCEL request, and exit on FINISHED
If another CANCEL request has already been made just exit from
uuid_cancel. If the build is FINISHED before it times out just exit,
don't remove the finished results.

Related: rhbz#1656691
2018-12-17 16:10:04 -08:00
Brian C. Lane
d18934775c lorax-composer: Handle packages with multiple builds
When the repository has multiple arches, eg. i686 and x86_64, it should
add a new entry to the project's builds list, not create a new project
in the list.

This handles that by adding a modified insort_left function and
examining the packages returned from dnf to make sure they aren't
already listed in the results. It also handles adding them in sorted
order so that no further sorting needs to be done on the results.

Resolves: rhbz#1656642
2018-12-06 15:48:58 -08:00
Brian C. Lane
4dd9004d13 lorax-composer: Check the queue and results at startup
If the system ran out of space, or was rebooted unexpectedly, the state
of the queue symlinks, or the results STATUS files may be inconsistent.
This checks them and:
 * Removes broken symlinks from queue/new and queue/run
 * Removes symlinks from run and sets the build to FAILED
 * Sets builds w/o a STATUS to FAILED
 * Sets builds with STATUS of RUNNING to FAILED
 * Creates missing queue/new symlinks to results with STATUS of WAITING

So, any builds that were running during the reboot will be FAILED, and
any that were waiting to be started will be started upon rebooting.

Resolves: rhbz#1647985
2018-12-06 15:13:28 -08:00
Brian C. Lane
8b11705ea0 Remove setfiles from mkrootfsimage
SELinux applies the correct labels, setfiles is no longer needed.
This allows lorax to run with SELinux in Enforcing mode.
2018-11-29 11:06:27 -08:00
Brian C. Lane
35b8957f12 Remove SELinux Permissive checks
Anaconda, Lorax, lorax-composer, and livemedia-creator can all now run
with SELinux in Enforcing mode. It does not need to be disabled and if
there are denials they should be reported as a bug.

Log the current state of SELinux when starting, update the
documentation.
2018-11-29 11:06:26 -08:00
Brian C. Lane
c11bffc357 Add --no-system-repos to lorax-composer
Running lorax-composer --no-system-repos will prevent it from copying
the dnf repositories from /etc/yum.repos.d/ into the lorax-composer repo
directory. It will *only* use repositories setup using the sources api
or written to /var/lib/lorax/composer/repos.d/

If lorax-composer has previously been run without this switch the system
repos will need to be removed from the composer/repos.d/ directory. It
would also be a good idea to remove the cached metadata in
/var/tmp/composer/

Resolves: rhbz#1650363
2018-11-26 17:05:08 -08:00
Brian C. Lane
d69f01d4a8 Create a kpartx_disk_img function
So that it can be used in tests as well as in imgutils
2018-11-12 11:13:37 -08:00
Brian C. Lane
8e749efbbf Fix make_appliance and the libvirt.tmpl
Apparently nobody has used these since the switch to py3, xrange is now
range and it needs to read the file in binary mode when generating the
sha256.
2018-11-12 11:13:37 -08:00
Brian C. Lane
a4783ba29f Add tests for ltmpl.py
This covers things like installing globbed package names from multiple
repos, pinned package versions, and ltmpl functions

Related: rhbz#1548586
2018-10-29 13:55:19 -07:00
Brian C. Lane
befa3128d7 Move get_dnf_base_object into a module
This allows it to be imported by tests.
2018-10-29 13:49:54 -07:00
Marek Marczykowski-Górecki
27e611629f Add --squashfs-only option to drop inner rootfs.img layer
Make runtime directly into squashfs image. This reduces largely
unreproducible ext4 layer, but requires anaconda's dracut module
modification to properly mount the image.
2018-10-29 13:16:38 -07:00
Brian C. Lane
e9e5139750 dnf changed the type of gpgkey to a tuple
And in an intermediate version it returns a VectorString object which
isn't serializable by the json or toml modules.

So convert it to a list so that the type is consistent in the sources
code.
2018-10-24 10:22:40 -07:00
Brian C. Lane
5409748e75 lorax: Log when SOURCE_DATE_EPOCH is used for the current time 2018-10-19 14:43:29 -07:00
Marek Marczykowski-Górecki
de8124366e Use SOURCE_DATE_EPOCH for volumeid of efi boot image
By default mkfs.mksdos choose volume id based on current time. If
SOURCE_DATE_EPOCH is set, use that instead.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
2018-10-19 14:43:29 -07:00
Marek Marczykowski-Górecki
e7f45d333f Preserve timestamps when building fs image
Even when FS do not support owner/modes, preserve timestamps.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
2018-10-19 14:43:29 -07:00
Marek Marczykowski-Górecki
876ec52215 Use SOURCE_DATE_EPOCH for metadata timestamps
This include .buildinfo, .treeinfo and .discinfo.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
2018-10-19 14:43:29 -07:00
Brian C. Lane
7e0a288f5b Update depsolving with suggestions from dnf (#1636239)
The previous method worked, but wasn't exactly idiomatic. This is more
correct, and appears to work the same (templates depsolve, version globs
work, multiple repos work).

Note that this does use a private dnf attribute ._goal, but the word is
that this is going to become a public api soon, so yes it is there on
purpose.
2018-10-12 12:00:25 -07:00
Brian C. Lane
faa65bca3c Disable false context-manager pylint error 2018-10-12 12:00:25 -07:00
Brian C. Lane
d7951e3a10 Fix directory creation for blueprints
Depending on how lorax-composer is run setting up an empty blueprints
directory can fail. So this moves checking/creation until after the
other directories are created and uses make_owned_dir to make sure
ownership is correct.
2018-10-12 11:59:32 -07:00