Commit Graph

1967 Commits

Author SHA1 Message Date
David Shea
cfff807df8 Allow customizations to be specified as a toml list
Support both

  [customizations]
  hostname = "whatever"

and

  [[customizations]]
  hostname = "whatever"

in the blueprint data. The [[ syntax matches the other customization
directives (user, group, sshkey), and as such it's easy to accidentally
use it for the hostname without even realizing it's specifying something
different.

Add some tests for converting customizations to kickstarts.

(cherry picked from commit 35ab6a1336)
2019-01-08 13:46:08 -08:00
Brian C. Lane
9af706dbe3 Revert "lorax-composer: Cancel running Anaconda process"
Drop running pkill. This causes problems if more than one is running on
a system (eg. in parallel using mock). It can kill off other processes
unrelated to this instance of anaconda.

This reverts commit 42addfc2b5.
2019-01-08 13:45:42 -08:00
Brian C. Lane
6be0ca9bb0 Make sure cancel_func is not None
(cherry picked from commit ca2c3d9e77)
2019-01-07 14:11:35 -08:00
Brian C. Lane
bef4d9f4a8 Automatic commit of package [lorax] release [29.23-1].
Created by command:

/bin/tito tag
2018-12-17 16:48:29 -08:00
Brian C. Lane
251306e1ab lorax: Save information about rootfs filesystem size and usage
Run df on the filesystem image after it has been created.
Output will be in program.log, eg:

Running... df /var/tmp/lorax.imgutils.wm04pg_v
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/loop0       1998672 1619508    362780  82% /var/tmp/lorax.imgutils.wm04pg_v
Return code: 0
2018-12-17 16:24:49 -08:00
Brian C. Lane
1607e33227 Turn on signed tags when using tito.
This requires a patched version of tito at the moment, and doesn't hurt
to have it set -- it will just continue not signing the tags.
2018-12-17 16:23:31 -08:00
Brian C. Lane
42addfc2b5 lorax-composer: Cancel running Anaconda process
It ends up that this isn't as easy as you'd think. Anaconda sets up some
signal handlers to handle cleanly exiting, but they are not being run
when sent a TERM after package installation has started. I think DNF
resets them causing it to get ignored.

When the cancel is sent it can take several minutes for it to have an
effect. In my testing it usually takes around 2 minutes for anaconda to
notice and exit.

This sends a TERM to the process and then waits for it to exit. When it
returns it then removed any device-mapper devices that were setup for
image installations, removes any hanging loop devices.

It then kills off any process with pyanaconda. in the cmdline, and
anaconda-bus.conf (because anaconda starts a bunch of helpers and if it
doesn't shut down cleanly they remain running).

Resolves: rhbz#1656691
2018-12-17 16:22:23 -08:00
Brian C. Lane
850ad9845a Add cancel_func to virt and novirt_install functions
In addition to monitoring the logs for errors, call a function (or
functions) that tell it to cancel the anaconda process and cleanup.

Also check for a cancel after creating the squashfs image for live-iso
since that's a long running process.

This required adding a new argument to a number of existing functions,
passing it down to QEMUInstall and novirt_install where the function is
called.

Resolves: rhbz#1656691
2018-12-17 16:22:23 -08:00
Brian C. Lane
df6d7654bd lorax-composer: Check for STATUS before deleting
When there is no run or new symlink do one last check to make sure no
STATUS file was written. If it is missing, go ahead and remove the
results directory.

Related: rhbz#1656691
2018-12-17 16:22:23 -08:00
Brian C. Lane
5bf223c21a Check for existing CANCEL request, and exit on FINISHED
If another CANCEL request has already been made just exit from
uuid_cancel. If the build is FINISHED before it times out just exit,
don't remove the finished results.

Related: rhbz#1656691
2018-12-17 16:22:23 -08:00
Vendula Poncova
8721ce765d Fix vhd images
Images don't work without these fixes:

* Enable Network Manager.
* Disable cloud-init.
* Add Hyper-V modules into initramfs.

Fixes specific for RHEL:

* Create ifcfg-eth0 required by waagent.
* Install python3 and net-tools required by waagent.

Recommended changes:

* Use recommended kernel boot args.
* Disable kdump.
2018-12-11 08:30:06 -08:00
Brian C. Lane
40cfaa515c Automatic commit of package [lorax] release [29.22-1].
Created by command:

/bin/tito tag
2018-12-06 16:14:24 -08:00
Brian C. Lane
3f034a981d lorax-composer: Handle packages with multiple builds
When the repository has multiple arches, eg. i686 and x86_64, it should
add a new entry to the project's builds list, not create a new project
in the list.

This handles that by adding a modified insort_left function and
examining the packages returned from dnf to make sure they aren't
already listed in the results. It also handles adding them in sorted
order so that no further sorting needs to be done on the results.

Resolves: rhbz#1656642
(cherry picked from commit d18934775c)
2018-12-06 16:06:45 -08:00
Brian C. Lane
574360f7d0 lorax-composer: Check the queue and results at startup
If the system ran out of space, or was rebooted unexpectedly, the state
of the queue symlinks, or the results STATUS files may be inconsistent.
This checks them and:
 * Removes broken symlinks from queue/new and queue/run
 * Removes symlinks from run and sets the build to FAILED
 * Sets builds w/o a STATUS to FAILED
 * Sets builds with STATUS of RUNNING to FAILED
 * Creates missing queue/new symlinks to results with STATUS of WAITING

So, any builds that were running during the reboot will be FAILED, and
any that were waiting to be started will be started upon rebooting.

Resolves: rhbz#1647985
(cherry picked from commit 4dd9004d13)
2018-12-06 15:26:40 -08:00
Brian C. Lane
1d21957036 Automatic commit of package [lorax] release [29.21-1].
Created by command:

/bin/tito tag
2018-11-29 14:46:42 -08:00
Brian C. Lane
0c042afd34 New lorax documentation - 29.21 2018-11-29 11:12:11 -08:00
Brian C. Lane
99d867db65 lorax-composer: Install selinux-policy-targeted in images
This is required to ensure that SELinux is configured properly while
building. It fixes the problem with building tar, and should be
installed in the other image types for consistency.

Resolves: rhbz#1645189
(cherry picked from commit 9ac4508135)
2018-11-29 11:11:28 -08:00
Brian C. Lane
4a4a415f88 Remove setfiles from mkrootfsimage
SELinux applies the correct labels, setfiles is no longer needed.
This allows lorax to run with SELinux in Enforcing mode.

(cherry picked from commit 8b11705ea0)
2018-11-29 11:11:28 -08:00
Brian C. Lane
080705e8e6 Remove SELinux Permissive checks
Anaconda, Lorax, lorax-composer, and livemedia-creator can all now run
with SELinux in Enforcing mode. It does not need to be disabled and if
there are denials they should be reported as a bug.

Log the current state of SELinux when starting, update the
documentation.

(cherry picked from commit 35b8957f12)
2018-11-29 11:11:28 -08:00
Brian C. Lane
43ff505804 Add --no-system-repos to lorax-composer
Running lorax-composer --no-system-repos will prevent it from copying
the dnf repositories from /etc/yum.repos.d/ into the lorax-composer repo
directory. It will *only* use repositories setup using the sources api
or written to /var/lib/lorax/composer/repos.d/

If lorax-composer has previously been run without this switch the system
repos will need to be removed from the composer/repos.d/ directory. It
would also be a good idea to remove the cached metadata in
/var/tmp/composer/

Resolves: rhbz#1650363
2018-11-29 11:11:28 -08:00
Brian C. Lane
a357c42f2d Automatic commit of package [lorax] release [29.20-1].
Created by command:

/usr/bin/tito tag
2018-10-29 15:52:25 -07:00
Brian C. Lane
5cf54c6f29 New lorax documentation - 29.20 2018-10-29 15:04:22 -07:00
Brian C. Lane
1006af4232 Add tests for ltmpl.py
This covers things like installing globbed package names from multiple
repos, pinned package versions, and ltmpl functions

Related: rhbz#1548586
(cherry picked from commit a4783ba29f)
2018-10-29 14:49:49 -07:00
Brian C. Lane
8a53d5e310 Move get_dnf_base_object into a module
This allows it to be imported by tests.

(cherry picked from commit befa3128d7)
2018-10-29 14:49:39 -07:00
Brian C. Lane
d9b282150e Build manpages for composer-cli and lorax-composer
Add manpage creation to make docs target to keep them updated.

(cherry picked from commit 7500a17f27)
2018-10-29 14:49:15 -07:00
Brian C. Lane
bdbd8258f0 Fix singlerepo test for f29 2018-10-29 14:48:40 -07:00
Brian C. Lane
88a0b475e9 Update the projects tests to use DNF Repo object
Stop using fake dnf object, use the real thing. This will help catch
problems with dnf returning unexpected types like VectorString.

(cherry picked from commit 27aff75aa3)
2018-10-29 14:27:23 -07:00
Brian C. Lane
af28b25e50 dnf changed the type of gpgkey to a tuple
And in an intermediate version it returns a VectorString object which
isn't serializable by the json or toml modules.

So convert it to a list so that the type is consistent in the sources
code.

(cherry picked from commit e9e5139750)
2018-10-29 14:27:13 -07:00
Brian C. Lane
4192dfc709 Install python3-librepo in the test container
(cherry picked from commit e1dc54ea8c)
2018-10-29 14:13:05 -07:00
Brian C. Lane
49db5403b8 Automatic commit of package [lorax] release [29.19-1].
Created by command:

/usr/bin/tito tag
2018-10-12 16:03:19 -07:00
Brian C. Lane
eafc2796a1 Update depsolving with suggestions from dnf (#1636239)
The previous method worked, but wasn't exactly idiomatic. This is more
correct, and appears to work the same (templates depsolve, version globs
work, multiple repos work).

Note that this does use a private dnf attribute ._goal, but the word is
that this is going to become a public api soon, so yes it is there on
purpose.
2018-10-12 11:02:13 -07:00
Brian C. Lane
f3cbeb8808 Disable false context-manager pylint error 2018-10-11 10:30:04 -07:00
Brian C. Lane
6fb5910acd Fix directory creation for blueprints
Depending on how lorax-composer is run setting up an empty blueprints
directory can fail. So this moves checking/creation until after the
other directories are created and uses make_owned_dir to make sure
ownership is correct.
2018-10-10 14:25:38 -07:00
Brian C. Lane
708cadee82 Update the tests for new make_dnf_dir arguments.
Use the uid and gid that the test is running as instead of hard-coding
0.
2018-10-10 14:25:38 -07:00
Brian C. Lane
3499426b2d Change make_dnf_dirs to be run as root
It needs to be root in order to set the ownership and permissions on the
directories that are under /var/lib/lorax/composer/

Refactor the directory creation into a utility function, and use a umask
of 0o006 to ensure that the parent directories created do not have o+rw
set on them (makedirs behavior is different between Python 3.6 and 3.7
so umask of 0 doesn't work consistently).
2018-10-10 14:25:38 -07:00
Brian C. Lane
5efce31772 Automatic commit of package [lorax] release [29.18-1].
Created by command:

/usr/bin/tito tag
2018-10-09 11:39:23 -07:00
Brian C. Lane
ce6fa9c05f Add an openstack image type
This is a qcow2 image with cloud-init in the template.
2018-10-09 11:07:46 -07:00
David Shea
a76aad36f6 Add cloud-init to vhd images.
cloud-init can be used in Azure now
2018-10-09 11:34:19 -04:00
David Shea
e116a5f4be Replace /etc/machine-id with an empty file
Since these images can be used to create multiple machines, they should
not have a unique machine-id attached to them. Replace /etc/machine-id
with an empty file so that it will be regenerated at boot time.
2018-10-09 11:34:17 -04:00
Brian C. Lane
f86aaa5d37 Automatic commit of package [lorax] release [29.17-1].
Created by command:

/usr/bin/tito tag
2018-10-08 16:03:05 -07:00
Brian C. Lane
d03e6e8c7e Update cli tests to use composer-cli name 2018-10-08 15:53:15 -07:00
Brian C. Lane
e63da65ad1 Revert "Rename composer-cli to composer"
This reverts commit d540ba3a23.

It conflicts with the PHP dependency manager project named 'composer'
2018-10-08 15:51:24 -07:00
Brian C. Lane
16dccde87f Automatic commit of package [lorax] release [29.16-1].
Created by command:

/usr/bin/tito tag
2018-10-05 14:55:28 -07:00
Brian C. Lane
f24f84282d Work around dnf problem with multiple repos
If a package is in multiple repos dnf may return more than 1 of them
when using best...glob so we pick the highest NEVRA one and install
that.

Related: rhbz#1636239
2018-10-05 09:44:16 -07:00
Lars Karlitski
8f1bc6e882 Add and enable cloud-init for ami images
Images don't work at all on AWS without cloud-init.

Fixes #492
2018-10-05 09:09:10 -07:00
Brian C. Lane
29fe174b6b Switch default platform id to f29 2018-10-05 09:09:10 -07:00
Brian C. Lane
d65a502861 Report an error if the blueprint doesn't exist
composer-cli uses TOML for 'blueprints save' which was returning an
empty 200 response if the blueprint didn't exist. Change this to return
a standard 400 error response if the blueprint doesn't exist.

composer-cli is already setup to handle receiving json when an error is
returned so just the toml API response for `blueprints/save` needed to
be changed.
2018-10-05 09:09:10 -07:00
Lars Karlitski
ec47fe12a9 cli: Clarify error message for unprivileged access
`os.path.exists("/run/weldr/api.socket")` returns False for users which have no
access. This leads to composer printing that the file does not exist, which is
misleading.

Since it's no possible to distinguish the two cases, fix this problem by
combining them and showing a single error message.
2018-10-05 09:09:10 -07:00
Brian C. Lane
c7b4dce2b6 Write a rootpw line if no root customizations in the blueprint
Anaconda requires the root password to be set or locked, so if there
isn't anything setting it we write out 'rootpw --lock'

Also adds tests for this.

Resolves: rhbz#1626122
2018-10-05 09:09:10 -07:00
Brian C. Lane
b35fe9d32a Add beakerlib to Dockerfile.test
Also kill the lorax-composer process and remove /run/weldr/api.socket
so that when this is run with podman you don't get an error about
attempting to tar up the socket.
2018-10-05 09:09:10 -07:00