Commit Graph

1927 Commits

Author SHA1 Message Date
Brian C. Lane
9431a85b85 Remove unneeded else from for/else loop. It confuses pylint
(cherry picked from commit 2950f2641b)
2019-01-08 14:13:22 -08:00
Brian C. Lane
e2a0c38e75 Turn off pylint warning about docstring with backslash
(cherry picked from commit 4fe21135e3)
2019-01-08 14:13:16 -08:00
Brian C. Lane
f51a940718 Turn off smartquotes in Sphinx documentation
It mangles the double dashes in the docstrings, and should close #155

(cherry picked from commit e85bcf4359)
2019-01-08 14:13:07 -08:00
Anthony F McInerney
cb03752810 fixes #543 qemu -nodefconfig deprecated
(cherry picked from commit f66bff5aa7)
2019-01-08 14:13:01 -08:00
Anthony F McInerney
4312408ade fix spinx build warnings
(cherry picked from commit 6bb64f94ff)
2019-01-08 14:12:56 -08:00
David Shea
bc574e2e2f Allow customizations to be specified as a toml list
Support both

  [customizations]
  hostname = "whatever"

and

  [[customizations]]
  hostname = "whatever"

in the blueprint data. The [[ syntax matches the other customization
directives (user, group, sshkey), and as such it's easy to accidentally
use it for the hostname without even realizing it's specifying something
different.

Add some tests for converting customizations to kickstarts.

(cherry picked from commit 35ab6a1336)
2019-01-08 14:12:50 -08:00
Brian C. Lane
a466f26c95 Revert "lorax-composer: Cancel running Anaconda process"
Drop running pkill. This causes problems if more than one is running on
a system (eg. in parallel using mock). It can kill off other processes
unrelated to this instance of anaconda.

This reverts commit e3e892ea43.
2019-01-08 14:12:14 -08:00
Brian C. Lane
8b59860217 Make sure cancel_func is not None
(cherry picked from commit ca2c3d9e77)
2019-01-07 14:11:59 -08:00
Brian C. Lane
970b787e68 Automatic commit of package [lorax] release [28.23-1].
Created by command:

/bin/tito tag
2018-12-17 16:55:19 -08:00
Brian C. Lane
2a86ed148e lorax: Save information about rootfs filesystem size and usage
Run df on the filesystem image after it has been created.
Output will be in program.log, eg:

Running... df /var/tmp/lorax.imgutils.wm04pg_v
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/loop0       1998672 1619508    362780  82% /var/tmp/lorax.imgutils.wm04pg_v
Return code: 0
2018-12-17 16:25:01 -08:00
Brian C. Lane
2aee00aea5 Turn on signed tags when using tito.
This requires a patched version of tito at the moment, and doesn't hurt
to have it set -- it will just continue not signing the tags.
2018-12-17 16:23:38 -08:00
Brian C. Lane
e3e892ea43 lorax-composer: Cancel running Anaconda process
It ends up that this isn't as easy as you'd think. Anaconda sets up some
signal handlers to handle cleanly exiting, but they are not being run
when sent a TERM after package installation has started. I think DNF
resets them causing it to get ignored.

When the cancel is sent it can take several minutes for it to have an
effect. In my testing it usually takes around 2 minutes for anaconda to
notice and exit.

This sends a TERM to the process and then waits for it to exit. When it
returns it then removed any device-mapper devices that were setup for
image installations, removes any hanging loop devices.

It then kills off any process with pyanaconda. in the cmdline, and
anaconda-bus.conf (because anaconda starts a bunch of helpers and if it
doesn't shut down cleanly they remain running).

Resolves: rhbz#1656691
2018-12-17 16:22:37 -08:00
Brian C. Lane
41658811dc Add cancel_func to virt and novirt_install functions
In addition to monitoring the logs for errors, call a function (or
functions) that tell it to cancel the anaconda process and cleanup.

Also check for a cancel after creating the squashfs image for live-iso
since that's a long running process.

This required adding a new argument to a number of existing functions,
passing it down to QEMUInstall and novirt_install where the function is
called.

Resolves: rhbz#1656691
2018-12-17 16:22:37 -08:00
Brian C. Lane
9f0c50ee83 lorax-composer: Check for STATUS before deleting
When there is no run or new symlink do one last check to make sure no
STATUS file was written. If it is missing, go ahead and remove the
results directory.

Related: rhbz#1656691
2018-12-17 16:22:37 -08:00
Brian C. Lane
63d6024230 Check for existing CANCEL request, and exit on FINISHED
If another CANCEL request has already been made just exit from
uuid_cancel. If the build is FINISHED before it times out just exit,
don't remove the finished results.

Related: rhbz#1656691
2018-12-17 16:22:37 -08:00
Vendula Poncova
3211284638 Fix vhd images
Images don't work without these fixes:

* Enable Network Manager.
* Disable cloud-init.
* Add Hyper-V modules into initramfs.

Fixes specific for RHEL:

* Create ifcfg-eth0 required by waagent.
* Install python3 and net-tools required by waagent.

Recommended changes:

* Use recommended kernel boot args.
* Disable kdump.
2018-12-11 08:30:28 -08:00
Brian C. Lane
286f4c1dce Automatic commit of package [lorax] release [28.22-1].
Created by command:

/bin/tito tag
2018-12-06 16:17:37 -08:00
Brian C. Lane
ed611933cb lorax-composer: Handle packages with multiple builds
When the repository has multiple arches, eg. i686 and x86_64, it should
add a new entry to the project's builds list, not create a new project
in the list.

This handles that by adding a modified insort_left function and
examining the packages returned from dnf to make sure they aren't
already listed in the results. It also handles adding them in sorted
order so that no further sorting needs to be done on the results.

Resolves: rhbz#1656642
(cherry picked from commit d18934775c)
2018-12-06 16:07:22 -08:00
Brian C. Lane
dfb2b962b3 lorax-composer: Check the queue and results at startup
If the system ran out of space, or was rebooted unexpectedly, the state
of the queue symlinks, or the results STATUS files may be inconsistent.
This checks them and:
 * Removes broken symlinks from queue/new and queue/run
 * Removes symlinks from run and sets the build to FAILED
 * Sets builds w/o a STATUS to FAILED
 * Sets builds with STATUS of RUNNING to FAILED
 * Creates missing queue/new symlinks to results with STATUS of WAITING

So, any builds that were running during the reboot will be FAILED, and
any that were waiting to be started will be started upon rebooting.

Resolves: rhbz#1647985
(cherry picked from commit 4dd9004d13)
2018-12-06 15:26:59 -08:00
Brian C. Lane
d262f6eaab Automatic commit of package [lorax] release [28.21-1].
Created by command:

/bin/tito tag
2018-11-29 14:58:43 -08:00
Brian C. Lane
c1977b8427 Adjust tmux version in the tests to 2.8 2018-11-29 12:18:41 -08:00
Brian C. Lane
6667521b0a New lorax documentation - 28.21 2018-11-29 11:36:02 -08:00
Brian C. Lane
91257da87e lorax-composer: Install selinux-policy-targeted in images
This is required to ensure that SELinux is configured properly while
building. It fixes the problem with building tar, and should be
installed in the other image types for consistency.

Resolves: rhbz#1645189
(cherry picked from commit 99d867db65)
2018-11-29 11:34:40 -08:00
Brian C. Lane
8105443bc6 Remove setfiles from mkrootfsimage
SELinux applies the correct labels, setfiles is no longer needed.
This allows lorax to run with SELinux in Enforcing mode.

(cherry picked from commit 4a4a415f88)
2018-11-29 11:34:35 -08:00
Brian C. Lane
e52d40216c Remove SELinux Permissive checks
Anaconda, Lorax, lorax-composer, and livemedia-creator can all now run
with SELinux in Enforcing mode. It does not need to be disabled and if
there are denials they should be reported as a bug.

Log the current state of SELinux when starting, update the
documentation.

(cherry picked from commit 080705e8e6)
2018-11-29 11:34:29 -08:00
Brian C. Lane
a5c5cb457e Add --no-system-repos to lorax-composer
Running lorax-composer --no-system-repos will prevent it from copying
the dnf repositories from /etc/yum.repos.d/ into the lorax-composer repo
directory. It will *only* use repositories setup using the sources api
or written to /var/lib/lorax/composer/repos.d/

If lorax-composer has previously been run without this switch the system
repos will need to be removed from the composer/repos.d/ directory. It
would also be a good idea to remove the cached metadata in
/var/tmp/composer/

Resolves: rhbz#1650363
(cherry picked from commit 43ff505804)
2018-11-29 11:33:16 -08:00
Brian C. Lane
8129e5a9f8 Automatic commit of package [lorax] release [28.20-1].
Created by command:

/usr/bin/tito tag
2018-10-29 15:48:24 -07:00
Brian C. Lane
bb1349cade New lorax documentation - 28.20 2018-10-29 15:25:10 -07:00
Brian C. Lane
f5732d21bf Build manpages for composer-cli and lorax-composer
Add manpage creation to make docs target to keep them updated.

(cherry picked from commit 7500a17f27)
(cherry picked from commit d9b282150e)
2018-10-29 15:22:38 -07:00
Brian C. Lane
4e46d776d5 Add tests for ltmpl.py
This covers things like installing globbed package names from multiple
repos, pinned package versions, and ltmpl functions

Related: rhbz#1548586
2018-10-29 13:53:59 -07:00
Brian C. Lane
71be466bde Move get_dnf_base_object into a module
This allows it to be imported by tests.
2018-10-25 11:20:42 -07:00
Brian C. Lane
6e57bfe11d lorax: Fix dnf problems with selecting highest NEVRA from multiple repos
When using package name globs and multiple repos dnf doesn't choose the
highest NEVRA as you would expect, work around this by applying max() to
the duplicate package names in installpkg.

Resolves: rhbz#1548586
2018-10-25 11:15:03 -07:00
Brian C. Lane
32156bd349 Automatic commit of package [lorax] release [28.19-1].
Created by command:

/usr/bin/tito tag
2018-10-12 15:51:00 -07:00
Brian C. Lane
9c758c7c4b Fix directory creation for blueprints
Depending on how lorax-composer is run setting up an empty blueprints
directory can fail. So this moves checking/creation until after the
other directories are created and uses make_owned_dir to make sure
ownership is correct.
2018-10-10 14:25:51 -07:00
Brian C. Lane
1a7b6c74b4 Update the tests for new make_dnf_dir arguments.
Use the uid and gid that the test is running as instead of hard-coding
0.
2018-10-10 14:25:51 -07:00
Brian C. Lane
d170622357 Change make_dnf_dirs to be run as root
It needs to be root in order to set the ownership and permissions on the
directories that are under /var/lib/lorax/composer/

Refactor the directory creation into a utility function, and use a umask
of 0o006 to ensure that the parent directories created do not have o+rw
set on them (makedirs behavior is different between Python 3.6 and 3.7
so umask of 0 doesn't work consistently).
2018-10-10 14:25:51 -07:00
Brian C. Lane
0a96c1eedb Disable false context-manager pylint error 2018-10-09 15:47:38 -07:00
Brian C. Lane
0037f10205 Add an openstack image type
This is a qcow2 image with cloud-init in the template.
2018-10-09 15:17:45 -07:00
David Shea
8908ae8a7a Add cloud-init to vhd images.
cloud-init can be used in Azure now
2018-10-09 15:17:34 -07:00
David Shea
e401f36496 Replace /etc/machine-id with an empty file
Since these images can be used to create multiple machines, they should
not have a unique machine-id attached to them. Replace /etc/machine-id
with an empty file so that it will be regenerated at boot time.
2018-10-09 15:17:24 -07:00
Brian C. Lane
9d73975c9f Update cli tests to use composer-cli name 2018-10-09 15:17:13 -07:00
Brian C. Lane
a0fc9eb7ad Work around dnf problem with multiple repos
If a package is in multiple repos dnf may return more than 1 of them
when using best...glob so we pick the highest NEVRA one and install
that.

Related: rhbz#1636239
2018-10-09 15:17:00 -07:00
Lars Karlitski
577618a1fc Add and enable cloud-init for ami images
Images don't work at all on AWS without cloud-init.

Fixes #492
2018-10-09 15:16:49 -07:00
David Shea
955631b872 Make no-virt generated images sparser
At the end of disk image installs, use fstrim on the generated filesystem to
discard any blocks that were allocated during the install and are now unused.
This will allow tools such as qemu-img to create images that do not include
deleted data.

For raw disk images that do not go through qemu-img, use fallocate --dig-holes
to create sparse holes in place of the unused blocks.
2018-10-09 15:16:38 -07:00
Brian C. Lane
fb1dfc9488 Report an error if the blueprint doesn't exist
composer-cli uses TOML for 'blueprints save' which was returning an
empty 200 response if the blueprint didn't exist. Change this to return
a standard 400 error response if the blueprint doesn't exist.

composer-cli is already setup to handle receiving json when an error is
returned so just the toml API response for `blueprints/save` needed to
be changed.
2018-10-09 15:16:20 -07:00
Lars Karlitski
bb40856f3e cli: Clarify error message for unprivileged access
`os.path.exists("/run/weldr/api.socket")` returns False for users which have no
access. This leads to composer printing that the file does not exist, which is
misleading.

Since it's no possible to distinguish the two cases, fix this problem by
combining them and showing a single error message.
2018-10-09 15:15:57 -07:00
Brian C. Lane
c2b0e8a8bb Write a rootpw line if no root customizations in the blueprint
Anaconda requires the root password to be set or locked, so if there
isn't anything setting it we write out 'rootpw --lock'

Also adds tests for this.

Resolves: rhbz#1626122
2018-10-09 15:15:42 -07:00
Brian C. Lane
558fdecde5 Add beakerlib to Dockerfile.test
Also kill the lorax-composer process and remove /run/weldr/api.socket
so that when this is run with podman you don't get an error about
attempting to tar up the socket.
2018-10-09 15:15:23 -07:00
Brian C. Lane
0a71478ac4 Adjust the new templates for locked root
also remote the auth line so that it uses the defaults.

Related: rhbz#1628645
Related: rhbz#1628646
Related: rhbz#1628647
Related: rhbz#1628648
2018-10-09 15:14:53 -07:00
Brian C. Lane
13e6a68154 Always update repo metadata when building an image
When the kickstart is handed off to Anaconda for building it will
download its own copy of the metadata and re-run the depsolve. So if the
dnf cache isn't current there will be a mismatch and the build will
fail to find some of the versions in final-kickstart.ks

This adds a new context to DNFLock, .lock_check, that will force a check
of the metadata. It also implements its own timeout and forces a
refresh of the metadata when that expires because the dnf expiration
doesn't always work as expected.

Resolves: rhbz#1631561
2018-10-09 15:13:48 -07:00