Commit Graph

1888 Commits

Author SHA1 Message Date
David Shea
252c9f6226 Allow customizations to be specified as a toml list
Support both

  [customizations]
  hostname = "whatever"

and

  [[customizations]]
  hostname = "whatever"

in the blueprint data. The [[ syntax matches the other customization
directives (user, group, sshkey), and as such it's easy to accidentally
use it for the hostname without even realizing it's specifying something
different.

Add some tests for converting customizations to kickstarts.

(cherry picked from commit 35ab6a1336)

Resolves: rhbz#1666517
2019-02-19 14:04:43 -08:00
Jiri Kortus
56d0e704c0 Make sure compose build tests run with SELinux in enforcing mode
Related: rhbz#1654795
2019-02-06 13:59:17 +01:00
Brian C. Lane
47fd6e85b2 Add tests for metapackages and package name globs
This makes sure that depsolving shim installs the shim-* package, and
that depsolving grub2-efi-*-cdboot installs a specific -cdboot package.

Related: rhbz#1641601
2019-01-24 09:57:08 -08:00
Alexander Todorov
2ac027c78b Upgrade pip & setuptools b/c they are rather old
and sometimes Python dependencies fail to install with:

SetuptoolsOutdatedWarning: your setuptools is too old (<12)
2019-01-09 10:55:58 +02:00
Alexander Todorov
177e62a896 Workaround openstacksdk dependency issue
See:
fd61b54679 (diff-b4ef698db8ca845e5845c4618278f29a)

Note: may also affect master/rhel8-branch but haven't seen it
so far. For master we can do:

	dnf install ansible python3-openstacksdk

Not so easy on RHEL 7
2018-12-20 10:55:04 +02:00
Alexander Todorov
7143456043 On Python 2 Azure needs the futures module
this will prevent
ImportError: No module named concurrent.futures

see
https://github.com/Azure/azure-sdk-for-python/issues/2166
2018-12-20 10:55:04 +02:00
Alexander Todorov
d63de76cb3 On RHEL 7 we have Python 2, not Python 3 2018-12-20 10:55:04 +02:00
Alexander Todorov
5dc19500ad On RHEL 7 we have yum instead of dnf 2018-12-20 10:55:04 +02:00
Alexander Todorov
808454b561 On RHEL 7 compose info is compose details 2018-12-20 10:55:04 +02:00
Brian C. Lane
74792f6b6e Report an error if the blueprint doesn't exist
composer-cli uses TOML for 'blueprints save' which was returning an
empty 200 response if the blueprint didn't exist. Change this to return
a standard 400 error response if the blueprint doesn't exist.

composer-cli is already setup to handle receiving json when an error is
returned so just the toml API response for `blueprints/save` needed to
be changed.
2018-12-20 10:55:04 +02:00
Alexander Todorov
aebc3bf280 Build the HTML docs before running tests
on master docs/html/ is part of the git repository, here it is not
2018-12-20 10:55:04 +02:00
Brian C. Lane
dccd283dd2 Disable pylint errors with Flask and gevent
pylint has trouble with Flask response objects, thinking they are tuples
and returning no-member errors. It also doesn't recognize gevent.socket
members like AF_UNIX.
2018-12-20 10:55:04 +02:00
Alexander Todorov
f8137b94db Backport cloud image tests from master 2018-12-20 10:55:04 +02:00
Brian C. Lane
63306c1f64 Fix compose_args for openstack image
RHEL7 arguments are slightly different from the RHEL8 version this was
ported from.

Related: rhbz#1656105
2018-12-14 15:23:43 -08:00
Brian C. Lane
7291c8f8c3 Fix compose_args for vmdk image
RHEL7 arguments are slightly different from the RHEL8 version this was
ported from.

Related: rhbz#1656105
2018-12-14 15:23:43 -08:00
Brian C. Lane
f426a357b1 Fix compose_args for vhd image
RHEL7 arguments are slightly different from the RHEL8 version this was
ported from.

Related: rhbz#1656105
2018-12-14 15:23:43 -08:00
Brian C. Lane
ab17674f75 Fix compose_args for ami image
RHEL7 arguments are slightly different from the RHEL8 version this was
ported from.

Related: rhbz#1656105
2018-12-14 14:19:54 -08:00
Brian C. Lane
5797099552 Update projects list to return only the unique projects
The projects_list function uses a different yum request, and doesn't
return a full project dict. Updated it to also return only the unique
project names.

Related: rhbz#1657055
2018-12-07 15:07:36 -08:00
Brian C. Lane
b449d1cd81 Change yaps_to_module to proj_to_module
It now gets a project dict instead of a yum object, so change it to use
the dict's "name" and update the tests.

Related: rhbz#1657055
2018-12-07 14:41:23 -08:00
Brian C. Lane
0e54983be8 lorax-composer: Handle packages with multiple builds
When the repository has multiple arches, eg. i686 and x86_64, it should
add a new entry to the project's builds list, not create a new project
in the list.

This handles that by adding a modified insort_left function and
examining the packages returned from dnf to make sure they aren't
already listed in the results. It also handles adding them in sorted
order so that no further sorting needs to be done on the results.

Resolves: rhbz#1657055
(cherry picked from commit 663a0dcd73)
2018-12-07 14:10:54 -08:00
Brian C. Lane
72bd52e061 lorax-composer: Check the queue and results at startup
If the system ran out of space, or was rebooted unexpectedly, the state
of the queue symlinks, or the results STATUS files may be inconsistent.
This checks them and:
 * Removes broken symlinks from queue/new and queue/run
 * Removes symlinks from run and sets the build to FAILED
 * Sets builds w/o a STATUS to FAILED
 * Sets builds with STATUS of RUNNING to FAILED
 * Creates missing queue/new symlinks to results with STATUS of WAITING

So, any builds that were running during the reboot will be FAILED, and
any that were waiting to be started will be started upon rebooting.

Resolves: rhbz#1657054
(cherry picked from commit f0bac40d7f)
2018-12-07 13:48:10 -08:00
Brian C. Lane
e1d2e168ed Add an openstack image type
This is a qcow2 image with cloud-init in the template.

(cherry picked from commit 98f8b23129)

Resolves: rhbz#1656105
2018-12-07 13:59:45 -05:00
David Shea
5e64051665 Replace /etc/machine-id with an empty file
Since these images can be used to create multiple machines, they should
not have a unique machine-id attached to them. Replace /etc/machine-id
with an empty file so that it will be regenerated at boot time.

(cherry picked from commit 6fab72d894)

Related: rhbz#1656105
2018-12-07 13:59:45 -05:00
David Shea
4f28d87a6d Add virt guest agents to the qcow2 compose
(cherry picked from commit d5a1993640)

Resolves: rhbz#1656105
2018-12-07 13:59:45 -05:00
David Shea
4862eccd88 Add a vmdk compose type.
This is similar to the AMI type, but also adds open-vm-tools and does not do
anything special to the partitioning

(cherry picked from commit 1056bfc25b)

Resolves: rhbz#1656105
2018-12-07 13:59:45 -05:00
David Shea
202814ae9d Add a vhd compose type for Azure images
This is similar to the AMI compose type, with a handful of additional
changes specific to Azure:

* Add waagent (but leave NetworkManager enabled, despite some of the
  docs)
* Disable cloud-init
* Add Hyper-V modules into initrams.

Fixes specific for RHEL:

* Create ifcfg-eth0 required by waagent.
* Install python3 and net-tools required by waagent.

Recommended changes:

* Use recommended kernel boot args.
* Disable kdump.

(cherry picked from commit e0c236ff36)
(cherry picked from commit da0435bc90)
(cherry picked from commit b594fa99bc)

Resolves: rhbz#1656105
2018-12-07 13:59:45 -05:00
David Shea
61d4353bc7 Add an ami compose type for AWS images
This differs from lmc's --make-ami in that creates a full disk image instead of
an fsimage. Create a raw disk image with a / and /boot partitions, and enable
sshd, chronyd, and cockpit by default.

(cherry picked from commit 18188bf6cf)
(cherry picked from commit 81d38b6445)

Resolves: rhbz#1656105
2018-12-07 13:59:45 -05:00
David Shea
1ca0393771 Remove --fstype from the generated part line
Instead of specifying the fstype, just let anaconda use the default.

(cherry picked from commit 847fff4e11)

Related: rhbz#1656105
2018-12-07 13:59:45 -05:00
Brian C. Lane
60c7ab3e91 lorax-composer: Install selinux-policy-targeted in images
This is required to ensure that SELinux is configured properly while
building. It fixes the problem with building tar, and should be
installed in the other image types for consistency.

Resolves: rhbz#1654795
2018-12-06 16:36:36 -08:00
Brian C. Lane
6b17850739 Remove setfiles from mkrootfsimage
SELinux applies the correct labels, setfiles is no longer needed.
This allows lorax to run with SELinux in Enforcing mode.

Resolves: rhbz#1654795
2018-12-06 16:36:24 -08:00
Brian C. Lane
482d6277e0 Remove SELinux Permissive checks
Anaconda, Lorax, lorax-composer, and livemedia-creator can all now run
with SELinux in Enforcing mode. It does not need to be disabled and if
there are denials they should be reported as a bug.

Log the current state of SELinux when starting, update the
documentation.

Resolves: rhbz#1654795
2018-12-06 16:36:03 -08:00
Brian C. Lane
a40fc22047 Automatic commit of package [lorax-composer] release [19.7.27-1].
Created by command:

/usr/bin/tito tag
2018-10-22 12:03:23 -07:00
Brian C. Lane
bc79b636bb Use matchPackageNames instead of searchNames
Some kickstart templates use globbing to match packages, searchNames
doesn't support this.

Add fallback code to find packages that are just a dep, matching the
behavior of yum.install()

This fixes things so that it can depsolve package names with globs in
them like grub2-efi-*-cdboot, and deps like shim and grub2-efi.

Results are still filtered by the version glob, with the highest NEVRA
selected for installation.

Resolves: rhbz#1641601
2018-10-22 11:03:40 -07:00
Brian C. Lane
e167e5d09f Automatic commit of package [lorax-composer] release [19.7.26-1].
Created by command:

/usr/bin/tito tag
2018-10-08 17:04:23 -07:00
Brian C. Lane
250e68840b Revert "Rename composer-cli to composer"
This reverts commit b8da0a8c8e.

It conflicts with the PHP dependency manager project named 'composer'

Related: rhbz#1635760
2018-10-08 16:55:57 -07:00
Brian C. Lane
b2f94675ce Automatic commit of package [lorax-composer] release [19.7.25-1].
Created by command:

/usr/bin/tito tag
2018-10-05 09:02:50 -07:00
Lars Karlitski
b8da0a8c8e Rename composer-cli to composer
Resolves: rhbz#1635760
2018-10-03 10:43:01 -07:00
Brian C. Lane
a7ab87832b Automatic commit of package [lorax-composer] release [19.7.24-1].
Created by command:

/usr/bin/tito tag
2018-10-01 13:50:24 -07:00
Brian C. Lane
0f130dd91c Add a test for repo metadata expiration
This tests to make sure that the metadata timer is working (by setting
it to 10s and adding a new package to the repo), and that
YumLock.lock_check immediately picks up a new package.

This depends on rpmfluff which is available from Fedora or EPEL repos.

Related: rhbz#1632962
2018-10-01 13:37:44 -07:00
Brian C. Lane
c9582a0468 Create a new YumBase object when repodata changes
The problem this solves is that yum really isn't designed to be part of\
a long running daemon. So when repodata changes upstream, even when
you force it to download the new metadata, it doesn't change in memory
so you end up with lorax-composer depsolving against old versions, and
anaconda depsolving against new versions (because it sets up its own
YumBase and cache) and then the kickstart is no longer valid.

To solve this I have
 - Added a 6h timeout to the metadata check (because yum's doesn't work
   in this situation).
 - Added a metadata check to the YumLock .lock property, but only when
   the timeout expires.
 - Added a new .lock_check property to YumLock that always checks the
   metadata and resets the timeout.

If it has changed it does its best to tear down the existing YumBase,
deleting as much as it can in hopes it doesn't leak memory. And then it
sets up a totally new YumBase with the new repodata.

Resolves: rhbz#1632962
2018-10-01 13:37:44 -07:00
Brian C. Lane
6fd0e71530 Fix projects_depsolve_with_size version globbing
Use a common _depsolve function for projects_depsolve and
projects_depsolve_with_size so that it always uses the correct version
glob support when depsolving blueprints and templates.

Resolves: rhbz#1628114
2018-10-01 11:09:20 -07:00
Brian C. Lane
f8f49159ad Add a version glob test forprojects_depsolve_with_size
Resolves: rhbz#1628114
2018-10-01 11:09:20 -07:00
Brian C. Lane
956b29eb64 Add tests for setting root password and ssh key with blueprints
Related: rhbz#1626120
2018-09-28 15:21:11 -07:00
Brian C. Lane
6e92d52c14 Use rootpw for setting the root password instead of user
Ends up you cannot use the kickstart user command on root, since it
already exists, so we have to translate that into a rootpw command.

So [[customizations.user]] with name = "root" only support key, which
will set the ssh key, and password which will use rootpw to set the
password. plain text or encrypted are supported.

Related: rhbz#1626120
2018-09-28 14:30:58 -07:00
Brian C. Lane
f113766f22 Lock the root account, except on live-iso
If we leave the root account w/o a password people will use it that way,
leading to insecure images. Also if we use a default password. So lock
the root account in the templates.

Users will need to do one of these things:
 1. Use [[customizations.user]] in their blueprint to configure root or
    another user.
 2. Use [[customizations.sshkey]] to set a key for root
 2. Install a package that configures a user at install time
 3. Install a package that sets up a user at boot time (eg. cloud-init)

This also drops the auth line from the kickstart templates, allowing it
to use the default password algoritm instead of md5.

Resolves: rhbz#1626120
2018-09-28 14:30:58 -07:00
Brian C. Lane
2876211b14 Automatic commit of package [lorax-composer] release [19.7.23-1].
Created by command:

/usr/bin/tito tag
2018-09-19 08:30:34 -07:00
Brian C. Lane
2a85694c9b Fix depsolve version globbing
The blueprint version glob was being applied to the whole package NEVRA
by yum (it lacks a separate API for just globbing versions), so this
implements that in filterVersionGlob using fnmatchcase on the package
names, and the yum package verGT comparison on the versions for the
selected package.

Also includes tests.

Resolves: rhbz#1628114
2018-09-19 08:19:59 -07:00
Brian C. Lane
9685fdd7aa Fix /compose/cancel API documentation
It said /blueprints/cancel which is incorrect.
2018-08-29 10:06:25 -07:00
Brian C. Lane
b9afb2eb6b Automatic commit of package [lorax-composer] release [19.7.22-1].
Created by command:

/usr/bin/tito tag
2018-08-27 15:06:25 -07:00
Brian C. Lane
fd54205a5f Fix composer-cli blueprints changes to get correct total
blueprints/changes is different, each blueprint has it's own total,
limited by the call's limit. So it needs to find the max total of all
the requested blueprints.
2018-08-23 16:21:54 -07:00