Commit Graph

100 Commits

Author SHA1 Message Date
Brian C. Lane
4b87dc09d0 Change customizations.firewall to append items instead of replace
To maintain consistency with the other options this changes firewall to
combine the existing settings from the image template with the settings
from the blueprint.

Also updated the docs, added a new test for it, and sorted the output
for consistency.

(cherry picked from commit 3e08389a0f)
(cherry picked from commit 9d48aa4d92)
2019-05-06 12:09:00 -07:00
Brian C. Lane
0945c9b81f lorax-composer: Add services support to blueprints
Add support for enabling and disabling systemd services in the
blueprint. It works like this:

    [customizations.services]
    enabled = ["sshd", "cockpit.socket", "httpd"]
    disabled = ["postfix", "telnetd"]

They are *added* to any existing settings in the kickstart templates.

(cherry picked from commit 1111aee92d)
(cherry picked from commit 69cc1b17f2)
2019-05-06 12:09:00 -07:00
Brian C. Lane
f39e965fb2 lorax-composer: Add firewall support to blueprints
You can now open ports in the firewall, using port numbers or service
names:

    [customizations.firewall]
    ports = ["22:tcp", "80:tcp", "imap:tcp", "53:tcp", "53:udp"]

Or enable/disable services registered with firewalld:

     [customizations.firewall.services]
     enabled = ["ftp", "ntp", "dhcp"]
     disabled = ["telnet"]

If the template contains firewall --disabled it cannot be overridden,
under the assumption that it is required for the image to boot in the
selected environment.

(cherry picked from commit 4d35668ab5)
(cherry picked from commit 9f1756cc27)
2019-05-06 12:09:00 -07:00
Brian C. Lane
725a0437c7 lorax-composer: Add locale support to blueprints
You can now set the keyboard layout and language. Eg.

[customizations.locale]
languages = ["en_CA.utf8", "en_HK.utf8"]
keyboard = "de (dvorak)"

Existing entries in the kickstart templates are replaced with the new
ones. If there are no entries then it will default to 'keyboard us' and
'lang en_US.UTF-8'

Includes tests, and leaves the existing keyboard and lang entries in the
templates with a note that they can be replaced by the blueprint.

(cherry picked from commit e5a8700bdf)
(cherry picked from commit 49259c607b)
2019-05-06 12:09:00 -07:00
Brian C. Lane
2c03c49149 lorax-composer: Add timezone support to blueprint
For example:

[customizations.timezone]
timezone = "US/Samoa"
ntpservers = ["0.pool.ntp.org"]

Also includes tests.

This removes the timezone kickstart command from all of the templates
except for google.ks which needs to set it's own ntp servers and timezone.

If timezone isn't included in the blueprint, and it is not already in a
template, it will be set to 'timezone UTC' by default.

If timezone is set in a template it is left as-is, under the assumption
that the image type requires it to boot correctly.

(cherry picked from commit 9bdbb29662)
(cherry picked from commit 40150508b8)
2019-05-06 12:09:00 -07:00
Brian C. Lane
5230b8d30e tests: Cleanup on failure of in_tempdir
Otherwise other tests will also fail when they try to run from the wrong
directory.

(cherry picked from commit 65b8a2be68)
(cherry picked from commit 94e2651cf6)
2019-05-06 12:08:17 -07:00
David Shea
9a4a45f05a Add a compose type for alibaba.
This compose type creates a partitioned disk as a qcow2 file, but with
only one partition instead of using a separate /boot.

(cherry picked from commit 44e14176bb)
(cherry picked from commit 0f59e576d8)
2019-05-06 12:07:29 -07:00
David Shea
0220077242 Add a new compose type for Hyper-V
This is based on the VHD compose type, with the following differences:

  * Use the vhdx format instead of vhd
  * No WALinuxAgent
  * Install hyperv-daemons

The hyperv-daemons are activated through udev rules, so there is no need
to add them to the services line.

(cherry picked from commit 434fe58c28)
(cherry picked from commit 367c2583e4)
2019-05-06 12:07:29 -07:00
David Shea
69caef3c2d Add a compose type for Google Compute Engine
(cherry picked from commit bd804c271b)
(cherry picked from commit 846d839980)
2019-05-06 12:07:29 -07:00
David Shea
b52877b63c Support compressing single files.
Modify imgutils.compress to allow the "rootdir" argument to be either a
directory or a single file to add to an archive.

(cherry picked from commit c585b91422)
(cherry picked from commit 5308e7bfec)
2019-05-06 12:07:29 -07:00
David Shea
c74e50d99d Add an option to align the image size to a multiplier.
If provided, round the disk image size up to a multiple of the value.
This allows for image formats with specific size-alignment requirements
(e.g., disk size must be in GiB).

(cherry picked from commit 8ef4f431d8)
(cherry picked from commit b2a33d5654)
2019-05-06 12:07:29 -07:00
Brian C. Lane
4fa08eb8e1 Add more tests for gitrpm.py
Make sure that dotfiles are installed when not directly under /
Make sure / is not packaged in the rpm (it will cause a conflict with
the filesystem package).
Make sure that using destination="/" works

(cherry picked from commit 6fd06c6931)
(cherry picked from commit d5d60ebe6d)
2019-05-06 12:07:06 -07:00
Brian C. Lane
7a86aa1d17 pylorax.ltmpl: Add a test for missing quotes
It should raise an error if a quote is missing in the template.

(cherry picked from commit df3aeee802)
(cherry picked from commit 1d4f47e26f)
2019-05-06 12:07:06 -07:00
Brian C. Lane
c5108bce9f Add tests using repos.git in blueprints
- Check final-kickstart.ks for the rpm source
- Check final-kickstart.ks for the rpm package name and version
- Make sure depsolve works
- Make sure errors from a bad repo are returned correctly
- Make sure errors from a bad reference are returned correctly

This moves _wait_for_status into a helper function so it can be shared
between the test classes.

(cherry picked from commit 8c2184d59e)
(cherry picked from commit 78f79a94ec)
2019-05-06 12:06:06 -07:00
Brian C. Lane
5042867d19 Move git repo creation into tests/lib.py
This way it can be shared with test_server.py

(cherry picked from commit 02f757d231)
(cherry picked from commit 136f9d877a)
2019-05-06 12:06:06 -07:00
Brian C. Lane
b25995d9d4 rpmgit: catch potential errors while running git
Log them and report them as RuntimeError. Also add a couple tests for
them.

(cherry picked from commit 61efa91a03)
(cherry picked from commit 087b0fe8c6)
2019-05-06 12:06:06 -07:00
Brian C. Lane
7a6ebb0ad3 tests: Add test for Recipe.freeze() function
The freeze function was not being tested. Add a test for it using the
repos.git test recipe.

(cherry picked from commit c26477a63c)
(cherry picked from commit bdf5572bc6)
2019-05-06 12:05:42 -07:00
Brian C. Lane
8696f197d4 Add repos.git support to lorax-composer builds
This hooks up creation of the rpm to the build, adds it to the
kickstart, and passes the url to Anaconda. The dnf repo with the rpms is
created under the results directory so it will be included when
downloading the build's results.

(cherry picked from commit cd8c884adb)
(cherry picked from commit 2e596010d3)
2019-05-06 12:05:34 -07:00
Brian C. Lane
0169422746 Add pylorax.api.gitrpm module and tests
This handles creating the rpm from the dictionary describing the
repository and rpm. Also adds tests for archive and rpm creation.

(cherry picked from commit f6f2308765)
(cherry picked from commit efc77c1d71)
2019-05-06 12:05:34 -07:00
Brian C. Lane
21b03c2108 Add support for [[repos.git]] section to blueprints
This adds support, documentation, and testing for a [[repos.git]]
blueprint section that can be used to install files from a git
repository. It will create an rpm that will be added to the build,
and included in the metadata that can be downloaded. This allows you to
accurately keep track of the source of configuration files and extra
metadata that is added to the build.

The source repo and reference will be listed in the rpm's summary making
it easy to discover on the installed system.

(cherry picked from commit d7b96c8f0f)
(cherry picked from commit 047f174dcf)
2019-05-06 12:05:34 -07:00
Brian C. Lane
2dd3dd54c5 lorax-composer: Return UnknownBlueprint errors when using deleted blueprints
Reading a blueprint wasn't checking to see if it had been deleted so it
was returning the most recent commit before it had been deleted. This
allowed things like starting a compose with a blueprint that technically
doesn't exist.

One exception to this is the /changes/ route, it must be available so
that you can use the commit hash to undo a delete.

This also adds tests for the various operations.

Resolves: rhbz#1682113
(cherry picked from commit d32f477e0b)
(cherry picked from commit 82aa9cdbc6)
2019-05-06 12:00:22 -07:00
Brian C. Lane
d0458750df lorax-composer: Delete workspace copy when deleting blueprint
Also extends the blueprint delete test to also check the workspace.

(cherry picked from commit 26bd2c1378)
(cherry picked from commit 5c0f127b3c)
2019-05-06 12:00:22 -07:00
Brian C. Lane
35faa61f7e lorax-composer: pass customization.kernel append to extra_boot_args
This allows iso builds to include the extra kernel boot parameters by
passing them to the arch-specific live/*tmpl template.

Also adds tests to make sure it is written to config.toml in the build
metadata.

(cherry picked from commit 5dea308080)
(cherry picked from commit 2861bdb95e)
2019-05-06 11:56:17 -07:00
Brian C. Lane
afa68df873 lorax-composer: Add the ability to append to the kernel command-line
Sometimes it is necessary to modify the kernel command-line of the
image, this adds support for a [customizations.kernel] section to the
blueprint:

[customizations.kernel]
append = "nosmt=force"

This will be appended to the kickstart's bootloader --append argument.

Includes tests for modifying the bootloader line, the kickstart
template, and examining the final-kickstart.ks created for a compose.

(cherry picked from commit 59464286f9)
(cherry picked from commit c5f4dfe113)
2019-05-06 11:54:56 -07:00
Brian C. Lane
1fd73cde52 Move the package requirements for live-iso setup out of the template
In order to support iso creation on multiple arches with the templates
we need to be able to select different packages based on arch.
lorax-composer uses the arch-specific Lorax templates in order to
generate the output iso so this patch:

1. Creates a new template and type to parse it, live-install.tmpl
   which contains only installpkg commands and #if clauses for arch
2. Removes bootloader related packages from the live-iso.ks
3. Remove dracut-config-rescue exclusion because it can cause problems
   with some blueprints.
4. Switch logo requirement to system-logos which is satisfied by
   generic-logos or fedora-logos. This prevents conflicts when a blueprint
   installs fedora-release-workstation.

So in the future, if x86.tmpl, etc. need a new package to support
creating the iso it should be added to the correct section in
./share/live/live-install.tmpl

(cherry picked from commit 3b8de2a233)
2019-05-06 11:45:32 -07:00
Brian C. Lane
19375b9d4e livemedia-creator: Add support for reqpart kickstart command
reqpart can be used to make kickstarts more platform agnostic, creating
needed partitions without lmc having to keep track of the arch-specific
needs. eg. ppc64 needs prepboot and /boot

This increases the size of the disk based on whether reqpart or
reqpart --add-boot is in the kickstart.

Note that this is only valid for partitioned disk output types, not
for filesystem images or live iso output.

(cherry picked from commit b47554d716)
2019-05-06 11:45:32 -07:00
Brian C. Lane
3d2c085cf0 Clarify the ks repo only error message
This also moves the run_creator kickstart checks into check_kickstart
so that tests may be added.

This will close #164

(cherry picked from commit 3676cb65bb)
2019-05-06 11:45:09 -07:00
Brian C. Lane
30098a52e3 tests: Fix makeFakeRPM calls
(cherry picked from commit 28bd68793b)
2019-05-03 17:05:57 -07:00
Brian C. Lane
51a9c28bc6 Add tests for partitioned disk images
The setup for this is a bit complex, so it really ends up testing things
twice.

(cherry picked from commit fb4e6f2588)
2019-05-03 16:39:18 -07:00
Brian C. Lane
e7310c7260 Add tests for pylorax.imgutils
Some of these can only run as root on a real system with access to loop
devices. They are skipped when running in a container.

(cherry picked from commit 063a1770e1)
2019-05-03 16:39:05 -07:00
Brian C. Lane
0aa206f661 Add tests to test_creator.py
Add a /.in-container file to the container root so that tests requiring root
and loop device support will be skipped when running in a container.

(cherry picked from commit bab4b20d0d)
2019-05-03 16:39:01 -07:00
Brian C. Lane
ef45d169b0 Add some tests for creator.py
This is complicated by the fact that much of this module requires mount.
So for now just test the things that don't need mount.

(cherry picked from commit 134a333d92)
2019-05-03 16:38:47 -07:00
Brian C. Lane
c63b2e3881 tests: Add executils test
(cherry picked from commit fb87db0e1e)
2019-05-03 16:38:38 -07:00
Brian C. Lane
c6b9a72778 tests: Add sysutils test
(cherry picked from commit cc69478249)
2019-05-03 16:38:33 -07:00
Brian C. Lane
c34a0fb535 tests: Add discinfo test
(cherry picked from commit 2e36e941ce)
2019-05-03 16:38:28 -07:00
Brian C. Lane
c5e839838f tests: Add treeinfo test
(cherry picked from commit e8fd8a496b)
2019-05-03 16:38:22 -07:00
Brian C. Lane
47a909209f Remove duplicate repositories from the sources list
In some cases when the host has, for whatever reason, multiple copies of
the same repo listed the build may fail with an error about running out
of space.

So this commit removes duplicate entries after the host's repos have been
loaded. It also adjusts some of the test repos to use different
temporary repo names for the tests.

(cherry picked from commit 98482e444d)
2019-01-30 08:43:37 -08:00
Brian C. Lane
76071db226 Remove unneeded else from for/else loop. It confuses pylint
(cherry picked from commit 2950f2641b)
2019-01-08 13:47:21 -08:00
David Shea
cfff807df8 Allow customizations to be specified as a toml list
Support both

  [customizations]
  hostname = "whatever"

and

  [[customizations]]
  hostname = "whatever"

in the blueprint data. The [[ syntax matches the other customization
directives (user, group, sshkey), and as such it's easy to accidentally
use it for the hostname without even realizing it's specifying something
different.

Add some tests for converting customizations to kickstarts.

(cherry picked from commit 35ab6a1336)
2019-01-08 13:46:08 -08:00
Brian C. Lane
3f034a981d lorax-composer: Handle packages with multiple builds
When the repository has multiple arches, eg. i686 and x86_64, it should
add a new entry to the project's builds list, not create a new project
in the list.

This handles that by adding a modified insort_left function and
examining the packages returned from dnf to make sure they aren't
already listed in the results. It also handles adding them in sorted
order so that no further sorting needs to be done on the results.

Resolves: rhbz#1656642
(cherry picked from commit d18934775c)
2018-12-06 16:06:45 -08:00
Brian C. Lane
574360f7d0 lorax-composer: Check the queue and results at startup
If the system ran out of space, or was rebooted unexpectedly, the state
of the queue symlinks, or the results STATUS files may be inconsistent.
This checks them and:
 * Removes broken symlinks from queue/new and queue/run
 * Removes symlinks from run and sets the build to FAILED
 * Sets builds w/o a STATUS to FAILED
 * Sets builds with STATUS of RUNNING to FAILED
 * Creates missing queue/new symlinks to results with STATUS of WAITING

So, any builds that were running during the reboot will be FAILED, and
any that were waiting to be started will be started upon rebooting.

Resolves: rhbz#1647985
(cherry picked from commit 4dd9004d13)
2018-12-06 15:26:40 -08:00
Brian C. Lane
1006af4232 Add tests for ltmpl.py
This covers things like installing globbed package names from multiple
repos, pinned package versions, and ltmpl functions

Related: rhbz#1548586
(cherry picked from commit a4783ba29f)
2018-10-29 14:49:49 -07:00
Brian C. Lane
bdbd8258f0 Fix singlerepo test for f29 2018-10-29 14:48:40 -07:00
Brian C. Lane
88a0b475e9 Update the projects tests to use DNF Repo object
Stop using fake dnf object, use the real thing. This will help catch
problems with dnf returning unexpected types like VectorString.

(cherry picked from commit 27aff75aa3)
2018-10-29 14:27:23 -07:00
Brian C. Lane
708cadee82 Update the tests for new make_dnf_dir arguments.
Use the uid and gid that the test is running as instead of hard-coding
0.
2018-10-10 14:25:38 -07:00
Brian C. Lane
ce6fa9c05f Add an openstack image type
This is a qcow2 image with cloud-init in the template.
2018-10-09 11:07:46 -07:00
Brian C. Lane
c7b4dce2b6 Write a rootpw line if no root customizations in the blueprint
Anaconda requires the root password to be set or locked, so if there
isn't anything setting it we write out 'rootpw --lock'

Also adds tests for this.

Resolves: rhbz#1626122
2018-10-05 09:09:10 -07:00
Brian C. Lane
1f514a3e55 Adjust projects test for DNF 3.6.1 tuple issue 2018-10-05 09:09:10 -07:00
Brian C. Lane
5184ccef9f Add a test for repo metadata expiration
This tests to make sure that the metadata timer is working (by setting
it to 10s and adding a new package to the repo), and that
DNFLock.lock_check immediately picks up a new package.

This depends on rpmfluff which is available from Fedora or EPEL repos.

Related: rhbz#1631561
2018-10-05 09:09:10 -07:00
Brian C. Lane
1c0705f4e0 Add tests for setting root password and ssh key with blueprints
Related: rhbz#1626120
2018-10-05 09:09:10 -07:00