Commit Graph

176 Commits

Author SHA1 Message Date
Brian C. Lane
69cc1b17f2 lorax-composer: Add services support to blueprints
Add support for enabling and disabling systemd services in the
blueprint. It works like this:

    [customizations.services]
    enabled = ["sshd", "cockpit.socket", "httpd"]
    disabled = ["postfix", "telnetd"]

They are *added* to any existing settings in the kickstart templates.

(cherry picked from commit 1111aee92d)
2019-05-03 14:22:23 -07:00
Brian C. Lane
9f1756cc27 lorax-composer: Add firewall support to blueprints
You can now open ports in the firewall, using port numbers or service
names:

    [customizations.firewall]
    ports = ["22:tcp", "80:tcp", "imap:tcp", "53:tcp", "53:udp"]

Or enable/disable services registered with firewalld:

     [customizations.firewall.services]
     enabled = ["ftp", "ntp", "dhcp"]
     disabled = ["telnet"]

If the template contains firewall --disabled it cannot be overridden,
under the assumption that it is required for the image to boot in the
selected environment.

(cherry picked from commit 4d35668ab5)
2019-05-03 14:22:23 -07:00
Brian C. Lane
49259c607b lorax-composer: Add locale support to blueprints
You can now set the keyboard layout and language. Eg.

[customizations.locale]
languages = ["en_CA.utf8", "en_HK.utf8"]
keyboard = "de (dvorak)"

Existing entries in the kickstart templates are replaced with the new
ones. If there are no entries then it will default to 'keyboard us' and
'lang en_US.UTF-8'

Includes tests, and leaves the existing keyboard and lang entries in the
templates with a note that they can be replaced by the blueprint.

(cherry picked from commit e5a8700bdf)
2019-05-03 14:22:23 -07:00
Brian C. Lane
40150508b8 lorax-composer: Add timezone support to blueprint
For example:

[customizations.timezone]
timezone = "US/Samoa"
ntpservers = ["0.pool.ntp.org"]

Also includes tests.

This removes the timezone kickstart command from all of the templates
except for google.ks which needs to set it's own ntp servers and timezone.

If timezone isn't included in the blueprint, and it is not already in a
template, it will be set to 'timezone UTC' by default.

If timezone is set in a template it is left as-is, under the assumption
that the image type requires it to boot correctly.

(cherry picked from commit 9bdbb29662)
2019-05-03 14:22:23 -07:00
Jiri Kortus
d78f58a0cb Add test for starting compose with deleted blueprint
(cherry picked from commit 21e141a84c)
2019-05-03 14:22:23 -07:00
Chris Roberts
03bbd030a0 Update VMware info for VMware testing
(cherry picked from commit 0fa5af2e30)
2019-05-03 14:22:23 -07:00
Brian C. Lane
94e2651cf6 tests: Cleanup on failure of in_tempdir
Otherwise other tests will also fail when they try to run from the wrong
directory.

(cherry picked from commit 65b8a2be68)
2019-05-03 14:22:23 -07:00
Alexander Todorov
eb4a8ac3b1 Change [[modules]] to [[packages]] in tests
> bcl: this should really be [[packages]] since [[modules]] is
> going to change to actual modules at some point

(cherry picked from commit 933fad9ded)
2019-05-03 14:22:23 -07:00
Alexander Todorov
31b260ccf2 Add new test to verify compose paths exist
because they are mentioned in the official documentation

(cherry picked from commit e45e645fd0)
2019-05-03 14:22:23 -07:00
Alexander Todorov
ce549d861d Add new sanity tests for blueprints
- verify SemVer .patch number will be automatically updated when
  we push the blueprint a second time without changing version
- verify show displays the content in TOML format and it matches
  what is on disk. Because of that also start with empty packages
  and groups fields in the initial toml. If they are missing they
  will be added automatically by lorax-composer and this simplifies
  the test
- verify delete works

(cherry picked from commit 7cbb324ae5)
2019-05-03 14:22:23 -07:00
Jiri Kortus
08ae3613fe Fixes for locked root account test
Fixes related to a few issues in the locked root account test I somehow
managed to overlook in the initial commit.

(cherry picked from commit 209bdd6911)
2019-05-03 14:22:23 -07:00
Jan Stodola
ae5475c178 Fix keeping files on Amazon s3
(cherry picked from commit b57de93468)
2019-05-03 14:22:23 -07:00
Jan Stodola
0029f23b0e Allow to keep objects in AWS
Instances, Volumes, Snapshots, AMIs and s3 objects with the "keep_me" tag will
not be deleted automatically even if they are older then the specified time limit.

(cherry picked from commit db7b1e4fcd)
2019-05-03 14:22:23 -07:00
David Shea
0f59e576d8 Add a compose type for alibaba.
This compose type creates a partitioned disk as a qcow2 file, but with
only one partition instead of using a separate /boot.

(cherry picked from commit 44e14176bb)
2019-05-03 14:22:23 -07:00
David Shea
367c2583e4 Add a new compose type for Hyper-V
This is based on the VHD compose type, with the following differences:

  * Use the vhdx format instead of vhd
  * No WALinuxAgent
  * Install hyperv-daemons

The hyperv-daemons are activated through udev rules, so there is no need
to add them to the services line.

(cherry picked from commit 434fe58c28)
2019-05-03 14:22:23 -07:00
David Shea
f35425795e Add a compose check for google cloud images.
(cherry picked from commit c6fcf9d1eb)
2019-05-03 14:22:23 -07:00
David Shea
846d839980 Add a compose type for Google Compute Engine
(cherry picked from commit bd804c271b)
2019-05-03 14:22:22 -07:00
David Shea
5308e7bfec Support compressing single files.
Modify imgutils.compress to allow the "rootdir" argument to be either a
directory or a single file to add to an archive.

(cherry picked from commit c585b91422)
2019-05-03 14:22:22 -07:00
David Shea
b2a33d5654 Add an option to align the image size to a multiplier.
If provided, round the disk image size up to a multiple of the value.
This allows for image formats with specific size-alignment requirements
(e.g., disk size must be in GiB).

(cherry picked from commit 8ef4f431d8)
2019-05-03 14:22:22 -07:00
Brian C. Lane
d5d60ebe6d Add more tests for gitrpm.py
Make sure that dotfiles are installed when not directly under /
Make sure / is not packaged in the rpm (it will cause a conflict with
the filesystem package).
Make sure that using destination="/" works

(cherry picked from commit 6fd06c6931)
2019-05-03 14:22:22 -07:00
Brian C. Lane
1d4f47e26f pylorax.ltmpl: Add a test for missing quotes
It should raise an error if a quote is missing in the template.

(cherry picked from commit df3aeee802)
2019-05-03 14:22:22 -07:00
Jiri Kortus
7c093ecbb4 Add checks for disabled root account
The root account checks are applied to generated and deployed images
to make sure that root account is locked, except for live ISO.

(cherry picked from commit 655e7e40c0)
2019-05-03 14:22:22 -07:00
Chris Roberts
7bf3371e33 Update datastore for VMware testing
(cherry picked from commit 32d5ff8615)
2019-05-03 14:22:22 -07:00
Brian C. Lane
78f79a94ec Add tests using repos.git in blueprints
- Check final-kickstart.ks for the rpm source
- Check final-kickstart.ks for the rpm package name and version
- Make sure depsolve works
- Make sure errors from a bad repo are returned correctly
- Make sure errors from a bad reference are returned correctly

This moves _wait_for_status into a helper function so it can be shared
between the test classes.

(cherry picked from commit 8c2184d59e)
2019-05-03 14:22:22 -07:00
Brian C. Lane
136f9d877a Move git repo creation into tests/lib.py
This way it can be shared with test_server.py

(cherry picked from commit 02f757d231)
2019-05-03 14:02:50 -07:00
Brian C. Lane
087b0fe8c6 rpmgit: catch potential errors while running git
Log them and report them as RuntimeError. Also add a couple tests for
them.

(cherry picked from commit 61efa91a03)
2019-05-03 14:02:38 -07:00
Brian C. Lane
bdf5572bc6 tests: Add test for Recipe.freeze() function
The freeze function was not being tested. Add a test for it using the
repos.git test recipe.

(cherry picked from commit c26477a63c)
2019-05-03 14:02:26 -07:00
Brian C. Lane
2e596010d3 Add repos.git support to lorax-composer builds
This hooks up creation of the rpm to the build, adds it to the
kickstart, and passes the url to Anaconda. The dnf repo with the rpms is
created under the results directory so it will be included when
downloading the build's results.

(cherry picked from commit cd8c884adb)
2019-05-03 14:02:09 -07:00
Brian C. Lane
efc77c1d71 Add pylorax.api.gitrpm module and tests
This handles creating the rpm from the dictionary describing the
repository and rpm. Also adds tests for archive and rpm creation.

(cherry picked from commit f6f2308765)
2019-05-03 14:02:00 -07:00
Brian C. Lane
047f174dcf Add support for [[repos.git]] section to blueprints
This adds support, documentation, and testing for a [[repos.git]]
blueprint section that can be used to install files from a git
repository. It will create an rpm that will be added to the build,
and included in the metadata that can be downloaded. This allows you to
accurately keep track of the source of configuration files and extra
metadata that is added to the build.

The source repo and reference will be listed in the rpm's summary making
it easy to discover on the installed system.

(cherry picked from commit d7b96c8f0f)
2019-05-03 14:01:47 -07:00
Alexander Todorov
1053c7477e Allow overriding $CLI outside test scripts
this will allow you to test against installed RPM like so:

    # export CLI="/usr/bin/composer-cli"
    # make test_images

If you already have lorax-composer running then you can directly
execute test scripts:

    # ./tests/cli/test_build_and_deploy_aws.sh

(cherry picked from commit 85dfbd7911)
2019-05-03 13:59:51 -07:00
Alexander Todorov
60daacfc38 New test: Build live-iso and boot with KVM
explicitly enables sshd for live-iso during testing

(cherry picked from commit eef11ef405)
2019-05-03 13:58:54 -07:00
Brian C. Lane
82aa9cdbc6 lorax-composer: Return UnknownBlueprint errors when using deleted blueprints
Reading a blueprint wasn't checking to see if it had been deleted so it
was returning the most recent commit before it had been deleted. This
allowed things like starting a compose with a blueprint that technically
doesn't exist.

One exception to this is the /changes/ route, it must be available so
that you can use the commit hash to undo a delete.

This also adds tests for the various operations.

Resolves: rhbz#1682113
(cherry picked from commit d32f477e0b)
2019-05-03 13:56:52 -07:00
Brian C. Lane
5c0f127b3c lorax-composer: Delete workspace copy when deleting blueprint
Also extends the blueprint delete test to also check the workspace.

(cherry picked from commit 26bd2c1378)
2019-05-03 13:56:38 -07:00
Alexander Todorov
3f000b1fe3 New test: Build qcow2 compose and test it with QEMU-KVM
(cherry picked from commit b936dfb198)
2019-05-03 13:55:41 -07:00
Brian C. Lane
2861bdb95e lorax-composer: pass customization.kernel append to extra_boot_args
This allows iso builds to include the extra kernel boot parameters by
passing them to the arch-specific live/*tmpl template.

Also adds tests to make sure it is written to config.toml in the build
metadata.

(cherry picked from commit 5dea308080)
2019-04-05 15:41:44 -07:00
Brian C. Lane
c5f4dfe113 lorax-composer: Add the ability to append to the kernel command-line
Sometimes it is necessary to modify the kernel command-line of the
image, this adds support for a [customizations.kernel] section to the
blueprint:

[customizations.kernel]
append = "nosmt=force"

This will be appended to the kickstart's bootloader --append argument.

Includes tests for modifying the bootloader line, the kickstart
template, and examining the final-kickstart.ks created for a compose.

(cherry picked from commit 59464286f9)
2019-04-05 15:41:38 -07:00
Brian C. Lane
d513747384 tests: Make it easier to update version globs
Use constants so we won't have to edit a dozen places in the test when
the package versions are bumped.

Also switch to using Fedora 31 GPG key now that it has branched for
Fedora 30.
2019-03-15 12:20:02 -07:00
Brian C. Lane
fc9ccb51ee Fix pylint problems with vmware_list_vms.py 2019-02-25 14:38:37 -08:00
Brian C. Lane
3b8de2a233 Move the package requirements for live-iso setup out of the template
In order to support iso creation on multiple arches with the templates
we need to be able to select different packages based on arch.
lorax-composer uses the arch-specific Lorax templates in order to
generate the output iso so this patch:

1. Creates a new template and type to parse it, live-install.tmpl
   which contains only installpkg commands and #if clauses for arch
2. Removes bootloader related packages from the live-iso.ks
3. Remove dracut-config-rescue exclusion because it can cause problems
   with some blueprints.
4. Switch logo requirement to system-logos which is satisfied by
   generic-logos or fedora-logos. This prevents conflicts when a blueprint
   installs fedora-release-workstation.

So in the future, if x86.tmpl, etc. need a new package to support
creating the iso it should be added to the correct section in
./share/live/live-install.tmpl
2019-02-25 13:49:12 -08:00
Brian C. Lane
b47554d716 livemedia-creator: Add support for reqpart kickstart command
reqpart can be used to make kickstarts more platform agnostic, creating
needed partitions without lmc having to keep track of the arch-specific
needs. eg. ppc64 needs prepboot and /boot

This increases the size of the disk based on whether reqpart or
reqpart --add-boot is in the kickstart.

Note that this is only valid for partitioned disk output types, not
for filesystem images or live iso output.
2019-02-25 13:49:12 -08:00
Jan Stodola
ce110afd22 Add script for removing old artifacts from VMware 2019-02-25 11:40:03 +02:00
Brian C. Lane
28bd68793b tests: Fix makeFakeRPM calls 2019-02-21 10:10:11 -08:00
Brian C. Lane
2272b2b189 Update bash to 5.0.*
so that the tests will pass again on rawhide.
2019-02-21 08:55:23 -08:00
Alexander Todorov
ea78cce882
New test: Verify tar images with Docker and systemd-nspawn
- on some arches (also Fedora x86_64) systemd-nspawn may not be
  available
- delete composes from other tests in rlPhaseStartCleanup because
  we're seeing the tar compose kind of hanging in Jenkins and that
  test script is executed last so the slave may be running out of
  disk space. Be a good citizen and clean up after the previous
  tests.
2019-02-18 19:14:56 +02:00
Alexander Todorov
c95d7084a6 Update OpenStack flavor and network settings in tests
b/c we've migrated to Upshift we must use different instance type,
specify the desired network to connect to and update how we get
the ip address of the launched VM.
2019-02-16 01:19:43 +02:00
Alexander Todorov
5dc895fbee Keep OpenStack VMs with Tag keep_me
so we can have some systems for debugging
2019-02-08 22:02:41 +01:00
Jiri Kortus
57be7313c4 Make sure compose build tests run with SELinux in enforcing mode 2019-02-06 18:54:16 +01:00
Jan Stodola
f26fcba902 Add script for removing old artifacts from Azure 2019-02-06 09:38:18 +01:00
Jan Stodola
8a8802aab7 Use existing storage account
To avoid creating a new storage account for every new VM.
2019-02-05 21:03:57 +01:00