Commit Graph

143 Commits

Author SHA1 Message Date
Brian C. Lane
8696f197d4 Add repos.git support to lorax-composer builds
This hooks up creation of the rpm to the build, adds it to the
kickstart, and passes the url to Anaconda. The dnf repo with the rpms is
created under the results directory so it will be included when
downloading the build's results.

(cherry picked from commit cd8c884adb)
(cherry picked from commit 2e596010d3)
2019-05-06 12:05:34 -07:00
Brian C. Lane
0169422746 Add pylorax.api.gitrpm module and tests
This handles creating the rpm from the dictionary describing the
repository and rpm. Also adds tests for archive and rpm creation.

(cherry picked from commit f6f2308765)
(cherry picked from commit efc77c1d71)
2019-05-06 12:05:34 -07:00
Brian C. Lane
21b03c2108 Add support for [[repos.git]] section to blueprints
This adds support, documentation, and testing for a [[repos.git]]
blueprint section that can be used to install files from a git
repository. It will create an rpm that will be added to the build,
and included in the metadata that can be downloaded. This allows you to
accurately keep track of the source of configuration files and extra
metadata that is added to the build.

The source repo and reference will be listed in the rpm's summary making
it easy to discover on the installed system.

(cherry picked from commit d7b96c8f0f)
(cherry picked from commit 047f174dcf)
2019-05-06 12:05:34 -07:00
Alexander Todorov
530279c44a Allow overriding $CLI outside test scripts
this will allow you to test against installed RPM like so:

    # export CLI="/usr/bin/composer-cli"
    # make test_images

If you already have lorax-composer running then you can directly
execute test scripts:

    # ./tests/cli/test_build_and_deploy_aws.sh

(cherry picked from commit 85dfbd7911)
(cherry picked from commit 1053c7477e)
2019-05-06 12:00:22 -07:00
Alexander Todorov
d0ac25d022 New test: Build live-iso and boot with KVM
explicitly enables sshd for live-iso during testing

(cherry picked from commit eef11ef405)
(cherry picked from commit 60daacfc38)
2019-05-06 12:00:22 -07:00
Brian C. Lane
2dd3dd54c5 lorax-composer: Return UnknownBlueprint errors when using deleted blueprints
Reading a blueprint wasn't checking to see if it had been deleted so it
was returning the most recent commit before it had been deleted. This
allowed things like starting a compose with a blueprint that technically
doesn't exist.

One exception to this is the /changes/ route, it must be available so
that you can use the commit hash to undo a delete.

This also adds tests for the various operations.

Resolves: rhbz#1682113
(cherry picked from commit d32f477e0b)
(cherry picked from commit 82aa9cdbc6)
2019-05-06 12:00:22 -07:00
Brian C. Lane
d0458750df lorax-composer: Delete workspace copy when deleting blueprint
Also extends the blueprint delete test to also check the workspace.

(cherry picked from commit 26bd2c1378)
(cherry picked from commit 5c0f127b3c)
2019-05-06 12:00:22 -07:00
Alexander Todorov
21975b44da New test: Build qcow2 compose and test it with QEMU-KVM
(cherry picked from commit b936dfb198)
(cherry picked from commit 3f000b1fe3)
2019-05-06 12:00:22 -07:00
Brian C. Lane
35faa61f7e lorax-composer: pass customization.kernel append to extra_boot_args
This allows iso builds to include the extra kernel boot parameters by
passing them to the arch-specific live/*tmpl template.

Also adds tests to make sure it is written to config.toml in the build
metadata.

(cherry picked from commit 5dea308080)
(cherry picked from commit 2861bdb95e)
2019-05-06 11:56:17 -07:00
Brian C. Lane
afa68df873 lorax-composer: Add the ability to append to the kernel command-line
Sometimes it is necessary to modify the kernel command-line of the
image, this adds support for a [customizations.kernel] section to the
blueprint:

[customizations.kernel]
append = "nosmt=force"

This will be appended to the kickstart's bootloader --append argument.

Includes tests for modifying the bootloader line, the kickstart
template, and examining the final-kickstart.ks created for a compose.

(cherry picked from commit 59464286f9)
(cherry picked from commit c5f4dfe113)
2019-05-06 11:54:56 -07:00
Brian C. Lane
850c35a613 Fix pylint problems with vmware_list_vms.py
(cherry picked from commit fc9ccb51ee)
2019-05-06 11:45:32 -07:00
Brian C. Lane
1fd73cde52 Move the package requirements for live-iso setup out of the template
In order to support iso creation on multiple arches with the templates
we need to be able to select different packages based on arch.
lorax-composer uses the arch-specific Lorax templates in order to
generate the output iso so this patch:

1. Creates a new template and type to parse it, live-install.tmpl
   which contains only installpkg commands and #if clauses for arch
2. Removes bootloader related packages from the live-iso.ks
3. Remove dracut-config-rescue exclusion because it can cause problems
   with some blueprints.
4. Switch logo requirement to system-logos which is satisfied by
   generic-logos or fedora-logos. This prevents conflicts when a blueprint
   installs fedora-release-workstation.

So in the future, if x86.tmpl, etc. need a new package to support
creating the iso it should be added to the correct section in
./share/live/live-install.tmpl

(cherry picked from commit 3b8de2a233)
2019-05-06 11:45:32 -07:00
Brian C. Lane
19375b9d4e livemedia-creator: Add support for reqpart kickstart command
reqpart can be used to make kickstarts more platform agnostic, creating
needed partitions without lmc having to keep track of the arch-specific
needs. eg. ppc64 needs prepboot and /boot

This increases the size of the disk based on whether reqpart or
reqpart --add-boot is in the kickstart.

Note that this is only valid for partitioned disk output types, not
for filesystem images or live iso output.

(cherry picked from commit b47554d716)
2019-05-06 11:45:32 -07:00
Jan Stodola
f02eab78a3 Add script for removing old artifacts from VMware
(cherry picked from commit ce110afd22)
2019-05-06 11:45:32 -07:00
Brian C. Lane
3d2c085cf0 Clarify the ks repo only error message
This also moves the run_creator kickstart checks into check_kickstart
so that tests may be added.

This will close #164

(cherry picked from commit 3676cb65bb)
2019-05-06 11:45:09 -07:00
Brian C. Lane
30098a52e3 tests: Fix makeFakeRPM calls
(cherry picked from commit 28bd68793b)
2019-05-03 17:05:57 -07:00
Alexander Todorov
ce361d6f48 New test: Verify tar images with Docker and systemd-nspawn
- on some arches (also Fedora x86_64) systemd-nspawn may not be
  available
- delete composes from other tests in rlPhaseStartCleanup because
  we're seeing the tar compose kind of hanging in Jenkins and that
  test script is executed last so the slave may be running out of
  disk space. Be a good citizen and clean up after the previous
  tests.

(cherry picked from commit ea78cce882)
2019-05-03 17:03:40 -07:00
Alexander Todorov
b7b8731c03 Update OpenStack flavor and network settings in tests
b/c we've migrated to Upshift we must use different instance type,
specify the desired network to connect to and update how we get
the ip address of the launched VM.

(cherry picked from commit c95d7084a6)
2019-05-03 17:03:40 -07:00
Alexander Todorov
ea63fbcc5b Keep OpenStack VMs with Tag keep_me
so we can have some systems for debugging

(cherry picked from commit 5dc895fbee)
2019-05-03 17:01:30 -07:00
Jiri Kortus
f7e71768fa Make sure compose build tests run with SELinux in enforcing mode
(cherry picked from commit 57be7313c4)
2019-05-03 17:01:30 -07:00
Jan Stodola
45710e435a Add script for removing old artifacts from Azure
(cherry picked from commit f26fcba902)
2019-05-03 17:01:30 -07:00
Jan Stodola
c6bff67549 Use existing storage account
To avoid creating a new storage account for every new VM.

(cherry picked from commit 8a8802aab7)
2019-05-03 17:01:30 -07:00
Jan Stodola
0d38224f12 Record date/time of VM creation
(cherry picked from commit e918ff807c)
2019-05-03 17:01:30 -07:00
Jan Stodola
43a79746fc Export OS_PROJECT_NAME variable in openstack scripts
The OS_PROJECT_NAME (or OS_TENANT_NAME) environment variable needs to be defined.
Use the OS_PROJECT_NAME, since it is recommended in the documentation instead of
the older OS_TENANT_NAME.

(cherry picked from commit cc6fdb2fac)
2019-05-03 16:57:37 -07:00
Jan Stodola
12cd27375a Expand parameters as separate words
(cherry picked from commit 961ea18145)
2019-05-03 16:57:37 -07:00
Jan Stodola
34c24117fe Add script for removing old artifacts from OpenStack
(cherry picked from commit 6680e2ffec)
2019-05-03 16:57:37 -07:00
Jan Stodola
8885d7eaca Add script for removing old artifacts from AWS
(cherry picked from commit 677c858f60)
2019-05-03 16:57:37 -07:00
Alexander Todorov
a038f7d904 tests: use the first IP address if more than 1 retruned from OpenStack
(cherry picked from commit 883a959442)
2019-05-03 16:52:44 -07:00
Alexander Todorov
963c7d8f89 tests: remove a debugging command
(cherry picked from commit 6563a33f63)
2019-05-03 16:52:44 -07:00
Alexander Todorov
4f3b81e4e7 Teach OpenStack test to distinguish between RHEL and Fedora
(cherry picked from commit 642b909d24)
2019-05-03 16:50:33 -07:00
Alexander Todorov
7b306e2e84 Use full path for Azure playbook as well
(cherry picked from commit 543a4d88d9)
2019-05-03 16:47:52 -07:00
Alexander Todorov
9584cbd96f Use a temporary dir for ssh keys during testing
resolves path problems with sudo, we already have the same change
in the AWS test script.

(cherry picked from commit 16260c1569)
2019-05-03 16:47:46 -07:00
Alexander Todorov
c4f3ceb0ec Update V_DATASTORE b/c defaults appear to have been changed
(cherry picked from commit 59050df710)
2019-05-03 16:47:38 -07:00
Alexander Todorov
41aa84ba04 Clone pyvmomi samples in the correct directory
(cherry picked from commit e0c8cdc5a8)
2019-05-03 16:47:10 -07:00
Alexander Todorov
a8ff02113b Use full path when pushing toml files during testing
(cherry picked from commit 8f430be2a6)
2019-05-03 16:47:00 -07:00
Alexander Todorov
efa9974609 Implicitly specify ssh key directory/files for testing
this is to work around identity/sudo problems

(cherry picked from commit 4b4f4bd0e0)
2019-05-03 16:46:48 -07:00
Alexander Todorov
a907a3757d [test] Clean up containers.json
(cherry picked from commit ed63611cb7)
2019-05-03 16:46:39 -07:00
Alexander Todorov
eda242623b Teach AWS test to distinguish between RHEL and Fedora
(cherry picked from commit ab048947cb)
2019-05-03 16:46:32 -07:00
Alexander Todorov
8c409c8263 Use a temporary shared dir when testing
otherwise composer-cli is unable to glob() the kickstart
files and we're left without supported compose types. Seen
during AWS testing for example.

Helps with running some of the tests via sudo b/c this is
what Jenkins requires.

(cherry picked from commit b88466fd74)
2019-05-03 16:43:41 -07:00
Alexander Todorov
9c17c0cc6c Copy blueprints used for testing to temporary directory
If trying to execute test_cli.sh inside a git checkout
we are going to get the following exception:

Traceback (most recent call last):
  File "./src/sbin/lorax-composer", line 251, in <module>
    repo = open_or_create_repo(server.config["REPO_DIR"])
  File "/home/jenkins/lorax/src/pylorax/api/recipes.py", line 306, in open_or_create_repo
gi.repository.GLib.Error: ggit-error: failed to stat '/home/jenkins/lorax/tests/pylorax/blueprints': Permission denied (-1)
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib64/python3.7/multiprocessing/popen_fork.py", line 54, in _send_signal
    os.kill(self.pid, sig)

From what I can tell open_or_create_repo() is trying to initialize
a git repository inside the blueprints directory which fails when
we have an active git checkout.

This doesn't happen when we run the tests in Travis CI because
rsync excludes .git/ inside the Docker container.

(cherry picked from commit c9d706a382)
2019-05-03 16:43:35 -07:00
Alexander Todorov
178eb927d6 Add make targets for Jenkins
these targets help hooking up things in Jenkins and enable us to
perform build & deploy tests for cloud images.

NOTE: use sudo -E to preserve the environment
(cherry picked from commit 366ae55abe)
2019-05-03 16:43:28 -07:00
Alexander Todorov
2184cff59c Teach test_cli.sh to execute test scripts via arguments
this will be used to invoke scripts that build/push cloud images
without having to duplicate the setup/teardown/report parts!

(cherry picked from commit af2ae790ce)
2019-05-03 16:41:00 -07:00
Alexander Todorov
63e35921fa new test: build an image and deploy it on Azure
(cherry picked from commit 13fbd0861b)
2019-05-03 16:40:54 -07:00
Alexander Todorov
9b31c3cad9 Fix typo in comment
(cherry picked from commit d6dcb38a78)
2019-05-03 16:40:48 -07:00
Alexander Todorov
3fd9699e6d For OpenStack build image with rng-tools installed
appears to be a decent workaround for slow boot likely caused
by lack of entropy for ssh-keygen, see:
https://bugzilla.redhat.com/show_bug.cgi?id=1639326

(cherry picked from commit 6ae78fbd47)
2019-05-03 16:39:25 -07:00
Brian C. Lane
51a9c28bc6 Add tests for partitioned disk images
The setup for this is a bit complex, so it really ends up testing things
twice.

(cherry picked from commit fb4e6f2588)
2019-05-03 16:39:18 -07:00
Brian C. Lane
e7310c7260 Add tests for pylorax.imgutils
Some of these can only run as root on a real system with access to loop
devices. They are skipped when running in a container.

(cherry picked from commit 063a1770e1)
2019-05-03 16:39:05 -07:00
Brian C. Lane
0aa206f661 Add tests to test_creator.py
Add a /.in-container file to the container root so that tests requiring root
and loop device support will be skipped when running in a container.

(cherry picked from commit bab4b20d0d)
2019-05-03 16:39:01 -07:00
Brian C. Lane
ef45d169b0 Add some tests for creator.py
This is complicated by the fact that much of this module requires mount.
So for now just test the things that don't need mount.

(cherry picked from commit 134a333d92)
2019-05-03 16:38:47 -07:00
Brian C. Lane
c63b2e3881 tests: Add executils test
(cherry picked from commit fb87db0e1e)
2019-05-03 16:38:38 -07:00