Commit Graph

2096 Commits

Author SHA1 Message Date
David Shea
69caef3c2d Add a compose type for Google Compute Engine
(cherry picked from commit bd804c271b)
(cherry picked from commit 846d839980)
2019-05-06 12:07:29 -07:00
David Shea
4e144e15b7 Add a new output type, tar-disk.
This option will create an optionally compressed tarball containing a
disk image. This format is used by Google's Compute Engine.

This also adds a new option, tar_disk_name, to set the name of the disk
image that will be wrapped in the final tarball. opts.image_name
continues to be the final output file name.

(cherry picked from commit c941b82b0c)
(cherry picked from commit 121717ac4a)
2019-05-06 12:07:29 -07:00
David Shea
b52877b63c Support compressing single files.
Modify imgutils.compress to allow the "rootdir" argument to be either a
directory or a single file to add to an archive.

(cherry picked from commit c585b91422)
(cherry picked from commit 5308e7bfec)
2019-05-06 12:07:29 -07:00
David Shea
c74e50d99d Add an option to align the image size to a multiplier.
If provided, round the disk image size up to a multiple of the value.
This allows for image formats with specific size-alignment requirements
(e.g., disk size must be in GiB).

(cherry picked from commit 8ef4f431d8)
(cherry picked from commit b2a33d5654)
2019-05-06 12:07:29 -07:00
Brian C. Lane
ce879af719 Add documentation references to lorax-composer service files
(cherry picked from commit ba2d834425)
(cherry picked from commit 93accbcfb9)
2019-05-06 12:07:29 -07:00
Brian C. Lane
4fa08eb8e1 Add more tests for gitrpm.py
Make sure that dotfiles are installed when not directly under /
Make sure / is not packaged in the rpm (it will cause a conflict with
the filesystem package).
Make sure that using destination="/" works

(cherry picked from commit 6fd06c6931)
(cherry picked from commit d5d60ebe6d)
2019-05-06 12:07:06 -07:00
Brian C. Lane
e84bf3a64c lorax-composer: Fix installing files from [[repos.git]] to /
rpmfluff was including / in the rpm, which conflicts with
filesystem.rpm

The rpm globs are pretty limited, and we don't actually know the file
paths until later, so we have to use a glob or a directory.

So when the destination is / it now uses /* to select all the files and
sub-directories in the archive. The limitation of this is that it cannot
support dotfiles directly under /, they will cause a rpmbuild error.

For destinations other than / it uses the name of the directory, so
dotfiles are fine in that situation.

(cherry picked from commit 049f68cb60)
(cherry picked from commit 4073dd4e4d)
2019-05-06 12:07:06 -07:00
Brian C. Lane
c9f076c86d Make it easier to generate docs for the next release
Change the docs-in-docker target to generate the docs for the NEXT
release, not the current one. Also pass in uid/gid so that the new files
can be set to the correct ownership instead of root.

Modify docs/conf.py to bump the version of the docs if
LORAX_VERSION=next is set in the environment.

(cherry picked from commit 2acd13d612)
(cherry picked from commit a71ef40dd5)
2019-05-06 12:07:06 -07:00
Brian C. Lane
7a86aa1d17 pylorax.ltmpl: Add a test for missing quotes
It should raise an error if a quote is missing in the template.

(cherry picked from commit df3aeee802)
(cherry picked from commit 1d4f47e26f)
2019-05-06 12:07:06 -07:00
Brian C. Lane
156e0ba7c9 Don't remove chmem and lsmem from install.img
(cherry picked from commit 6ccbfe0585)
(cherry picked from commit 389fd487da)
2019-05-06 12:07:06 -07:00
Jiri Kortus
4c313acc64 Add checks for disabled root account
The root account checks are applied to generated and deployed images
to make sure that root account is locked, except for live ISO.

(cherry picked from commit 655e7e40c0)
(cherry picked from commit 7c093ecbb4)
2019-05-06 12:07:06 -07:00
Chris Roberts
3339048a66 Update datastore for VMware testing
(cherry picked from commit 32d5ff8615)
(cherry picked from commit 7bf3371e33)
2019-05-06 12:07:06 -07:00
Brian C. Lane
c5108bce9f Add tests using repos.git in blueprints
- Check final-kickstart.ks for the rpm source
- Check final-kickstart.ks for the rpm package name and version
- Make sure depsolve works
- Make sure errors from a bad repo are returned correctly
- Make sure errors from a bad reference are returned correctly

This moves _wait_for_status into a helper function so it can be shared
between the test classes.

(cherry picked from commit 8c2184d59e)
(cherry picked from commit 78f79a94ec)
2019-05-06 12:06:06 -07:00
Brian C. Lane
5042867d19 Move git repo creation into tests/lib.py
This way it can be shared with test_server.py

(cherry picked from commit 02f757d231)
(cherry picked from commit 136f9d877a)
2019-05-06 12:06:06 -07:00
Brian C. Lane
b25995d9d4 rpmgit: catch potential errors while running git
Log them and report them as RuntimeError. Also add a couple tests for
them.

(cherry picked from commit 61efa91a03)
(cherry picked from commit 087b0fe8c6)
2019-05-06 12:06:06 -07:00
Brian C. Lane
7a6ebb0ad3 tests: Add test for Recipe.freeze() function
The freeze function was not being tested. Add a test for it using the
repos.git test recipe.

(cherry picked from commit c26477a63c)
(cherry picked from commit bdf5572bc6)
2019-05-06 12:05:42 -07:00
Brian C. Lane
8696f197d4 Add repos.git support to lorax-composer builds
This hooks up creation of the rpm to the build, adds it to the
kickstart, and passes the url to Anaconda. The dnf repo with the rpms is
created under the results directory so it will be included when
downloading the build's results.

(cherry picked from commit cd8c884adb)
(cherry picked from commit 2e596010d3)
2019-05-06 12:05:34 -07:00
Brian C. Lane
0169422746 Add pylorax.api.gitrpm module and tests
This handles creating the rpm from the dictionary describing the
repository and rpm. Also adds tests for archive and rpm creation.

(cherry picked from commit f6f2308765)
(cherry picked from commit efc77c1d71)
2019-05-06 12:05:34 -07:00
Brian C. Lane
21b03c2108 Add support for [[repos.git]] section to blueprints
This adds support, documentation, and testing for a [[repos.git]]
blueprint section that can be used to install files from a git
repository. It will create an rpm that will be added to the build,
and included in the metadata that can be downloaded. This allows you to
accurately keep track of the source of configuration files and extra
metadata that is added to the build.

The source repo and reference will be listed in the rpm's summary making
it easy to discover on the installed system.

(cherry picked from commit d7b96c8f0f)
(cherry picked from commit 047f174dcf)
2019-05-06 12:05:34 -07:00
Brian C. Lane
d9b55b78cb tests: Add python3-mock and python3-sphinx_rtd_theme
Whatever was pulling them in has stopped so we need to add them to the
list.

(cherry picked from commit 5f530bd501)
(cherry picked from commit de99a43485)
2019-05-06 12:00:22 -07:00
Alexander Todorov
7a1e443aff Use make ci inside test-in-copy target
this minimizes the possibility of these two to diverge over time.
make ci is the default for Jenkins and will also be used for
internal gating tests.

(cherry picked from commit f5286e1243)
(cherry picked from commit 331acbf3fe)
2019-05-06 12:00:22 -07:00
Alexander Todorov
530279c44a Allow overriding $CLI outside test scripts
this will allow you to test against installed RPM like so:

    # export CLI="/usr/bin/composer-cli"
    # make test_images

If you already have lorax-composer running then you can directly
execute test scripts:

    # ./tests/cli/test_build_and_deploy_aws.sh

(cherry picked from commit 85dfbd7911)
(cherry picked from commit 1053c7477e)
2019-05-06 12:00:22 -07:00
Alexander Todorov
d0ac25d022 New test: Build live-iso and boot with KVM
explicitly enables sshd for live-iso during testing

(cherry picked from commit eef11ef405)
(cherry picked from commit 60daacfc38)
2019-05-06 12:00:22 -07:00
Brian C. Lane
2dd3dd54c5 lorax-composer: Return UnknownBlueprint errors when using deleted blueprints
Reading a blueprint wasn't checking to see if it had been deleted so it
was returning the most recent commit before it had been deleted. This
allowed things like starting a compose with a blueprint that technically
doesn't exist.

One exception to this is the /changes/ route, it must be available so
that you can use the commit hash to undo a delete.

This also adds tests for the various operations.

Resolves: rhbz#1682113
(cherry picked from commit d32f477e0b)
(cherry picked from commit 82aa9cdbc6)
2019-05-06 12:00:22 -07:00
Brian C. Lane
d0458750df lorax-composer: Delete workspace copy when deleting blueprint
Also extends the blueprint delete test to also check the workspace.

(cherry picked from commit 26bd2c1378)
(cherry picked from commit 5c0f127b3c)
2019-05-06 12:00:22 -07:00
Alexander Todorov
21975b44da New test: Build qcow2 compose and test it with QEMU-KVM
(cherry picked from commit b936dfb198)
(cherry picked from commit 3f000b1fe3)
2019-05-06 12:00:22 -07:00
Brian C. Lane
35faa61f7e lorax-composer: pass customization.kernel append to extra_boot_args
This allows iso builds to include the extra kernel boot parameters by
passing them to the arch-specific live/*tmpl template.

Also adds tests to make sure it is written to config.toml in the build
metadata.

(cherry picked from commit 5dea308080)
(cherry picked from commit 2861bdb95e)
2019-05-06 11:56:17 -07:00
Brian C. Lane
78ccea2231 Improve logging for template syntax errors
The shlex splitting can fail, resulting in error messages like:

ERROR livemedia-creator: No closing quotation

without any context in the log files. This logs the line that failed to
be split and expanded.

(cherry picked from commit f9665940bb)
(cherry picked from commit 4c0e632b93)
2019-05-06 11:56:17 -07:00
Brian C. Lane
6aa0a3fdf6 Add extra boot args to the livemedia-creator iso templates
This adds a new livemedia-creator argument, --extra-boot-args, which can
be used to add arguments to the kernel command-line in the templates.

(cherry picked from commit 235813212f)
(cherry picked from commit 48548722b3)
2019-05-06 11:56:09 -07:00
Brian C. Lane
afa68df873 lorax-composer: Add the ability to append to the kernel command-line
Sometimes it is necessary to modify the kernel command-line of the
image, this adds support for a [customizations.kernel] section to the
blueprint:

[customizations.kernel]
append = "nosmt=force"

This will be appended to the kickstart's bootloader --append argument.

Includes tests for modifying the bootloader line, the kickstart
template, and examining the final-kickstart.ks created for a compose.

(cherry picked from commit 59464286f9)
(cherry picked from commit c5f4dfe113)
2019-05-06 11:54:56 -07:00
Brian C. Lane
935f66662b qemu wasn't restoring the terminal if it was terminated early
You would need to run reset to regain control of your terminal after
this happened, so this turns off the monitor and serial port mux to
stdout.

(cherry picked from commit 1accce819afac96a6d58f24a4908a4913e46624c)
(cherry picked from commit ae07d8d888)
2019-05-06 11:54:17 -07:00
Brian C. Lane
53c4daa529 Switch the --virt-uefi method to use SecureBoot
This updates the qemu arguments so that it will actually work, and
switches to using SecureBoot OVMF firmware.

(cherry picked from commit f2b19cfcf7e23dfdb7176fcb1fa8b0335da5aa9a)
(cherry picked from commit 79c38687f3)
2019-05-06 11:51:32 -07:00
Brian C. Lane
850c35a613 Fix pylint problems with vmware_list_vms.py
(cherry picked from commit fc9ccb51ee)
2019-05-06 11:45:32 -07:00
Brian C. Lane
0dd9ffab92 Makefile: Make the .test-results directory
podman won't run if it is missing.

(cherry picked from commit a7e96ab828)
2019-05-06 11:45:32 -07:00
Brian C. Lane
208941afd8 Add a ppc64le template for live iso creation
(cherry picked from commit f3c147448a)
2019-05-06 11:45:32 -07:00
Brian C. Lane
1fd73cde52 Move the package requirements for live-iso setup out of the template
In order to support iso creation on multiple arches with the templates
we need to be able to select different packages based on arch.
lorax-composer uses the arch-specific Lorax templates in order to
generate the output iso so this patch:

1. Creates a new template and type to parse it, live-install.tmpl
   which contains only installpkg commands and #if clauses for arch
2. Removes bootloader related packages from the live-iso.ks
3. Remove dracut-config-rescue exclusion because it can cause problems
   with some blueprints.
4. Switch logo requirement to system-logos which is satisfied by
   generic-logos or fedora-logos. This prevents conflicts when a blueprint
   installs fedora-release-workstation.

So in the future, if x86.tmpl, etc. need a new package to support
creating the iso it should be added to the correct section in
./share/live/live-install.tmpl

(cherry picked from commit 3b8de2a233)
2019-05-06 11:45:32 -07:00
Brian C. Lane
85708340eb Remove exclusions from lorax-composer templates
If a package is excluded in the template and later added by a blueprint
or dependency, anaconda will fail to finish the installation. So remove
the -dracut-config-rescue exclusion and instead remove the rescue
artifacts in %post

(cherry picked from commit 822bc7ac68)
2019-05-06 11:45:32 -07:00
Brian C. Lane
4f07db73a5 Add LiveTemplateRunner to parse per-arch live-iso package requirements
This template runner only uses the installpkg command to gather a list
of package NEVRA to be installed to support live iso creation.

(cherry picked from commit 6c5e89ed14)
2019-05-06 11:45:32 -07:00
Brian C. Lane
54fe00d16e Move the run part of LoraxTemplateRunner into new TemplateRunner class
This will make it easier to add a new subclass that only handles
installpkg for use with livemedia-creator and lorax-composer iso
creation.

(cherry picked from commit 8eaad3bc5e)
2019-05-06 11:45:32 -07:00
Brian C. Lane
883bc07fc8 lorax-composer: Use reqpart --add-boot for partitioned disk templates
Some platforms, like ppc64, require that the /boot partition be present.
It doesn't hurt to have it there on other platforms so instead of trying
to add per-arch kickstart templates just use reqpart --add-boot
everywhere.

(cherry picked from commit a0ff34dcb1)
2019-05-06 11:45:32 -07:00
Brian C. Lane
19375b9d4e livemedia-creator: Add support for reqpart kickstart command
reqpart can be used to make kickstarts more platform agnostic, creating
needed partitions without lmc having to keep track of the arch-specific
needs. eg. ppc64 needs prepboot and /boot

This increases the size of the disk based on whether reqpart or
reqpart --add-boot is in the kickstart.

Note that this is only valid for partitioned disk output types, not
for filesystem images or live iso output.

(cherry picked from commit b47554d716)
2019-05-06 11:45:32 -07:00
Brian C. Lane
ab74164d83 Make the lorax-composer ks templates more generic
Some platforms do not have grub2, and some require other partitions.

Anaconda will add platform specific partitions if the 'reqpart' command
is included, and it will add bootloader specific packages to the list if
they are needed.

(cherry picked from commit d27b4fcbd4)
2019-05-06 11:45:32 -07:00
Jan Stodola
f02eab78a3 Add script for removing old artifacts from VMware
(cherry picked from commit ce110afd22)
2019-05-06 11:45:32 -07:00
Brian C. Lane
3d2c085cf0 Clarify the ks repo only error message
This also moves the run_creator kickstart checks into check_kickstart
so that tests may be added.

This will close #164

(cherry picked from commit 3676cb65bb)
2019-05-06 11:45:09 -07:00
Brian C. Lane
30098a52e3 tests: Fix makeFakeRPM calls
(cherry picked from commit 28bd68793b)
2019-05-03 17:05:57 -07:00
Brian C. Lane
b537d653c4 Drop _unique_dicts function
It is not actually needed. projects_info deduplicates the package list,
placing other builds into the builds list instead of making a new
package entry. So it returns a sorted and deduped list of packages, as
expected.

(cherry picked from commit 6443f34337)
2019-05-03 17:05:36 -07:00
Jiri Konecny
36b0985265 Remove unsupported anaconda-docker-addon (#619)
It is failing to load on Fedora for quite some time and there no-one
complaining about this so it would be easier to disable it instead of
fixing it.

(cherry picked from commit d64a320ba1)
2019-05-03 17:04:52 -07:00
Alexander Todorov
ce361d6f48 New test: Verify tar images with Docker and systemd-nspawn
- on some arches (also Fedora x86_64) systemd-nspawn may not be
  available
- delete composes from other tests in rlPhaseStartCleanup because
  we're seeing the tar compose kind of hanging in Jenkins and that
  test script is executed last so the slave may be running out of
  disk space. Be a good citizen and clean up after the previous
  tests.

(cherry picked from commit ea78cce882)
2019-05-03 17:03:40 -07:00
Alexander Todorov
b7b8731c03 Update OpenStack flavor and network settings in tests
b/c we've migrated to Upshift we must use different instance type,
specify the desired network to connect to and update how we get
the ip address of the launched VM.

(cherry picked from commit c95d7084a6)
2019-05-03 17:03:40 -07:00
Brian C. Lane
a08dd1585c Remove 3G minimum from lorax-composer
The reason for the 3G minimum was because anaconda had a bug with how it
calculated minimum disk size when using kickstart. The gix for this has
been in Anaconda since 29.19-1, so we can now remove our limit and
create somewhat smaller disk images.

(cherry picked from commit 7e78dc368f)
2019-05-03 17:02:28 -07:00