2015-08-10 07:25:04 +00:00
List of variables usable in tests
=================================
This list contains variables that are possible to set/redefine in WebUI and get their values in tests.
This document merges relevant variables from
[official documentation ](https://github.com/os-autoinst/os-autoinst/blob/master/doc/backend_vars.asciidoc )
(backend variables), variables that control flow through `main.pm` (test variables) and variables that
are specified when ISO is posted into OpenQA (run variables).
Backend variables
-----------------
These variables control settings of underlying virtual machine where tests runs on. Some of them (`ARCH`, `QEMURAM` , ...) should be set per machine, others (`HDD_$i`, `HDDMODEL` , ...) should be set per test and some of them (`KEEPHDDS`, `MAKETESTSNAPSHOTS` , `SKIPTO` ) should be set only during development and debugging.
| Variable | Values allowed | Default value | Explanation |
| -------- | :------------: | :-----------: | ----------- |
| `ARCH` | `x86_64` , `i686` , `aarch64` , ... | depends on tested medium | architecture of VM |
| `BIOS` | filename | `ovmf-x86_64-ms.bin` for `x86_64` UEFI, not set otherwise | set different BIOS |
| `BOOTFROM` | characters (see man qemu, `-boot order=` option) | not set | boot from different medium than CD |
| `CDMODEL` | see `qemu-kvm -device help` | not set/Qemu default | type of device for CD |
2015-08-11 11:30:42 +00:00
| `HDDMODEL` | see `qemu-kvm -device help` (e. g. PATA = `id-hd` , SATA = `ide-hd,bus-ahci0.0` , SCSI = `virtio-scsi-pci` ) | `virtio-blk` | type of device for HDD |
2015-08-10 07:25:04 +00:00
| `HDDSIZEGB` | integer | `10` | size (in GBs) for disks that are created automatically |
| `HDD_$i` (`HDD_1, HDD_2`, ...) | filename | not set | attach additional HDD to VM |
| `ISO` | filename | not set | attach CD drive with ISO inserted |
| `ISO_$i` (`ISO_1`, `ISO_2` , ...) | filename | not set | additional CD drives to by attached |
| `KEEPHDDS` | boolean | `false` /not set | don't delete HDD after test finishes |
| `LAPTOP` | boolean or filename | `false` /not set | if `true` , Dell E6330 DMI is used; if set to filename, that file is used for DMI |
| `MAKETESTSNAPSHOTS` | boolean | `false` /not set | save snapshot for each test |
| `NOVIDEO` | boolean | `false` /not set | don't create video output if set |
| `NUMDISKS` | integer | 1 | number of disks to be created and attached to VM |
2015-10-15 14:07:00 +00:00
| `PART_TABLE_TYPE` | `mbr` , `gpt` | not set | what type of partition table should attached HDD have |
2015-08-10 07:25:04 +00:00
| `PXEBOOT` | boolean | `false` /not set | boot VM from PXE (network) |
| `QEMU` | filename, path to Qemu binary | not set | filename of Qemu binary |
| `QEMUCPU` | see `qemu-kvm -cpu help` | not set | CPU to emulate |
| `QEMUCPUS` | integer | 1 | number of processor cores to use for Qemu |
| `QEMURAM` | integer | 1024 | size of RAM to use (in MiB) |
| `QEMUTHREADS` | integer | 0 | number of CPU threads to use for Qemu |
| `QEMUVGA` | see `qemu-kvm -device help` | `cirrus` (should be set to `std` on Fedora, cirrus driver [was retired ](https://lists.fedoraproject.org/pipermail/devel/2014-May/199459.html )) | display device to use for VM |
| `QEMU_COMPRESS_QCOW2` | boolean | `false` /not set | compress qcow2 images |
| `SKIPTO` | name of snapshot | not set | restore VM from given snapshot - better used with `MAKETESTSNAPSHOTS` |
| `UEFI` | boolean | `false` /not set | whether to use UEFI (UEFI BIOS files should be installed) |
| `UEFI_BIOS` | filename | `false` /not set, `ovmf-x86_64-ms.bin` when `UEFI` is set | filename of UEFI BIOS |
| `USBBOOT` | boolean | `false` /not set | if set, mount ISO as USB and boot from it |
For additional (kvm2usb, IPMI...) as well as for other Qemu variables, see
[official documentation ](https://github.com/os-autoinst/os-autoinst/blob/master/doc/backend_vars.asciidoc ).
Test variables
--------------
These variables control flow through `main.pm` - by setting them, you can alter what parts will be loaded.
There are certain tests that conflict each other. Note that conflict is always mutual, but for simplification
purposes, only one side of this conflict is shown (that means, if it's shown that `A` conflicts `B` ,
it also means that `B` conflicts `A` even if not shown in the table).
| Variable | Values allowed | Default value/behavior | Conflicts | Explanation |
| -------- | :------------: | ---------------------- | :-------: | ----------- |
| `LIVE` | boolean | not set | technically `PACKAGE_SET` , `KICKSTART` , `MIRRORLIST_GRAPHICAL` , `REPOSITORY_GRAPHICAL` , `REPOSITORY_VARIATION` | specify that the test is running on Live system - additional steps are required (starting Anaconda...) |
| `PACKAGE_SET` | string (`minimal`, `default` , ...) | `minimal` ; `default` when `LIVE` is set | nothing | sets packageset to install - you have to add appropriate needles, see `tests/_software_selection.pm` |
2016-08-16 07:33:10 +00:00
| `ENTRYPOINT` | filename | not set | N/A | set `ENTRYPOINT` to load specific test directly (bypass modular structure, mainly usable for testing). If you want to run more than one test, separate them with space. |
2015-08-10 07:25:04 +00:00
| `UPGRADE` | string (`minimal`, `desktop` , `encrypted` , ...) | not set | all except of `LIVE` and `USER_PASSWORD` | when set, do an upgrade test of specified type |
| `KICKSTART` | boolean | `false` /not set | all | when specified, do an kickstart installation (you should use `GRUB` variable to specify where kickstart file resides) |
| `MIRRORLIST_GRAPHICAL` | boolean | `false` /not set | `REPOSITORY_GRAPHICAL` | sets installation source to mirrorlist |
| `REPOSITORY_GRAPHICAL` | url to repository (without arch and `/os` , for example `http://dl.fedoraproject.org/pub/fedora/linux/development` ) | not set | `MIRRORLIST_GRAPHICAL` | sets installation source to repository url in Anaconda |
| `REPOSITORY_VARIATION` | url to repository (without arch and `/os` , for example `http://dl.fedoraproject.org/pub/fedora/linux/development` ) | not set | nothing | sets installation source to repository url in GRUB |
2015-08-10 18:01:12 +00:00
| `PARTITIONING` | string (`custom_software_raid`, `guided_delete_all` , ...) | `guided_empty` | nothing | load specified test for partitioning part (when `PARTITIONING=guided_delete_all` , `tests/disk_guided_delete_all.pm` is loaded) and optionally post-install partitioning check (if `tests/disk_guided_delete_all_postinstall.pm` exists, it will be loaded after login to the installed system). Also, if value starts with `custom_` , the `select_disks()` method will check the custom partitioning box |
2015-08-10 07:25:04 +00:00
| `ENCRYPT_PASSWORD` | string | not set | nothing | if set, encrypt disk with given password |
| `DESKTOP` | boolean | `false` /not set | nothing | set to indicate that Fedora is running with GUI (so for example OpenQA should expect graphical login screen) |
| `ROOT_PASSWORD` | string | `weakpassword` | nothing | root password is set to this value |
| `GRUB` | string | not set | nothing | when set, append this string to kernel line in GRUB |
rename BOOT_UPDATES_IMG_URL to TEST_UPDATES, add GRUBADD
Summary:
BOOT_UPDATES_IMG_URL is a pretty misleading name - it used to
be the actual URL, but now it's simply a boolean that decides
whether we look for the effect of the openQA updates image or
not. TEST_UPDATES seems clearer.
GRUBADD does the same thing as GRUB, on top of it. The point of
this is so we can add an option to the scheduler CLI that lets
you say 'run the normal tests, but with this updates image' -
so we can easily (albeit manually triggered) check the impact
of some anaconda change that needs testing. It should never be
set in the templates or the tests, it's there strictly for the
scheduler (whether that's fedora_openqa_schedule or literally a
person calling `client isos post`) to use as a kind of override.
The tests that test updates image loading will probably fail
when doing this, but all other tests should work as intended,
including ones that specify GRUB, becase the extra params will
just get added on top. That's why I invented a new var instead
of just letting the scheduler override GRUB's value when POST
ing.
Test Plan:
Check the rename didn't break anything (updates tests
still work). Run tests with GRUBADD param, make sure value is
correctly appended to cmdline both when GRUB is also specified
and when it is not.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D801
2016-04-08 20:21:29 +00:00
| `GRUBADD` | string | not set | nothing | when set, append this string to kernel line in GRUB, after the `GRUB` string if that one is set too. This is never set by tests; instead it's provided as a way to e.g. run the normal tests with an `updates.img` to test some anaconda change (the scheduler CLI can help you use this feature) |
2016-03-23 20:52:00 +00:00
| `USER_LOGIN` | string | not set | should be used with `USER_PASSWORD` (unless `false` ) | when set, user login is set to this value. If not set, default value `test` is used for console installs, no login is done for graphical installs. If set to `false` , no user login will be done |
| `USER_PASSWORD` | string | not set | should be used with `USER_LOGIN` | when set, user password is set to this value. If not set, default value `weakpassword` is used for console installs, no login is done for graphical installs |
rename BOOT_UPDATES_IMG_URL to TEST_UPDATES, add GRUBADD
Summary:
BOOT_UPDATES_IMG_URL is a pretty misleading name - it used to
be the actual URL, but now it's simply a boolean that decides
whether we look for the effect of the openQA updates image or
not. TEST_UPDATES seems clearer.
GRUBADD does the same thing as GRUB, on top of it. The point of
this is so we can add an option to the scheduler CLI that lets
you say 'run the normal tests, but with this updates image' -
so we can easily (albeit manually triggered) check the impact
of some anaconda change that needs testing. It should never be
set in the templates or the tests, it's there strictly for the
scheduler (whether that's fedora_openqa_schedule or literally a
person calling `client isos post`) to use as a kind of override.
The tests that test updates image loading will probably fail
when doing this, but all other tests should work as intended,
including ones that specify GRUB, becase the extra params will
just get added on top. That's why I invented a new var instead
of just letting the scheduler override GRUB's value when POST
ing.
Test Plan:
Check the rename didn't break anything (updates tests
still work). Run tests with GRUBADD param, make sure value is
correctly appended to cmdline both when GRUB is also specified
and when it is not.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D801
2016-04-08 20:21:29 +00:00
| `TEST_UPDATES` | boolean | `false` /not set | set to indicate that this test checks updates.img loading, so we should check for the expected effect of the updates image used for this testing |
2016-03-24 21:07:23 +00:00
| `POSTINSTALL` | string | not set | nothing | If set, `tests/(value)_postinstall.pm` will be loaded after install, boot, login, and other postinstall tests
2015-09-15 09:04:01 +00:00
| `UEFI` | boolean | `false` /not set | nothing | whether to use UEFI, this variable isn't usually set in test suites but in machine definition |
2016-09-07 08:34:54 +00:00
| `ANACONDA_TEXT` | boolean | `false` /not set | all | when specified, anaconda will run in text mode |
Add support for testing updates
Summary:
This adds an entirely new workflow for testing distribution
updates. The `ADVISORY` variable is introduced: when set,
`main.pm` will load an early post-install test that sets up
a repository containing the packages from the specified update,
runs `dnf -y update`, and reboots. A new templates file is
added, `templates-updates`, which adds two new flavors called
`updates-server` and `updates-workstation`, each containing
job templates for appropriate post-install tests. Scheduler is
expected to post `ADVISORY=(update ID) HDD_1=(base image)
FLAVOR=updates-(server|workstation)`, where (base image) is one
of the stable release base disk images produced by `createhdds`
and usually used for upgrade testing. This will result in the
appropriate job templates being loaded.
We rejig postinstall test loading and static network config a
bit so that this works for both the 'compose' and 'updates' test
flows: we have to ensure we bring up networking for the tap
tests before we try and install the updates, but still allow
later adjustment of the configuration. We take advantage of the
openQA feature that was added a few months back to run the same
module multiple times, so the `_advisory_update` module can
reboot after installing the updates and the modules that take
care of bootloader, encryption and login get run again. This
looks slightly wacky in the web UI, though - it doesn't show the
later runs of each module.
We also use the recently added feature to specify `+HDD_1` in
the test suites which use a disk image uploaded by an earlier
post-install test, so the test suite value will take priority
over the value POSTed by the scheduler for those tests, and we
will use the uploaded disk image (and not the clean base image
POSTed by the scheduler) for those tests.
My intent here is to enhance the scheduler, adding a consumer
which listens out for critpath updates, and runs this test flow
for each one, then reports the results to ResultsDB where Bodhi
could query and display them. We could also add a list of other
packages to have one or both sets of update tests run on it, I
guess.
Test Plan:
Try a post something like:
HDD_1=disk_f25_server_3_x86_64.img DISTRI=fedora VERSION=25
FLAVOR=updates-server ARCH=x86_64 BUILD=FEDORA-2017-376ae2b92c
ADVISORY=FEDORA-2017-376ae2b92c CURRREL=25 PREVREL=24
Pick an appropriate `ADVISORY` (ideally, one containing some
packages which might actually be involved in the tests), and
matching `FLAVOR` and `HDD_1`. The appropriate tests should run,
a repo with the update packages should be created and enabled
(and dnf update run), and the tests should work properly. Also
test a regular compose run to make sure I didn't break anything.
Reviewers: jskladan, jsedlak
Reviewed By: jsedlak
Subscribers: tflink
Differential Revision: https://phab.qa.fedoraproject.org/D1143
2017-01-25 16:16:12 +00:00
| `ANACONDA_STATIC` | string (IPv4 address) | not set | `ANACONDA_TEXT` | If set, will set up static networking using the chosen IP address during install |
| `POST_STATIC` | string (space-separated IPv4 address and hostname) | not set | nothing | If set, will set up static networking using the chosen IP address and hostname during early post-install |
2015-08-10 07:25:04 +00:00
Run variables
-------------
These variables should be set when tests are scheduled (when running `isos post` ).
| Variable | Explanation |
| -------- | ----------- |
Add support for testing updates
Summary:
This adds an entirely new workflow for testing distribution
updates. The `ADVISORY` variable is introduced: when set,
`main.pm` will load an early post-install test that sets up
a repository containing the packages from the specified update,
runs `dnf -y update`, and reboots. A new templates file is
added, `templates-updates`, which adds two new flavors called
`updates-server` and `updates-workstation`, each containing
job templates for appropriate post-install tests. Scheduler is
expected to post `ADVISORY=(update ID) HDD_1=(base image)
FLAVOR=updates-(server|workstation)`, where (base image) is one
of the stable release base disk images produced by `createhdds`
and usually used for upgrade testing. This will result in the
appropriate job templates being loaded.
We rejig postinstall test loading and static network config a
bit so that this works for both the 'compose' and 'updates' test
flows: we have to ensure we bring up networking for the tap
tests before we try and install the updates, but still allow
later adjustment of the configuration. We take advantage of the
openQA feature that was added a few months back to run the same
module multiple times, so the `_advisory_update` module can
reboot after installing the updates and the modules that take
care of bootloader, encryption and login get run again. This
looks slightly wacky in the web UI, though - it doesn't show the
later runs of each module.
We also use the recently added feature to specify `+HDD_1` in
the test suites which use a disk image uploaded by an earlier
post-install test, so the test suite value will take priority
over the value POSTed by the scheduler for those tests, and we
will use the uploaded disk image (and not the clean base image
POSTed by the scheduler) for those tests.
My intent here is to enhance the scheduler, adding a consumer
which listens out for critpath updates, and runs this test flow
for each one, then reports the results to ResultsDB where Bodhi
could query and display them. We could also add a list of other
packages to have one or both sets of update tests run on it, I
guess.
Test Plan:
Try a post something like:
HDD_1=disk_f25_server_3_x86_64.img DISTRI=fedora VERSION=25
FLAVOR=updates-server ARCH=x86_64 BUILD=FEDORA-2017-376ae2b92c
ADVISORY=FEDORA-2017-376ae2b92c CURRREL=25 PREVREL=24
Pick an appropriate `ADVISORY` (ideally, one containing some
packages which might actually be involved in the tests), and
matching `FLAVOR` and `HDD_1`. The appropriate tests should run,
a repo with the update packages should be created and enabled
(and dnf update run), and the tests should work properly. Also
test a regular compose run to make sure I didn't break anything.
Reviewers: jskladan, jsedlak
Reviewed By: jsedlak
Subscribers: tflink
Differential Revision: https://phab.qa.fedoraproject.org/D1143
2017-01-25 16:16:12 +00:00
| `ISO` | contains filename of ISO that is used for booting. If `ISO_URL` is not set, file must exist in /var/lib/openqa/share/factory/iso. If `ISO_URL` is set, it will be downloaded to this name instead of its original name |
2016-02-24 02:04:09 +00:00
| `ISO_URL` | contains URL for ISO to boot, openQA will download it |
Add support for testing updates
Summary:
This adds an entirely new workflow for testing distribution
updates. The `ADVISORY` variable is introduced: when set,
`main.pm` will load an early post-install test that sets up
a repository containing the packages from the specified update,
runs `dnf -y update`, and reboots. A new templates file is
added, `templates-updates`, which adds two new flavors called
`updates-server` and `updates-workstation`, each containing
job templates for appropriate post-install tests. Scheduler is
expected to post `ADVISORY=(update ID) HDD_1=(base image)
FLAVOR=updates-(server|workstation)`, where (base image) is one
of the stable release base disk images produced by `createhdds`
and usually used for upgrade testing. This will result in the
appropriate job templates being loaded.
We rejig postinstall test loading and static network config a
bit so that this works for both the 'compose' and 'updates' test
flows: we have to ensure we bring up networking for the tap
tests before we try and install the updates, but still allow
later adjustment of the configuration. We take advantage of the
openQA feature that was added a few months back to run the same
module multiple times, so the `_advisory_update` module can
reboot after installing the updates and the modules that take
care of bootloader, encryption and login get run again. This
looks slightly wacky in the web UI, though - it doesn't show the
later runs of each module.
We also use the recently added feature to specify `+HDD_1` in
the test suites which use a disk image uploaded by an earlier
post-install test, so the test suite value will take priority
over the value POSTed by the scheduler for those tests, and we
will use the uploaded disk image (and not the clean base image
POSTed by the scheduler) for those tests.
My intent here is to enhance the scheduler, adding a consumer
which listens out for critpath updates, and runs this test flow
for each one, then reports the results to ResultsDB where Bodhi
could query and display them. We could also add a list of other
packages to have one or both sets of update tests run on it, I
guess.
Test Plan:
Try a post something like:
HDD_1=disk_f25_server_3_x86_64.img DISTRI=fedora VERSION=25
FLAVOR=updates-server ARCH=x86_64 BUILD=FEDORA-2017-376ae2b92c
ADVISORY=FEDORA-2017-376ae2b92c CURRREL=25 PREVREL=24
Pick an appropriate `ADVISORY` (ideally, one containing some
packages which might actually be involved in the tests), and
matching `FLAVOR` and `HDD_1`. The appropriate tests should run,
a repo with the update packages should be created and enabled
(and dnf update run), and the tests should work properly. Also
test a regular compose run to make sure I didn't break anything.
Reviewers: jskladan, jsedlak
Reviewed By: jsedlak
Subscribers: tflink
Differential Revision: https://phab.qa.fedoraproject.org/D1143
2017-01-25 16:16:12 +00:00
| `ADVISORY` | A Bodhi update ID. If set, the 'update testing' flow will be used: post-install tests will be run with the packages from the update, starting from the stable release base disk images |
2015-08-10 07:25:04 +00:00
| `DISTRI` | contains distribution name (should be same as in WebUI, probably `fedora` ) |
| `VERSION` | contains version of distribution |
2017-03-13 19:43:54 +00:00
| `DEVELOPMENT` | set for update tests if the update is for a development release (not a stable release) |
Pungi 4 conversion: handle Pungi-derived BUILD and FLAVOR
With the arrival of Pungi 4, the scheduler is no longer using
fedfind-provided BUILD and FLAVOR values, but ones derived from
Pungi properties. BUILD is now simply the Pungi compose_id.
FLAVOR is produced by joining the Pungi variant, type, and
format with '-' characters as the separators.
Pungi, unfortunately, does not treat 'Rawhide' as a release, it
synthesizes a release number for Rawhide composes and places
that in the compose ID. To cope with that, for now, the
scheduler will set RAWHIDE to '1' if the compose is a Rawhide
one. As we have to adapt all places where we parse the release
in any case, this commit consolidates them into a fedorabase
subroutine.
For the one place where we also used to parse the 'milestone'
from fedfind, there is a placeholder get_milestone subroutine
which currently returns an empty string, as I don't yet have a
good handle on how to draw the kinds of distinctions fedfind
mapped to 'milestone' from Pungi metadata.
2016-01-28 04:48:04 +00:00
| `FLAVOR` | indicates what type of distribution is used. Three Pungi properties, joined with `-` : `variant` , `type` , and `format` . e.g.: `Server-dvd-iso` . Special value `universal` is used to schedule the group of tests that should be run once each per arch per compose, against the 'best' available ISO |
2015-08-10 07:25:04 +00:00
| `ARCH` | is set to architecture that will be used (`x86_64`, `i686` ) |
Pungi 4 conversion: handle Pungi-derived BUILD and FLAVOR
With the arrival of Pungi 4, the scheduler is no longer using
fedfind-provided BUILD and FLAVOR values, but ones derived from
Pungi properties. BUILD is now simply the Pungi compose_id.
FLAVOR is produced by joining the Pungi variant, type, and
format with '-' characters as the separators.
Pungi, unfortunately, does not treat 'Rawhide' as a release, it
synthesizes a release number for Rawhide composes and places
that in the compose ID. To cope with that, for now, the
scheduler will set RAWHIDE to '1' if the compose is a Rawhide
one. As we have to adapt all places where we parse the release
in any case, this commit consolidates them into a fedorabase
subroutine.
For the one place where we also used to parse the 'milestone'
from fedfind, there is a placeholder get_milestone subroutine
which currently returns an empty string, as I don't yet have a
good handle on how to draw the kinds of distinctions fedfind
mapped to 'milestone' from Pungi metadata.
2016-01-28 04:48:04 +00:00
| `BUILD` | contains Pungi compose_id (something like `Fedora-24-20160121.n.3` ) |
2016-09-01 17:46:49 +00:00
| `CURRREL` | the current stable Fedora release at the time of the test run |
| `PREVREL` | the previous stable Fedora release at the time of the test run |
| `LOCATION` | contains Pungi base compose location (something like `https://kojipkgs.fedoraproject.org/compose/branched/Fedora-25-20160901.n.0/compose/` ) |
