srbala/os-autoinst-distri-fedora
1
0
Fork 0
mirror of https://pagure.io/fedora-qa/os-autoinst-distri-fedora.git synced 2024-11-30 17:33:07 +00:00
Code Issues Releases Activity
serialfixes
os-autoinst-distri-fedora/VARIABLES.md

112 lines
13 KiB
Markdown
Raw Permalink Normal View History

add VARIABLES file Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D488
2015-08-10 07:25:04 +00:00
List of variables usable in tests
=================================
This list contains variables that are possible to set/redefine in WebUI and get their values in tests.
This document merges relevant variables from
[official documentation](https://github.com/os-autoinst/os-autoinst/blob/master/doc/backend_vars.asciidoc)
(backend variables), variables that control flow through `main.pm` (test variables) and variables that
are specified when ISO is posted into OpenQA (run variables).
Backend variables
-----------------
These variables control settings of underlying virtual machine where tests runs on. Some of them (`ARCH`, `QEMURAM`, ...) should be set per machine, others (`HDD_$i`, `HDDMODEL`, ...) should be set per test and some of them (`KEEPHDDS`, `MAKETESTSNAPSHOTS`, `SKIPTO`) should be set only during development and debugging.
| Variable | Values allowed | Default value | Explanation |
| -------- | :------------: | :-----------: | ----------- |
| `ARCH` | `x86_64`, `i686`, `aarch64`, ... | depends on tested medium | architecture of VM |
Run the install_default_update test on UEFI as well Obviously an installer image may work fine for a BIOS install but fail for a UEFI install, so we should test it both ways. This requires the invention of a new (terribly named, but I couldn't think of anything better...) variable to allow us to use the 'run test on machine A after test run on machine B' mechanism without just hardcoding '64bit' as the machine name in the START_AFTER setting (which would break if we ever wanted to run update tests on any *other* arch). Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-29 13:23:22 +00:00
| `ARCH_BASE_MACHINE` | `uefi`, `64bit`, `ARM`, ... | depends on machine | 'machine' value to be used for tests where we might want to start after a test run on a different machine. For instance, we run tests that produce x86_64 ISOs on the '64bit' machine, so `ARCH_BASE_MACHINE` for the 'uefi' machine is set to '64bit', so that we can run UEFI install tests on the 'uefi' machine with an image generated by a test run on the '64bit' machine |
add VARIABLES file Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D488
2015-08-10 07:25:04 +00:00
| `BIOS` | filename | `ovmf-x86_64-ms.bin` for `x86_64` UEFI, not set otherwise | set different BIOS |
| `BOOTFROM` | characters (see man qemu, `-boot order=` option) | not set | boot from different medium than CD |
| `CDMODEL` | see `qemu-kvm -device help` | not set/Qemu default | type of device for CD |
mention main.pm architecture in README Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D495
2015-08-11 11:30:42 +00:00
| `HDDMODEL` | see `qemu-kvm -device help` (e. g. PATA = `id-hd`, SATA = `ide-hd,bus-ahci0.0`, SCSI = `virtio-scsi-pci`) | `virtio-blk` | type of device for HDD |
add VARIABLES file Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D488
2015-08-10 07:25:04 +00:00
| `HDDSIZEGB` | integer | `10` | size (in GBs) for disks that are created automatically |
| `HDD_$i` (`HDD_1, HDD_2`, ...) | filename | not set | attach additional HDD to VM |
| `ISO` | filename | not set | attach CD drive with ISO inserted |
| `ISO_$i` (`ISO_1`, `ISO_2`, ...) | filename | not set | additional CD drives to by attached |
| `KEEPHDDS` | boolean | `false`/not set | don't delete HDD after test finishes |
| `LAPTOP` | boolean or filename | `false`/not set | if `true`, Dell E6330 DMI is used; if set to filename, that file is used for DMI |
| `MAKETESTSNAPSHOTS` | boolean | `false`/not set | save snapshot for each test |
| `NOVIDEO` | boolean | `false`/not set | don't create video output if set |
| `NUMDISKS` | integer | 1 | number of disks to be created and attached to VM |
Adds variable PART_TABLE_TYPE to machines PART_TABLE_TYPE variable says which type of partition table type should be on attached HDDs. Some tests with uefi have to use disks with gpt. Tests are amended to use right disks. Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D623
2015-10-15 14:07:00 +00:00
| `PART_TABLE_TYPE` | `mbr`, `gpt` | not set | what type of partition table should attached HDD have |
add VARIABLES file Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D488
2015-08-10 07:25:04 +00:00
| `PXEBOOT` | boolean | `false`/not set | boot VM from PXE (network) |
| `QEMU` | filename, path to Qemu binary | not set | filename of Qemu binary |
| `QEMUCPU` | see `qemu-kvm -cpu help` | not set | CPU to emulate |
| `QEMUCPUS` | integer | 1 | number of processor cores to use for Qemu |
| `QEMURAM` | integer | 1024 | size of RAM to use (in MiB) |
| `QEMUTHREADS` | integer | 0 | number of CPU threads to use for Qemu |
| `QEMUVGA` | see `qemu-kvm -device help` | `cirrus` (should be set to `std` on Fedora, cirrus driver [was retired](https://lists.fedoraproject.org/pipermail/devel/2014-May/199459.html)) | display device to use for VM |
| `QEMU_COMPRESS_QCOW2` | boolean | `false`/not set | compress qcow2 images |
| `SKIPTO` | name of snapshot | not set | restore VM from given snapshot - better used with `MAKETESTSNAPSHOTS` |
| `UEFI` | boolean | `false`/not set | whether to use UEFI (UEFI BIOS files should be installed) |
| `UEFI_BIOS` | filename | `false`/not set, `ovmf-x86_64-ms.bin` when `UEFI` is set | filename of UEFI BIOS |
| `USBBOOT` | boolean | `false`/not set | if set, mount ISO as USB and boot from it |
For additional (kvm2usb, IPMI...) as well as for other Qemu variables, see
[official documentation](https://github.com/os-autoinst/os-autoinst/blob/master/doc/backend_vars.asciidoc).
Test variables
--------------
These variables control flow through `main.pm` - by setting them, you can alter what parts will be loaded.
There are certain tests that conflict each other. Note that conflict is always mutual, but for simplification
purposes, only one side of this conflict is shown (that means, if it's shown that `A` conflicts `B`,
it also means that `B` conflicts `A` even if not shown in the table).
| Variable | Values allowed | Default value/behavior | Conflicts | Explanation |
| -------- | :------------: | ---------------------- | :-------: | ----------- |
| `LIVE` | boolean | not set | technically `PACKAGE_SET`, `KICKSTART`, `MIRRORLIST_GRAPHICAL`, `REPOSITORY_GRAPHICAL`, `REPOSITORY_VARIATION` | specify that the test is running on Live system - additional steps are required (starting Anaconda...) |
| `PACKAGE_SET` | string (`minimal`, `default`, ...) | `minimal`; `default` when `LIVE` is set | nothing | sets packageset to install - you have to add appropriate needles, see `tests/_software_selection.pm` |
enable ENTRYPOINT to be more than one test, ensure shutdown when on ARM This adds possiblity to specify more than one test in ENTRYPOINT variable - test names should be separated with spaces and loading is done as with POSTINSTALL. Add _console_shutdown test to ARM image deployment. Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D973
2016-08-16 07:33:10 +00:00
| `ENTRYPOINT` | filename | not set | N/A | set `ENTRYPOINT` to load specific test directly (bypass modular structure, mainly usable for testing). If you want to run more than one test, separate them with space. |
add VARIABLES file Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D488
2015-08-10 07:25:04 +00:00
| `UPGRADE` | string (`minimal`, `desktop`, `encrypted`, ...) | not set | all except of `LIVE` and `USER_PASSWORD` | when set, do an upgrade test of specified type |
| `KICKSTART` | boolean | `false`/not set | all | when specified, do an kickstart installation (you should use `GRUB` variable to specify where kickstart file resides) |
| `MIRRORLIST_GRAPHICAL` | boolean | `false`/not set | `REPOSITORY_GRAPHICAL` | sets installation source to mirrorlist |
| `REPOSITORY_GRAPHICAL` | url to repository (without arch and `/os`, for example `http://dl.fedoraproject.org/pub/fedora/linux/development`) | not set | `MIRRORLIST_GRAPHICAL` | sets installation source to repository url in Anaconda |
| `REPOSITORY_VARIATION` | url to repository (without arch and `/os`, for example `http://dl.fedoraproject.org/pub/fedora/linux/development`) | not set | nothing | sets installation source to repository url in GRUB |
add ext3, btrfs and LVM thin tests, tweaks to custom methods Summary: This adds three new custom storage tests and some needles to support them, and tweaks the custom storage methods a bit to address some things that cropped up in writing the tests. A new method is added for changing the filesystem, as that's a distinct operation from changing the device type. This also restores the previous behaviour of select_disks() where it handled selecting custom partitioning when needed. Turns out it's pretty common to use regex'es in perl! Who'd'a thought. A corresponding commit to add the tests to openqa_fedora_tools is coming. There's no post-install step for the tests yet; I'll try and write those up and add them soon. Test Plan: Do a full run, including the new tests, on Alpha RC2 and check all are scheduled correctly and run correctly. The LVM thinp test is expected to fail as it catches a genuine bug. Reviewers: jskladan, garretraziel Reviewed By: garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D490
2015-08-10 18:01:12 +00:00
| `PARTITIONING` | string (`custom_software_raid`, `guided_delete_all`, ...) | `guided_empty` | nothing | load specified test for partitioning part (when `PARTITIONING=guided_delete_all`, `tests/disk_guided_delete_all.pm` is loaded) and optionally post-install partitioning check (if `tests/disk_guided_delete_all_postinstall.pm` exists, it will be loaded after login to the installed system). Also, if value starts with `custom_`, the `select_disks()` method will check the custom partitioning box |
add VARIABLES file Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D488
2015-08-10 07:25:04 +00:00
| `ENCRYPT_PASSWORD` | string | not set | nothing | if set, encrypt disk with given password |
| `DESKTOP` | boolean | `false`/not set | nothing | set to indicate that Fedora is running with GUI (so for example OpenQA should expect graphical login screen) |
More tweaking for this damn no root password spoke situation Previous approach wouldn't work for tests that run after the install test...let's just set a password from a chroot after install completes. Don't really like this as it changes the 'real' install process a bit, but it's the least invasive short term fix at least. We can maybe do something more sudo-y later with a bit more thought. Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-03-09 04:31:14 +00:00
| `ROOT_PASSWORD` | string | `weakpassword` | nothing | root password is set to this value |
| `INSTALLER_NO_ROOT` | boolean | `false`/not set | nothing | if set, root password is not set via the anaconda spoke (but, for now, will be set by chrooting into the installed system at the end of installation) |
add VARIABLES file Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D488
2015-08-10 07:25:04 +00:00
| `GRUB` | string | not set | nothing | when set, append this string to kernel line in GRUB |
rename BOOT_UPDATES_IMG_URL to TEST_UPDATES, add GRUBADD Summary: BOOT_UPDATES_IMG_URL is a pretty misleading name - it used to be the actual URL, but now it's simply a boolean that decides whether we look for the effect of the openQA updates image or not. TEST_UPDATES seems clearer. GRUBADD does the same thing as GRUB, on top of it. The point of this is so we can add an option to the scheduler CLI that lets you say 'run the normal tests, but with this updates image' - so we can easily (albeit manually triggered) check the impact of some anaconda change that needs testing. It should never be set in the templates or the tests, it's there strictly for the scheduler (whether that's fedora_openqa_schedule or literally a person calling `client isos post`) to use as a kind of override. The tests that test updates image loading will probably fail when doing this, but all other tests should work as intended, including ones that specify GRUB, becase the extra params will just get added on top. That's why I invented a new var instead of just letting the scheduler override GRUB's value when POST ing. Test Plan: Check the rename didn't break anything (updates tests still work). Run tests with GRUBADD param, make sure value is correctly appended to cmdline both when GRUB is also specified and when it is not. Reviewers: jskladan, garretraziel Reviewed By: garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D801
2016-04-08 20:21:29 +00:00
| `GRUBADD` | string | not set | nothing | when set, append this string to kernel line in GRUB, after the `GRUB` string if that one is set too. This is never set by tests; instead it's provided as a way to e.g. run the normal tests with an `updates.img` to test some anaconda change (the scheduler CLI can help you use this feature) |
add firewall kickstart tests (disabled and configured) Summary: these together test QA:Testcase_kickstart_firewall from the Server matrix. I'll have to come up with some kinda way to handle reporting that, might be tricky. Couple of tweaks to overall test flow: tests can now specify a POSTINSTALL variable which will load a post-install test following a naming convention, and tests can specify USER_LOGIN as 'false' to disable the 'log in as a user' step entirely. We could easily adjust the kickstarts to create a user so the test could log in as one, but it seems like an unnecessary step and I liked the idea of allowing the user login to be skipped. Test Plan: Schedule 'universal' tests, check the new tests run and pass or fail as they should, check no other test is broken by the logic flow changes. Reviewers: jskladan, garretraziel Reviewed By: garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D792
2016-03-23 20:52:00 +00:00
| `USER_LOGIN` | string | not set | should be used with `USER_PASSWORD` (unless `false`) | when set, user login is set to this value. If not set, default value `test` is used for console installs, no login is done for graphical installs. If set to `false`, no user login will be done |
| `USER_PASSWORD` | string | not set | should be used with `USER_LOGIN` | when set, user password is set to this value. If not set, default value `weakpassword` is used for console installs, no login is done for graphical installs |
rename BOOT_UPDATES_IMG_URL to TEST_UPDATES, add GRUBADD Summary: BOOT_UPDATES_IMG_URL is a pretty misleading name - it used to be the actual URL, but now it's simply a boolean that decides whether we look for the effect of the openQA updates image or not. TEST_UPDATES seems clearer. GRUBADD does the same thing as GRUB, on top of it. The point of this is so we can add an option to the scheduler CLI that lets you say 'run the normal tests, but with this updates image' - so we can easily (albeit manually triggered) check the impact of some anaconda change that needs testing. It should never be set in the templates or the tests, it's there strictly for the scheduler (whether that's fedora_openqa_schedule or literally a person calling `client isos post`) to use as a kind of override. The tests that test updates image loading will probably fail when doing this, but all other tests should work as intended, including ones that specify GRUB, becase the extra params will just get added on top. That's why I invented a new var instead of just letting the scheduler override GRUB's value when POST ing. Test Plan: Check the rename didn't break anything (updates tests still work). Run tests with GRUBADD param, make sure value is correctly appended to cmdline both when GRUB is also specified and when it is not. Reviewers: jskladan, garretraziel Reviewed By: garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D801
2016-04-08 20:21:29 +00:00
| `TEST_UPDATES` | boolean | `false`/not set | set to indicate that this test checks updates.img loading, so we should check for the expected effect of the updates image used for this testing |
New install_repository_hd_variation use a new PREINSTALL to execute a rescue script before install. This is covering the Hard_drive_variation test as per: https://fedoraproject.org/wiki/QA:Testcase_install_repository_Hard_drive_variation Signed-off-by: Guy Menanteau <menantea@linux.vnet.ibm.com>
2019-02-04 14:46:31 +00:00
| `PREINSTALL` | string | not set | nothing | If set, specified module will be loaded before reboot and install; module supposed to be starting as rescue mode |
Create a test suite for Evince. This PR fixes issue #188. It adds a test suite to test basic functionality of Evince and brings the following features: * test scripts for various Evince functions. * needles to support the Evince test scripts * new template variables `TESTPATH` and `POSTINSTALL_LOAD_ALL` (see below) * new logic in `main.py` (see below) The new variables and the new logic make it easier to create test suites for post-installation tests. If TESTPATH is used, OpenQA will take all tests mentioned in POSTINSTALL from that specified TESTPATH. If both TESTPATH and POSTINSTALL_LOAD_ALL are used, then OpenQA will run all tests it can find at the TESTPATH location. If POSTINSTALL and POSTINSTALL_LOAD_ALL are set simultaneously, then only POSTINSTALL will be taken into account and OpenQA will only load tests mentioned there.
2021-07-14 16:04:16 +00:00
| `POSTINSTALL` | string | not set | `POSTINSTALL_PATH` | If set, `tests/(value)_postinstall.pm` will be loaded after install, boot, login, and other postinstall tests
Small fixes.
2021-07-28 07:02:02 +00:00
| `POSTINSTALL_PATH` | boolean | not set | `POSTINSTALL` | If set, `all tests on this location will be loaded as postinstall tests after install, boot, login, and other postinstall tests |
Adds uefi support to tests where it makes sense What changed: * There is a new needle for uefi bootloader. * UEFI postinstallation phase * UEFI tests and machine added to template
2015-09-15 09:04:01 +00:00
| `UEFI` | boolean | `false`/not set | nothing | whether to use UEFI, this variable isn't usually set in test suites but in machine definition |
add anaconda text UI test Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D980
2016-09-07 08:34:54 +00:00
| `ANACONDA_TEXT` | boolean | `false`/not set | all | when specified, anaconda will run in text mode |
Automate the QA:Testcase_Anaconda_help. This PR automates the mentioned testcase to test that Help can be displayed in Anaconda during the installation. It navigates through the available Help screens and if it can see it, it finishes. This test runs after `install_default_upload` to override the installation defaults defined for all primary tests. Delete a duplicated needle. Reformat list extensions to make it nicer. Get rid of wrong export and an empty line. Delete empty line. Use _boot_to_anaconda for booting and move subroutine accordingly. Add variable to templates.fif.json Delete trailing whitespace. Fix calling the pretest. Move help checking to another place.
2021-05-03 15:37:46 +00:00
| `HELPCHECK` | boolean | `false`/not set | all | when specified, Anaconda Help will be called on each available pane. |
Add support for testing updates Summary: This adds an entirely new workflow for testing distribution updates. The `ADVISORY` variable is introduced: when set, `main.pm` will load an early post-install test that sets up a repository containing the packages from the specified update, runs `dnf -y update`, and reboots. A new templates file is added, `templates-updates`, which adds two new flavors called `updates-server` and `updates-workstation`, each containing job templates for appropriate post-install tests. Scheduler is expected to post `ADVISORY=(update ID) HDD_1=(base image) FLAVOR=updates-(server|workstation)`, where (base image) is one of the stable release base disk images produced by `createhdds` and usually used for upgrade testing. This will result in the appropriate job templates being loaded. We rejig postinstall test loading and static network config a bit so that this works for both the 'compose' and 'updates' test flows: we have to ensure we bring up networking for the tap tests before we try and install the updates, but still allow later adjustment of the configuration. We take advantage of the openQA feature that was added a few months back to run the same module multiple times, so the `_advisory_update` module can reboot after installing the updates and the modules that take care of bootloader, encryption and login get run again. This looks slightly wacky in the web UI, though - it doesn't show the later runs of each module. We also use the recently added feature to specify `+HDD_1` in the test suites which use a disk image uploaded by an earlier post-install test, so the test suite value will take priority over the value POSTed by the scheduler for those tests, and we will use the uploaded disk image (and not the clean base image POSTed by the scheduler) for those tests. My intent here is to enhance the scheduler, adding a consumer which listens out for critpath updates, and runs this test flow for each one, then reports the results to ResultsDB where Bodhi could query and display them. We could also add a list of other packages to have one or both sets of update tests run on it, I guess. Test Plan: Try a post something like: HDD_1=disk_f25_server_3_x86_64.img DISTRI=fedora VERSION=25 FLAVOR=updates-server ARCH=x86_64 BUILD=FEDORA-2017-376ae2b92c ADVISORY=FEDORA-2017-376ae2b92c CURRREL=25 PREVREL=24 Pick an appropriate `ADVISORY` (ideally, one containing some packages which might actually be involved in the tests), and matching `FLAVOR` and `HDD_1`. The appropriate tests should run, a repo with the update packages should be created and enabled (and dnf update run), and the tests should work properly. Also test a regular compose run to make sure I didn't break anything. Reviewers: jskladan, jsedlak Reviewed By: jsedlak Subscribers: tflink Differential Revision: https://phab.qa.fedoraproject.org/D1143
2017-01-25 16:16:12 +00:00
| `ANACONDA_STATIC` | string (IPv4 address) | not set | `ANACONDA_TEXT` | If set, will set up static networking using the chosen IP address during install |
| `POST_STATIC` | string (space-separated IPv4 address and hostname) | not set | nothing | If set, will set up static networking using the chosen IP address and hostname during early post-install |
Fix French install test on aarch64 by skipping `uefi_postinstall` The French install/boot test fails on aarch64 due to a bit of an ordering issue: https://openqa.stg.fedoraproject.org/tests/665124 we run `uefi_postinstall`, which does `loadkeys us`, before we run `_console_login`, which still expects the French layout, so it breaks. The safest way to solve this I think is to add a new variable that lets us skip `uefi_postinstall` - I don't want to change the ordering so we load `uefi_postinstall` after we read `POSTINSTALL` in case that breaks things somewhere else. Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-11-08 19:21:37 +00:00
| `NO_UEFI_POST` | boolean | `false`/not set | nothing | If set, `uefi_postinstall` test will not be loaded even if `UEFI` is set (can be useful for non-English tests to avoid `uefi_postinstall` running loadkeys) |
Automate the Anaconda_Save_Traceback_to_Bugzilla_Text test case. This PR adds a new test that automates the above mentioned test case. It starts the installation in text mode using the `install_text` test case, which it interrupts using the Anaconda crash trigger. When the crash happens, it goes through the process of reporting the bug to Bugzilla, checks that Bugzilla sends a positive confirmation of the action, but also performs some REST API calls to do a proper check and then it closes the bug to clean up.
2021-04-27 11:02:53 +00:00
| `CRASH_REPORT` | boolean | not set | nothing | If set, Anaconda will perorm a simulated crash during installation, which will be used to create a report in the Bugzilla. Currently, this only affects Anaconda when it runs in text mode. |
| `BUGZILLA_LOGIN` | string | not set | used with `_SECRET_BUGZILLA_PASSWORD` | This is used to store a login string which does not get exposed in log files. |
| `_SECRET_BUGZILLA_PASSWORD` | string | not set | used with `BUGZILLA_LOGIN` | This is used to store a password string which does not get exposed in log files. |
| `_SECRET_BUGZILLA_APIKEY` | string | not set | used with other secrets | This is used to store an API key which does not get exposed in log files. |
Modify Desktop Printing test to use two different methods. The PR introduces an improved logic to the desktop_printing.pm that allows to use the USE_CUPS variable in templates to trigger the installation of cups-pdf prior to the actual test. The cups-pdf is then used as an alternative PDF printer instead the built-in Save As PDF method.
2022-02-08 08:57:27 +00:00
| `USE_CUPS` | boolean | `false`/not set | if set, the desktop printing test will use cups-pdf instead of the generic method |
add VARIABLES file Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D488
2015-08-10 07:25:04 +00:00
Run variables
-------------
These variables should be set when tests are scheduled (when running `isos post`).
| Variable | Explanation |
| -------- | ----------- |
Add support for testing updates Summary: This adds an entirely new workflow for testing distribution updates. The `ADVISORY` variable is introduced: when set, `main.pm` will load an early post-install test that sets up a repository containing the packages from the specified update, runs `dnf -y update`, and reboots. A new templates file is added, `templates-updates`, which adds two new flavors called `updates-server` and `updates-workstation`, each containing job templates for appropriate post-install tests. Scheduler is expected to post `ADVISORY=(update ID) HDD_1=(base image) FLAVOR=updates-(server|workstation)`, where (base image) is one of the stable release base disk images produced by `createhdds` and usually used for upgrade testing. This will result in the appropriate job templates being loaded. We rejig postinstall test loading and static network config a bit so that this works for both the 'compose' and 'updates' test flows: we have to ensure we bring up networking for the tap tests before we try and install the updates, but still allow later adjustment of the configuration. We take advantage of the openQA feature that was added a few months back to run the same module multiple times, so the `_advisory_update` module can reboot after installing the updates and the modules that take care of bootloader, encryption and login get run again. This looks slightly wacky in the web UI, though - it doesn't show the later runs of each module. We also use the recently added feature to specify `+HDD_1` in the test suites which use a disk image uploaded by an earlier post-install test, so the test suite value will take priority over the value POSTed by the scheduler for those tests, and we will use the uploaded disk image (and not the clean base image POSTed by the scheduler) for those tests. My intent here is to enhance the scheduler, adding a consumer which listens out for critpath updates, and runs this test flow for each one, then reports the results to ResultsDB where Bodhi could query and display them. We could also add a list of other packages to have one or both sets of update tests run on it, I guess. Test Plan: Try a post something like: HDD_1=disk_f25_server_3_x86_64.img DISTRI=fedora VERSION=25 FLAVOR=updates-server ARCH=x86_64 BUILD=FEDORA-2017-376ae2b92c ADVISORY=FEDORA-2017-376ae2b92c CURRREL=25 PREVREL=24 Pick an appropriate `ADVISORY` (ideally, one containing some packages which might actually be involved in the tests), and matching `FLAVOR` and `HDD_1`. The appropriate tests should run, a repo with the update packages should be created and enabled (and dnf update run), and the tests should work properly. Also test a regular compose run to make sure I didn't break anything. Reviewers: jskladan, jsedlak Reviewed By: jsedlak Subscribers: tflink Differential Revision: https://phab.qa.fedoraproject.org/D1143
2017-01-25 16:16:12 +00:00
| `ISO` | contains filename of ISO that is used for booting. If `ISO_URL` is not set, file must exist in /var/lib/openqa/share/factory/iso. If `ISO_URL` is set, it will be downloaded to this name instead of its original name |
ISO_URL not ISOURL, now...
2016-02-24 02:04:09 +00:00
| `ISO_URL` | contains URL for ISO to boot, openQA will download it |
Add support for testing updates Summary: This adds an entirely new workflow for testing distribution updates. The `ADVISORY` variable is introduced: when set, `main.pm` will load an early post-install test that sets up a repository containing the packages from the specified update, runs `dnf -y update`, and reboots. A new templates file is added, `templates-updates`, which adds two new flavors called `updates-server` and `updates-workstation`, each containing job templates for appropriate post-install tests. Scheduler is expected to post `ADVISORY=(update ID) HDD_1=(base image) FLAVOR=updates-(server|workstation)`, where (base image) is one of the stable release base disk images produced by `createhdds` and usually used for upgrade testing. This will result in the appropriate job templates being loaded. We rejig postinstall test loading and static network config a bit so that this works for both the 'compose' and 'updates' test flows: we have to ensure we bring up networking for the tap tests before we try and install the updates, but still allow later adjustment of the configuration. We take advantage of the openQA feature that was added a few months back to run the same module multiple times, so the `_advisory_update` module can reboot after installing the updates and the modules that take care of bootloader, encryption and login get run again. This looks slightly wacky in the web UI, though - it doesn't show the later runs of each module. We also use the recently added feature to specify `+HDD_1` in the test suites which use a disk image uploaded by an earlier post-install test, so the test suite value will take priority over the value POSTed by the scheduler for those tests, and we will use the uploaded disk image (and not the clean base image POSTed by the scheduler) for those tests. My intent here is to enhance the scheduler, adding a consumer which listens out for critpath updates, and runs this test flow for each one, then reports the results to ResultsDB where Bodhi could query and display them. We could also add a list of other packages to have one or both sets of update tests run on it, I guess. Test Plan: Try a post something like: HDD_1=disk_f25_server_3_x86_64.img DISTRI=fedora VERSION=25 FLAVOR=updates-server ARCH=x86_64 BUILD=FEDORA-2017-376ae2b92c ADVISORY=FEDORA-2017-376ae2b92c CURRREL=25 PREVREL=24 Pick an appropriate `ADVISORY` (ideally, one containing some packages which might actually be involved in the tests), and matching `FLAVOR` and `HDD_1`. The appropriate tests should run, a repo with the update packages should be created and enabled (and dnf update run), and the tests should work properly. Also test a regular compose run to make sure I didn't break anything. Reviewers: jskladan, jsedlak Reviewed By: jsedlak Subscribers: tflink Differential Revision: https://phab.qa.fedoraproject.org/D1143
2017-01-25 16:16:12 +00:00
| `ADVISORY` | A Bodhi update ID. If set, the 'update testing' flow will be used: post-install tests will be run with the packages from the update, starting from the stable release base disk images |
Handle running update tests on Koji tasks We quite often want to run the update tests on a Koji task (not a Bodhi update) for some reason - usually to test a potential fix for an issue, or at a maintainer's request to test a change before it is merged upstream and officially sent out as an update. Up till now I've always hacked up utils.pm on the staging server by hand to do this, which is horrible. Together with a commit to fedora_openqa, this should allow us to do it in a nice, sane way via the CLI. It's mostly just tweaking the "updates" repo setup in utils.pm as you'd expect, but there's a bit of subtlety to it because of the installer tests that use %ADVISORY% as a variable substitution in the disk image name; you can't do something like `%ADVISORY or KOJITASK%`, sadly, so I had to have almost-redundant variables ADVISORY, KOJITASK and ADVISORY_OR_TASK (we could kinda just live with ADVISORY_OR_TASK except I didn't want to drop ADVISORY as it's an unnecessary change from previous behavior). Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-29 09:06:16 +00:00
| `KOJITASK` | A Koji task ID. If set, the modified 'update testing' flow for testing scratch builds will be used: post-install tests will be run with the packages from the update, starting from the stable release base disk images |
add VARIABLES file Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D488
2015-08-10 07:25:04 +00:00
| `DISTRI` | contains distribution name (should be same as in WebUI, probably `fedora`) |
| `VERSION` | contains version of distribution |
Pungi 4 conversion: handle Pungi-derived BUILD and FLAVOR With the arrival of Pungi 4, the scheduler is no longer using fedfind-provided BUILD and FLAVOR values, but ones derived from Pungi properties. BUILD is now simply the Pungi compose_id. FLAVOR is produced by joining the Pungi variant, type, and format with '-' characters as the separators. Pungi, unfortunately, does not treat 'Rawhide' as a release, it synthesizes a release number for Rawhide composes and places that in the compose ID. To cope with that, for now, the scheduler will set RAWHIDE to '1' if the compose is a Rawhide one. As we have to adapt all places where we parse the release in any case, this commit consolidates them into a fedorabase subroutine. For the one place where we also used to parse the 'milestone' from fedfind, there is a placeholder get_milestone subroutine which currently returns an empty string, as I don't yet have a good handle on how to draw the kinds of distinctions fedfind mapped to 'milestone' from Pungi metadata.
2016-01-28 04:48:04 +00:00
| `FLAVOR` | indicates what type of distribution is used. Three Pungi properties, joined with `-`: `variant`, `type`, and `format`. e.g.: `Server-dvd-iso`. Special value `universal` is used to schedule the group of tests that should be run once each per arch per compose, against the 'best' available ISO |
add VARIABLES file Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D488
2015-08-10 07:25:04 +00:00
| `ARCH` | is set to architecture that will be used (`x86_64`, `i686`) |
Pungi 4 conversion: handle Pungi-derived BUILD and FLAVOR With the arrival of Pungi 4, the scheduler is no longer using fedfind-provided BUILD and FLAVOR values, but ones derived from Pungi properties. BUILD is now simply the Pungi compose_id. FLAVOR is produced by joining the Pungi variant, type, and format with '-' characters as the separators. Pungi, unfortunately, does not treat 'Rawhide' as a release, it synthesizes a release number for Rawhide composes and places that in the compose ID. To cope with that, for now, the scheduler will set RAWHIDE to '1' if the compose is a Rawhide one. As we have to adapt all places where we parse the release in any case, this commit consolidates them into a fedorabase subroutine. For the one place where we also used to parse the 'milestone' from fedfind, there is a placeholder get_milestone subroutine which currently returns an empty string, as I don't yet have a good handle on how to draw the kinds of distinctions fedfind mapped to 'milestone' from Pungi metadata.
2016-01-28 04:48:04 +00:00
| `BUILD` | contains Pungi compose_id (something like `Fedora-24-20160121.n.3`) |
Revise release variable handling, prerelease checks, os-release I started out trying to fix os-release for the recent change to add "Prerelease" tags to the VERSION and PRETTY_NAME fields, then things spiralled. It got me thinking about the awkward DEVELOPMENT variable we use, so I decided to get rid of it and refactor the few things that use it. I refactored the anaconda prerelease tag check, and wrote a new giant comment that gives details about exactly how anaconda decides whether to show those tags, to give context to our choices about when to expect them. This check now uses a new LABEL variable the scheduler now sets. I also wound up creating new UP1REL and UP2REL vars to define the 'source' release for upgrade tests, separate from CURRREL and PREVREL, which are now never lies - they really are the current stable and previous stable release, even for update upgrade tests. Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-05-07 22:42:08 +00:00
| `LABEL` | contains Pungi compose label, if it has one (otherwise should be unset) - e.g `RC-1.5` |
update VARIABLES a bit for newly added ones
2016-09-01 17:46:49 +00:00
| `CURRREL` | the current stable Fedora release at the time of the test run |
| `PREVREL` | the previous stable Fedora release at the time of the test run |
Revise release variable handling, prerelease checks, os-release I started out trying to fix os-release for the recent change to add "Prerelease" tags to the VERSION and PRETTY_NAME fields, then things spiralled. It got me thinking about the awkward DEVELOPMENT variable we use, so I decided to get rid of it and refactor the few things that use it. I refactored the anaconda prerelease tag check, and wrote a new giant comment that gives details about exactly how anaconda decides whether to show those tags, to give context to our choices about when to expect them. This check now uses a new LABEL variable the scheduler now sets. I also wound up creating new UP1REL and UP2REL vars to define the 'source' release for upgrade tests, separate from CURRREL and PREVREL, which are now never lies - they really are the current stable and previous stable release, even for update upgrade tests. Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-05-07 22:42:08 +00:00
| `UP1REL` | the source release for "1-release" upgrade tests (usually but not always same as `CURRREL`) |
| `UP2REL` | the source release for "2-release" upgrade tests (currently always same as `PREVREL`) |
update VARIABLES a bit for newly added ones
2016-09-01 17:46:49 +00:00
| `LOCATION` | contains Pungi base compose location (something like `https://kojipkgs.fedoraproject.org/compose/branched/Fedora-25-20160901.n.0/compose/`) |
Reference in New Issue Copy Permalink