srbala/os-autoinst-distri-fedora
1
0
Fork 0
mirror of https://pagure.io/fedora-qa/os-autoinst-distri-fedora.git synced 2024-11-09 09:04:20 +00:00
Code Issues Releases Activity
ddb6f8e416
os-autoinst-distri-fedora/README.md
Adam Williamson 92d588f245 Add support for testing updates 
Summary:
This adds an entirely new workflow for testing distribution
updates. The `ADVISORY` variable is introduced: when set,
`main.pm` will load an early post-install test that sets up
a repository containing the packages from the specified update,
runs `dnf -y update`, and reboots. A new templates file is
added, `templates-updates`, which adds two new flavors called
`updates-server` and `updates-workstation`, each containing
job templates for appropriate post-install tests. Scheduler is
expected to post `ADVISORY=(update ID) HDD_1=(base image)
FLAVOR=updates-(server|workstation)`, where (base image) is one
of the stable release base disk images produced by `createhdds`
and usually used for upgrade testing. This will result in the
appropriate job templates being loaded.

We rejig postinstall test loading and static network config a
bit so that this works for both the 'compose' and 'updates' test
flows: we have to ensure we bring up networking for the tap
tests before we try and install the updates, but still allow
later adjustment of the configuration. We take advantage of the
openQA feature that was added a few months back to run the same
module multiple times, so the `_advisory_update` module can
reboot after installing the updates and the modules that take
care of bootloader, encryption and login get run again. This
looks slightly wacky in the web UI, though - it doesn't show the
later runs of each module.

We also use the recently added feature to specify `+HDD_1` in
the test suites which use a disk image uploaded by an earlier
post-install test, so the test suite value will take priority
over the value POSTed by the scheduler for those tests, and we
will use the uploaded disk image (and not the clean base image
POSTed by the scheduler) for those tests.

My intent here is to enhance the scheduler, adding a consumer
which listens out for critpath updates, and runs this test flow
for each one, then reports the results to ResultsDB where Bodhi
could query and display them. We could also add a list of other
packages to have one or both sets of update tests run on it, I
guess.

Test Plan:
Try a post something like:
HDD_1=disk_f25_server_3_x86_64.img DISTRI=fedora VERSION=25
FLAVOR=updates-server ARCH=x86_64 BUILD=FEDORA-2017-376ae2b92c
ADVISORY=FEDORA-2017-376ae2b92c CURRREL=25 PREVREL=24

Pick an appropriate `ADVISORY` (ideally, one containing some
packages which might actually be involved in the tests), and
matching `FLAVOR` and `HDD_1`. The appropriate tests should run,
a repo with the update packages should be created and enabled
(and dnf update run), and the tests should work properly. Also
test a regular compose run to make sure I didn't break anything.

Reviewers: jskladan, jsedlak

Reviewed By: jsedlak

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1143
2017-02-22 11:33:32 -08:00

96 lines
8.4 KiB
Markdown
Raw Blame History

openQA tests for the Fedora distribution
========================================
This repository contains tests and images for testing [Fedora](https://getfedora.org/) with [openQA](http://os-autoinst.github.io/openQA/). The [fedora_openqa library and CLI](https://pagure.io/fedora-qa/fedora_openqa) are used for scheduling tests, and [createhdds](https://pagure.io/fedora-qa/createhdds) is used for creating base disk images for the test. For openQA installation instructions, see [the Fedora openQA wiki page](https://fedoraproject.org/wiki/OpenQA).
Issues
------
For the present, issues (and pull requests - 'diffs') are tracked in [Fedora QA Phabricator](https://phab.qa.fedoraproject.org/). You can open issues against the `os-autoinst-distri-fedora` project [here](https://phab.qa.fedoraproject.org/maniphest/task/edit/form/default/?tags=os-autoinst-distri-fedora). See [the QA Phabricator wiki page](https://fedoraproject.org/wiki/QA:Phabricator#issues-diffs) for detailed instructions on filing issues and diffs. Note that this repository does not use the 'gitflow' system, so the main development branch is `master`: please branch from `master` and submit diffs against it. This is not a Python repository and has no tests or linting, so you do not need to follow the instructions regarding `virtualenv`.
Test development
----------------
See official documentation on:
* [basic concept](https://github.com/os-autoinst/openQA/blob/master/docs/GettingStarted.asciidoc)
* [test development (including API specification)](https://github.com/os-autoinst/openQA/blob/master/docs/WritingTests.asciidoc)
* [needles specification](https://github.com/os-autoinst/os-autoinst/blob/master/doc/needles.txt)
* [supported variables for backend](https://github.com/os-autoinst/os-autoinst/blob/master/doc/backend_vars.asciidoc).
See [this example repo](https://github.com/os-autoinst/os-autoinst-distri-example) on how tests should be structured.
### main.pm modular architecture
Since openQA uses only one entrypoint for all tests (main.pm), we have decided to utilize this feature and make tests modular. It means that basic passing through main.pm (without any variables set) results in most basic installation test executed. Developer can customize it with additional variables (for example by setting `PACKAGE_SET=minimal` to do installation only with minimal package set).
Make your test modular, so that it utilizes `_boot_to_anaconda.pm`, `_software_selection.pm` and `_do_install_and_reboot.pm` tests (that are loaded automatically). Break your test into smaller parts, each dealing with one specific feature (e. g. partitioning, user creation...) and add their loading into main.pm based on reasonable variable setting (so they can be used in other tests also).
Fedora installation (and consequently main.pm) consists of several parts:
#### Booting into Anaconda or booting live image and starting Anaconda
Since there isn't much variation between tests in this step, we have developed universal `_boot_to_anaconda.pm` test that is loaded automatically each time except when `ENTRYPOINT` or `UPGRADE` is set (see VARIABLES.md).
To customize this step, you can set following variables:
- `GRUB` is appended to kernel line before boot. You can set for example `inst.updates` here.
- If `KICKSTART` is set, this part of installation ends here (program doesn't wait for Anaconda to appear). Note that you should set `inst.ks` yourself by setting `GRUB` variable.
- If `LIVE` is set, program waits for desktop to appear and then clicks on "Install to Hard Drive" button.
#### Customizing installation by interacting with Anaconda spokes
Most of the differences between tests take place in this part. If you want to add another installation test, you will probably put your variable checking and test loading here. All tests in this part should start on Anaconda's main hub and after they done its part, they should go back to Anaconda's main hub so that next test could be executed. In this phase, universal `_software_selection.pm` test is loaded that handles selecting what software to install.
To customize this step, you can set following variables:
- Set `PACKAGE_SET` to install required package set on "Software selection spoke" - you have to provide correct needles with the name of `anaconda_${PACKAGE_SET}_highlighted` and `anaconda_${PACKAGE_SET}_selected`.
- Set `ENCRYPT_PASSWORD` to encrypt disk, value of this variable is used as an actual password.
#### Installing Fedora and waiting for Fedora to reboot
After all customizations are finished, `_do_install_and_reboot.pm` test is automatically loaded. It starts installation, creates user and sets root password when required, waits for installation to finish and reboots into installed system. Only variables that control flow in this part are these:
- `ROOT_PASSWORD` to set root password to this value.
- When set, `USER_LOGIN` and `USER_PASSWORD` are used to create user in Anaconda.
#### Post-install phase
After installation is finished and installed system is fully booted, you can run additional tests as checks that installed system has correct attributes - that correct file system is used, that RAID is used etc.
### Test inheritance
Your test can inherit from `basetest`, `installedtest` or `anacondatest`. Each provides relevant methods that are documented in-line, so read the files (`lib/anacondatest.pm`, `lib/installedtest.pm`) for information on these.
- `basetest`: A base class provided by os-autoinst - it has empty `post_fail_hook()` and doesn't set any flags.
- `anacondatest`: should be used in tests where Anaconda is running. It uploads Anaconda logs (for example `anaconda.log` or `packaging.log`) in `post_fail_hook()`.
- `installedtest`: should be used in tests that are running on installed system (either in postinstall phase or in upgrade tests).
There are also several modules that export utility functions, currently `utils`, `anaconda`, `freeipa`, `packagetest` and `tapnet`. Your test can `use` any of these modules and then directly call the functions they export. Again, the functions are documented in-line.
### New test development workflow
1. Select test from [phabricator page](https://phab.qa.fedoraproject.org/w/openqa/tests/).
2. Put each part of your test as a separate file into `tests/` directory, reimplementing `run()` method
and `test_flags()` method, inheriting from one of the classes mentioned above.
3. Set correct variables (so that all test parts you have made are executed) in [WebUI -> Test suites](https://localhost:8080/admin/test_suites).
4. Link your newly created Test suite to medium type in [WebUI -> Job groups](https://localhost:8080/admin/groups).
5. Run test (see [openqa_fedora_tools repository](https://bitbucket.org/rajcze/openqa_fedora_tools)).
6. Create needles (images) by using interactive mode and needles editor in WebUI.
7. Add new Job template and Test suite into `templates` file (and `templates-updates`, if the test is applicable to the update testing workflow)
8. Add new Test suite and Test case into [`conf_test_suites.py`](https://pagure.io/fedora-qa/fedora_openqa/blob/master/f/fedora_openqa/conf_test_suites.py) file in fedora_openqa repository.
9. Open differential request via phabricator, set os-autoinst-disti-fedora as a project and repository.
10. Open differential request for fedora_openqa change if necessary.
10. Mark your test in [phabricator page](https://phab.qa.fedoraproject.org/w/openqa/tests/) as done.
### Language handling
Tests can run in different languages. To set the language which will be used for a test, set the `LANGUAGE` variable for the test suite. The results of this will be:
1. The value set will be typed into the language search box in anaconda.
2. Any needle with at least one tag that starts with `LANGUAGE` will be unregistered unless it has the tag `LANGUAGE-(LANGUAGE)` (where `(LANGUAGE)` is the value set, forced to upper-case).
3. As a consequence, the chosen language will be selected at the anaconda Welcome screen.
It is very important, therefore, that needles have the correct tags. Any needle which is expected to match for tests run in *any* language must have no `LANGUAGE` tags. Other needles must have the appropriate tag(s) for the languages they are expected to match. The safest option if you are unsure is to set no `LANGUAGE` tag(s). The only danger of this is that missing translations may not be caught.
Note that tags of the form `ENV-INSTLANG-(anything)` are useless artefacts and should be removed.
Reference in New Issue View Git Blame Copy Permalink