srbala/os-autoinst-distri-fedora
1
0
Fork 0
mirror of https://pagure.io/fedora-qa/os-autoinst-distri-fedora.git synced 2024-11-21 21:43:08 +00:00
Code Issues Releases Activity
0b5aabcc57
os-autoinst-distri-fedora/README.md

107 lines
10 KiB
Markdown
Raw Normal View History

Add fifloader tests, template schemas, update README This adds a test suite for fifloader (renamed fifloader.py for test sanity). It adds JSON Schema form schemas for both FIF and upstream openQA template data, and has fifloader (optionally, but by default) validate both input and output data against the schemas. It also adds a tox.ini configured to run the fifloader tests, use fifloader to validate the template files, and do diff coverage and lint checks. It also adjusts the Zuul config to run tox instead of the test job. There are also some pylint cleanups, since the new tests run pylint. fifcheck, fifconverter and tojson.pm are removed, as they were mainly only needed for one-time conversion of the old format templates; now they are in the git history we can always recover them if we need them. Along with all this I updated the README a bit to explain some of it (and explain FIF better), and to explicitly state that this repo is GPLv2+ licensed, and added GPL headers to some of the files. Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-25 15:35:26 +00:00
# openQA tests for the Fedora distribution
add comments and documentation Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D481
2015-08-05 06:23:59 +00:00
Yet more README updates
2017-02-15 00:12:17 +00:00
This repository contains tests and images for testing [Fedora](https://getfedora.org/) with [openQA](http://os-autoinst.github.io/openQA/). The [fedora_openqa library and CLI](https://pagure.io/fedora-qa/fedora_openqa) are used for scheduling tests, and [createhdds](https://pagure.io/fedora-qa/createhdds) is used for creating base disk images for the test. For openQA installation instructions, see [the Fedora openQA wiki page](https://fedoraproject.org/wiki/OpenQA).
add comments and documentation Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D481
2015-08-05 06:23:59 +00:00
Add fifloader tests, template schemas, update README This adds a test suite for fifloader (renamed fifloader.py for test sanity). It adds JSON Schema form schemas for both FIF and upstream openQA template data, and has fifloader (optionally, but by default) validate both input and output data against the schemas. It also adds a tox.ini configured to run the fifloader tests, use fifloader to validate the template files, and do diff coverage and lint checks. It also adjusts the Zuul config to run tox instead of the test job. There are also some pylint cleanups, since the new tests run pylint. fifcheck, fifconverter and tojson.pm are removed, as they were mainly only needed for one-time conversion of the old format templates; now they are in the git history we can always recover them if we need them. Along with all this I updated the README a bit to explain some of it (and explain FIF better), and to explicitly state that this repo is GPLv2+ licensed, and added GPL headers to some of the files. Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-25 15:35:26 +00:00
## Issues
Go back to documenting PRs via Phab for now Moving to Pagure for PRs and issues will require figuring out a way to migrate existing ones over, so let's just stick with Phab for now.
2017-01-12 22:53:18 +00:00
One more Phab -> Pagure change (dangit)
2017-09-04 18:46:51 +00:00
[Issues](https://pagure.io/fedora-qa/os-autoinst-distri-fedora/issues) and [pull requests](https://pagure.io/fedora-qa/os-autoinst-distri-fedora/pull-requests) are tracked in [os-autoinst-distri-fedora Pagure](https://pagure.io/fedora-qa/os-autoinst-distri-fedora). Pagure uses a Github-like pull request workflow, so if you're familiar with that, you can easily submit Pagure pull requests. If not, you can read up in the [Pagure documentation](https://docs.pagure.org/pagure/usage/index.html).
More Phab -> Pagure updates
2017-09-04 18:46:15 +00:00
Add fifloader tests, template schemas, update README This adds a test suite for fifloader (renamed fifloader.py for test sanity). It adds JSON Schema form schemas for both FIF and upstream openQA template data, and has fifloader (optionally, but by default) validate both input and output data against the schemas. It also adds a tox.ini configured to run the fifloader tests, use fifloader to validate the template files, and do diff coverage and lint checks. It also adjusts the Zuul config to run tox instead of the test job. There are also some pylint cleanups, since the new tests run pylint. fifcheck, fifconverter and tojson.pm are removed, as they were mainly only needed for one-time conversion of the old format templates; now they are in the git history we can always recover them if we need them. Along with all this I updated the README a bit to explain some of it (and explain FIF better), and to explicitly state that this repo is GPLv2+ licensed, and added GPL headers to some of the files. Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-25 15:35:26 +00:00
## Requirements
Note new test perl library requirements in README Signed-off-by: Adam Williamson <awilliam@redhat.com>
2021-06-18 16:48:24 +00:00
Obviously, this repository is little use without access to an openQA installation. Also, the tests themselves require the perl libraries JSON and REST::Client to be installed on the worker host; in Fedora, the package names are `perl-JSON` and `perl-REST-Client`. To load templates from this repository, you will need the upstream client tools (packaged as `openqa-client` in Fedora) and the dependencies of `fifloader.py` (see below for more on this tool) installed. Those dependencies are Python 3 and the `jsonschema` library. For running the unit tests, you will additionally need `pytest` and `tox`.
Add fifloader tests, template schemas, update README This adds a test suite for fifloader (renamed fifloader.py for test sanity). It adds JSON Schema form schemas for both FIF and upstream openQA template data, and has fifloader (optionally, but by default) validate both input and output data against the schemas. It also adds a tox.ini configured to run the fifloader tests, use fifloader to validate the template files, and do diff coverage and lint checks. It also adjusts the Zuul config to run tox instead of the test job. There are also some pylint cleanups, since the new tests run pylint. fifcheck, fifconverter and tojson.pm are removed, as they were mainly only needed for one-time conversion of the old format templates; now they are in the git history we can always recover them if we need them. Along with all this I updated the README a bit to explain some of it (and explain FIF better), and to explicitly state that this repo is GPLv2+ licensed, and added GPL headers to some of the files. Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-25 15:35:26 +00:00
## Test development
Go back to documenting PRs via Phab for now Moving to Pagure for PRs and issues will require figuring out a way to migrate existing ones over, so let's just stick with Phab for now.
2017-01-12 22:53:18 +00:00
Update and unwrap README, move function docs in-line The README looks pretty ugly on Pagure. So let's unwrap it. Let's also move the function docs into the source files. We're much more likely to keep them up to date that way, I think. We should probably change over to proper perl POD documentation at some point, but comments in-line are OK for now I think.
2017-01-12 22:27:42 +00:00
See official documentation on:
* [basic concept](https://github.com/os-autoinst/openQA/blob/master/docs/GettingStarted.asciidoc)
* [test development (including API specification)](https://github.com/os-autoinst/openQA/blob/master/docs/WritingTests.asciidoc)
* [needles specification](https://github.com/os-autoinst/os-autoinst/blob/master/doc/needles.txt)
* [supported variables for backend](https://github.com/os-autoinst/os-autoinst/blob/master/doc/backend_vars.asciidoc).
See [this example repo](https://github.com/os-autoinst/os-autoinst-distri-example) on how tests should be structured.
add comments and documentation Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D481
2015-08-05 06:23:59 +00:00
Add fifloader tests, template schemas, update README This adds a test suite for fifloader (renamed fifloader.py for test sanity). It adds JSON Schema form schemas for both FIF and upstream openQA template data, and has fifloader (optionally, but by default) validate both input and output data against the schemas. It also adds a tox.ini configured to run the fifloader tests, use fifloader to validate the template files, and do diff coverage and lint checks. It also adjusts the Zuul config to run tox instead of the test job. There are also some pylint cleanups, since the new tests run pylint. fifcheck, fifconverter and tojson.pm are removed, as they were mainly only needed for one-time conversion of the old format templates; now they are in the git history we can always recover them if we need them. Along with all this I updated the README a bit to explain some of it (and explain FIF better), and to explicitly state that this repo is GPLv2+ licensed, and added GPL headers to some of the files. Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-25 15:35:26 +00:00
### FIF template format
The test templates in this repository (files ending in `fif.json`) are not in the same format as expected by and are not directly compatible with the upstream template loader. They are in a format referred to as 'FIF' ('Fedora Intermediate Format') which is parsed into the upstream format by the `fifloader.py` utility found in this repository. This format is intended to be more convenient for human reading and editing. It is more fully explained in the docstring at the top of `fifloader.py`. Please refer to this when adding new tests to the templates. A command like `./fifloader.py --load templates.fif.json templates-updates.fif.json` can be used to load templates in the FIF format (this converts them to the upstream format, and calls the upstream template loader on the converted data). See `./fifloader.py -h` for further details on `fifloader.py`.
mention main.pm architecture in README Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D495
2015-08-11 11:30:42 +00:00
### main.pm modular architecture
Go back to documenting PRs via Phab for now Moving to Pagure for PRs and issues will require figuring out a way to migrate existing ones over, so let's just stick with Phab for now.
2017-01-12 22:53:18 +00:00
Update and unwrap README, move function docs in-line The README looks pretty ugly on Pagure. So let's unwrap it. Let's also move the function docs into the source files. We're much more likely to keep them up to date that way, I think. We should probably change over to proper perl POD documentation at some point, but comments in-line are OK for now I think.
2017-01-12 22:27:42 +00:00
Since openQA uses only one entrypoint for all tests (main.pm), we have decided to utilize this feature and make tests modular. It means that basic passing through main.pm (without any variables set) results in most basic installation test executed. Developer can customize it with additional variables (for example by setting `PACKAGE_SET=minimal` to do installation only with minimal package set).
add comments and documentation Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D481
2015-08-05 06:23:59 +00:00
Update and unwrap README, move function docs in-line The README looks pretty ugly on Pagure. So let's unwrap it. Let's also move the function docs into the source files. We're much more likely to keep them up to date that way, I think. We should probably change over to proper perl POD documentation at some point, but comments in-line are OK for now I think.
2017-01-12 22:27:42 +00:00
Make your test modular, so that it utilizes `_boot_to_anaconda.pm`, `_software_selection.pm` and `_do_install_and_reboot.pm` tests (that are loaded automatically). Break your test into smaller parts, each dealing with one specific feature (e. g. partitioning, user creation...) and add their loading into main.pm based on reasonable variable setting (so they can be used in other tests also).
mention main.pm architecture in README Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D495
2015-08-11 11:30:42 +00:00
Update and unwrap README, move function docs in-line The README looks pretty ugly on Pagure. So let's unwrap it. Let's also move the function docs into the source files. We're much more likely to keep them up to date that way, I think. We should probably change over to proper perl POD documentation at some point, but comments in-line are OK for now I think.
2017-01-12 22:27:42 +00:00
Fedora installation (and consequently main.pm) consists of several parts:
mention main.pm architecture in README Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D495
2015-08-11 11:30:42 +00:00
Update and unwrap README, move function docs in-line The README looks pretty ugly on Pagure. So let's unwrap it. Let's also move the function docs into the source files. We're much more likely to keep them up to date that way, I think. We should probably change over to proper perl POD documentation at some point, but comments in-line are OK for now I think.
2017-01-12 22:27:42 +00:00
#### Booting into Anaconda or booting live image and starting Anaconda
mention main.pm architecture in README Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D495
2015-08-11 11:30:42 +00:00
Update and unwrap README, move function docs in-line The README looks pretty ugly on Pagure. So let's unwrap it. Let's also move the function docs into the source files. We're much more likely to keep them up to date that way, I think. We should probably change over to proper perl POD documentation at some point, but comments in-line are OK for now I think.
2017-01-12 22:27:42 +00:00
Since there isn't much variation between tests in this step, we have developed universal `_boot_to_anaconda.pm` test that is loaded automatically each time except when `ENTRYPOINT` or `UPGRADE` is set (see VARIABLES.md).
mention main.pm architecture in README Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D495
2015-08-11 11:30:42 +00:00
Update and unwrap README, move function docs in-line The README looks pretty ugly on Pagure. So let's unwrap it. Let's also move the function docs into the source files. We're much more likely to keep them up to date that way, I think. We should probably change over to proper perl POD documentation at some point, but comments in-line are OK for now I think.
2017-01-12 22:27:42 +00:00
To customize this step, you can set following variables:
mention main.pm architecture in README Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D495
2015-08-11 11:30:42 +00:00
Update and unwrap README, move function docs in-line The README looks pretty ugly on Pagure. So let's unwrap it. Let's also move the function docs into the source files. We're much more likely to keep them up to date that way, I think. We should probably change over to proper perl POD documentation at some point, but comments in-line are OK for now I think.
2017-01-12 22:27:42 +00:00
- `GRUB` is appended to kernel line before boot. You can set for example `inst.updates` here.
- If `KICKSTART` is set, this part of installation ends here (program doesn't wait for Anaconda to appear). Note that you should set `inst.ks` yourself by setting `GRUB` variable.
- If `LIVE` is set, program waits for desktop to appear and then clicks on "Install to Hard Drive" button.
mention main.pm architecture in README Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D495
2015-08-11 11:30:42 +00:00
Update and unwrap README, move function docs in-line The README looks pretty ugly on Pagure. So let's unwrap it. Let's also move the function docs into the source files. We're much more likely to keep them up to date that way, I think. We should probably change over to proper perl POD documentation at some point, but comments in-line are OK for now I think.
2017-01-12 22:27:42 +00:00
#### Customizing installation by interacting with Anaconda spokes
mention main.pm architecture in README Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D495
2015-08-11 11:30:42 +00:00
Update and unwrap README, move function docs in-line The README looks pretty ugly on Pagure. So let's unwrap it. Let's also move the function docs into the source files. We're much more likely to keep them up to date that way, I think. We should probably change over to proper perl POD documentation at some point, but comments in-line are OK for now I think.
2017-01-12 22:27:42 +00:00
Most of the differences between tests take place in this part. If you want to add another installation test, you will probably put your variable checking and test loading here. All tests in this part should start on Anaconda's main hub and after they done its part, they should go back to Anaconda's main hub so that next test could be executed. In this phase, universal `_software_selection.pm` test is loaded that handles selecting what software to install.
mention main.pm architecture in README Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D495
2015-08-11 11:30:42 +00:00
Update and unwrap README, move function docs in-line The README looks pretty ugly on Pagure. So let's unwrap it. Let's also move the function docs into the source files. We're much more likely to keep them up to date that way, I think. We should probably change over to proper perl POD documentation at some point, but comments in-line are OK for now I think.
2017-01-12 22:27:42 +00:00
To customize this step, you can set following variables:
mention main.pm architecture in README Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D495
2015-08-11 11:30:42 +00:00
Update and unwrap README, move function docs in-line The README looks pretty ugly on Pagure. So let's unwrap it. Let's also move the function docs into the source files. We're much more likely to keep them up to date that way, I think. We should probably change over to proper perl POD documentation at some point, but comments in-line are OK for now I think.
2017-01-12 22:27:42 +00:00
- Set `PACKAGE_SET` to install required package set on "Software selection spoke" - you have to provide correct needles with the name of `anaconda_${PACKAGE_SET}_highlighted` and `anaconda_${PACKAGE_SET}_selected`.
- Set `ENCRYPT_PASSWORD` to encrypt disk, value of this variable is used as an actual password.
mention main.pm architecture in README Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D495
2015-08-11 11:30:42 +00:00
Update and unwrap README, move function docs in-line The README looks pretty ugly on Pagure. So let's unwrap it. Let's also move the function docs into the source files. We're much more likely to keep them up to date that way, I think. We should probably change over to proper perl POD documentation at some point, but comments in-line are OK for now I think.
2017-01-12 22:27:42 +00:00
#### Installing Fedora and waiting for Fedora to reboot
mention main.pm architecture in README Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D495
2015-08-11 11:30:42 +00:00
Update and unwrap README, move function docs in-line The README looks pretty ugly on Pagure. So let's unwrap it. Let's also move the function docs into the source files. We're much more likely to keep them up to date that way, I think. We should probably change over to proper perl POD documentation at some point, but comments in-line are OK for now I think.
2017-01-12 22:27:42 +00:00
After all customizations are finished, `_do_install_and_reboot.pm` test is automatically loaded. It starts installation, creates user and sets root password when required, waits for installation to finish and reboots into installed system. Only variables that control flow in this part are these:
mention main.pm architecture in README Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D495
2015-08-11 11:30:42 +00:00
Update and unwrap README, move function docs in-line The README looks pretty ugly on Pagure. So let's unwrap it. Let's also move the function docs into the source files. We're much more likely to keep them up to date that way, I think. We should probably change over to proper perl POD documentation at some point, but comments in-line are OK for now I think.
2017-01-12 22:27:42 +00:00
- `ROOT_PASSWORD` to set root password to this value.
- When set, `USER_LOGIN` and `USER_PASSWORD` are used to create user in Anaconda.
mention main.pm architecture in README Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D495
2015-08-11 11:30:42 +00:00
Update and unwrap README, move function docs in-line The README looks pretty ugly on Pagure. So let's unwrap it. Let's also move the function docs into the source files. We're much more likely to keep them up to date that way, I think. We should probably change over to proper perl POD documentation at some point, but comments in-line are OK for now I think.
2017-01-12 22:27:42 +00:00
#### Post-install phase
mention main.pm architecture in README Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D495
2015-08-11 11:30:42 +00:00
Update and unwrap README, move function docs in-line The README looks pretty ugly on Pagure. So let's unwrap it. Let's also move the function docs into the source files. We're much more likely to keep them up to date that way, I think. We should probably change over to proper perl POD documentation at some point, but comments in-line are OK for now I think.
2017-01-12 22:27:42 +00:00
After installation is finished and installed system is fully booted, you can run additional tests as checks that installed system has correct attributes - that correct file system is used, that RAID is used etc.
add comments and documentation Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D481
2015-08-05 06:23:59 +00:00
### Test inheritance
Update and unwrap README, move function docs in-line The README looks pretty ugly on Pagure. So let's unwrap it. Let's also move the function docs into the source files. We're much more likely to keep them up to date that way, I think. We should probably change over to proper perl POD documentation at some point, but comments in-line are OK for now I think.
2017-01-12 22:27:42 +00:00
Move all remaining utility functions into exporter modules Summary: This adds a couple of new exporter modules, renames main_common to utils (this is a better name: openSUSE's main_common is functions used in main.pm, utils is what they call their module full of miscellaneous commonly-used functions), and moves a bunch of utility functions that were previously needlessly implemented as instance methods in base classes into the exporter modules. That means we can get rid of all the annoying $self-> syntax for calling them. We get rid of `fedorabase` entirely, as it's no longer useful for anything. Other base classes keep the 'standard' methods (like `post_fail_hook`) and methods which actually need to be methods (like `root_console`, whose behaviour is different in anacondatest and installedtest). Test Plan: Do a full test suite run and check everything lines up. There should be no functional differences from before at all, this is just a re-org. Reviewers: jskladan, garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames Reviewed By: garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames Subscribers: tflink Differential Revision: https://phab.qa.fedoraproject.org/D1080
2017-01-18 07:15:44 +00:00
Your test can inherit from `basetest`, `installedtest` or `anacondatest`. Each provides relevant methods that are documented in-line, so read the files (`lib/anacondatest.pm`, `lib/installedtest.pm`) for information on these.
Update and unwrap README, move function docs in-line The README looks pretty ugly on Pagure. So let's unwrap it. Let's also move the function docs into the source files. We're much more likely to keep them up to date that way, I think. We should probably change over to proper perl POD documentation at some point, but comments in-line are OK for now I think.
2017-01-12 22:27:42 +00:00
- `basetest`: A base class provided by os-autoinst - it has empty `post_fail_hook()` and doesn't set any flags.
- `anacondatest`: should be used in tests where Anaconda is running. It uploads Anaconda logs (for example `anaconda.log` or `packaging.log`) in `post_fail_hook()`.
- `installedtest`: should be used in tests that are running on installed system (either in postinstall phase or in upgrade tests).
Move all remaining utility functions into exporter modules Summary: This adds a couple of new exporter modules, renames main_common to utils (this is a better name: openSUSE's main_common is functions used in main.pm, utils is what they call their module full of miscellaneous commonly-used functions), and moves a bunch of utility functions that were previously needlessly implemented as instance methods in base classes into the exporter modules. That means we can get rid of all the annoying $self-> syntax for calling them. We get rid of `fedorabase` entirely, as it's no longer useful for anything. Other base classes keep the 'standard' methods (like `post_fail_hook`) and methods which actually need to be methods (like `root_console`, whose behaviour is different in anacondatest and installedtest). Test Plan: Do a full test suite run and check everything lines up. There should be no functional differences from before at all, this is just a re-org. Reviewers: jskladan, garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames Reviewed By: garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames Subscribers: tflink Differential Revision: https://phab.qa.fedoraproject.org/D1080
2017-01-18 07:15:44 +00:00
There are also several modules that export utility functions, currently `utils`, `anaconda`, `freeipa`, `packagetest` and `tapnet`. Your test can `use` any of these modules and then directly call the functions they export. Again, the functions are documented in-line.
add comments and documentation Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D481
2015-08-05 06:23:59 +00:00
mention main.pm architecture in README Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D495
2015-08-11 11:30:42 +00:00
### New test development workflow
add comments and documentation Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D481
2015-08-05 06:23:59 +00:00
Update README for retirement of Phabricator (use Pagure now)
2017-09-04 18:39:29 +00:00
1. Put each part of your test as a separate file into `tests/` directory, reimplementing `run()` method
add comments and documentation Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D481
2015-08-05 06:23:59 +00:00
and `test_flags()` method, inheriting from one of the classes mentioned above.
Update README for retirement of Phabricator (use Pagure now)
2017-09-04 18:39:29 +00:00
2. Set correct variables (so that all test parts you have made are executed) in [WebUI -> Test suites](https://localhost:8080/admin/test_suites).
3. Link your newly created Test suite to medium type in [WebUI -> Job groups](https://localhost:8080/admin/groups).
Update two links in README (#179, thanks @defolos) Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-07-10 23:05:00 +00:00
4. Run test (see [fedora_openqa repository](https://pagure.io/fedora-qa/fedora_openqa)).
Update README for retirement of Phabricator (use Pagure now)
2017-09-04 18:39:29 +00:00
5. Create needles (images) by using interactive mode and needles editor in WebUI.
Add a whole intermediate template format ('FIF') and tools I and @lruzicka (and I think @jskladan and @jsedlak and @michelmno and everyone else who's ever touched it...) are being gradually driven nuts by manually editing the test templates. The bigger the files get the more awkward it is to keep them straight and be sure we're doing it right. Upstream doesn't do things the same way we do (they mostly edit in the web UI and dump to file for the record), but we do still think making changes in the repo and posting to the web UI is the right way around to do it, we just wish the format was saner. Upstream has actually recently introduced a YAML-based approach to storing job templates which tries to condense things a bit, and you can dump to that format with dump-templates --json, but @lruzicka and I agree that that format is barely better for hand editing in a text editor than the older one our templates currently use. So, this commit introduces...Fedora Intermediate Format (FIF) - an alternative format for representing job templates - and some tools for working with it. It also contains our existing templates in this new format, and removes the old template files. The format is documented in the docstrings of the tools, but briefly, it keeps Machines, Products and TestSuites but improves their format a bit (by turning dicts-of-lists into dicts-of- dicts), and adds Profiles, which are combinations of Machines and Products. TestSuites can indicate which Profiles they should be run on. The intermediate format converter (`fifconverter`) converts existing template data (in JSON format; use tojson.pm to convert our perl templates to JSON) to the intermediate format and writes it out. As this was really intended only for one-time use (the idea is that after one-time conversion, we will edit the templates in the intermediate format from now on), its operation is hardcoded and relies on specific filenames. The intermediate format loader (`fifloader`) generates JobTemplates from the TestSuites and Profiles, reverses the quality-of-life improvements of the intermediate format, and produces template data compatible with the upstream loader, then can write it to disk and/or call the upstream loader directly. The check script (`fifcheck`) runs existing template data through both the converter and the loader, then checks that the result is equivalent to the input. Again this was mostly written for one- time use so is fairly rough and hard-coded, but I'm including it in the commit so others can check the work and so on. Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-23 14:20:10 +00:00
6. Add new test suite and profiles into `templates.fif.json` file (and/or `templates-updates.fif.json`, if the test is applicable to the update testing workflow)
Update two links in README (#179, thanks @defolos) Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-07-10 23:05:00 +00:00
7. Add new Test suite and Test case into [`conf_test_suites.py`](https://pagure.io/fedora-qa/fedora_openqa/blob/master/f/src/fedora_openqa/conf_test_suites.py) file in fedora_openqa repository.
Add fifloader tests, template schemas, update README This adds a test suite for fifloader (renamed fifloader.py for test sanity). It adds JSON Schema form schemas for both FIF and upstream openQA template data, and has fifloader (optionally, but by default) validate both input and output data against the schemas. It also adds a tox.ini configured to run the fifloader tests, use fifloader to validate the template files, and do diff coverage and lint checks. It also adjusts the Zuul config to run tox instead of the test job. There are also some pylint cleanups, since the new tests run pylint. fifcheck, fifconverter and tojson.pm are removed, as they were mainly only needed for one-time conversion of the old format templates; now they are in the git history we can always recover them if we need them. Along with all this I updated the README a bit to explain some of it (and explain FIF better), and to explicitly state that this repo is GPLv2+ licensed, and added GPL headers to some of the files. Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-25 15:35:26 +00:00
8. Run `tox`. This will check the templates are valid.
9. Open pull request for the os-autoinst-distri-fedora changes in [Pagure](https://pagure.io/fedora-qa/os-autoinst-distri-fedora). Pagure uses a Github-style workflow (summary: fork the project via the web interface, push your changes to a branch on your fork, then use the web interface to submit a pull request). See the [Pagure documentation](https://docs.pagure.org/pagure/usage/index.html) for more details.
10. Open a pull request in [fedora_openqa Pagure](https://pagure.io/fedora-qa/fedora_openqa) for any necessary fedora_openqa changes.
add a french (encrypted) test Summary: this handles Non-English European Language Install. Basically it's a bunch of new screenshots for existing tag names, plus a bit of configurability in _boot_to_anaconda and tweaking some existing needles to do non-text matches. The weird 'half-the- icon' needles are for cases where there may or may not be a warning triangle but we want to click it either way (saves duplicating the needle). This also sets up a convention for tagging what languages a needle is appropriate for. If it's specifically appropriate for one or more languages, a tag ENV-LANGUAGE-(LANGUAGE) should be applied for each language, where (LANGUAGE) is the install language in upper-case ('LANGUAGE' variable, which should also be the string that will be typed into the language selection screen). If the needle ought to be used for *all* languages - i.e. it's not a text match, or any text in the match is known not to be translated - the tag ENV-INSTLANG-ALL should be applied. To back this, main.pm now unregisters all needles that are not tagged with either ENV-LANGUAGE-ALL or the tag for the language actually being used (if the LANGUAGE var is not set, we assume english). The point of this is to check the install is actually translated; if we allow all needles to match, the test would pass even if no translations appeared at all. Test Plan: Run all tests and make sure you get the expected results. You can schedule a run against 23 Beta TC1 to see the French test fails 'correctly' when translations are missing. Reviewers: jskladan, garretraziel Reviewed By: garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D577
2015-09-15 01:08:58 +00:00
### Language handling
Update and unwrap README, move function docs in-line The README looks pretty ugly on Pagure. So let's unwrap it. Let's also move the function docs into the source files. We're much more likely to keep them up to date that way, I think. We should probably change over to proper perl POD documentation at some point, but comments in-line are OK for now I think.
2017-01-12 22:27:42 +00:00
Tests can run in different languages. To set the language which will be used for a test, set the `LANGUAGE` variable for the test suite. The results of this will be:
add a french (encrypted) test Summary: this handles Non-English European Language Install. Basically it's a bunch of new screenshots for existing tag names, plus a bit of configurability in _boot_to_anaconda and tweaking some existing needles to do non-text matches. The weird 'half-the- icon' needles are for cases where there may or may not be a warning triangle but we want to click it either way (saves duplicating the needle). This also sets up a convention for tagging what languages a needle is appropriate for. If it's specifically appropriate for one or more languages, a tag ENV-LANGUAGE-(LANGUAGE) should be applied for each language, where (LANGUAGE) is the install language in upper-case ('LANGUAGE' variable, which should also be the string that will be typed into the language selection screen). If the needle ought to be used for *all* languages - i.e. it's not a text match, or any text in the match is known not to be translated - the tag ENV-INSTLANG-ALL should be applied. To back this, main.pm now unregisters all needles that are not tagged with either ENV-LANGUAGE-ALL or the tag for the language actually being used (if the LANGUAGE var is not set, we assume english). The point of this is to check the install is actually translated; if we allow all needles to match, the test would pass even if no translations appeared at all. Test Plan: Run all tests and make sure you get the expected results. You can schedule a run against 23 Beta TC1 to see the French test fails 'correctly' when translations are missing. Reviewers: jskladan, garretraziel Reviewed By: garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D577
2015-09-15 01:08:58 +00:00
1. The value set will be typed into the language search box in anaconda.
revise language tag handling to be easier to use (T617) Summary: T617 makes some good points about the language tags; this is my suggestion for an improvement. It requires a bit of cleverness in unregister_prefix_tags(), but the upshot is that you don't need to know to set any special tags when creating needles, a needle with no language-related tags will be considered as valid for all languages. You have to explicitly add LANGUAGE- tag(s) to a needle for the language filtering to 'kick in' in any way. If a needle has at least one LANGUAGE- tag, it will be filtered unless it has the appropriate tag for the job's specified language (default is still 'english'). With this approach, only needles which we specifically want to *only* match their tagged language(s) need the tags, so we can drop all those -ALL tags. We're using LANGUAGE- instead of ENV-LANGUAGE- now because the ENV- tag names denote tags that are treated slightly specially by openQA, and this is not one. We cannot cleanly use ENV-INSTLANG because openQA has a hardwired default of 'en_US' for that. Test Plan: Check both English and French tests still work as intended. Reviewers: jskladan, garretraziel Reviewed By: garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D589
2015-09-29 22:52:50 +00:00
2. Any needle with at least one tag that starts with `LANGUAGE` will be unregistered unless it has the tag `LANGUAGE-(LANGUAGE)` (where `(LANGUAGE)` is the value set, forced to upper-case).
add a french (encrypted) test Summary: this handles Non-English European Language Install. Basically it's a bunch of new screenshots for existing tag names, plus a bit of configurability in _boot_to_anaconda and tweaking some existing needles to do non-text matches. The weird 'half-the- icon' needles are for cases where there may or may not be a warning triangle but we want to click it either way (saves duplicating the needle). This also sets up a convention for tagging what languages a needle is appropriate for. If it's specifically appropriate for one or more languages, a tag ENV-LANGUAGE-(LANGUAGE) should be applied for each language, where (LANGUAGE) is the install language in upper-case ('LANGUAGE' variable, which should also be the string that will be typed into the language selection screen). If the needle ought to be used for *all* languages - i.e. it's not a text match, or any text in the match is known not to be translated - the tag ENV-INSTLANG-ALL should be applied. To back this, main.pm now unregisters all needles that are not tagged with either ENV-LANGUAGE-ALL or the tag for the language actually being used (if the LANGUAGE var is not set, we assume english). The point of this is to check the install is actually translated; if we allow all needles to match, the test would pass even if no translations appeared at all. Test Plan: Run all tests and make sure you get the expected results. You can schedule a run against 23 Beta TC1 to see the French test fails 'correctly' when translations are missing. Reviewers: jskladan, garretraziel Reviewed By: garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D577
2015-09-15 01:08:58 +00:00
3. As a consequence, the chosen language will be selected at the anaconda Welcome screen.
Update and unwrap README, move function docs in-line The README looks pretty ugly on Pagure. So let's unwrap it. Let's also move the function docs into the source files. We're much more likely to keep them up to date that way, I think. We should probably change over to proper perl POD documentation at some point, but comments in-line are OK for now I think.
2017-01-12 22:27:42 +00:00
It is very important, therefore, that needles have the correct tags. Any needle which is expected to match for tests run in *any* language must have no `LANGUAGE` tags. Other needles must have the appropriate tag(s) for the languages they are expected to match. The safest option if you are unsure is to set no `LANGUAGE` tag(s). The only danger of this is that missing translations may not be caught.
revise language tag handling to be easier to use (T617) Summary: T617 makes some good points about the language tags; this is my suggestion for an improvement. It requires a bit of cleverness in unregister_prefix_tags(), but the upshot is that you don't need to know to set any special tags when creating needles, a needle with no language-related tags will be considered as valid for all languages. You have to explicitly add LANGUAGE- tag(s) to a needle for the language filtering to 'kick in' in any way. If a needle has at least one LANGUAGE- tag, it will be filtered unless it has the appropriate tag for the job's specified language (default is still 'english'). With this approach, only needles which we specifically want to *only* match their tagged language(s) need the tags, so we can drop all those -ALL tags. We're using LANGUAGE- instead of ENV-LANGUAGE- now because the ENV- tag names denote tags that are treated slightly specially by openQA, and this is not one. We cannot cleanly use ENV-INSTLANG because openQA has a hardwired default of 'en_US' for that. Test Plan: Check both English and French tests still work as intended. Reviewers: jskladan, garretraziel Reviewed By: garretraziel Subscribers: tflink Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D589
2015-09-29 22:52:50 +00:00
Update and unwrap README, move function docs in-line The README looks pretty ugly on Pagure. So let's unwrap it. Let's also move the function docs into the source files. We're much more likely to keep them up to date that way, I think. We should probably change over to proper perl POD documentation at some point, but comments in-line are OK for now I think.
2017-01-12 22:27:42 +00:00
Note that tags of the form `ENV-INSTLANG-(anything)` are useless artefacts and should be removed.
Add fifloader tests, template schemas, update README This adds a test suite for fifloader (renamed fifloader.py for test sanity). It adds JSON Schema form schemas for both FIF and upstream openQA template data, and has fifloader (optionally, but by default) validate both input and output data against the schemas. It also adds a tox.ini configured to run the fifloader tests, use fifloader to validate the template files, and do diff coverage and lint checks. It also adjusts the Zuul config to run tox instead of the test job. There are also some pylint cleanups, since the new tests run pylint. fifcheck, fifconverter and tojson.pm are removed, as they were mainly only needed for one-time conversion of the old format templates; now they are in the git history we can always recover them if we need them. Along with all this I updated the README a bit to explain some of it (and explain FIF better), and to explicitly state that this repo is GPLv2+ licensed, and added GPL headers to some of the files. Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-25 15:35:26 +00:00
## Licensing and credits
The contents of this repository are available under the GPL, version 3 or any later version. A copy is included as COPYING. Note that we do not include the full GPL header in every single test file as they are quite short and this would waste a lot of space.
The tools and tests in this repository are maintained by the [Fedora QA team](https://fedoraproject.org/wiki/QA). We are grateful to the [openSUSE](https://opensuse.org) team for developing openQA, and for the [openSUSE tests](https://github.com/os-autoinst/os-autoinst-distri-opensuse) on which this repository was initially based (and from which occasional pieces are still borrowed).
Reference in New Issue Copy Permalink