Add fifloader tests, template schemas, update README
This adds a test suite for fifloader (renamed fifloader.py for
test sanity). It adds JSON Schema form schemas for both FIF and
upstream openQA template data, and has fifloader (optionally,
but by default) validate both input and output data against the
schemas. It also adds a tox.ini configured to run the fifloader
tests, use fifloader to validate the template files, and do diff
coverage and lint checks. It also adjusts the Zuul config to run
tox instead of the test job.
There are also some pylint cleanups, since the new tests run
pylint.
fifcheck, fifconverter and tojson.pm are removed, as they were
mainly only needed for one-time conversion of the old format
templates; now they are in the git history we can always recover
them if we need them.
Along with all this I updated the README a bit to explain some
of it (and explain FIF better), and to explicitly state that this
repo is GPLv2+ licensed, and added GPL headers to some of the
files.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-25 15:35:26 +00:00
|
|
|
[tox]
|
|
|
|
|
skipsdist = True
|
Make the update non-matching package check smarter
With Rawhide updates, we quite often run into a situation where
a test runs after a *later* version of the package has already
gone stable. This even happens for stable releases too, though
less often. The current shell-based check just always fails on
this case, but it's usually OK, and manually marking every case
like this with an "it's OK!" comment gets tiring. Instead, let's
use a smarter Python script to do the check. We compare the EVR
of all installed update packages with the EVR of the package
from the update. If it's the same, fine. If the installed package
is lower-versioned, that's always an error, and we fail. If the
installed package is higher-versioned, we check whether the
update already went stable. If it did, then we soft fail, because
probably nothing can go wrong at this point (this is the usual
Rawhide case). If the update did not yet go stable, we still
hard fail, because something can go wrong in this case: if the
update *now* goes stable, the older version from the update may
be tagged over the newer version the test got (presumably from
current stable).
If anything goes wrong with the Bodhi check, or the test is
running on a task not an advisory, we treat both cases as fatal.
The script also gives easier-to-understand output than the old
approach, which should be a bonus.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2023-01-31 01:37:06 +00:00
|
|
|
envlist = py37,py38,py39,py310,py311,coverage-report
|
Add fifloader tests, template schemas, update README
This adds a test suite for fifloader (renamed fifloader.py for
test sanity). It adds JSON Schema form schemas for both FIF and
upstream openQA template data, and has fifloader (optionally,
but by default) validate both input and output data against the
schemas. It also adds a tox.ini configured to run the fifloader
tests, use fifloader to validate the template files, and do diff
coverage and lint checks. It also adjusts the Zuul config to run
tox instead of the test job.
There are also some pylint cleanups, since the new tests run
pylint.
fifcheck, fifconverter and tojson.pm are removed, as they were
mainly only needed for one-time conversion of the old format
templates; now they are in the git history we can always recover
them if we need them.
Along with all this I updated the README a bit to explain some
of it (and explain FIF better), and to explicitly state that this
repo is GPLv2+ licensed, and added GPL headers to some of the
files.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-25 15:35:26 +00:00
|
|
|
skip_missing_interpreters = true
|
|
|
|
|
[testenv]
|
|
|
|
|
deps =
|
|
|
|
|
pytest
|
|
|
|
|
jsonschema
|
|
|
|
|
coverage
|
|
|
|
|
pytest-cov
|
|
|
|
|
commands=
|
2021-11-18 23:15:36 +00:00
|
|
|
python ./fifloader.py --clean templates.fif.json templates-updates.fif.json
|
|
|
|
|
python ./check-needles.py
|
Add fifloader tests, template schemas, update README
This adds a test suite for fifloader (renamed fifloader.py for
test sanity). It adds JSON Schema form schemas for both FIF and
upstream openQA template data, and has fifloader (optionally,
but by default) validate both input and output data against the
schemas. It also adds a tox.ini configured to run the fifloader
tests, use fifloader to validate the template files, and do diff
coverage and lint checks. It also adjusts the Zuul config to run
tox instead of the test job.
There are also some pylint cleanups, since the new tests run
pylint.
fifcheck, fifconverter and tojson.pm are removed, as they were
mainly only needed for one-time conversion of the old format
templates; now they are in the git history we can always recover
them if we need them.
Along with all this I updated the README a bit to explain some
of it (and explain FIF better), and to explicitly state that this
repo is GPLv2+ licensed, and added GPL headers to some of the
files.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-25 15:35:26 +00:00
|
|
|
py.test unittests/
|
|
|
|
|
py.test --cov-report term-missing --cov-report xml --cov fifloader unittests/
|
Make the update non-matching package check smarter
With Rawhide updates, we quite often run into a situation where
a test runs after a *later* version of the package has already
gone stable. This even happens for stable releases too, though
less often. The current shell-based check just always fails on
this case, but it's usually OK, and manually marking every case
like this with an "it's OK!" comment gets tiring. Instead, let's
use a smarter Python script to do the check. We compare the EVR
of all installed update packages with the EVR of the package
from the update. If it's the same, fine. If the installed package
is lower-versioned, that's always an error, and we fail. If the
installed package is higher-versioned, we check whether the
update already went stable. If it did, then we soft fail, because
probably nothing can go wrong at this point (this is the usual
Rawhide case). If the update did not yet go stable, we still
hard fail, because something can go wrong in this case: if the
update *now* goes stable, the older version from the update may
be tagged over the newer version the test got (presumably from
current stable).
If anything goes wrong with the Bodhi check, or the test is
running on a task not an advisory, we treat both cases as fatal.
The script also gives easier-to-understand output than the old
approach, which should be a bonus.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2023-01-31 01:37:06 +00:00
|
|
|
setenv =
|
|
|
|
|
PYTHONPATH = {toxinidir}
|
|
|
|
|
|
|
|
|
|
[testenv:coverage-report]
|
|
|
|
|
skip_install = true
|
|
|
|
|
deps =
|
|
|
|
|
diff-cover
|
|
|
|
|
pylint
|
|
|
|
|
commands =
|
2022-07-12 23:15:32 +00:00
|
|
|
diff-cover coverage.xml --fail-under=90 --compare-branch=origin/main
|
|
|
|
|
diff-quality --violations=pylint --fail-under=90 --compare-branch=origin/main
|
Add fifloader tests, template schemas, update README
This adds a test suite for fifloader (renamed fifloader.py for
test sanity). It adds JSON Schema form schemas for both FIF and
upstream openQA template data, and has fifloader (optionally,
but by default) validate both input and output data against the
schemas. It also adds a tox.ini configured to run the fifloader
tests, use fifloader to validate the template files, and do diff
coverage and lint checks. It also adjusts the Zuul config to run
tox instead of the test job.
There are also some pylint cleanups, since the new tests run
pylint.
fifcheck, fifconverter and tojson.pm are removed, as they were
mainly only needed for one-time conversion of the old format
templates; now they are in the git history we can always recover
them if we need them.
Along with all this I updated the README a bit to explain some
of it (and explain FIF better), and to explicitly state that this
repo is GPLv2+ licensed, and added GPL headers to some of the
files.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2020-01-25 15:35:26 +00:00
|
|
|
setenv =
|
|
|
|
|
PYTHONPATH = {toxinidir}
|