Summary:
this is following a couple of upstream changes I noticeed while
playing with dump_templates. Upstream has completely got rid of
the 'variables' keys from all dicts, and when you run dump_
templates the output does not contain any 'variables' keys, so
it seems reasonable to ditch these entirely (I think it was some
old thing that got subsumed into 'settings'). Also, when you
run dump_templates the 'prio' values come out in JobTemplates,
not in TestSuites; if you look at load_templates there's a bit
marked "we have to migrate the prio from the TestSuite to the
JobTemplate" which does exactly what it sounds like, so it
seems a good idea to move these instead of relying on that to
always be there.
While I had to re-do all the priorities anyway I tried to clean
them up a bit. The idea is roughly this:
10 - 19: most-critical sanity tests (i.e. default_install)
20 - 29: Alpha tests
30 - 39: Beta tests
40 - 49: Final tests
50+ : Optional tests
within each group I ordered 64-bit first, UEFI second, and
32-bit last (usually just as x0 / x1 / x2).
Test Plan:
Check that the file is valid and loads correctly,
and that everything works more or less the same except the
order of tests run is a bit different.
Reviewers: jskladan, garretraziel
Reviewed By: jskladan, garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D692
add KDE package set testcase - this only adds needles and
new testcase configuration, because with needles cleanup from D670,
it should work without change in code.
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D673
Let's use the same principle with DESKTOP tag as we use with
LANGUAGE tag. Where there are any GNOME-only (or KDE-only) needles, tag
them with `DESKTOP-gnome` and delete then when different `DESKTOP` value
is specified. This will help with `QA:Testcase_Package_Sets_KDE_Package_Install`
testcase - we will be able to use almost the same code and check that KDE
got installed properly.
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D670
damnit, that was still wrong. This should do the trick: we can
do 32-bit F23 upgrade tests now (virt-builder added the base
image we need), so re-add that, and put things in the best
order and fix the priorities.
Noticed some problems with the commit while testing it: we
weren't doing 32-bit minimal (to avoid having too many tests,
I think) so let's not do it for upgrade_2 for now, and also
the arch on one test was wrong.
Summary:
This bumps the existing upgrade tests to F23, and drops the
32-bit ones for now, as there is no 32-bit F23 base image for
virt-builder - RHBZ #1288733. It then adds new tests named
'upgrade_2_(etc)' and associates them with the F22 images. The
intent is that we should always have two sets of upgrade tests,
one for each of the currently-supported stable releases; when
we bump to testing F25, the 'upgrade' tests will be bumped to
F24 and the 'upgrade_2' tests to F23, and so on. There will be
a matching diff for openqa_fedora_tools.
Test Plan:
Execute a test run and make sure all the upgrade
tests run; of course you need to make sure you've built the
required disk images.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D679
As we saw with F23 testing, qemu32 isn't really supported CPU. Also,
we cannot be sure that `std` is really supported graphics. This changes MACHINE
variables to use `host` CPU with 64bit machines (moreover, this is the case in
BOS now). It also deletes 32bit machine and schedules 32bit tests on 64bit
instead. It also changes graphics to `qxl`. Even though we aren't using SPICE, qxl has
better support (and higher priority) and it seems to work OK with VNC. Fixes T637.
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D655
There is a bug (it is fixed in newest unstable version though) in openQA
4.2-1.2 that when you use iSCI HDD device, it fails to boot. Workaround is to
set CDMODEL=scsi-cd (and it won't hurt even when this gets fixed in stable).
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D642
PART_TABLE_TYPE variable says which type of partition table type
should be on attached HDDs.
Some tests with uefi have to use disks with gpt.
Tests are amended to use right disks.
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D623
Summary: simple enough. scheduler should already have the necessary bits.
Test Plan:
Kick off a run and see if we get tests, and the results are
reported to the wiki (Final TC1 should work).
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D607
Summary:
We have these 'atomic installer' images (so far just Cloud),
and maxamillion wanted to get them tested. Turns out it's
pretty trivial - they look much like other installs. Only
little wrinkle is they have a reduced hub (no repository
needles) like live images, but are not like live images in
any other way, so I rejigged the 'small hub needle filtering'
handling a bit.
There will be an accompanying diff for tools, and also some
changes in fedfind (these images are getting built nightly
for *current stable*, and it'd be good to test those).
Because we'd like to test the 22 nightlies, I had to add some
needles for 'olddpi' versions of a few screens. See 2e4c1c2 -
the 22 Atomic installer images still have the old GTK+ code
meaning they run at 96.09dpi. I only retook the necessary
needles for the default-install test, if we add any others we
made need to retake a few more needles.
Test Plan:
Schedule jobs for a compose with the atomic installer
image. You will need the matching openqa_fedora_tools diff and
the very latest git fedfind. Check the test for that image runs,
all other tests run as usual, excessive images are not
downloaded, and the atomic installer is not used for running
universal tests.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D595
Summary:
So test runs are getting very long on BOS, and we have UEFI
tests coming. Try to help out by reducing the 32-bit test load
a bit. I tried to strategically drop the tests that are least
likely to differ, e.g. different storage layouts (but not
filesystem/device types), kickstart delivery, and only doing
one upgrade test.
Test Plan:
Check the templates file loads and there are no
obvious errors. See if you agree with the tests I cut.
Reviewers: garretraziel, jskladan
Reviewed By: jskladan
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D580
Summary:
this handles Non-English European Language Install. Basically
it's a bunch of new screenshots for existing tag names, plus
a bit of configurability in _boot_to_anaconda and tweaking some
existing needles to do non-text matches. The weird 'half-the-
icon' needles are for cases where there may or may not be a
warning triangle but we want to click it either way (saves
duplicating the needle).
This also sets up a convention for tagging what languages a
needle is appropriate for. If it's specifically appropriate for
one or more languages, a tag ENV-LANGUAGE-(LANGUAGE) should be
applied for each language, where (LANGUAGE) is the install
language in upper-case ('LANGUAGE' variable, which should also
be the string that will be typed into the language selection
screen). If the needle ought to be used for *all* languages -
i.e. it's not a text match, or any text in the match is known
not to be translated - the tag ENV-INSTLANG-ALL should be
applied.
To back this, main.pm now unregisters all needles that are not
tagged with either ENV-LANGUAGE-ALL or the tag for the language
actually being used (if the LANGUAGE var is not set, we assume
english). The point of this is to check the install is actually
translated; if we allow all needles to match, the test would
pass even if no translations appeared at all.
Test Plan:
Run all tests and make sure you get the expected
results. You can schedule a run against 23 Beta TC1 to see the
French test fails 'correctly' when translations are missing.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D577
Summary:
This is a first cut which more or less works for now. Issues:
1) We're not really testing the BUILD, here. All the test does
is try and upgrade to the specified VERSION - so it'll be using
the latest 'stable' for the given VERSION at the time the test
runs. This isn't really that terrible, but especially for TC/RC
validation, we might want to make things a bit more elaborate
and set up the repo for the actual BUILD (and disable the main
repos).
2) We'd actually need --nogpgcheck for non-Rawhide, at one
specific point in the release cycle - after Branching but
before Bodhi activation (which is when we can be sure all
packages are signed). This won't matter until 24 branches, and
maybe releng will have it fixed by then...if not, I'll tweak
it.
3) We don't really test that the upgrade actually *happened*
for desktop, at the moment - the only thing in the old test
that really checked that was where we checked for the fedup
boot menu entry, but that has no analog in dnf. What we should
probably do is check that GUI login works, then switch to a
console and check /etc/fedora-release just as the minimal test
does.
Test Plan:
Run the tests. Note that creating the desktop disk
image doesn't work ATM, so I can't verify the desktop test
works, but the minimal one seems to (with D565). There'll be
a matching diff for openqa_fedora_tools to update the test
case names there.
Reviewers: jskladan, garretraziel
Reviewed By: jskladan, garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D567
Summary:
since we did this live at Flock today, I figured I'd tidy it
up and submit it. This is an 'optional' test, but some people
do run this way so it'd be nice to have it. This adds another
little helper method in anacondatest.pm, for deleting partitions,
which works much like the others added in previous commits.
Test Plan: Schedule a test run, see if the test runs and works.
Reviewers: jskladan, garretraziel
Reviewed By: jskladan, garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D503
Summary:
This duplicates basically the entire test suite for 32-bit. We
could choose to run only a few tests for 32-bit if we wanted,
but I figure we may as well do as much testing as we can. Only
a few of the results will actually wind up 'counting' separately
in the wiki, but we can always see the results for all the tests
in openQA itself.
Next we can duplicate the whole set again for UEFI!
Test Plan:
Schedule a full test run and make sure all the tests
run for both arches. Also check result submission works
correctly. Requires the corresponding change to tools (or
you can just use one of the trigger commands which lets you
specify arch with a parameter).
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D500
Summary:
This adds three new custom storage tests and some needles to
support them, and tweaks the custom storage methods a bit to
address some things that cropped up in writing the tests. A
new method is added for changing the filesystem, as that's
a distinct operation from changing the device type.
This also restores the previous behaviour of select_disks()
where it handled selecting custom partitioning when needed.
Turns out it's pretty common to use regex'es in perl! Who'd'a
thought.
A corresponding commit to add the tests to openqa_fedora_tools
is coming.
There's no post-install step for the tests yet; I'll try and
write those up and add them soon.
Test Plan:
Do a full run, including the new tests, on Alpha RC2 and check
all are scheduled correctly and run correctly. The LVM thinp
test is expected to fail as it catches a genuine bug.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D490
Summary:
This contains several tweaks to storage handling. It adds a
method for disk selection which all the storage tests can
share. It sets up a more extensible approach for main.pm to
run the storage tests, instead of an ever-growing forest of
'else' clauses. Finally it sets up a couple of methods for
changing partitioning schemes on the custom part screen and
uses one of them in the software RAID test; the other will
be used for other custom storage tests.
This kills the two_disks needle. I could keep it and work
it into select_disks, but it doesn't fit naturally and I
really just don't see the point of the needle. The only thing
we lose is we don't check that anaconda actually sees two
disks in the 'attach two disks, only install to one' test
(that's server_sata_multi), but the other multi-disk tests
will serve to catch that case failing for some reason.
What I actually intended to do was add some more tests for
different custom part storage types, but it seemed a good
idea to do some of this cleanup so that can be implemented
efficiently. I'll have followups for that.
Test Plan:
Run all tests and ensure they work exactly as
before (not just that they still pass, but that the correct
test steps are actually scheduled in each case.)
Reviewers: garretraziel, jskladan
Reviewed By: garretraziel, jskladan
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D475
The flavor names changed and we started using the * version
wild card in the 'live' merge, so we need to correct the jobs
that were added to master in the meantime.
This requires adding products, flavors and needles and test
cases, and tweaking some existing ones to handle the
slightly different behaviour of live images in shared tests.
To handle the different main hub screens in live and non-live,
a less stringent needle is added which is unregistered for
non-live tests, so they don't match on it before they've
finished updating repository metadata.
There are a few small bugfix tweaks in this too, like some
delays in user creation to try and avoid intermittent failures
there.
A new root_logged_in needle is also included, to handle a new
console font in Rawhide - that has nothing strictly to do with
live testing, it just happened to come up while working on
this.
