Summary:
T617 makes some good points about the language tags; this is my
suggestion for an improvement. It requires a bit of cleverness
in unregister_prefix_tags(), but the upshot is that you don't
need to know to set any special tags when creating needles, a
needle with no language-related tags will be considered as valid
for all languages. You have to explicitly add LANGUAGE- tag(s)
to a needle for the language filtering to 'kick in' in any way.
If a needle has at least one LANGUAGE- tag, it will be filtered
unless it has the appropriate tag for the job's specified
language (default is still 'english').
With this approach, only needles which we specifically want to
*only* match their tagged language(s) need the tags, so we can
drop all those -ALL tags.
We're using LANGUAGE- instead of ENV-LANGUAGE- now because the
ENV- tag names denote tags that are treated slightly specially
by openQA, and this is not one. We cannot cleanly use
ENV-INSTLANG because openQA has a hardwired default of 'en_US'
for that.
Test Plan:
Check both English and French tests still work as
intended.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D589
Summary:
We have these 'atomic installer' images (so far just Cloud),
and maxamillion wanted to get them tested. Turns out it's
pretty trivial - they look much like other installs. Only
little wrinkle is they have a reduced hub (no repository
needles) like live images, but are not like live images in
any other way, so I rejigged the 'small hub needle filtering'
handling a bit.
There will be an accompanying diff for tools, and also some
changes in fedfind (these images are getting built nightly
for *current stable*, and it'd be good to test those).
Because we'd like to test the 22 nightlies, I had to add some
needles for 'olddpi' versions of a few screens. See 2e4c1c2 -
the 22 Atomic installer images still have the old GTK+ code
meaning they run at 96.09dpi. I only retook the necessary
needles for the default-install test, if we add any others we
made need to retake a few more needles.
Test Plan:
Schedule jobs for a compose with the atomic installer
image. You will need the matching openqa_fedora_tools diff and
the very latest git fedfind. Check the test for that image runs,
all other tests run as usual, excessive images are not
downloaded, and the atomic installer is not used for running
universal tests.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D595
Summary:
This removes a couple of unused needles. password_required
needles were dupes of console_password_required, and no tests
actually use password_required. tmp_failed_proc just isn't
used by anything at all (I think it was at some point used in
fedup tests).
This also drops the ENV-OFW-1 tag from all needles, which I
think got in because someone based a needle on a SUSE needle
which has that tag, then it spread. It's of no use or meaning
in Fedora.
Test Plan: Check all tests still run as normal.
Reviewers: garretraziel, jskladan
Reviewed By: jskladan
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D579
Summary:
So test runs are getting very long on BOS, and we have UEFI
tests coming. Try to help out by reducing the 32-bit test load
a bit. I tried to strategically drop the tests that are least
likely to differ, e.g. different storage layouts (but not
filesystem/device types), kickstart delivery, and only doing
one upgrade test.
Test Plan:
Check the templates file loads and there are no
obvious errors. See if you agree with the tests I cut.
Reviewers: garretraziel, jskladan
Reviewed By: jskladan
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D580
Summary:
this handles Non-English European Language Install. Basically
it's a bunch of new screenshots for existing tag names, plus
a bit of configurability in _boot_to_anaconda and tweaking some
existing needles to do non-text matches. The weird 'half-the-
icon' needles are for cases where there may or may not be a
warning triangle but we want to click it either way (saves
duplicating the needle).
This also sets up a convention for tagging what languages a
needle is appropriate for. If it's specifically appropriate for
one or more languages, a tag ENV-LANGUAGE-(LANGUAGE) should be
applied for each language, where (LANGUAGE) is the install
language in upper-case ('LANGUAGE' variable, which should also
be the string that will be typed into the language selection
screen). If the needle ought to be used for *all* languages -
i.e. it's not a text match, or any text in the match is known
not to be translated - the tag ENV-INSTLANG-ALL should be
applied.
To back this, main.pm now unregisters all needles that are not
tagged with either ENV-LANGUAGE-ALL or the tag for the language
actually being used (if the LANGUAGE var is not set, we assume
english). The point of this is to check the install is actually
translated; if we allow all needles to match, the test would
pass even if no translations appeared at all.
Test Plan:
Run all tests and make sure you get the expected
results. You can schedule a run against 23 Beta TC1 to see the
French test fails 'correctly' when translations are missing.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D577
Summary:
This is a first cut which more or less works for now. Issues:
1) We're not really testing the BUILD, here. All the test does
is try and upgrade to the specified VERSION - so it'll be using
the latest 'stable' for the given VERSION at the time the test
runs. This isn't really that terrible, but especially for TC/RC
validation, we might want to make things a bit more elaborate
and set up the repo for the actual BUILD (and disable the main
repos).
2) We'd actually need --nogpgcheck for non-Rawhide, at one
specific point in the release cycle - after Branching but
before Bodhi activation (which is when we can be sure all
packages are signed). This won't matter until 24 branches, and
maybe releng will have it fixed by then...if not, I'll tweak
it.
3) We don't really test that the upgrade actually *happened*
for desktop, at the moment - the only thing in the old test
that really checked that was where we checked for the fedup
boot menu entry, but that has no analog in dnf. What we should
probably do is check that GUI login works, then switch to a
console and check /etc/fedora-release just as the minimal test
does.
Test Plan:
Run the tests. Note that creating the desktop disk
image doesn't work ATM, so I can't verify the desktop test
works, but the minimal one seems to (with D565). There'll be
a matching diff for openqa_fedora_tools to update the test
case names there.
Reviewers: jskladan, garretraziel
Reviewed By: jskladan, garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D567
Summary:
To understand this change see test 1092 on BOS. When you click
Done once and then get the 'click Done again to proceed' warning,
the Done button seems to grows a little dotted line (I think
this indicates it's selected), which causes the match to drop to
94% and fail. To counter that, shrink the match area to the bit
of the button inside the dotted line, which will be the same
whether it's there or not.
Test Plan:
See if no_swap stops failing sometimes now, maybe,
at last? Yeesh. Also see if all other tests still run OK.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D549
I've seen no_swap fail several times with the 'Click for details
or press Done again to continue.' message showing (e.g. job
967 on BOS). I think this is just because sometimes we try and
click Done too fast, so introduce a 1-second sleep between Done
clicks to try and solve that.
Summary:
since we did this live at Flock today, I figured I'd tidy it
up and submit it. This is an 'optional' test, but some people
do run this way so it'd be nice to have it. This adds another
little helper method in anacondatest.pm, for deleting partitions,
which works much like the others added in previous commits.
Test Plan: Schedule a test run, see if the test runs and works.
Reviewers: jskladan, garretraziel
Reviewed By: jskladan, garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D503
Summary:
This duplicates basically the entire test suite for 32-bit. We
could choose to run only a few tests for 32-bit if we wanted,
but I figure we may as well do as much testing as we can. Only
a few of the results will actually wind up 'counting' separately
in the wiki, but we can always see the results for all the tests
in openQA itself.
Next we can duplicate the whole set again for UEFI!
Test Plan:
Schedule a full test run and make sure all the tests
run for both arches. Also check result submission works
correctly. Requires the corresponding change to tools (or
you can just use one of the trigger commands which lets you
specify arch with a parameter).
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D500
Summary:
This adds three new custom storage tests and some needles to
support them, and tweaks the custom storage methods a bit to
address some things that cropped up in writing the tests. A
new method is added for changing the filesystem, as that's
a distinct operation from changing the device type.
This also restores the previous behaviour of select_disks()
where it handled selecting custom partitioning when needed.
Turns out it's pretty common to use regex'es in perl! Who'd'a
thought.
A corresponding commit to add the tests to openqa_fedora_tools
is coming.
There's no post-install step for the tests yet; I'll try and
write those up and add them soon.
Test Plan:
Do a full run, including the new tests, on Alpha RC2 and check
all are scheduled correctly and run correctly. The LVM thinp
test is expected to fail as it catches a genuine bug.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D490
Summary:
I discovered another fun font issue today. Current anaconda
images don't use the intended 'default' console font, eurlatgr.
Neither do live images, but installed systems *do*.
The font they use is the system BIOS font, which in openQA
cases means the qemu firmware font. The easiest way to spot the
difference is the @ character; the shorter version is from the
system BIOS, the slightly taller one is what it looks like in
eurlatgr and latarcyrheb-sun16 (the old default).
In a test image I built, for some reason, I *did* get eurlatgr
in the tmux console, and that broke some needle matches. After
figuring all this out, bcl has sent a lorax patch to use
eurlatgr in the installer, so it makes sense to add these fixes
to the repo for when that kicks in.
We shrink the match on root_logged_in.json by one line. This
screenshot is taken from a post-install case where the prompt
appears in the middle of the screen, and has three black rows
above the prompt; in anaconda, when the prompt appears right at
the top of the screen, there's only *two* rows of black above
it, so the match fails. This fixes that. It's been working so
far because installs have been matching root_logged_in_
rawhide20150311, which is taken with the firmware font, but
once the installer starts using eurlatgr, that won't match any
more.
We also add a new needle for the anaconda_install_source_check
_repo_added tag, taken with eurlatgr. The existing screenshot
was taken either with the firmware font or with latarcyrheb.
They both use a curly glyph for a single quote ('), while
eurlatgr uses a straight line.
This also renames the root_logged_in variant needle to be
clearer about why it's there. We'll probably need variants of
some needles until we're sure lives, anaconda env, and installed
systems are all using eurlatgr. RHBZ #1250262 is a bug I filed
for the live images not using eurlatgr.
Test Plan:
Run the tests with both BIOS font and eurlatgr as
the anaconda font and make sure they all work. The latter
might be a bit tricky till the change lands upstream, I've no
idea how it worked out that way in my test boot.iso.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D483
Summary:
I actually have no idea why I created this, now. Examining it
closely there is no difference in the matched area between it
and user_logged_in. Both are evidently using the eurlatgr font
(tall @ sign). I tested that the kickstart_user_creation test
still works on F23 Alpha TC2 with this needle gone, and the
kickstart tests are the only ones that use it.
Test Plan: Run the kickstart tests and verify they still work.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D484
Summary:
This contains several tweaks to storage handling. It adds a
method for disk selection which all the storage tests can
share. It sets up a more extensible approach for main.pm to
run the storage tests, instead of an ever-growing forest of
'else' clauses. Finally it sets up a couple of methods for
changing partitioning schemes on the custom part screen and
uses one of them in the software RAID test; the other will
be used for other custom storage tests.
This kills the two_disks needle. I could keep it and work
it into select_disks, but it doesn't fit naturally and I
really just don't see the point of the needle. The only thing
we lose is we don't check that anaconda actually sees two
disks in the 'attach two disks, only install to one' test
(that's server_sata_multi), but the other multi-disk tests
will serve to catch that case failing for some reason.
What I actually intended to do was add some more tests for
different custom part storage types, but it seemed a good
idea to do some of this cleanup so that can be implemented
efficiently. I'll have followups for that.
Test Plan:
Run all tests and ensure they work exactly as
before (not just that they still pass, but that the correct
test steps are actually scheduled in each case.)
Reviewers: garretraziel, jskladan
Reviewed By: garretraziel, jskladan
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D475
Summary:
Missed this one in D456. It causes the software RAID test to
fail.
Test Plan: Run the software RAID test on Alpha TC2.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D476
Summary:
I was having a weird failure that I finally figured out: when
software source config runs then storage config, the 'begin_
installation' needle can match while the 'slide down from the
top' animation is still playing, and by the time os-autoinst
positions the cursor to click where the button was when the
match happened, it's moved down and we wind up clicking outside
the button area. So, wait a sec before clicking to avoid this.
Test Plan: Run the server_sata_multi test.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D464
Summary:
Depending on the disks attached to the system and some other
idiosyncracries (like what the disk 'description' currently
happens to be), the distance between the two disk icons can
vary, so a needle that matches on the two disk icons is fragile
(whenever the distance differs, the match fails). So instead of
that, let's have a needle that matches on a single icon with
a lot of grey space to its left, plus the two little matches
we later use in the select_disk(1/2) needles. This should be
robust enough and doesn't need us to keep duplicating it when
the distance between the icons changes. With this tweak, the
'2' variant is no longer needed.
Test Plan:
Run the various tests that use two disks and check this
needle always matches. You'll also want to have D456 applied.
Reviewers: garretraziel, jskladan
Reviewed By: garretraziel, jskladan
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D463
Summary:
Root console in anaconda got broken by RHBZ #1222413 - no
shell on tty2. Decided to clean up console use in general as
part of fixing it.
This creates a class 'fedorabase' and has 'anacondalog' and
'fedoralog' both inherit from it. boot_to_login_screen is
moved there (as it seems appropriate) and it has a new
method, console_login, which basically handles 'get me a
shell on a console': if we're already at one it returns,
if not it'll type the user name and the password *if
necessary* (sometimes it's not) and return once it sees a
prompt. It takes a hash of named parameters for user,
password and 'check', which is whether it should die if it
fails to reach a console or not (some users don't want it
to).
anacondalog and fedoralog both get 'root_console' methods
which do something appropriate and then call
console_login; both have a hash of named parameters,
anacondalog's version only bothers with 'check', while
fedoralog's also accepts 'tty' to pick the tty to use.
This also adjusts all things which try to get to a console
prompt to use either root_console or console_login as
appropriate.
It also tweaks the needle tags a bit, drops some unneeded
needles, and adds a new 'user console prompt' needle; we
really just need two versions of the root prompt needle
and two of the user prompt needle (one for <F23, one for
F23+ - the console font changed in F23, and the @ character
at least doesn't match between the two). I think we still
need the <F23 case for upgrade tests, for now.
Test Plan:
Do a full test run and see that more tests
succeed. I've done a run on happyassassin with a hack to
workaround the SELinux issue for interactive installs,
and the results look good. I also fiddled about a bit to
test some different cases, like forcing a failure in a
live test to test post_fail_hook (and hence root_console)
in that scenario, and forcing failures after some console
commands had been run to check that it DTRT when we've
already reached a console, etc.
Reviewers: jskladan, garretraziel
Reviewed By: jskladan, garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D462
Summary:
Most of the anaconda screenshots were taken with older netinst
images, at 96.09dpi. Workstation lives use 96dpi exactly. This
led to some needles failing on the Workstation live as fonts
render slightly differently at 96 and 96.09dpi. LONG STORY
SHORT, we got a change in GTK+ which means traditional install
images now use 96dpi exactly too; so we should be consistent
between lives and trad images now, but we need to retake all
the screenshots taken at 96.09dpi. That's what this is.
I also cleaned up a couple of other things. The anaconda_error
needle was effectively a duplicate of anaconda_report_btn.
Neither is actually used atm, but I kept report_btn in case we
use it in future.
anaconda_main_hub_installdest_selected isn't needed any more,
since my tweaks to the matching in anaconda_main_hub_nonlive;
it will now match the case this needle was created for initially.
anaconda_software_select_box is no longer used by any test. The
software selection tests were changed to use key sequences and
dropped their use of this needle, but the needle wasn't removed.
anaconda_select_install_lang_english_filtered_english_selected_
as_default was never really necessary, the situation it's meant
for can be handled by simply adding the anaconda_select_install_
lang_english_filtered tag to the anaconda_select_install_lang_
english_selected needle (as this commit does). That achieves
the needed effect and doesn't break the case where the UI is in
Czech.
Most of the match area changes are just because the layout
changed a bit since the screenshot was taken. I did reduce some
over-precision in some needles, notably the multi-disk needles
- there is no reason we need to match on the disk descriptions
and sizes when we're just selecting the appropriate disks. We
aren't testing anaconda's description of the disks (which may
change unexpectedly), only that it *works* with the disk
types used in the test.
After this commit a little under half of the tests pass with
the 23 Branched 20150717 nightly. Most of the ones that fail
fail because of the anaconda tty2 bug (there's no console on
tty2) or the selinux bug (#1243168) that prevents console login
post-install with SELinux enabled. There are a few other fails
too, but none of them seems to be caused by a needle mismatch,
so I'll investigate and fix them separately.
Test Plan:
Run the full test suite on a current nightly, see that some
tests actually pass now!
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D456
Summary:
There's no need to include the radio button state here, which
causes the match to fail if Minimal is highlighted and already
selected. Hitting Space in this case won't de-select it (it's
a radio button, not a checkbox), and will give the desired
result. So we should accept it.
Test Plan:
Run tests against Rawhide (and maybe F22 as a regression check)
and ensure the package set selection step works, in tests where
it's included.
Reviewers: garretraziel, jskladan
Reviewed By: jskladan
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D385
Summary:
Two needles (at least, there may be more) fail to match properly
with Rawhide (F23) because the top part of some screens now has
branding, while before it was plain grey. On root_password_screen
we can just throw that match away, we don't really need it, the
text 'Root Password:' seems specific enough to be reliable. On
user_creation_screen let's replace the branded area match with a
couple of the text box labels.
This also tightens the 'Done' button match area on both spokes;
the original area is too loose and covers some of the now-branded
area outside the button, so the match fails.
This fixes the default_install test for 2015-06-07 nightly.
Test Plan:
Try a test run against the 2015-06-07 nightly and see if the
default_install test passes. Maybe also check that it still
passes for an F22 build.
Reviewers: garretraziel, jskladan
Reviewed By: jskladan
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D386
The flavor names changed and we started using the * version
wild card in the 'live' merge, so we need to correct the jobs
that were added to master in the meantime.
This requires adding products, flavors and needles and test
cases, and tweaking some existing ones to handle the
slightly different behaviour of live images in shared tests.
To handle the different main hub screens in live and non-live,
a less stringent needle is added which is unregistered for
non-live tests, so they don't match on it before they've
finished updating repository metadata.
There are a few small bugfix tweaks in this too, like some
delays in user creation to try and avoid intermittent failures
there.
A new root_logged_in needle is also included, to handle a new
console font in Rawhide - that has nothing strictly to do with
live testing, it just happened to come up while working on
this.
