We want to get accurate results from the rest of the upgrade
test and we know about this bug, so let's make it a soft fail
so we can see how the test is functioning otherwise.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Investigation suggests that the Firefox certificate validation
failure we sometimes get when loading the FreeIPA web UI in the
freeipa_webui test module is a race issue in Firefox itself:
https://bugzilla.mozilla.org/show_bug.cgi?id=1530429
refreshing the page seems to work around the problem, so let's
do that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The 'largetext' version of the needle is needed when there's
some notification displaying over the window title, it seems
F28's current font rendering doesn't match the current version
of that needle...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We're getting an intermittent case where FreeIPA tests fail
because of a web server certificate issue. Click 'Advanced' in
Firefox when this happens so we can get a bit more info on the
problem.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
ctrl-t seems to be failing often in Firefox tests, and I can't
figure out why. Let's try clicking the 'new tab' button instead
as a workaround.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The Adwaita icon theme got some changes in a git snapshot build
for Rawhide recently. These changed icons break various needles,
we need new ones.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Fedora logo is missing from GDM in today's Rawhide. Not going
all the way and making this a soft fail as the bug is known and
should be fixed soon, I'll take the needle out again then.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We're getting a spurious "Authentication required" for a
background refresh of the system repos sometimes, depending on
whether it happens while we're at a tty. This is now accepted
by upstream, so let's handle it and make it a soft failure for
now.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
New GTK+ changed something so the background of many interface
elements is a slightly lighter grey, this broke a bunch of
needles. Here are the retakes. Includes one not-strictly-related
SDDM update and a rename of the one @lruzicka did to match the
others.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The needle could sometimes fail because the item it was looking
for only appeared in a limited set of situations. The new
location should be safer to check and the test should not
fail as a result.
This adds a test which builds a netinst image potentially with
the package(s) from the update, and uploads that image. It also
adds a test which runs a default install using that image. This
is intended to check whether the update breaks the creation or
use of install images; particularly this will let us test
anaconda etc. updates. We also update the minimal disk image
name, as we have to make it bigger to accommodate this test,
and making it bigger changes its name - the actual change to
the disk image itself is in createhdds. We also have to redo a
bunch of installer needles for F28 fonts, after I removed them
a month or so back...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
If a test fails to the dracut shell, we currently don't do
anything useful. This should recognize when that happens, and
upload rdsosreport.txt.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It's my second least favorite day of the year again: Stale
Needle Cleanup Day!
This should get rid of all non-anaconda needles that definitely
are not being used any more. Cleanup of all anaconda needles was
in the previous commit.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It's my second least favorite day of the year again: Stale
Needle Cleanup Day!
This should get rid of all anaconda needles that definitely are
not being used any more. A few borderline cases (where I'm not
100% sure if they may still be useful in odd corner cases, like
running universal tests on non-server images, and runs outside
of the US, and stuff) are kept around.
Cleanup of all non-anaconda needles will be in the next commit.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This was part of a workaround for a Fedora Atomic Workstation
(old name for Silverblue) issue that got resolved a while back;
we forgot to take the needle out.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It seems Rawhide auto-signing is working fine now: openQA claims
this needle has 'never' been matched (which really means 'not
for a long time'), and I can't find any test fail in the last
year which looks like it landed on this 'authenticate' screen
but the needle failed to match, or anything like that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There's really no point having separate error and error_report
needles. Just match on error_report as well as clicking on it.
Also add a new error_report needle for latest Rawhide fonts.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
GNOME Software 3.30.5 split the offline update process into two
separate 'download' and 'apply' phases. So we need to handle
clicking 'download' before 'apply', if that happens.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Somehow, recently, FreeIPA tests are running into Firefox not
quitting because it's showing a warning about closing multiple
tabs. (I think we didn't *get* multiple tabs before but now we
do, for some reason). So let's work around this by clicking
"Close tabs" if the warning appears.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We're not seeing *exactly* #1314991 any more, but we're seeing
something that looks quite similar: the first attempt to find
updates just doesn't find any. No error message, no updates. I
have reported a bug for this and am investigating it, in the
meantime, let's restore the workaround, elaborated a bit, and
looking for the 'Software is up to date' screen instead of the
error message.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I rather suspect the *bug* is still basically present and it's
why this test often fails, but we no longer seem to see the
*error message* which lets us detect the bug happening. This
needle has not been hit by any test for six months. So let's
remove the workaround as it adds complexity.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
An selinux alert can hide the 'getting started' window title, so
let's have a variant needle which matches on the big 'Started'
in the window instead. We may as well have both for maximum
match potential.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The --MORE-- text looks different with the UEFI firmware font,
so we need another needle. Also I named the original needle
wrong, bad me.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Per Neal Gompa boot will proceed if we just page through the
error(?) messages displayed when #1618928 happens, so let's do
that to let the tests get further and see what else is broken.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
So that we get the crash info uploaded. Note these are not
language-tagged as we *don't* want to 'test' translation of
these, we want the best possible chance of matching any error
dialog in any language...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
anaconda's new hub layout makes all the icons smaller, so all
these needles needed re-doing, even though the same icons are
present...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Seems sometimes tests are showing up with the date/time spoke
in a non-active highlighted state or something, don't really
know why, but it's not broken and we should just accept it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Seems on x86_64 the text cursor is at the left of the text entry
box, but on aarch64 it's at the right. I have no idea why.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Sometimes on aarch64 clicking the partition scheme drop-down
just doesn't seem to make the menu appear, instead the button
goes active but that's all. It's very unlikely we'll be able
to track down why as this doesn't happen in manual testing on
aarch64 (according to @pwhalen), so instead let's just work
around it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
These all are caused by a new desktop background and bits of
KDE desktop chrome apparently becoming translucent.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The background is now blue for ppc64le, so add related needles as bypass.
no such problem with ppc64 (BE)
problem initially detected on Rawhide compose 20180204
but still present on f28 compose 20180302
and Rawhide compose 20180513
Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
https://bugzilla.redhat.com/show_bug.cgi?id=1403365 has been
around approximately forever and I still haven't managed to
debug it; let's just make needles for it, as it's not really a
critical bug, the system still *works*.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
For quite a while on i386 the 'enter passphrase' console screen
has used bright white text, for some reason. Let's just have a
variant needle for this instead of worrying too hard about why.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We've been seeing an odd case lately where the language select
screen is not foregrounded when it appears (so all text is
grey). It happens very occasionally on x86_64, but a lot on
ppc64. To work around this, let's add a needle that matches the
inactive screen, and click on the screen when it appears just
to make sure it's active.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
KDE in F28+ seems to show a network connection notification on
live boot, for some reason. Just dismiss it to help the test
pass.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Seems like the indicator arrow's vertical position changed in
F28, for some reason. Anyway, new needles.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
...I hope. This is necessary as we now have a case where it
needs to match post-install (aarch64 support_server, since
aarch64 is always UEFI).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
In the most recent Rawhide compose, anaconda has changed how it
names volume groups (again), and now the VGs in openQA installs
have rather longer names. This winds up causing a problem for
this needle: the column where the 'Device Type' dropdown is
placed actually gets narrower (because the column to the right,
which has the volume group name in it, is wider), and so the
dropdown box is narrower, and the arrow protrudes into the area
which the needle expects not to have an arrow in it. So, let's
make the match area slightly narrower.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Rawhide has a bit of a problem where its 'description' of an
iSCSI disk is so long that the other columns that should appear
in the CONFIGURE MOUNT POINT dialog don't. This means our
device_sda_selected needle doesn't match, because the column
where 'sda' should appear isn't visible.
So add a soft-fail needle to cover this case; we know what the
description for the disk that's 'sda' in this case looks like,
so match on that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Time for an annual spring clean. Based on the admin UI's list
of needles that haven't been matched for a long time, but with
some manual tweaking (some are actually still needed).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This somehow snuck in (in the wrong directory) with the ppc64
merges, but it's not needed. I've verified with the admin tab
for checking on needle use; this needle has never been matched,
and all the ppc64 text install tests match the non-variant
needle just fine. So I'm removing this unneeded variant.
The way GNOME Software displays an error after running into RHBZ
1314991 has changed, it seems, so we need a variant needle to
cope with that. Also, when an upgrade notification is visible,
the 'Restart & Update' button for doing a regular update is
shown in grey (not blue), so we need a variant needle to handle
that too.
Because "Rescue Mount" now replaced by "Rescue Shell" string
in expected rescue screen head.
Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
The font Firefox uses when we don't ensure dejavu is installed
seems to bounce around a bit, so let's ensure the dejavu fonts
are there before we start Firefox. Also update a needle for
this.
Seems like it doesn't display "Initial setup for Fedora 27" any
more, so the old needle doesn't match. We should look into
whether that's a bug, but for now, let's update the needle.
Matching on the user name really isn't doing much good. It just
means we need more variant needles. Let's ditch that part of the
match and just match on the distinctive character sequence ~]$,
which doesn't really occur for any other reason. With this we
can drop the separate 'qwerty' needle (since the qwerty case
will match the regular needle now) and should also handle the
FreeIPA tests that are failing in Rawhide because a logged in
FreeIPA user doesn't just have a sh prompt now.
We can deal with this annoying bug by looking out for the error
we see when it happens, hitting the 'refresh' button again, and
resetting the loop counter to 1 (requires changing the loop to
a C-style loop).
This is required because anaconda is still checking for it
even if not mandatory. Already tracked by bug
https://bugzilla.redhat.com/show_bug.cgi?id=1172791
Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
related to known bug
https://bugzilla.gnome.org/show_bug.cgi?id=771127
like other needles with same bgo#771127 reference
do not set it as Workaround needle.
Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
* New OFW variable to identify Open Firmware (used by PowerPC)
* Few needles changes for PowerPC support
* as requested do not change the timers value below for PowerPC
tests/install_source_graphical.pm (300 to 600)
tests/_boot_to_anaconda.pm (300 to 1200)
This will be handled by TIMEOUT_SCALE in templates
Signed-off-by: Guy Menanteau <menantea@linux.vnet.ibm.com>
There's a bug in current Rawhide causing sourcing of /etc/bashrc
to fail when logging in as a regular user. This results in the
bash prompt looking different, which is currently a hard fail,
and causes most tests to die. It's better to treat this as a
soft fail so the rest of the test can run. So add a needle to
spot this case, and a little finish function the console login
function calls whenever it's successfully logged in, to check
whether it got the no-profile prompt and register a soft fail.
That other one didn't help, so let's try this - try and spot if
the spoke is in the unexpected state (the needle should only
match if the spoke is done processing and still in warning
state, it shouldn't match while the needle is still thinking)
and click through it again if so.
Well, that OCR needle isn't working out so great, as it seems
to match when it shouldn't:
https://openqa.fedoraproject.org/tests/119217#step/_graphical_wait_login/5
So let's try another approach. Ditch the OCR needle and have a
function for checking we're at a clean desktop. It does the
normal needle match, but if we're on GNOME, it also tries
hitting alt+f1 and seeing if we're at the overview; if so, it
hits alt+f1 again (to go back to the desktop) and returns.
https://openqa.fedoraproject.org/tests/117131 shows a spurious
match of the anaconda warning_bar needle - it's matching in an
ad for LibreOffice. Add a bit of the grey above the bar, so
hopefully this won't happen any more.
The F26 background does not change with the time of day, so that
needle should serve as-is, it's just no use for Rawhide. For
Rawhide, let's try a needle with an OCR match on 'Activities',
plus a tiny match area for a pure white block (expected to be
matched in the user menu area), since os-autoinst requires all
needles to include a match area for some reason.
These are no use :/ The Rawhide default background is a 'changes
with the time of day' one, and with the transparent top bar,
that makes a proper needle match impossible to reliably achieve,
as there is not a single pixel of the 'clean desktop' which
will always be the same any more. We'll have to come up with
something else. I'm trying out the use of an OCR match now.
The GNOME top bar is now transparent in Rawhide, which broke the
'clean desktop' needle. It also means it'll break every time
the background changes, from now on :( Update the needle.
Seems like the top '[patch]' link on kernel.org may now be a
raw patch file which the browser just displays, not a compressed
patch it offers to download. So tweak the needle so we should
click on the *bottom* link instead. openQA looks for matches
as close as possible to the location in the needle.
When running universal tests with a non-Server image, which is
not common but *does* happen (e.g. manually built test image),
the pre-selected filesystem in blivet-gui will be ext4 not xfs
(just as with custom partitioning). So we need xfs and ext4
variants of this needle. Renamed the XFS one for consistency.
Summary:
This adds a new test suite, run for Workstation and KDE live
images, which does not create a user during install. It then
expects initial-setup (KDE) or gnome-initial-setup (Workstation)
to appear after install, creates a user, and proceeds with
normal boot.
Note the ARM image test already covers the initial-setup text
mode, and the ARM minimal image is the only case where that
actually matters (it's not included in Server).
Test Plan:
Run the new tests, check they work. Run all old
tests, check the changes didn't break them.
Reviewers: jsedlak, jskladan
Reviewed By: jsedlak
Subscribers: tflink
Differential Revision: https://phab.qa.fedoraproject.org/D1185
They got rid of the 'Dashboard' text we were matching on, so
let's change this needle. This 'Hardware' text should show up
in all cockpit versions, I think.
Summary:
This is to handle cases like #1414904 , where the system boots
to emergency mode. We really need logs to try and debug this.
Test Plan:
Force a test to hit emergency mode somehow (right now
you can just run base_services_start on Rawhide over and over
until you hit #1414904, but there's probably an easier way to
do it, I think there's a systemd boot arg to tell it which target
to boot for e.g.) and check logs get uploaded. Also check this
doesn't break log upload for a 'normal' failure.
Reviewers: garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames
Reviewed By: garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames
Subscribers: tflink
Differential Revision: https://phab.qa.fedoraproject.org/D1103
openQA has some problems handling needles with identical names
in different subdirectories. We haven't had the cycles to send
fixes for this yet, so for now, let's just rename all such
needles we have.
Not sure if the color change will occur in Fedora 25 nightlies
or not, so not sure if we can remove the old needles. But we
*can* remove the F24 needles now I think.
Summary:
Since 26.17, anaconda shows a warning when the user password
contains non-ASCII characters, and requires a second Done click
to confirm. This change should handle that.
On the 'catch cases where password typing went wrong and re-try'
bit: to keep that, but not re-type the password *every single
time* on the Russian install test, we'd have to make the needle
match the text of the warning. This is problematic because then
that needle will be able to break without us easily noticing;
that's why I wanted to keep the 'warning bar' needle text-free.
Unfortunately, that means we have to skip the protection for
switched-layout installs.
Note the protection was actually not working for any non-English
install anyhow, because the needle had `LANGUAGE-english` as a
tag. We never noticed that. Failed password typing is pretty
rare now, so we can live without the protection - it's just nice
to have it for the English install tests because there's so many
of them.
Test Plan:
Run the Russian install with a recent Rawhide image,
check it clicks 'Done' twice. Note, it will still fail, because
of RHBZ #1413813.
Reviewers: jskladan, garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames
Reviewed By: garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames
Subscribers: tflink
Differential Revision: https://phab.qa.fedoraproject.org/D1084
The F25 'desktopencrypt' images built by createhdds boot to a
console passphrase entry screen, not graphical. F24 images
built the same way boot to a graphical passphrase entry screen.
I'm not sure why, but it's not really worth spending a lot of
energy on, I don't think - let's just add a needle to cover
this case.
Summary:
Include some basic testing of Japanese input, and split the
input testing (including Russian) into a separate module, since
it's not really part of 'login' testing.
Test Plan:
Run the test, and the Russian and French tests too to
make sure they didn't break. Tested on staging. Note the Japanese
test soft fails, intentionally, at present, as I discovered a bug
while working on it:
https://bugzilla.gnome.org/show_bug.cgi?id=776189
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1072
Summary:
This isn't in the criteria, but it's commonly used, so we ought
to test this way. Require authentication for the iSCSI target
and have the test provide the appropriate auth info.
Test Plan:
Run the iscsi test and check it works (you need the
recent fixes for support_server to make *that* work). Nothing
else should be affected.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1070
