We get a 96% match on this because the "A" in "Activities" moved
a bit. Instead of a new needle let's just be OK with the lower
match.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
So it doesn't fail if there's a dotted line around the entry
(indicating keyboard active, I think).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This seems to happen every cycle: when we get to testing an
RC, the layout indicators in anaconda move a bit (the text
moves relative to the little keyboard icon). I think the fact
that the prerelease text is no longer present is what causes
this. Anyway, we need to handle it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The dropped needles are all of the Beta art, before it was
fuzzed out a bit for Final. We drop the match area on the
Final needles to catch more transitions (saw a 94% today).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
...this reminds me I was naming them by time of day for F33,
which was a good idea. Should've remembered to keep doing that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We'll need a whole farm of variants for this revised artwork.
Joy. Once it goes stable we can ditch the older versions.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The Japanese "accept your fate" button changed slightly, and
the new root password screen got a French translation.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Somehow this button looks slightly different depending on where
the arrow is or something, even if you don't match on the arrow.
Sigh. So we need two.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Don't match on the arrow, another app has it on the other side.
The color and text should be enough to avoid false matches.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
kgpg had its own, which doesn't make any sense. There are grey
and blue background variants that weren't consistently named.
This should rationalize things sensibly, and adds a new needle
for the new Plasma in Rawhide, with a lighter blue background.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The cross got bigger. This needle might be a dupe with something
but I couldn't be bothered finding it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There's some issue with font kerning or something on this page
which means we constantly get slightly different rendering of
this text. Rather annoying.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
For background translucency (sigh) and also a color difference
that I guess indicates a critical update or something.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Some changes were made in the icon theme used by the installer
and GNOME, update various needles for this.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The other tag the Help test uses is created by a concatenation
that isn't easy to change, so just add the tag to the other needle.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Similar to the 'open' needle, adjust match area to work even
when there's a notification obscuring the top bar.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We somehow hit a transition point that didn't match any existing
needle. Oh background transitions, how I love you.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We do this slightly differently on KDE and GNOME for whatever
reason, so this needle needs updating too.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
LibreOffice adjusted layout a bit and dropped a template, one
icon changed in the utilities menu compilation.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Arabic doesn't match the English needle even though it's not
translated, because it's RTL, so the radio button is on the
other side of the text.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The screen is not translated in any language yet, so let's just
make the needle language-independent at least for now. We could
change this after translations show up.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This often seems to get a 90-95% match for some reason. Let's
just drop the threshold. I think it may be a kerning difference
based on the width of the MAC address?
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Really this just boils down to needing an extra click. We can
even just do the click in the old UI as well, it's not needed
but won't hurt anything and keeps the code simple.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
See https://github.com/cockpit-project/cockpit/issues/16243 .
This is a fairly minor issue upstream knows about but will not
be fixed immediately, so we'll add a workaround for it for now.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
As suggested by @kparal, this adds a test that specifies an
additional repository using a metalink. The repository contains
a single package, 'testpackage', that supplements glibc (so it
should always get installed). The test runs an install then
checks that testpackage got installed.
We also deduplicate a pair of needles which were matching on the
same anaconda UI feature (an "add" button) and use that same
needle in this test.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This also requires a new needle for accounts.fedoraproject.org, as the webpage
is completely different now. The old needle for browser_fas_home is obsolete via
this change and can be removed
Remove a whole chunk of needles that haven't matched for more
than 3 months. Also move a few needles to appropriate locations,
simplify some code chunks that relied on removed needles (if
we're not matching the needles, we don't need those chunks any
more), and drop some other no-longer-needed conditionals for
older releases.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
As of today's Rawhide, KDE seems to be using a different icon
theme for anaconda. Not sure if this is intentional or not, but
we need to cope with it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This adds a test for fingerprint-based login, as requested by
@benzea in #223. We use the fprintd dummy device to let us
simulate scanning a fingerprint, and check various scenarios
recommended by @benzea.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Now we have two areas, openQA wants to click in the wrong one.
Let's tell it which one to click in.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The main screen now also has identical "Japanese" (that's what it
says) text. To avoid false matching before the picker opens, add
another match area.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The open dialog on Silverblue (which is apparently not at all
the same thing as the open dialog on Workstation, though they
look the same) does not default to the Documents folder, so we
have to open it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
These all failed to match first time the test was run in
production. I guess Lukas was working from an older release.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This PR fixes issue #188. It adds a test suite to test basic
functionality of Evince and brings the following features:
* test scripts for various Evince functions.
* needles to support the Evince test scripts
* new template variables `TESTPATH` and `POSTINSTALL_LOAD_ALL` (see
below)
* new logic in `main.py` (see below)
The new variables and the new logic make it easier to create test
suites for post-installation tests. If TESTPATH is used, OpenQA
will take all tests mentioned in POSTINSTALL from that specified
TESTPATH. If both TESTPATH and POSTINSTALL_LOAD_ALL are used, then
OpenQA will run all tests it can find at the TESTPATH location.
If POSTINSTALL and POSTINSTALL_LOAD_ALL are set simultaneously,
then only POSTINSTALL will be taken into account and OpenQA will
only load tests mentioned there.
Recent git os-autoinst no longer downsamples screenshots as far
as it did before comparison. This makes a lot of needles where
colors have changed slightly no longer match, so they all needed
updating.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Again, to combat os-autoinst's new behaviour without adding yet
more needles. We're getting 96% match on stg.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Also delete ones for F32 and earlier. This should hopefully help
us not need to have too many needles for the newer stricter
matching.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Latest Firefox has moved the underline down slightly for some
dumb reason. This tweaks the match area to only the text, so it
matches wherever the underline is.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
In #235 we accidentally included an entire disks.pm test that
wasn't meant to be there - the infocenter module test is the
right thing to exercise plasma-disks, there is no standalone
app, running 'disks' just gets you the KDE Partition Manager,
which we already test. So this removes that test and renames a
needle that looks like it's for that test but is actually for
the kinfocenter module, to make this clearer.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This PR adds a new test that automates the above mentioned test case.
It starts the installation in text mode using the `install_text` test
case, which it interrupts using the Anaconda crash trigger.
When the crash happens, it goes through the process of reporting
the bug to Bugzilla, checks that Bugzilla sends a positive
confirmation of the action, but also performs some REST API
calls to do a proper check and then it closes the bug to clean up.
GNOME dropped the g-i-s new user mode in F34, so on a Japanese
install with user created in the installer, you don't get an
input source configured out of the box or on first boot. So
we'll just have to do it manually after booting, before we test
if it works.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Looks like the latest Rawhide got a permanent update notification
for KDE again. F34 is still around, though, so we can't just
revert to the old code, I don't think.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The 'desktop_pacakge_tool_update-kde-detected' needles and
'desktop_update_notification_systray-kde' needles are actually
matching on exactly the same thing, so drop the redundancy. We
need to have the desktop_package_tool_update tag on the older
(F33) version of this needle because on F33 we click on it to
launch the update tool in the desktop_update_graphical test; from
F34 onwards this is *not* what we want to do so the needle should
not have that tag to avoid throwing the test off. When F33 goes
EOL we can drop that tag from the needle and simplify the
destop_update_graphical test. Also add a needle for the Discover
app's 'update' icon when no updates are found.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There's some cheating/sloppiness going on here, with the same tag
and sometimes same needle being used to match "LVM2 Volume Group"
and "LVM2 Logical Volume". Today this caused us to pick the thin
pool entry instead in a test, so let's just clean this up and do
it right, with separate needles for matching each thing.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The needle match seems to have changed when bug #1957858 showed
up, but it's actually just a text rendering change in the window
title, it's not exactly caused by the tiny window. So not marking
as a workaround needle.
Maximizing the window makes the test work faster when we hit that
bug, as type_safely needs to be able to see the results of its
typing.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Some fixups for anaconda_help. Two runs of it failed today around
handoff from the root password screen to the install progress
screen; add a couple of wait_still_screens there to make it
safer. Drop the added nonlive needles, because they're too
permissive, causing problems for other tests (they're matching
before they should); instead we solve the problem of spokes being
highlighted by just pressing shift-tab a few times. And fix some
tabs to be spaces.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This PR automates the mentioned testcase to test that Help can be
displayed in Anaconda during the installation. It navigates through
the available Help screens and if it can see it, it finishes.
This test runs after `install_default_upload` to override the
installation defaults defined for all primary tests.
Delete a duplicated needle.
Reformat list extensions to make it nicer.
Get rid of wrong export and an empty line.
Delete empty line.
Use _boot_to_anaconda for booting and move subroutine accordingly.
Add variable to templates.fif.json
Delete trailing whitespace.
Fix calling the pretest.
Move help checking to another place.
Not sure why these changed, but oh well. Utilities menu was
highlighted in a test run for some reason, so let's just handle
that. Other needles changed very slightly.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We're not using them, so they fail the unused needles check. We
can just revert this commit when we want them back.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The background color of the generic left bar needle has changed
to match the new logo base color. The top bar's background color
has similarly changed, but this also caused us to notice a bug
in fedora-logos - that topbar image file seems to be basically
empty (just a transparent rectangle) so we see no 'image' in the
top bar, just solid electric blue. This needle matches that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Boxes dropped VNC functionality. It's supposed to be replaced by
Connections, but we can't use that until it has fullscreen:
https://gitlab.gnome.org/GNOME/connections/-/issues/5
so use Vinagre for now. We do also prepare some needles for
Connections in anticipation of being able to use it later (since
I already did the work and don't want to waste it...)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
OK, so sometimes we have a Reboot button, sometimes a Restart
button, sometimes both, but never *always* one or the other. So
we need both needles. Sigh.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There's a spurious warning on the Updates page, but we don't
want to fail tests for non-related updates on that, and I've
already reported it so it should get fixed.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The 'reboot' button isn't always there in this case, it seems,
but the 'restart services' one is more likely to be. So let's
switch.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Stupid Firefox survey means the thing we usually check isn't
always on the screen. This one checks for PRIORITY.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
KDE switched to using Noto fonts by default (and title bars seem
to be blue again), many needles need to be updated.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
On current F34 we get no permanent update notification in the
notifications view, we only get a *transient* one plus the
systray icon. This tweaks things so on F34 we check both of
those things correctly, behaviour on <F34 should be unchanged.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This dialog gets cut off sometimes, it seems. That's a bug, but
we just want to handle it, because we want to quit cleanly.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
KDE update was still often failing on #1943943, so this tries a
bit harder to work around it. We add a 'refresh' needle for KDE,
and tweak the 'retry' logic to click it if we get to that point.
Note adding the needle also changes behaviour slightly - we may
click this needle if we see it on first entering the screen. So
either change may be helping. Either way, this does make the test
more reliable.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Design team tweaked the new logo a bit in anaconda. We probably
won't need the old needle any more, but I'll take that out later.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Some apps moved around, others the needles stopped matching for
some reason, some kind of slight scale change or something.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The package with the new logo is not submitted as an update yet,
but we ran the tests on the Koji build and these are the new
needles. We'll need more when we run the full set of compose
tests on the change.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
When we have to restart Cockpit during the update, we don't get
the usual "successful" screen after logging back in, we'll have
to match on something indicating an update did happen. I wanted
to use the little icon next to the package count but it seems to
have some kind of problem with anti-aliasing or something, I've
created two needles for it and neither matched on the next run
of the test. So let's match on this "reboot needed" text instead,
I hope it'll always be there when we hit this case.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Make the 'deactivate overview if it's active' thing a bit more
robust by asserting the inactive state after deactivating it,
and add new needles for the new RC (text got a bit brighter).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
For consistency, let's just return to the desktop right away. We
also need to handle closing the overview before running installer
on live image boot.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We worked around it in a different way (installing a Plymouth
theme in the base disk image) so these shouldn't be needed any
more.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There's a bug in F34/Rawhide lately where the bootsplash isn't
fully cleared when displaying text consoles, this causes all the
login needles not to match. Add several workarounds to make the
most common cases at least work with the bug, failing on it is
not much use.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We need to hit 'restart' after applying updates, and we also
need the 'done' needle *not* to match the restart message, so
change that to match on the text (unfortunately). That also
means we have to add another variant of the needle for F32 as
the background of the text is a different color there.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This PR adds a test that uses the Blivet interface to create an LVM
layout with ext4 filesystem as well as a postinstall test that checks
that the LVM layout has been created correctly.
We finally saw a test where there were *no* errors logged by the
time Cockpit reached the log screen, so there were no entries to
click. Let's just make the test set log level to info before
looking for entries - I prefer this to 'click entry if found,
otherwise change log level' as that's twice as many branches to
look after. Of course, it means the warning triangle entry needle
is useless now :(
Signed-off-by: Adam Williamson <awilliam@redhat.com>
KDE in F34+ is now placing sleep, restart and shutdown buttons
right on the system menu, not in a submenu. So we need to sort of
tweak this logic. The approach here is: we count the GNOME
submenu as both a "power" and "leave" menu, so the needle to
enter it has both tags. KDE still has a "leave" submenu, but the
power options are not in a submenu any more, so the new "leave"
needle only has the leave tag, not the power tag. For "leave"
actions we just unconditionally expect the "leave" tag; for
power actions we first match on *either* the submenu tag (for
GNOME and earlier KDE) *or* the action tag, click whatever we
found, and then if we matched the submenu (not the action), we
assert and click the action. After that all paths should be in
sync again and we can continue.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Cockpit 237 has misaligned radio buttons here. We know about the
issue and want the test to not fail on unrelated updates, so let's
make it a soft fail.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This PR uses the Anaconda Blivet partitioning to recreate a partition
layout while preserving the content of the /home subvolume.
It also adds the postinstall test to check that the home has been
preserved.
This PR adds the `install_btrfs_upload` to install the btrfs based
image, the `btrfs_preserve_home_extras` to prepare and test the data
on the home partition, as well as the `custom_btrfs_preserve_home` that
uses the preinstalled btrfs image and uses its current partitioning to
preserve the home partition and the data on it.
We don't always get a smartd error any more, so add one for a
different error that's more consistent ATM.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The background seems to be different in different tests, even
the reduced area I tried most recently seems to be different in
some runs. So let's just match in the text area instead. This
captures some "Fedora" text so it should still notice if branding
is broken.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Required because ppc64le has a PReP partition
before boot partition.
PReP partition must not be changed by this script.
Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
The font of the identifier text got a bit smaller. I think it
looks kinda bad now, but it's not an outright bug, so just
living with it. I did file an upstream issue:
https://github.com/cockpit-project/cockpit/issues/15111
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There's a race issue with just treating it as a next button: it's
not in the same place as a next button. Sometimes in the g-i-s
code we actually get ahead of ourselves and click early, which
isn't really a problem when the buttons are all in the same
place, but if we click "Start Setup" in the middle of transition
to the Privacy screen - as in
https://openqa.fedoraproject.org/tests/745034#step/_graphical_wait_login/4
- the click effectively gets lost. So let's make it its own tag
and have the initial assert look for it too. That way we won't
match on it again in the main loop over "@nexts".
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There seem to be some weird shenanigans going on with the pipe
bit here (it looks different on different test runs? what?) so
let's just match on the plain grey area. Should still be enough.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Just matching the Overview entry isn't really enough, the app
hasn't really run yet. This makes the test more robust and also
helps out on aarch64 desktop tests where the app window takes a
long time to appear.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
When Fedora went to BTRFS as a default, we lost the LVM based image to
run LVM resize tests with.
This PR introduces the `install_lvm.pm` installation test that creates
an LVM based qcow2 image to be used by follow-up tests.
Styling on the KDE base disk image seems to have gone wrong
again, which throws this test off. We need to figure out what's
gone wrong in the base image, but it's a Friday night and I don't
want the test failing all weekend, and there's nothing wrong in
the *test itself*, so let's create some workaround needles and
figure it out next week.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Update a needle with slightly different text rendering, and add
a workaround to hit tab three times rather than once on entering
the "Join a domain" screen, see
https://github.com/cockpit-project/cockpit/issues/14895 .
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We've been getting failures lately on the first page load, I
think because Firefox is getting even more grindy on startup. So
turn the 'sleep' into a 'wait_still_screen', extend another wait,
and tweak the 'browser' needle so it only matches after the
bookmark bar has loaded rather than as soon as half the chrome
appears. Also make all the wait_still_screens use similarity 45
for consistency (flashing cursor could be there on any of them).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Dropped the use of this in the recent notification simplification,
but forgot to remove the needle. Should've run tox...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is the best option I can come up with to deal with #195.
Update notifications seem to have become transient in KDE lately
(even in F31 and F32, if I'm looking at these screenshots right).
This actually simplifies things a lot to do more or less the
same in the KDE and GNOME paths: open the 'permanent' store of
notifications (in GNOME you get to it by clicking on the clock,
in KDE via the systray) and then look for no notifications (live
path) or only an update notification (post-install path). We
only run this test for composes so we shouldn't need to worry
about anything older than F32, and I believe this should work
for KDE in F32 and F33. I left out click_unwanted_notifications
for now as I'm hoping it should be unnecessary, but we can add
it back in if necessary.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We don't need a separate 'welcome' needle because it just matches
on an OK button anyway. So turn that needle into an OK needle
(we don't have any existing 'blue OK button' needle) and simplify
the logic to a single loop for kde_ok and krusader_settings_close.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It had krusader_settings_close as its tag, not kde_ok. That's
why the krusader test module was failing weirdly.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The 'universal' tests have quite a few failures if you run them
on an image where btrfs is the default (currently they usually
run on the Server DVD, where xfs-on-LVM is still the default).
This fixes some of them, the others would need code fixes.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Even though we have subdirs, we actually usually make needle
names unique across subdirs due to limitations of openQA's UI
upstream.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
OTP field was moved into the last position in the password change dialog
to prevent issues with OTP code expiring while users enter their
passwords.
Signed-off-by: Alexander Bokovoy <abokovoy@redhat.com>
In g-i-s 3.37.91, the first screen has a 'Start Setup' button
rather than a 'Next' button. Easiest thing for us to do here is
just to add a new needle which has the 'next_button' tag even
though it's clearly not a 'Next' button, because then the code
still works :) So do that, but give the file a suggestive name
and explain the situation in a code comment.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
GNOME now also splits 'Restart...' and 'Power Off...' as KDE
does, so we need to tweak the conditional and add some needles.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is a bit complex to automate, because we cannot really use
the production Zezere server (provision.fedoraproject.org) as
the test case shows, as we'd have to solve authentication and
we also don't really want to constantly keep registering new
hosts to it that are going to disappear and never be seen again.
So, instead we'll do it by setting up our *own* Zezere, and
provisioning our IoT system in that. We run two tests. The
'ignition' test is the actual IoT 'device'; all it really does
is boot up, sit around, and wait to be provisioned. The 'server'
test first sets up a Zezere server, then logs into it, adds an
ssh key, claims the IoT device, provisions it, and connects to
it to create a special file which tells the 'ignition' test
everything worked and it can close out.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
See https://pagure.io/background-logo-extension/issue/26 - in
current Rawhide, the search box in the overview is not active
when the overview is opened, so you can't just open the
overview and type, you have to click it first.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
See https://bugzilla.redhat.com/show_bug.cgi?id=1854225 . A new
fontawesome release broke nearly all font-based interface
elements in the FreeIPA web UI, which breaks several of our
needles. This adds workaround needles for each one (plus a couple
of non-workaround needles for cases which don't seem to be the
icon bug, but somehow changed anyway).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
In some recent tests the text is rendered slightly differently
for some reason. Not sure why, but I don't see a problem with
just adding a variant needle.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Recent live respins were built with livecd-creator rather than
livemedia-creator because the maintainer had trouble with LMC.
This results in grub looking a bit different, we need a variant
needle for it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Not sure if the needles changed or just the way they're rendered
in the overview, but either way, we need to update a bunch.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The little triangle that's used on drop-down menus and stuff got
bigger. That breaks all these needles.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This goes back to when we called this needle desktop_clean, but
there's really no point in having it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
In Cockpit 220, the Updates entry is off the bottom of the screen
so we need to scroll the left bar down before we can click it.
Also update some other needles.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It seems like sometimes the machine ID is off the bottom of the
screen now, so let's just match on the section title instead.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Unfortunately only the entry name is clickable now so we can't
have a 'generic' needle that'll work for any service.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Since GDM shows the "system-menu-button", it could not correctly
switch users on a locked screen. I added a check to see
if we are on a locked screen and behave accordingly.
This adds a new test that implementsQA:Testcase_desktop_login
on both GNOME and KDE.
While working on this, we realized that the "desktop_clean"
needles were really "app menu" needles, and for KDE, this was
a duplication with the new "system menu" needles, because on KDE
the app menu and the system menu are the same. So I (Adam)
started to de-duplicate that, but also realized that "app menu
button" is a much more accurate name for these needles, so I was
renaming the old desktop_clean needles to app_menu_button. That
led me to the realization that "check_desktop_clean" is itself a
dumb name, because we don't (at least, any more, way back in the
mists of time we may have done) do anything to check that the
desktop is "clean" - we're really just asserting that we're at a
desktop *at all*. While thinking *that* through, I *also* realized
that the whole "open the overview and look for the app grid icon"
workaround it did is no longer necessary, because GNOME doesn't
use a translucent top bar any more. That went away in GNOME 3.32,
which is in Fedora 30, our oldest supported release.
So I threw that away, renamed the function "check_desktop",
cleaned up all the needle naming and tagging, and also added an
app menu needle for GNOME in Japanese because we were missing
one (the Japanese tests have been using the "app grid icon"
workaround the whole time).
This adds a needle for the 'day' background from 32.1.1, which
is the non-animated default, and renames the needle for the
background from 32.0.0 to 'old' as it's no longer present at all
in the new build. Depending on whether animation actually works
after the update we may later need to add needles for the other
images plus possibly some transition needles too. Sigh.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There's this weird thing where the vertical alignment of the
language name in the keyboard layout indicator is sometimes
different. I never can figure out why. It may be to do with
presence or absence of the pre-release indicator.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I call this...The @lruzicka Catcher!
It's a script that checks for needles that aren't actually used
anywhere. It also checks for cases where we have a needle JSON
file but no image, or an image file but no JSON file (and wipes
one case of the latter). It also adds a run of the script to tox
so we get it in CI.
You could make this script a lot more elaborate if you like, by
being fancier about parsing the test code and templates, but I
don't think it's really warranted, I think it just needs to be
'good enough'. It's not the end of the world if it misses the
odd thing or the whitelisting goes stale.
Quite a lot of the removed needles are remnants of different
approaches to app start/stop testing which weren't caught in the
initial PR review. The short-name partitioning ones are odd; they
were introduced in the commit that moved needles into subdirs,
but at least some of them don't actually appear to be moves. They
may have been non-tracked files Josef had lying around that got
into the commit by mistake, or they may just be old needles we
really used at some point but aren't using any more.
reclaim_space_second_partition was introduced as part of the
shrink test (along with reclaim_space_first_partition) but was
never actually used by that test - I guess, again, the test got
re-written during review but we forgot to remove the needle. We
rejigged user creation to use tab presses not a needle match a
while back, which made user_creation_password_input unnecessary.
The various cockpit_updates_* needles are I think remnants of
rewrites of the cockpit update tests that again were missed in
PR review, the tests as merged never used them.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This reverts commit d784bf54ca.
It turned out that Locations are not connected to Konqueror
at all. The reason why the test is failing is that the
application has been removed to limit the number of
web browsers.
Remove a bunch of needles that have not been used for some time,
plus a few workarounds that are similarly stale.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I think what happens here is the kerning and/or subpixel hinting
changes depending on the column position, and the column position
keeps changing as upstream releases new versions on different
dates and stuff. Hopefully eventually we'll have enough needles
to cover all possibilities...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This test has been failing forever, now the bug is fixed, we need
to update the needles for font changes...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This reverts commit adf3f91818.
The bug has been fixed in anaconda, so we can drop this, which
is good as it has timing issues producing false positives on
Rawhide...
Rawhide seems to have developed a bug where a single disk is no
longer automatically selected as the install target when you
enter the INSTALLATION DESTINATION spoke. We need to work around
this (but register it as a soft failure).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Seems some changes to symbolic icons and widgets have happened
in GTK+ or something, this updates anaconda needles for them.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
IoT is becoming a release-blocking edition for F32, so we should
be testing it for sure. We may add specific tests, but for now
let's run the install and base tests on it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I updated this last month without looking closely enough; one of
the matches was bogus. We now don't seem to get the 'added to
Firefox' popup any more, so we can't match on it, but I guess we
can match on the 'remove' button as an indicator that the add
worked.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Again, no reason not to run these on updates. Includes adding
oldcantarell versions of several needles for current cockpit,
as they're needed for the tests to pass. Also tweak a couple of
needles to avoid false matches (add more empty space).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Note we need two variants of most of these, one for Rawhide with
Cantarell 0.201, one for F30/F31 with Cantarell 0.111.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This one from ppc64le test. The difference between all the
recent needles here is the amount of white space between the
'Domain' and 'domain.local'. I don't want to change it to two
match areas as there's a chance we may get a false positive
match when the join actually failed (it may match on 'Domain'
in the right place, and 'domain.local' somewhere else just as
part of the system's hostname). I suppose if this keeps happening
we could try two match areas but include some white space to the
left of 'domain.local' to avoid matching on it as part of the
hostname...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The text we used to get has been replaced with a spinner, which
is difficult and unreliable to match on. This match was only
here to make the test fail a bit faster if it was broken, so
let's just live without it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This match keeps giving us problems, and now I look at the test
again, there is actually no need for it at all. Clicking it
doesn't do anything, and we already confirm that we're on the
right page at the next step, where we look for a log entry and
click on that - that will fail if we aren't actually on the
Logs page.
I don't remember what Cockpit used to look like when we first
put this line and needle in, presumably there's a reason we had
them, but they're clearly unnecessary now.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I am so freaking pissed off with these bizarre 'not quite'
matches, what the hell is Firefox doing
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Sigh, I don't know why we're having so much trouble with fonts
in browsers lately. Drop the match requirement on the needles
that have been matching or nearly matching the most, and remove
older ones that aren't matching.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We need a couple of new needles, plus the 'join domain' button
has disappeared from the front page due to the very inefficient
UI redesign, so we need to scroll down to find it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is getting annoying. We're getting, like, 94-95% matches
for the recent needles sometimes. Instead of continuing to make
more and more needles, let's try dropping the required match
threshold a bit. The kde-20191206 needle doesn't *seem* to be
needed so let's get rid of it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is for KDE. KDE and GNOME have different font rendering,
and they both look slightly different between a few days ago
and today, I think because of a change to the width of the
columns on the page itself.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Slight font rendering changes from the last GNOME version of
this needle, not sure if there's actually a font library change
or it's just hinting changes caused by the column being a bit
further right than before...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Seems the latest Firefox builds in F30 and F31 updates render
fonts a little differently again, not sure why this is.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Not sure what changed; it seems like mostly browser needles got
broken, but there's a few installer needles too.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We still have a 'apps_run_firefox_stop' needle tag which is for
the same thing as 'firefox_close_tabs'. That's dumb. Get rid of
it and only have the firefox_close_tabs tag and needles. Also
clean up some old firefox_close_tabs needles that haven't matched
for months and all the 'apps_run_firefox_stop' needles.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
On Rawhide Cloud_Base boots, there's some SSH key and network
information printed above the 'login:' prompt, so we can't
expect empty space there. Also tweak console_login() to clear
the screen after logging in, so the login prompt is cleared and
doesn't confuse things on subsequent runs (like it did first
time we tried this). And add a new user logged in needle, as it
seems after we clear the screen the tilde appears in a slightly
different position and the existing needle doesn't match.
https://openqa.fedoraproject.org/tests/489003#step/_console_wait_login/7
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This reverts commit f66f510832.
Turns out this breaks lots of stuff, so it needs to go back.
I'll have to figure another way to handle the cloud login.
On Rawhide Cloud_Base boots, there's some SSH key and network
information printed above the 'login:' prompt, so we can't
expect empty space there. Let's just hope not looking for the
empty space doesn't break anything else.
https://openqa.fedoraproject.org/tests/489003#step/_console_wait_login/7
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This adds a test for QA:Testcase_Anaconda_User_Interface_VNC -
the VNC install test case. It's implemented as a server/client
pair, with the server booting from the Server DVD image with
`inst.vnc` and the client booting from the desktop base disk
image, setting up networking, then running Boxes to connect to
the server and run the install.
There are various little tweaks to test loading and logic to
handle this, mostly pretty clear. We also move the workaround
for 'spurious auth prompt appears on desktop after you switch
away to a VT and back' out of the desktop update test and into
the `desktop_vt` helper function, since now this test can hit
it as well. We enhance _graphical_wait_login to handle the boot
loader if needed (it has never needed to until now).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This handles a case where KDE shows a notification saying
'PIM Maintenance (Finished)', like this:
https://openqa.fedoraproject.org/tests/477345#step/desktop_notifications/34
we need to click it away for the desktop_notification test to
pass. It also clarifies the difference between this notification
and the eternal 'akonadi_migration_agent is doing something'
popup in the needle names and comments. It also replaces the
'check_screen then assert_and_click if found' pattern in several
notifications-related places with the better 'check_screen then
click_lastmatch if found' pattern now available upstream.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The login screen background changed in 205.1 (it's actually the
same image, but it's now zoomed rather than just a chunk of it
being shown on a smaller screen).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Once again, the text is at a slightly different height to the
needles we already have. I do not know why this sometimes
happens. It's a mystery.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This reduces the coverage of the identification test a bit but
also *substantially* simplifies it. We run into a ton of problems
when we try to check the version and prerelease text on screens
where it appears on banners:
* The banners differ between variants
* The pre-release text is translated
* The banners have gradients so for RTL languages, even if some
text is untranslated (e.g. 'Fedora 31') it appears on a
different background color than on LTR languages
* The prerelease text is dark red; if it appears on a dark blue
area of the banner this can trigger an os-autoinst needle
comparison bug: https://progress.opensuse.org/issues/56822
All of this together means we wind up continually fighting these
checks and we have a whole forest of needles just for them, and
it doesn't seem worthwhile. So let's drop all the places where
we were checking version and prerelease on banners, and only
check them in two places where they appear on a grey background,
which avoids most of the problems (we just need one version
needle per release, and one prerelease needle per language).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Now one of the strings we check for is translated, but the other
is not...handle this, again with a workaround needle so the
missing translation triggers a soft fail.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
* server-cockpit-updates tests that Cockpit can be used to update the system.
* server-cockpit-autoupdate tests that users can use dnf-automatic for system
updates.
* cockpit functions were removed from utils.pm and put into an extra library
for cockpit - cockpit.pm which all cockpit tests are now using.
Review cockpit.pm
Review autoupdate test.
Review the update test.
Fix typo in cockpit.pm
Add sleep.
Add missing command.
Delete an unused needle.
The new background seems to be throwing us off here. I think
the menu is *very slightly* transparent so the needles usually
work for most Fedora backgrounds as they tend to have a similar
base color, but the 31 one is unusually light.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
For RTL languages the keyboard icon and language indicator are
the other way around, so we have different needles even for the
US layout indicator which is otherwise the same in all langs.
This is a variant needle for the new keyboard icon that showed
up in Rawhide recently.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This checkbox sometimes renders slightly differently, especially
on non-x86_64. We're not really sure why, but we just add more
needles to handle it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Existing one doesn't match if the background is too different,
shrink it to just the center of the icon.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We need another needle for French installs, and we need to use
ISO_URL for checking the ISO file name if ISO isn't set.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
These showed up because I did a manual universal flavor run on
an ISO today, but we'll need them as soon as we do a candidate
compose too.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
These only get used when we run universal tests on a non-Server
image, which is pretty rare (these days, as Server DVD is a
critical image and composes fail if it fails, it only really
happens when we do it manually for some reason, like I did
today). So they get stale and aren't updated for font rendering
changes and stuff. As I said I had to do a run like that today,
so I had to update all these needles...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is doing slightly less to exercise the launcher menus and see
whether icons appear or disappear, but it's much faster and more
reliable. We do still use menu launch for one app, just to check
the mechanism works in general.
The top bar differs in non-English languages - for RTL the
version ident is on the left (so background is different), and
the width of the area with no text differs depending on the
length of the translations.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This adds a whole wodge of stuff to support_server to make it
act as a PXE server, then adds a new test which boots from PXE
and so should hit the PXE server. We use the NFS install repo as
that can be relied on to work for a support_server install.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Seems like there's been a change to freetype or something, text
rendering changed in all of these. Note there's one workaround
needle for an untranslated message in Japanese.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Split this out of install_default, because it really is not a
part of that test and we do not want that test to fail because
the desktop background is wrong. Make it its own test module
and test suite instead. Don't do it on Rawhide, because we
really can't assert anything worthwhile about Rawhide at the
moment at least (this means the test runs but is a no-op and
will always pass on Rawhide, unfortunately). Move the needles
to a more appropriate location (this has nothing to do with
anaconda) and use 'background' not 'wallpaper' naming (that's
the name we use elsewhere in the project, e.g. package names).
Also, run the test on updates, and add an F29 needle for this
purpose.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The backgrounds we have are F30 backgrounds, not F31: there are
no F31 backgrounds yet, F31 images are using F30 backgrounds
(which is a bug we should file). Also we really only need one
F30 background needle to match both KDE and GNOME if we pick a
sensible area of the screen to use, and let's use one that has
a bit more contrast for safer matching.
Note: F30 background is *meant* to be animated, but in fact
neither GNOME nor KDE seems to use the animated version by
default. Which makes our lives easier! Sucks for whoever put in
the work to animate it, though.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Cockpit 198 comes with a UI overhaul, so almost all needles
need an update.
The 'auditd' service is no longer on the first page. To make
this less fragile (at the cost of not testing that clicking on
a service actually opens the detail page *for that service*,
tweak the needles to just look for *any* running service, click
on it, and check we got to a 'details' page. We also redo the
existing needles for this design.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
@lruzicka added this to the KDE app tests in 71c4e273, but there
was no need for a new needle as I'd already done the same thing
in the desktop updates tests; let's just use the same needle.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is really a 'generic' needle since it'd be needed for any
RTL language test, it's not specific to arabic.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Mainly because the GDM background became a lighter shade of
grey, for some reason, but also some dialog and icon changes.
Also put all forms of layout_us_ltr-gdm in the same directory.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Removing this broke the GNOME apps test entirely because there
was no GNOME 'workspace' needle any more.
I don't like this needle much, we should probably use
check_desktop_clean or something instead. But for now let's just
put it back.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The way KDE does update notifications has changed - it's now a
permanent pop-up notification. This is a bit awkward for our
logic; it's hard to define a needle that proves this pop-up is
the only notification. Instead, let's dismiss it, then open the
notification tray and assert that there aren't any others. But
we also retain the old behaviour (more or less) for testing old
releases.
The popup notification also blocks the 'refresh' needle in the
systray and so breaks the desktop update test, so we deal with
that too.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Since the KDE menus have transparency set, any time the wallpaper
changes the menus will look different, and the app tests may
fail. This sets the desktop wallpaper to black at the start of
the test suite to avoid this problem.
In the last few weeks Boxes is showing an auth dialog on start.
I've filed a bug on this; let's have the test handle it as a
soft failure, since this isn't a fatal problem. Do this by
making an existing needle for this dialog a bit more generic
in name and using it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The KDE and GNOME 'apps' tests for Firefox both invented their
own 'oh look Firefox is running' needle, even though we already
had one. The GNOME one was broken by the removal of the app
title bar in Firefox 66.
Instead of having three needles for the same thing, let's just
throw the 'apps' ones out and use the pre-existing one from
needles/firefox for all cases.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Utilities came up different *again* in a Rawhide test today, so
this adds a needle which matches on the text. Usually we prefer
image matches to text, but this image seems to keep changing...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Add png files as a background for the needles.
Rewrite the test handling methods to make them ready for KDE
Add the ABRT startstop test.
Make changes and corrections to the abrt test.
Add test for Firewall.
Add test for DNF dragora.
Add wait time for dnfdragora test.
Correct syntax.
Add Language test.
Make some changes to the DNF dragora test.
Add Users test case.
Add needles for DBUS viewer test.
Add Dbusviewer test.
Add Mahjong test and needles for games.
Add Minesweeper tests.
Add Patience test.
Add test for Document Viewer.
add test for gwenview
Add test for koulourpaint.
Add test for Kruler
Add test for Kcolorchooser
Reneedle failing needle.
Add ktorrent tests.
Add tests for CPT editors.
Add test for Krfb
Fix names for those files.
Add test for Kget
Add Akregator test.
Add test for Konversation.
Make Konversation end really.
Add tests for Kmail
Add test for PIM exporter.
Add test for KTnef and Krdc.
Fix problems after test runs.
Make more tries.
Fix needle to be found better.
Fix more errors.
Add test for Falkon.
Add tests for browsers.
Add support for closing tabs into Firefox test.
Add tests for K3B
move needles to correct directory
Add Kaddressbook test.
Add Kontakt text.
Add test for korganizer.
Add menu office needle and correct konqueror needle.
Add test for calligra stage.
Add test for Calligra.
Add test for network connections.
Modify needle for kaddressbook to prevent failing.
Add test for system settings and fix others.
Add test for FMW.
Add test for Dolphin
Add test for Infocenter.
Add test for kparted.
Fix a wrong needle.
Test relnotes.
Fix some errors in tests.
Add test for Discover.
Add test for Ksysguard.
Add tests for Konsole.
Add tests for KDE wallet.
Add tests for several utilities.
Add Krusader test.
Finish utility tests.
Fix some errors.
Fix needle for spectacle.
Add wait time to let Dragora wait for network.
I don't know why the ones I *just created* don't always match,
but they don't. Let's just have a bunch of needles and in a few
weeks I'll look and see if any are never matching, and get rid
of them.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
In latest Branched, 'Close Tabs' button has an upper-case T, and
the addon install button is a darker blue.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Dunno why we need yet *another*, but even the new one didn't
match on the most recent Branched compose. For now just add
another, we can look in a few weeks which can be thrown away.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
As of Fedora-30-20190316.n.1, gnome-abrt is back under the
Utilities menu and the 'Sundry' menu seems to have disappeared.
Change the test back to using utilities and drop the sundry
needle.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We want to get accurate results from the rest of the upgrade
test and we know about this bug, so let's make it a soft fail
so we can see how the test is functioning otherwise.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Investigation suggests that the Firefox certificate validation
failure we sometimes get when loading the FreeIPA web UI in the
freeipa_webui test module is a race issue in Firefox itself:
https://bugzilla.mozilla.org/show_bug.cgi?id=1530429
refreshing the page seems to work around the problem, so let's
do that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The 'largetext' version of the needle is needed when there's
some notification displaying over the window title, it seems
F28's current font rendering doesn't match the current version
of that needle...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We're getting an intermittent case where FreeIPA tests fail
because of a web server certificate issue. Click 'Advanced' in
Firefox when this happens so we can get a bit more info on the
problem.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
ctrl-t seems to be failing often in Firefox tests, and I can't
figure out why. Let's try clicking the 'new tab' button instead
as a workaround.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The Adwaita icon theme got some changes in a git snapshot build
for Rawhide recently. These changed icons break various needles,
we need new ones.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Fedora logo is missing from GDM in today's Rawhide. Not going
all the way and making this a soft fail as the bug is known and
should be fixed soon, I'll take the needle out again then.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We're getting a spurious "Authentication required" for a
background refresh of the system repos sometimes, depending on
whether it happens while we're at a tty. This is now accepted
by upstream, so let's handle it and make it a soft failure for
now.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
New GTK+ changed something so the background of many interface
elements is a slightly lighter grey, this broke a bunch of
needles. Here are the retakes. Includes one not-strictly-related
SDDM update and a rename of the one @lruzicka did to match the
others.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The needle could sometimes fail because the item it was looking
for only appeared in a limited set of situations. The new
location should be safer to check and the test should not
fail as a result.
This adds a test which builds a netinst image potentially with
the package(s) from the update, and uploads that image. It also
adds a test which runs a default install using that image. This
is intended to check whether the update breaks the creation or
use of install images; particularly this will let us test
anaconda etc. updates. We also update the minimal disk image
name, as we have to make it bigger to accommodate this test,
and making it bigger changes its name - the actual change to
the disk image itself is in createhdds. We also have to redo a
bunch of installer needles for F28 fonts, after I removed them
a month or so back...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
If a test fails to the dracut shell, we currently don't do
anything useful. This should recognize when that happens, and
upload rdsosreport.txt.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It's my second least favorite day of the year again: Stale
Needle Cleanup Day!
This should get rid of all non-anaconda needles that definitely
are not being used any more. Cleanup of all anaconda needles was
in the previous commit.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It's my second least favorite day of the year again: Stale
Needle Cleanup Day!
This should get rid of all anaconda needles that definitely are
not being used any more. A few borderline cases (where I'm not
100% sure if they may still be useful in odd corner cases, like
running universal tests on non-server images, and runs outside
of the US, and stuff) are kept around.
Cleanup of all non-anaconda needles will be in the next commit.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This was part of a workaround for a Fedora Atomic Workstation
(old name for Silverblue) issue that got resolved a while back;
we forgot to take the needle out.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It seems Rawhide auto-signing is working fine now: openQA claims
this needle has 'never' been matched (which really means 'not
for a long time'), and I can't find any test fail in the last
year which looks like it landed on this 'authenticate' screen
but the needle failed to match, or anything like that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There's really no point having separate error and error_report
needles. Just match on error_report as well as clicking on it.
Also add a new error_report needle for latest Rawhide fonts.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
GNOME Software 3.30.5 split the offline update process into two
separate 'download' and 'apply' phases. So we need to handle
clicking 'download' before 'apply', if that happens.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Somehow, recently, FreeIPA tests are running into Firefox not
quitting because it's showing a warning about closing multiple
tabs. (I think we didn't *get* multiple tabs before but now we
do, for some reason). So let's work around this by clicking
"Close tabs" if the warning appears.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We're not seeing *exactly* #1314991 any more, but we're seeing
something that looks quite similar: the first attempt to find
updates just doesn't find any. No error message, no updates. I
have reported a bug for this and am investigating it, in the
meantime, let's restore the workaround, elaborated a bit, and
looking for the 'Software is up to date' screen instead of the
error message.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I rather suspect the *bug* is still basically present and it's
why this test often fails, but we no longer seem to see the
*error message* which lets us detect the bug happening. This
needle has not been hit by any test for six months. So let's
remove the workaround as it adds complexity.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
An selinux alert can hide the 'getting started' window title, so
let's have a variant needle which matches on the big 'Started'
in the window instead. We may as well have both for maximum
match potential.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The --MORE-- text looks different with the UEFI firmware font,
so we need another needle. Also I named the original needle
wrong, bad me.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Per Neal Gompa boot will proceed if we just page through the
error(?) messages displayed when #1618928 happens, so let's do
that to let the tests get further and see what else is broken.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
So that we get the crash info uploaded. Note these are not
language-tagged as we *don't* want to 'test' translation of
these, we want the best possible chance of matching any error
dialog in any language...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
anaconda's new hub layout makes all the icons smaller, so all
these needles needed re-doing, even though the same icons are
present...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Seems sometimes tests are showing up with the date/time spoke
in a non-active highlighted state or something, don't really
know why, but it's not broken and we should just accept it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Seems on x86_64 the text cursor is at the left of the text entry
box, but on aarch64 it's at the right. I have no idea why.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Sometimes on aarch64 clicking the partition scheme drop-down
just doesn't seem to make the menu appear, instead the button
goes active but that's all. It's very unlikely we'll be able
to track down why as this doesn't happen in manual testing on
aarch64 (according to @pwhalen), so instead let's just work
around it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
These all are caused by a new desktop background and bits of
KDE desktop chrome apparently becoming translucent.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The background is now blue for ppc64le, so add related needles as bypass.
no such problem with ppc64 (BE)
problem initially detected on Rawhide compose 20180204
but still present on f28 compose 20180302
and Rawhide compose 20180513
Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
https://bugzilla.redhat.com/show_bug.cgi?id=1403365 has been
around approximately forever and I still haven't managed to
debug it; let's just make needles for it, as it's not really a
critical bug, the system still *works*.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
For quite a while on i386 the 'enter passphrase' console screen
has used bright white text, for some reason. Let's just have a
variant needle for this instead of worrying too hard about why.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We've been seeing an odd case lately where the language select
screen is not foregrounded when it appears (so all text is
grey). It happens very occasionally on x86_64, but a lot on
ppc64. To work around this, let's add a needle that matches the
inactive screen, and click on the screen when it appears just
to make sure it's active.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
KDE in F28+ seems to show a network connection notification on
live boot, for some reason. Just dismiss it to help the test
pass.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Seems like the indicator arrow's vertical position changed in
F28, for some reason. Anyway, new needles.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
...I hope. This is necessary as we now have a case where it
needs to match post-install (aarch64 support_server, since
aarch64 is always UEFI).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
In the most recent Rawhide compose, anaconda has changed how it
names volume groups (again), and now the VGs in openQA installs
have rather longer names. This winds up causing a problem for
this needle: the column where the 'Device Type' dropdown is
placed actually gets narrower (because the column to the right,
which has the volume group name in it, is wider), and so the
dropdown box is narrower, and the arrow protrudes into the area
which the needle expects not to have an arrow in it. So, let's
make the match area slightly narrower.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Rawhide has a bit of a problem where its 'description' of an
iSCSI disk is so long that the other columns that should appear
in the CONFIGURE MOUNT POINT dialog don't. This means our
device_sda_selected needle doesn't match, because the column
where 'sda' should appear isn't visible.
So add a soft-fail needle to cover this case; we know what the
description for the disk that's 'sda' in this case looks like,
so match on that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Time for an annual spring clean. Based on the admin UI's list
of needles that haven't been matched for a long time, but with
some manual tweaking (some are actually still needed).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This somehow snuck in (in the wrong directory) with the ppc64
merges, but it's not needed. I've verified with the admin tab
for checking on needle use; this needle has never been matched,
and all the ppc64 text install tests match the non-variant
needle just fine. So I'm removing this unneeded variant.
The way GNOME Software displays an error after running into RHBZ
1314991 has changed, it seems, so we need a variant needle to
cope with that. Also, when an upgrade notification is visible,
the 'Restart & Update' button for doing a regular update is
shown in grey (not blue), so we need a variant needle to handle
that too.
Because "Rescue Mount" now replaced by "Rescue Shell" string
in expected rescue screen head.
Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
The font Firefox uses when we don't ensure dejavu is installed
seems to bounce around a bit, so let's ensure the dejavu fonts
are there before we start Firefox. Also update a needle for
this.
Seems like it doesn't display "Initial setup for Fedora 27" any
more, so the old needle doesn't match. We should look into
whether that's a bug, but for now, let's update the needle.
Matching on the user name really isn't doing much good. It just
means we need more variant needles. Let's ditch that part of the
match and just match on the distinctive character sequence ~]$,
which doesn't really occur for any other reason. With this we
can drop the separate 'qwerty' needle (since the qwerty case
will match the regular needle now) and should also handle the
FreeIPA tests that are failing in Rawhide because a logged in
FreeIPA user doesn't just have a sh prompt now.
We can deal with this annoying bug by looking out for the error
we see when it happens, hitting the 'refresh' button again, and
resetting the loop counter to 1 (requires changing the loop to
a C-style loop).
This is required because anaconda is still checking for it
even if not mandatory. Already tracked by bug
https://bugzilla.redhat.com/show_bug.cgi?id=1172791
Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
related to known bug
https://bugzilla.gnome.org/show_bug.cgi?id=771127
like other needles with same bgo#771127 reference
do not set it as Workaround needle.
Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
* New OFW variable to identify Open Firmware (used by PowerPC)
* Few needles changes for PowerPC support
* as requested do not change the timers value below for PowerPC
tests/install_source_graphical.pm (300 to 600)
tests/_boot_to_anaconda.pm (300 to 1200)
This will be handled by TIMEOUT_SCALE in templates
Signed-off-by: Guy Menanteau <menantea@linux.vnet.ibm.com>
There's a bug in current Rawhide causing sourcing of /etc/bashrc
to fail when logging in as a regular user. This results in the
bash prompt looking different, which is currently a hard fail,
and causes most tests to die. It's better to treat this as a
soft fail so the rest of the test can run. So add a needle to
spot this case, and a little finish function the console login
function calls whenever it's successfully logged in, to check
whether it got the no-profile prompt and register a soft fail.
That other one didn't help, so let's try this - try and spot if
the spoke is in the unexpected state (the needle should only
match if the spoke is done processing and still in warning
state, it shouldn't match while the needle is still thinking)
and click through it again if so.
Well, that OCR needle isn't working out so great, as it seems
to match when it shouldn't:
https://openqa.fedoraproject.org/tests/119217#step/_graphical_wait_login/5
So let's try another approach. Ditch the OCR needle and have a
function for checking we're at a clean desktop. It does the
normal needle match, but if we're on GNOME, it also tries
hitting alt+f1 and seeing if we're at the overview; if so, it
hits alt+f1 again (to go back to the desktop) and returns.
https://openqa.fedoraproject.org/tests/117131 shows a spurious
match of the anaconda warning_bar needle - it's matching in an
ad for LibreOffice. Add a bit of the grey above the bar, so
hopefully this won't happen any more.
The F26 background does not change with the time of day, so that
needle should serve as-is, it's just no use for Rawhide. For
Rawhide, let's try a needle with an OCR match on 'Activities',
plus a tiny match area for a pure white block (expected to be
matched in the user menu area), since os-autoinst requires all
needles to include a match area for some reason.
These are no use :/ The Rawhide default background is a 'changes
with the time of day' one, and with the transparent top bar,
that makes a proper needle match impossible to reliably achieve,
as there is not a single pixel of the 'clean desktop' which
will always be the same any more. We'll have to come up with
something else. I'm trying out the use of an OCR match now.
The GNOME top bar is now transparent in Rawhide, which broke the
'clean desktop' needle. It also means it'll break every time
the background changes, from now on :( Update the needle.
Seems like the top '[patch]' link on kernel.org may now be a
raw patch file which the browser just displays, not a compressed
patch it offers to download. So tweak the needle so we should
click on the *bottom* link instead. openQA looks for matches
as close as possible to the location in the needle.
When running universal tests with a non-Server image, which is
not common but *does* happen (e.g. manually built test image),
the pre-selected filesystem in blivet-gui will be ext4 not xfs
(just as with custom partitioning). So we need xfs and ext4
variants of this needle. Renamed the XFS one for consistency.
Summary:
This adds a new test suite, run for Workstation and KDE live
images, which does not create a user during install. It then
expects initial-setup (KDE) or gnome-initial-setup (Workstation)
to appear after install, creates a user, and proceeds with
normal boot.
Note the ARM image test already covers the initial-setup text
mode, and the ARM minimal image is the only case where that
actually matters (it's not included in Server).
Test Plan:
Run the new tests, check they work. Run all old
tests, check the changes didn't break them.
Reviewers: jsedlak, jskladan
Reviewed By: jsedlak
Subscribers: tflink
Differential Revision: https://phab.qa.fedoraproject.org/D1185
They got rid of the 'Dashboard' text we were matching on, so
let's change this needle. This 'Hardware' text should show up
in all cockpit versions, I think.
Summary:
This is to handle cases like #1414904 , where the system boots
to emergency mode. We really need logs to try and debug this.
Test Plan:
Force a test to hit emergency mode somehow (right now
you can just run base_services_start on Rawhide over and over
until you hit #1414904, but there's probably an easier way to
do it, I think there's a systemd boot arg to tell it which target
to boot for e.g.) and check logs get uploaded. Also check this
doesn't break log upload for a 'normal' failure.
Reviewers: garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames
Reviewed By: garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames
Subscribers: tflink
Differential Revision: https://phab.qa.fedoraproject.org/D1103
openQA has some problems handling needles with identical names
in different subdirectories. We haven't had the cycles to send
fixes for this yet, so for now, let's just rename all such
needles we have.
Not sure if the color change will occur in Fedora 25 nightlies
or not, so not sure if we can remove the old needles. But we
*can* remove the F24 needles now I think.
Summary:
Since 26.17, anaconda shows a warning when the user password
contains non-ASCII characters, and requires a second Done click
to confirm. This change should handle that.
On the 'catch cases where password typing went wrong and re-try'
bit: to keep that, but not re-type the password *every single
time* on the Russian install test, we'd have to make the needle
match the text of the warning. This is problematic because then
that needle will be able to break without us easily noticing;
that's why I wanted to keep the 'warning bar' needle text-free.
Unfortunately, that means we have to skip the protection for
switched-layout installs.
Note the protection was actually not working for any non-English
install anyhow, because the needle had `LANGUAGE-english` as a
tag. We never noticed that. Failed password typing is pretty
rare now, so we can live without the protection - it's just nice
to have it for the English install tests because there's so many
of them.
Test Plan:
Run the Russian install with a recent Rawhide image,
check it clicks 'Done' twice. Note, it will still fail, because
of RHBZ #1413813.
Reviewers: jskladan, garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames
Reviewed By: garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames
Subscribers: tflink
Differential Revision: https://phab.qa.fedoraproject.org/D1084
The F25 'desktopencrypt' images built by createhdds boot to a
console passphrase entry screen, not graphical. F24 images
built the same way boot to a graphical passphrase entry screen.
I'm not sure why, but it's not really worth spending a lot of
energy on, I don't think - let's just add a needle to cover
this case.
Summary:
Include some basic testing of Japanese input, and split the
input testing (including Russian) into a separate module, since
it's not really part of 'login' testing.
Test Plan:
Run the test, and the Russian and French tests too to
make sure they didn't break. Tested on staging. Note the Japanese
test soft fails, intentionally, at present, as I discovered a bug
while working on it:
https://bugzilla.gnome.org/show_bug.cgi?id=776189
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1072
Summary:
This isn't in the criteria, but it's commonly used, so we ought
to test this way. Require authentication for the iSCSI target
and have the test provide the appropriate auth info.
Test Plan:
Run the iscsi test and check it works (you need the
recent fixes for support_server to make *that* work). Nothing
else should be affected.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1070
Summary:
The non-English tests so far did not test that graphical login
worked as expected, which is a fairly large hole. With this
change, they should do a Workstation install and test login to
both GNOME and the console works as expected. KDE is not yet
tested.
As part of this we tweak the implementation of keyboard layout
switching in graphical environments to use a generic function
in main_common which can handle both anaconda and desktops
(just GNOME at present, but should extend easily to any desktop
with a known switcher key and a visible layout indicator),
replacing the anacondatest class method. I kinda don't like that
the test has to specifically tell the function when it's in
anaconda, but I don't think I want to start experimenting with
a global 'test phase' openQA variable or anything like that at
present.
Fixes T842.
Test Plan:
Run the French and Russian install tests and check
they work as expected. Also run an English Workstation install
if you like, and make sure that didn't break. This change is
live on staging ATM, seems to work fine.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Maniphest Tasks: T842
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1071
Summary:
GNOME's update notification criteria are pretty braindead: it
fires the update check timer once on login then once every hour
thereafter, but only actually checks for and notifies of updates
once a day if it's after 6am(?!?!?!). So we have to do a bunch
of fiddling around to ensure we reliably get a notification.
Move the clock to 6am if it's earlier than that, and reset the
'last update check' timer to 48 hours ago, then log in to GNOME
after that.
Note: I thought this still wasn't fully reliable, but I've looked
into all the recent failures of either test on staging and
there's only one which was really 'no update notification came
up', and the logs clearly indicate PK did run an update check,
so I don't think that was a test bug (I think something went
wrong with the update check). The other failures are all 'GNOME
did something wacky', plus one case where the needle didn't quite
match because I think the match area is slightly too tall; I'll
fix that in a second.
Test Plan:
Run the tests on both KDE and GNOME and check they
work properly now (assuming nothing unrelated breaks, like KDE
crashing...)
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1039
OK, we once again hit an annoying case where the button match
that doesn't include text got matched in the sidebar:
https://openqa.fedoraproject.org/tests/39576#step/_do_install_and_reboot/13
so I think it's time to give up on that approach and just go
ahead and deal with the button text varying between live and
traditional installs. So let's change the existing needles to
include the button text and add one new needle to cover the
English live F25/F26 case. I don't think we need a 'quit'
needle for F24 (as the only thing we test for F24 is the Atomic
installer image, which says 'Reboot') or for French or Russian
(as we only test a traditional installer image for those, not
a live image).
Summary:
I started out wanting to fix an issue I noticed today where
graphical upgrade tests were failing because they didn't wait
for the graphical login screen properly; the test was sitting
at the 'full Fedora logo' state of plymouth for a long time,
so the current boot_to_login_screen's wait_still_screen was
triggered by it and the function wound up failing on the
assert_screen, because it was still some time before the real
login screen appeared.
So I tweaked the boot_to_login_screen implementation to work
slightly differently (look for a login screen match, *then* -
if we're dealing with a graphical login - wait_still_screen
to defeat the 'old GPU buffer showing login screen' problem
and assert the login screen again). But while working on it,
I figured we really should consolidate all the various places
that handle the bootloader -> login, we were doing it quite
differently in all sorts of different places. And as part of
that, I converted the base tests to use POSTINSTALL (and thus
go through the shared _wait_login tests) instead of handling
boot themselves. As part of *that*, I tweaked main.pm to not
require all POSTINSTALL tests have the _postinstall suffix on
their names, as it really doesn't make sense, and renamed the
tests.
Test Plan: Run all tests, see if they work.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1015
When a package is unsigned, KDE will prompt for authentication.
Let's handle this. But count it as a soft fail, because
puiterwijk claims that Rawhide packages will be autosigned
from next week, so this *should* not happen and would indicate
an unsigned package in the repos. We make the KDE 'update
complete' needle area smaller because the wider area includes
some transparency and so will only match if the update applet
is open; this area will match whether it's open (no auth case)
or not open (auth case - the applet seems to disappear after
you provide the password in the auth prompt).
Pushing without review as the test is in production so I want
to make sure it works correctly.
(Also, hey, check out that array match for assert_screen and
that match_has_tag! This is gonna make some things so much
easier...thanks upstream)
Summary:
this more or less covers desktop_error_checks and desktop_
update_notification, though it can't really distinguish
between them easily. All we know is that if both the live and
postinstall versions of this test pass, both of those tests
pass. Any fails will have to be investigated manually.
Test Plan:
Run the tests for both KDE and Workstation, see
what happens. Workstation will fail for F25 and Rawhide at
present, due to SELinux/abrt notifications.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1004
Summary:
Very similar to the CLI update test, but using the desktops'
update applications. This is based off the CLI update test
branch as it uses the shared functions that branch introduced.
We do not use the fake update packages, as they don't really
do anything useful for these tests; for dnf they can help us
distinguish between issues with the dnf mechanism and issues
with the repos, but we can't really tell that in the graphical
case. So we only use the python3-kickstart package here.
Test Plan:
Run the test on both KDE and GNOME and ensure it
performs as intended. I've been testing it on staging, so you
can see it there.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1010
Summary:
by waiting for the bootloader in _boot_to_anaconda rather than
_console_wait_login, we can ensure that we use the anaconda
post-fail hook and thus get logs uploaded when a kickstart
install fails.
Test Plan:
Run a kickstart install test that fails and check
anaconda logs get uploaded. Then run one that works and make
sure it...still works.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1005
seems to affect the network spoke a lot but not anywhere else.
Odd. Anyhow, here we go. Keeping the old needles for now as
F25 still has the old rendering.
Summary:
I've been wanting to do this for a while, but
https://github.com/os-autoinst/openQA/issues/786 is making it
difficult. Still, I think the quantity of needles is slowing
down openQA, so I'm doing it the old-fashioned way - looking
through test results and seeing what needles are actually used
now.
Test Plan:
Run full test suite for 24 Atomic, 25 and Rawhide and
make sure all tests still work. This is currently deployed on
staging (along with key-fixes) and I'm testing it there.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D994
Summary:
the main thing this does is try and type slower in X - this
should cover nearly everywhere we type anything in X, and make
it type slower. We also add a bit more safety checking to some
old tests which didn't have it (mainly _do_install_and_reboot)
- wait_still_screen after typing to make sure all the keypresses
were registered before continuing.
This is an attempt to mitigate the problems we've seen where
the wrong text gets typed into the wrong places and the tests
break.
This branch is live on staging atm. It still has *some* issues,
but I do think it's an improvement.
Test Plan:
run the tests (probably several times), compare to
runs without the change, see if it's better or worse...
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D993
so what seems to be going on on the software selection screen
is some kind of GTK+ bug:
https://bugzilla.gnome.org/show_bug.cgi?id=771127
the radio button bullets don't always render correctly - I think
they're not always completing a transition they go through on
selection. I think they can get stuck in any state between 'grey
and small' and 'black and big', but for now these are the needles
I've managed to create from failures; we're missing a 'greyer
KDE' variant, but if that happens, we can add it. If the bug
gets fixed we should be able to drop all these.
these rarely get used any more. they're only used when we're
running the filesystem tests on a non-Server image, where the
default fs is ext4 not xfs. With the old nightly process, where
we only got a generic boot.iso, we used them all the time, but
now we get product-ized nightlies, we rarely do. But I did hit
them today running the universal tests with a generic boot.iso
I hand-built to test an anaconda update, so in case we need to
do that kind of thing again in future, we may as well update
the needles.
Summary:
we have a long-standing problem with all the tests that hit
the repositories. The tests are triggered as soon as a compose
completes. At this point in time, the compose is not synced to
the mirrors, where the default 'fedora' repo definition looks;
the sync happens after the compose completes, and there is also
a metadata sync step that must happen after *that* before any
operation that uses the 'fedora' repository definition will
actually use the packages from the new compose. Thus all net
install tests and tests that installed packages have been
effectively testing the previous compose, not the current one.
We have some thoughts about how to fix this 'properly' (such
that the openQA tests wouldn't have to do anything special,
but their 'fedora' repository would somehow reflect the compose
under test), but none of them is in place right now or likely
to happen in the short term, so in the mean time this should
deal with most of the issues. With this change, everything but
the default_install tests for the netinst images should use
the compose-under-test's Everything tree instead of the 'fedora'
repository, and thus should install and test the correct
packages.
This relies on a corresponding change to openqa_fedora_tools
to set the LOCATION openQA setting (which is simply the base
location of the compose under test).
Test Plan:
Do a full test run, check (as far as you can) tests run sensibly
and use appropriate repositories.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D989
Summary:
the dictionary error bug was fixed some time back, so drop this
workaround for it.
Test Plan:
Run all tests for F25 and Rawhide and verify they don't need
this workaround any longer.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D988
this needle can actually also match the 'Delete all' button,
resulting in the test not doing what it should. openQA should
prefer the match closer to the area's location in the needle,
but https://openqa.fedoraproject.org/tests/31136 seems to show
this not happening, so let's make the match area wider in all
versions of the needle so it should no longer match the Delete
all button.
font rendering has changed somehow in both Rawhide and F25
updates-testing, retake a bunch of needles for this. Some more
may be needed later in tests that are currently failed or
skipped for other reasons.
this isn't actually f25 specific, I don't think...the tests
that failed are the Workstation network install tests, which
use updates-testing, so I think they're getting the same font
rendering change as just showed up in Rawhide. So these should
actually be correct for both branches, let's just leave the
name as the date.
looks like some font change occurred at some point, this test
hasn't reached this point for a long time so I can't really pin
down what changed, so just name it for the date.
seems like the overlay of the little 'speech bubble' triangle
changed a bit, with a recent GTK+ or something I guess. Since
this test is quite new we shouldn't need to keep the old
version, so just replace the screenshot.
Summary:
pretty simple stuff here. The distinction between 'firefox' and
'browser' is that the 'browser' needles I expect would also be
correct for other default browsers, while the 'firefox' needles
are specific to Firefox. We need '-kde' variants of some Firefox
needles where interface text is included, because the font is
Cantarell in GNOME but whatever the default 'sans' font is in
KDE - I suppose we should really use -thatfontsname rather than
-kde, but I can't think what it's called...
I couldn't do the 'log in to FAS' bit of the test since we don't
really have a sane way to provide a password while not exposing
it to the public.
Test Plan:
Run the test, check it works - for both KDE and
Workstation.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D938
Summary:
again, added as a non-fatal module for realmd_join_cockpit as
it's convenient to do it here. Also abstract a couple of ipa
bits into a new exporter package in the style of SUSE's
mm_network, rather than using ill-fitting class inheritance as
we have before - we should probably convert our existing class
based stuff to work this way.
Also a few minor tweaks and clean-ups of the other tests:
The path in console_login() where we detect login of a regular
user when we want root or vice versa and log out was actually
broken because it would 'wait' for the result of the 'exit'
command, which obviously doesn't work (as it relies on running
another command afterwards, and we're no longer at a shell).
This commit no longer actually uses that path, but I spotted
the bug with an earlier version of this which did, and we may
as well keep the fix.
/var/log/lastlog is an apparently-extremely-large sparse file.
A couple of times it seemed to cause tar to run very slowly
while creating the /var/log archive for upload on failure. It's
no use for diagnosing bugs, so we may as well exclude it from
the archive.
I caught cockpit webUI login failing one time when testing the
test, so threw in a wait_still_screen before starting to type
the URL, as we have for the FreeIPA webUI.
I also caught a timing issue with the openQA webUI policy add
step; the test flips from the Users screen to the HBAC screen
then clicks the 'add' button, but there's actually an identical
'add' button on *both* screens, so it could wind up trying to
click the one on the Users screen instead, if the web UI took
a few milliseconds to switch. So we throw in a needle match to
make sure we're actually on the HBAC screen before clicking the
button.
We make the freeipa_webui test a 'milestone' so that if the
new test fails, restoring to the last-known-good milestone
doesn't take so long; it actually seems like openQA can get
confused and try to cancel the test if restoring the milestone
takes a *really* long time, and wind up with a zombie qemu
process, which isn't good. This seems to avoid that happening.
Test Plan:
In the simple case, just run all the FreeIPA-related
tests on Fedora 24 (as Rawhide is broken) and make sure they all
work properly. To get a bit more advanced you can throw in an
`assert_script_run 'false'` in either of the non-fatal tests to
break it and make sure things go properly when that happens (the
last milestone should be restored - which should be right after
freeipa_webui, sitting at tty1 - and run properly; things are
set up so each test starts with root logged in on tty1).
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D935
Summary:
This requires us to handle decryption each time we reboot in
the upgrade process, so factor that little block out into the
base class so we don't have to keep pasting it. It's also a
bit tricky to integrate into the 'catch a boot loop' code we
have to deal with #1349721, but I think this should work. There
is a matching openqa_fedora_tools diff to generate the disk
image.
Test Plan:
Run the tests, check that they work, run the other
upgrade and encrypted install tests and check they still work
properly too.
Reviewers: garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D922
Summary:
as a new, non-fatal test step in the cockpit enrolment test,
because it kinda fits in there; we have an enrolled system with
a web browser *right there*. This will require making the wiki
reporting stuff slightly cleverer so we can say 'report a pass
for this wiki test instance if this test step passed', but that
should be possible. Making this non-fatal means the rest of the
cockpit enrolment test will go ahead even if the freeipa web UI
fails.
The 'check if we can log in' stuff is identical to freeipa_
client_postinstall except with different user names, so we could
potentially factor that out somehow, but I couldn't think of a
super clean way to do it so for now it's just copied.
Note this diff is on top of the freeipa-realmd branch which
is for D894, it's not on top of develop.
Test Plan:
Run the modified test and see if it works. No other
tests are modified, so they should be OK.
Reviewers: garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D895
now we're getting composes with the product name 'Fedora-Atomic'
this needle doesn't match them. Tweak it so it should match both
'Fedora' and 'Fedora-Atomic'.
Summary:
Set up the support server to provide DHCP/DNS functionality and
an NFS server, providing a kickstart. Add a kickstart test just
like the other root-user-crypted-net kickstart tests except it
gets the kickstart from the support server via NFS. Also add NFS
repository tests and a second support server for Server-dvd-iso
flavor: this test must run on that flavor to ensure that packages
are actually available. The support server just mounts the
attached 'DVD' and exports it via NFS.
Note we don't need to do anything clever to avoid IP conflicts
between the two support servers, because os-autoinst-openvswitch
ensures each worker group is on its own VLAN.
As part of adding the NFS repo tests, I did a bit of cleanup,
moving little things we were repeating a lot into anacondatest,
and sharing the 'check if the repo was used' logic between all
the tests (by making it into a test step that's loaded for all
of them). I also simplified the 'was repo used' checks a bit,
it seems silly to run a 'grep' command inside the VM then have
os-autoinst do a grep on the output (which is effectively what
we were doing before), instead we'll just use a single grep
within the VM, and clean up the messy quoting/escaping a bit.
Test Plan:
Run all tests - at least all repository tests - and
check they work (make sure the tests are actually still sane,
not just that they pass). I've done runs of all the repo tests
and they look good to me, but please double-check. I'm currently
re-running the whole 24-20160609.n.0 test on staging with these
changes.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D888
Summary:
this is following a SUSE model for tests where we need a server
end but don't want setting up the server to constitute a real
test in itself, we want it to be stable. The 'support_server'
test just boots a pre-built (by createhdds) disk image, sets up
networking, and runs the iSCSI server.
To run the iSCSI test we need to handle networking config in
anaconda (or we would need to set the support server up as a
DHCP server, which may be worth considering), so this adds that.
We also need to be able to specify the target device for a
volume in custom partitioning, so this adds that too.
Test Plan:
Build the necessary support server disk image (use
D883), then run the test and make sure it works. Also make sure
all other tests continue to work.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D884
