These tests weren't doing it, I guess it's just an oversight;
this is probably why they often fail on slow repos.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This makes the two rpm-ostree tests written for IoT - overlaying
and rebasing - work across all rpm-ostree-based flavors we
currently test (IoT, CoreOS and Silverblue) and runs them on
all those flavors.
This requires some other changes. For the Workstation ostree
installer update tests, we have install_default_update_ostree
upload a disk image and run these tests on that image. That means
install_default_update_ostree cannot use a scratch disk (as if
we boot it with two disks but only upload one, the subsequent
tests fail to boot, looking for the missing second disk), but its
specified disk size should be large enough for all updates.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
What's supposed to happen here is the `do_bootloader` invocation
a few lines back boots to the installer, then here, we wait for
the install to complete and the system to reboot, and match the
bootloader again. However, on PXE installs, the bootloader screen
can hang around for quite a long time here, and if it does, we
can match it again before the installer starts up, and move on
too early. Hence the sleep.
It seems on current Rawhide 20 seconds isn't long enough - we're
still matching the installer bootloader after the sleep, see
e.g. https://openqa.stg.fedoraproject.org/tests/2431660#step/_boot_to_anaconda/3
This is causing the test to almost always fail (it'll only pass
if the install+reboot takes less than five minutes). Let's bump
it to 60 seconds and hope that's long enough.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Sadly, dropping this sleep caused the test to start failing
again at least on F36, so we still need it - update the note.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Thankfully this all calmed down a bit so we can simplify it a
lot. Clean things up a bit at the same time; escaping nested
single quotes is a lot clearer than concatening blocks with
different quote marks.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The bug seems to have gone away, at least I don't see that this
soft failure has been hit much for the last two months.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It's been on 1 so long now I kinda don't want to change it to 3
or 4 or anything. That might break something. As long as it's not
causing any trouble let's just leave it on 1.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
PXE install on UEFI (incl. aarch64) is failing at present, this
seems to be due to a grub bug:
https://bugzilla.redhat.com/show_bug.cgi?id=2152763
we're really intending to test the client side here, not the
server end, so let's work around this problem on the server end
by installing a grub2 scratch build that's the package from just
before the bad change, but with the release and epoch bumped,
from a side repo.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This turns out to be overcomplicated. We don't need the special
handling for updates any more, because a few months after we
implemented it, we had to make sure the affected update tests had
an empty START_AFTER_TEST anyway, or else openQA would refuse to
schedule them. So we can just rely on the START_AFTER_TEST
condition for those now. We also don't need the additional
INSTALL_NO_USER condition; the only case where it's actually used
is for install_arm_image_deployment_upload on Workstation, and
that test does not have START_AFTER_TEST set, so the other
condition catches it for welcome screen handling purposes. There
is no need to nest the IMAGE_DEPLOY conditional inside a check
for the desktop and the INSTALL_NO_USER var either; we don't
test any other desktop on ARM, and the IMAGE_DEPLOY var is only
set for that one install_arm_image_deployment_upload test.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We recently started using the buildroot repo for Rawhide update
tests, but weren't including it in the image build tests. This
should include it in all the image build tests.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We currently snapshot after every run of _console_wait_login or
_graphical_wait_login, which means we snapshot *twice* on most
update tests as those modules get run twice. However, we almost
never use those snapshots. Snapshotting takes quite some time,
and hits the disk pretty hard, so we should avoid it unless it
is really needed.
We only have a few modules that are not fatal (and so might use
the snapshots), and most of those don't run after one of these
tests, or run after a later module that's also a milestone. Best
I can tell, only two test suites really need to use a snapshot
from a login test: server_cockpit_updates and modularity_tests.
To handle these and potential future cases, we'll add a new
module that does nothing, but is marked 'milestone', so it will
take a snapshot, and load that test after the login test if the
var LOGIN_SNAPSHOT is set, and set that var for those two suites.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We started using this in real composes a year or two back, so
openQA should do the same. It drops the nesting of an ext4 fs
image inside a squashfs image, just using a single squashfs
image instead. This results in smaller images - missing this
is why the images built by openQA were coming out larger than
the real ones.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
* Scarborough provided quite a messy map that resulted
in frequent needle failure. Changing the location
for something better to make it more reliable.
* The zoom test could have failed with a low resolution
image. Adding some timeout to the needle give more
time to load the proper image.
The check_screen function checks for the existing tag
but it only waits 1 second by default. In this time,
Abrt will not even start so we need to prolong
the check_screen timeout to make sure the application
has started (or at least give it enough time to try).
Instead of just redirecting it to a log file, let's tee it, so
simple errors can be read off a screenshot without bothering to
download the file. Also, let's timestamp it (via `ts` from
moreutils) so we can see which bits of it take a long time...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is like the existing tests that build network install and
live images then install them, only for Silverblue. First we
build an ostree, using the standard configuration for the release
and subvariant but with the 'advisory' and 'workarounds' repos
included, so it will contain current stable packages plus the
packages from the update and any workarounds. Then we build an
ostree installer image with the ostree embedded, again including
advisory and workarounds repos in the installer build config so
packages from them will be included in the installer environment.
The image is uploaded, which completes the _ostree_build test.
Then an install_default_update_ostree test runs, which does a
standard install and boot from the installer image.
We do make a change that affects other tests, too. We now run
_advisory_post on live image install tests, as well as this new
ostree install image install test. It was skipped before because
of an exception that's really only needed for the netinst image
install test. In that test, packages from the update won't be
included in the installed system, so we can't run _advisory_post
on it. But for ostree and live image build/install tests, the
installed system *should* include packages from the update, so
we should check and make sure that it does.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There's one point in the tests where we may log into cockpit for
the second time in one run (it depends how a package update
process goes). When this happens, we don't get prompted again
for admin access, so we need to *not* expect that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This reverts commit 3208d15725 and
the two follow-ups. I'm hoping
https://bugzilla.redhat.com/show_bug.cgi?id=2133829 is now
resolved; this was intended to help with that (though I'm not
sure it ever really did), and so we can hopefully ditch it, which
simplifies this code.
This reverts the last few commits which worked around a focus bug
in GTK. This bug is now (I hope) fixed, so I'm dropping the
workarounds so the tests will confirm whether it's fixed.
We have a big problem with Rawhide KDE update tests getting OOM
killed during this phase. Stopping the desktop before we install
updates should save some RAM and help avoid this.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Boy this seems slow in Rawhide currently. This has the effect
of being more defensive around the services page load.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We've seen some failures of the weather test at the start of
weather_report, where the test expects to be at the hourly view,
and instead it seems to be at a sort of broken state:
https://openqa.fedoraproject.org/tests/1505080#step/weather_report/3
I'm guessing this may be because currently aaa_setup clicks the
city name then is immediately complete, so it will immediately
snapshot. I guess this can result in things being stuck in a kind
of intermediate state on snapshot restore. So, to try and avoid
this, let's assert that we reach the hourly view after clicking
the city name, then wait_still_screen for a few seconds to make
sure things are settled down, before we complete and snapshot.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We seem to be hitting very long loads on the Services and Logs
pages of Cockpit in recent Rawhide testing especially. As I don't
have time to deeply debug this at the moment, let's just give it
longer (but make it a soft failure when it takes longer than
expected).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Sometimes the windows are displayed in the reversed order, which
prevents the checks to find the needles and the test fail
even if it should pass. This change should address this case.
We use variables to track test state across modules, sometimes.
As this is all internal to the test logic I didn't bother always
making these variables upper-case, but os-autoinst now treats
lower-case variables as a fatal error, so we have to change.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The change to read-only sysroot for IoT in F37 causes problems
with this rebase test. It's not supported to rebase from an RO
release (37 or 38) to a non-RO release (36). So we need to make
sure we don't try and do that. This uses some quick hack logic,
but it should be OK and sufficiently specific not to break
anything even if we forget to remove it in future.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It seems we still have trouble with this turned on :( About 60%
of tests fail with the client unable to resolve names.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The cluster of bugs for F37/Rawhide should all be resolved now,
and I'm hoping the old upgrade bug is no longer relevant.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This issue appeared when we started testing Rawhide updates, but
I only noticed it today. When testing Rawhide updates after
Branch point, the upgrade tests upgrade from Branched to Rawhide.
On Branched, updates-testing is enabled by default. We only
disable it when we reach `upgrade_run`, but by that point we've
already done a `dnf -y update` in `upgrade_preinstall` and
potentially installed other packages in steps between
`upgrade_preinstall` and `upgrade_run`. That can cause problems,
like today all FreeIPA upgrade tests on Rawhide are failing
because there's a newer freeipa in updates-testing for F37 than
is in the current Rawhide compose.
Solve this by disabling updates-testing before we do the update
in `upgrade_preinstall`. To avoid excessive code duplication,
factor out the repo disabling code.
We'll do this twice on upgrade tests now, but it shouldn't be a
problem.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Also stop re-doing get_var("DESKTOP") because that's dumb. This
should only, at worst, make things slower if unexpected things
happen - it shouldn't cause failures that wouldn't happen anyway.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This will make them slower, but lately type_safely is just not
reliable, particularly in the new_file test, it's constantly
typoing.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Rawhide live image builds are still taking an awful long time
and often failing. I will look more into why later, but for now,
let's bump the timeouts even more just to try and get through
the job backlog.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I'd prefer not to have to do this, but having the tests fail on
every compose and Rawhide update test is just too distracting
to live with.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
From anaconda-37.12.1, anaconda defaults to GPT for all BIOS
installs. So we need to create a BIOS boot partition when doing
a BIOS install. I think all other potential configs (x86_64
UEFI, aarch64 (UEFI), ppc64le (OFW)) are covered under the other
two paths, so just making this `else` should be OK.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We need to move the mouse out of the way so we don't need two
needles for "X not highlighted" and "X highlighted", and give
the check_screen a few seconds to update for the cursor move.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
"Star" was removed from the file context menu, so we have to
star the file from the main view now.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We had a ton of needles all covering something very similar
(press a "Credits" button in a GNOME app). There are about four
real variations: old-style regular face white-on-black (eog),
old-style regular face (nautilus and evince before recent
libadwaita ports), old-style bold face (GTE and Clocks before
new libadwaita), and new-style (everything that's been ported
to use libadwaita for its About page). Let's just rationalize
it down to those, using the same needle tag for all of them.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The city is called Reykjavík (note the í). Previously our search
worked because the search function also looked in the name of the
timezone, and the *timezone* name is "Atlantic/Reykjavik" - i.e.
it's really Latin-ized in the name of the timezone. However,
upstream intentionally stopped including the timezone name in
search matches:
https://gitlab.gnome.org/GNOME/gnome-clocks/-/merge_requests/199
so that doesn't work any more. Just searching for "Reykjav"
should solve the problem.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
In 43-alpha it's gone, the replacement is Troubleshooting ->
Debugging Information. But it would be a pain to write two
forks in the code for handling both cases, and there's a ton
more stuff in the new-style About dialog that we're not
checking either. I don't think we really need to click on all
of it, and this bit of it isn't super important. So on the
whole I'd rather just keep things simple and drop this check.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
the new_file test failed today because it seems g-t-e now pre-
fills a suggested filename, with extension, and pre-selects the
name part but not the extension part. So when we type 'list.md'
we wound up saving the file as 'list.md.md'. I think hitting
ctrl-a should fix this, and not break when run on older versions
of g-t-e if we ever do that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
aarch64 looks like it often needs more time to settle after
restoring from snapshot before trying a key combo.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
That last commit to 'fix' the Clocks tests when Silverblue needs
location access to be granted wasn't complete, I left the needle
out. D'oh. Take the chance to give it a better name too.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
If there are too many categories, we don't see the Updates entry
on the left, and this has been breaking Rawhide and F36 tests.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This PR changes the way to download the test data into the VM.
Although it does not use a disk image as suggested in one
of the review, it does not clone the entire repository, but
a simple tar.gz file that holds the data which will be
distributed into the directory structure.
This way, the amount of data needed to be downloaded dropped
from approximately 50MB to below 2MB.
Also, the existing test suites were adapted to this situation.
Clocks' aaa_setup did not ever actually check the app launched
properly. It also doesn't handle granting permissions if
necessary, which the apps_startstop test for Clocks does do.
This makes the permission check in the apps_startstop test more
efficient, and adds it to the Clocks app aaa_setup test too.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This was just doing something silly before, but with recent
os-autoinst having function signatures, it actually causes the
test to fail because '5' isn't a sane value for the argument
this was setting before. Fix it to set the timeout, as it was
trying to do all along.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It seems when we quit Firefox back to a VT on ppc64le, the system
hangs. Not sure why, but we can deal with it by rebooting the
system and logging back in as root if it happens. Also take the
opportunity to clean up the flow of quit_firefox so we always
check that we get back to a console then wait 5 seconds for the
console to settle, so all the tests that call it can stop doing
that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
On Rawhide update tests we often don't seem to get to the login
prompt in 10 seconds, so tweak the code a bit to let us specify
a timeout in root_console, and use 30 seconds here.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Let's not have the reference files in one person's fedorapeople
space, in case that person leaves. Let's upload the text.txt
for checking (it's easier to be able to just read what's in it
than try and figure it out from the diff output), and let's use
diff -u because non-unified diff output is awful.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is the automation of the optional testcase https://fedoraproject.org/wiki/QA:Testcase_i18n_default_fonts.
The test implementation runs the same commands as the mentioned test
case and checks the expected output. It is designed to run in the scope
of postinstall tests when the language is set to "japanese".
I'd prefer not to do this, but I don't see a better way to deal
with the stupid 'welcome' screens. I'm not maintaining needles
to click them away forever.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Firefox 97 is now stable on all releases, so we can forget about
handling browser_download_save and just assume download will
happen automatically.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Up to F33 we could check that an error message was shown when we
tried to log in with the wrong password on GNOME, but since F34
it's transient and disappears too quick to reliably catch, so
we don't check it any more. Now F33 is EOL, drop the conditional.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The g-i-s new user mode was removed in F34, so we don't need to
do this any more (as the now-removed comment said).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Before F34 we had to launch the update tool from the systray.
This isn't the case any more, so we can throw all this code and
these needles away.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We seem to be solidly back to always getting a permanent update
notification in current F36/Rawhide, so we don't need this more
complex path any more. We also don't need these needles any more,
they haven't matched for months.
Logging of additional repo setup changed recently in F37, we
need to handle the new message format here.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2 seconds isn't enough to be sure it opened, which is causing
some weird failures in KDE Rawhide update tests.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
aarch64 tests are failing here because anaconda's still thinking
about the delete confirmation, it seems.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
When testing Rawhide updates we set VERSION to the Rawhide
release number, not "Rawhide". This post-upgrade version check
needs adjusting to handle that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Seeing a failure quite often lately where when we try and hit
ctrl-p to print, we just type a p. There's no wait between
maximizing and trying to print, and only a short wait on
launch, so let's try being a bit more defensive there.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This can actually take a bit of time, it seems, especially with
debug kernels. Let's give it a minute.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Lately anaconda can take up to 10 seconds to exit the root pw
spoke, which can defeat the subsequent `wait_still_screen` that's
meant to wait out the 'slide-in-from-the-top' animation of the
hub. So let's assert the hub after we click Done, then the still
screen wait will only happen *once the hub is visible* and should
really do its job.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
OK, this is annoying. GNOME Software intentionally does *not*
clear the 'download' or 'reboot and update' button when you hit
the refresh button, it just leaves them sitting there while the
refresh happens. So let's specifically require the 'refreshing'
text to appear and go away before we try and click on download
or apply.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Agh, GNOME's UI is *not* helpful here at all. The Apply button
remains visible for a long time after you hit Refresh.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is yet another twiddle to that same damn bit where we try
to apply updates. This more or less reverts the last tweak to
it, where we skipped hitting 'refresh' if 'download' or 'apply'
are already visible. The trouble with that is that the app may
have already found and prepared updates before we got our
"prepared" python3-kickstart update in place - so the update
operation might work perfectly, but not update the package we
expect it to update, and the test may fail.
This time, let's try *always* refreshing, then wait a bit after
hitting the refresh button before we start looking for apply
or download to try and avoid the 'race' we were trying to solve
with the last tweak (where we hit refresh then immediately try
to hit a download or apply button which vanishes before we can
hit it). I think this should be safe as both KDE and GNOME
should always show a refresh button now (this wasn't the case
before, I think, F34).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There's a bug in the Save As... dialog on the flatpak version
of evince currently where the existing filename is not pre-
selected, so when the test types 'alternative', it gets
prepended to the existing filename instead of overwriting it,
and we wind up with alternativeevince.pdf, not alternative.pdf.
Let's treat this as a soft failure rather than a hard failure.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
On upgraded systems, gedit might still be present; remove it so
we don't launch it by accident and try to test it instead.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Firefox 97+ don't ask you what to do with downloads any more,
they just...download them. For now we'll handle both workflows,
once 97+ is stable everywhere we can drop handling the old one.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
At the end of _podman_client we create a mutex that we want the
server to pick up. If we exit too soon, the mutex goes away and
the server test fails. 5 seconds turns out to be not enough of a
wait because, although the server retries the lock call every
5 seconds, it can hit `api_call_2 failed` and wait 10 seconds
before retrying. Let's wait 30 seconds just to be safe.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Otherwise, if there's an update to fXX-backgrounds pending, we'll
get it on reboot in the middle of the test, and the real
backgrounds will comes back and replace our black fake ones and
break the test.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
