The 'universal' flavor has been kinda pointless for some time
now. It dates back to the earliest days of openQA, before Pungi
4 was a thing, when composes were very different; we only built
a boot.iso and some live images nightly for Rawhide, these
weren't even formally grouped as a 'compose' at all (fedfind had
to invent the concept). The TCs/RCs had DVD installer images
(not *Server* DVD, at the time, just a universal DVD installer).
We wanted to run some tests on the DVD image if it was available,
but we still wanted to run them for the nightlies, so we invented
a whole mechanism for that - this 'universal' flavor, with some
complicated logic in fedora_openqa which schedules universal on
the 'best available' image it can find in the compose.
All this is functionally obsolete now. All composes we test are
now run through Pungi (except the live respins, but they aren't
relevant here). In current config, the Server DVD is non-failable
on x86_64 and aarch64, which means it will *always be there* -
if it fails to build, the compose itself fails, so we won't test
it. It's failable for ppc64le, but we don't care that much about
ppc64le; I'm fine with these tests just not running if the Server
DVD happens to fail in a ppc64le compose.
As a cherry on top, some of the 'universal' tests aren't really
universal anyway, they fail if you run them on a netinst (off
the top of my head, all the NFS install tests are like this, as
we use the ISO to populate the NFS share on the server end).
So let's just move all the tests that actually need an installer
image to the Server-dvd-iso flavor. Left over in the 'universal'
flavor are upgrade tests, which don't need an ISO at all - they
boot from hard disk images and run an upgrade using repos. We
can change the scheduler logic to be more simple for these, and
just always schedule them, with no ISO attached. We could even
rename this flavor 'upgrade', but it might not be worth it.
One slight complication is that the split happened to be helping
us avoid too many tests in a single support_server cluster; we
have a cluster of five support_server tests on Server-dvd-iso
and five support_server tests on universal. I try to avoid the
clusters getting too big as you need as many worker instances on
at least one worker host as your largest cluster; if you don't
have that many, the cluster's tests simply never get scheduled.
Requiring folks to have at least ten worker instances on one
host to run these tests is a bit of a big ask. So, to handle
that, we create a support_server_2 and have the former universal
tests use that one instead, so we'll have two separate clusters
on Server-dvd-iso now.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Upstream https://github.com/os-autoinst/openQA/pull/4973 requires
us to poke things here a bit. This only works with the newer
os-autoinst and openQA (there may be a way to conditionalize it
to work with both, but I can't be bothered figuring it out, let's
just update).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This keeps failing on the accessibility section, and looking at
the screenshots I realized why. When you press 'down', GNOME
doesn't just 'snap' to the new view, it does a smooth downward
scroll. We're often matching *while it's scrolling*, so the
needle match is right at the bottom of the screen. But then the
animation continues, so when we get to the click action (even
though we use click_lastmatch it's not *instant* in openQA),
the thing we're trying to click (the "Accessibility" section
title) is a bit further up the screen, and the click 'misses'.
So, we need to wait out the scroll then re-assert and click.
This unfortunately will make the test take about 30 seconds
longer, but I don't see another way to do it. We could maybe
shave the wait_still_screen to one second...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Similar to the dedicated tests for these apps, the app can appear
for a split second before the access request, so we match on the
app and don't realize we need to click through the access
request. Handle this the same way we do in the dedicated tests.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Another message changed format a bit, and all the messages are
now showing up in syslog instead of packaging.log, so handle
all possibilities here. I had to split the first check into two
commands because I can't seem to make it work if I try and do it
all in one command with bracket groups :/
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We actually get a softfail because we're expecting it; now it's
been fixed not to show up, drop the code that expects it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We're constantly seeing this test fail on an odd problem where
text editor starts *behind* characters. To handle this, check
if we see text editor and if not, hit alt-tab.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It always launches in basic mode anyway, and sometimes this key
press doesn't work right and leaves a stray 'b' in the entry
field, which messes things up when we get to the calculation
tests.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This group was only added to comps today, so it's not in the
comps in the rawhide repo yet (will be after next compose). The
Koji repo doesn't have normal comps so it's not there either.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Try and make sure maximize actually works - wait for still screen
after hitting Done and before trying to maximize.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There's a failure mode we hit quite often where, when we run
the text editor after copying the character, it pops up *behind*
the character page. So let's close the page first. When we click
the 'Copy Character' button a notification that the character was
copied is briefly displayed and if we hit esc while it's visible
we dismiss *that* not the character page, so hit esc twice to be
safe. If we miss dismissing the notification, the 'extra' press
is safe, it doesn't quit the app.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
A change to how anaconda handles NFS repos changed the log
messages we get when we use one. We may need further changes for
using NFS as a base repo when this change hits Rawhide nightly.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It looks like the desktop_notifications postinstall test on KDE
on F38 is failing currently because the notification is being
shown during the install_default_upload test that precedes it,
so KDE decides not to show it again yet. So, unset the setting
that stores the timestamp of the last time the notification was
shown. This is similar to a thing we already do for GNOME above.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is needed to force the rebase on current CoreOS (because it
has zincati registered by default). It should be harmless in all
other cases.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Reasoning:
1. pandoc is not in critpath so will not itself be tested
2. pandoc is widely used and actively maintained
3. package is noarch
4. package has minimal deps
Hopefully this will work for everything. For some reason, the
"use python3-blivet for pykickstart tests" fails mysteriously
sometimes, see e.g.
https://openqa.stg.fedoraproject.org/tests/2672282
Signed-off-by: Adam Williamson <awilliam@redhat.com>
See e.g. https://openqa.fedoraproject.org/tests/1829593
sometimes we see the app UI briefly before the access prompt
appears. Handle that case by waiting a few seconds and doing
the match again.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Somehow, the dummy package being python3-kickstart causes the
graphical update tests (only) to fail for pykickstart updates
(that's the source package of python3-kickstart). The CLI and
Cockpit update tests are fine with this and pass.
To workaround this, use python3-blivet as the dummy package for
the graphical update tests when testing an update that contains
python3-kickstart. I've updated the test repo to contain both
dummy packages.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This reverts commit 2fecb70468.
Sadly, clicking on the right menu entry...doesn't work. Let's
try going back to the old way, but add an 'enter' press once
the entry we want is selected.
Move the xauth disablement and the disabling of studies into
_setup_browser, instead of repeating it in a couple of other
places (but *not* doing it in the zezere test, where we should
be doing it). Drop some explicit package installs that should
no longer be needed as Firefox and/or X.org now depend on those
things. Install the current default fonts (Noto), not the old
ones (DejaVu).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There's a weird issue with the rpmostree_rebase test for this
update:
https://bodhi.fedoraproject.org/updates/FEDORA-2023-f6afa6f9e5#comment-2919613
it doesn't reproduce locally (I can type fine after doing the
same things the test does up to the rollback) and I can't think
of any possible cause, and I don't want to hold the update up.
So we're just gonna work around it and hope this doesn't start
happening to all F38 update tests after this goes stable. If it
does, we'll have to do the workaround for all of them.
The workaround is just to rollback and reboot 'blindly', instead
of checking the rollback command works. The drawback is that if
the rollback command fails we'll wait 7.5 minutes before giving
up on it, and it'll be a bit less clear exactly what happened.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Plasma 5.27.1 is going all the way back to F36 (in
FEDORA-2023-d7dcc38129), so we'll have a welcome tour on both
desktops we actually test on, on all supported releases. So we
can just drop the desktop conditional entirely here now.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There's this awkward path for the live image install tests on
updates. We run the 'are the correct versions of all the packages
installed' check on these tests to ensure the right versions
actually made it onto the live image. So we don't run
`dnf -y update` at the end of repo_setup_updates on that path,
because if we did that, even if the packages on the live image
were old, we'd update them there and hide the problem.
However, this causes a bit of an ordering issue, because in
order to set up the advisory repo, we need to install a few
packages. What if the update under test includes one of those
packages, or a dependency that wasn't already installed? In
that case, we wind up with the older stable version of the
package (because obviously we can't install the newer version
from the advisory repo *before we've set up the advisory repo*),
don't update it later, and so the 'correct version' check at
the end of the test fails. See:
https://openqa.fedoraproject.org/tests/1778707 for a case of
this happening with a python-cryptography update.
Up till now I was trying to handle this by just updating the
specific packages we install, but that doesn't account for
*dependencies* of them. I looked down the path of trying to
generate a list of all those dependencies and update all of
them but it looks a bit mad. So instead let's try this. On that
specific path, we'll generate the "all installed packages" list
*before* we run repo_setup, so it just doesn't include anything
that gets installed during repo_setup. The implementation is a
bit icky but not too horrible.
We *could* just *always* generate the all installed packages
list earlier, but then that would mean we *wouldn't* catch dep
issues in this kind of package on the other test paths, whereas
currently we do. I don't want to lose that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
5.27.1 is going to F37, and adds it. In the short term this
will waste a minute and a half and cause soft fails on all other
F37 updates until the update that adds this goes stable, but
I don't really feel like working around this, let's just live
with it till the update goes stable.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
OK, neither 'input' nor 'keyboard' actually gives us the Keyboard
pane, they both give results for uninstalled apps from Software
:(. 'hotkey' (which is one of the keywords in the .desktop file)
does seem to work, for now at least, let's try that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Contacts now has two burger menus, which is awkward. We need
specific needles to identify each, we can't rely on the generic
needle any more as it won't always open the right menu. We also
need to still work with the old UI for the flatpak.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
KDE has a welcome tour now, on F38 and Rawhide at least. Let's
"handle" it with extreme prejudice...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
On GNOME 44, typing 'input' is now giving us the Software page
for PulseAudio Volume Control, for some reason. Let's try typing
'keyboard' again and hope that works again now...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Logout seems to be taking a long time in Rawhide currently. Give
it longer to run, but soft fail. I'll add a bug link once I've
investigated and filed one.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
In today's F38 and Rawhide, changes to the persistent overlay
stuff result in a boot warning you have to spam through. Let's
handle this as a soft fail so we don't have floods of failed
tests till it's fixed.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Tested that this is not necessary on KDE or GNOME, and on current
KDE it actually seems to break stuff. It's better to just start
typing the password.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Typing a partial binary name no longer seems to work. Typing the
full binary name works, but differently from before; seems best
to do a partial entry name search so we launch the actual entry,
not the executable directly.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I've seen a few cases of this test failing because there was
some running operation when it tries to do `rpm-ostree install`.
I think this is GNOME Software checking for updates or something,
I've seen it on my own Silverblue install too. Let's just throw
some `rpm-ostree cancel`s at it and hope that helps.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
"Additional repositories" is now hidden behind a dropdown we
have to open first. This will make the test fail on anything
older than Fedora-Rawhide-20230121.n.0, but I don't think we
run this test anywhere that would be a problem.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This PR adds a small test suite to test the Characters applications.
It displays several different groups of characters and then tries
to copy one of the characters and place it into a text editor.
GNOME Software no longer has a welcome screen in any current
Fedora (it was dropped between 35 and 36), but in Rawhide it now
has a popup that prompts you to enable third-party repos which
we need to get rid of, so just convert the welcome screen check
to handle that, and drop all the welcome screen needles.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It seems ending the test right after we create the mutex can
cause the client not to catch it, sometimes. So let's sleep for
a few seconds after creating it to make sure it does.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
These tests weren't doing it, I guess it's just an oversight;
this is probably why they often fail on slow repos.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This makes the two rpm-ostree tests written for IoT - overlaying
and rebasing - work across all rpm-ostree-based flavors we
currently test (IoT, CoreOS and Silverblue) and runs them on
all those flavors.
This requires some other changes. For the Workstation ostree
installer update tests, we have install_default_update_ostree
upload a disk image and run these tests on that image. That means
install_default_update_ostree cannot use a scratch disk (as if
we boot it with two disks but only upload one, the subsequent
tests fail to boot, looking for the missing second disk), but its
specified disk size should be large enough for all updates.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
What's supposed to happen here is the `do_bootloader` invocation
a few lines back boots to the installer, then here, we wait for
the install to complete and the system to reboot, and match the
bootloader again. However, on PXE installs, the bootloader screen
can hang around for quite a long time here, and if it does, we
can match it again before the installer starts up, and move on
too early. Hence the sleep.
It seems on current Rawhide 20 seconds isn't long enough - we're
still matching the installer bootloader after the sleep, see
e.g. https://openqa.stg.fedoraproject.org/tests/2431660#step/_boot_to_anaconda/3
This is causing the test to almost always fail (it'll only pass
if the install+reboot takes less than five minutes). Let's bump
it to 60 seconds and hope that's long enough.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Sadly, dropping this sleep caused the test to start failing
again at least on F36, so we still need it - update the note.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Thankfully this all calmed down a bit so we can simplify it a
lot. Clean things up a bit at the same time; escaping nested
single quotes is a lot clearer than concatening blocks with
different quote marks.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The bug seems to have gone away, at least I don't see that this
soft failure has been hit much for the last two months.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It's been on 1 so long now I kinda don't want to change it to 3
or 4 or anything. That might break something. As long as it's not
causing any trouble let's just leave it on 1.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
PXE install on UEFI (incl. aarch64) is failing at present, this
seems to be due to a grub bug:
https://bugzilla.redhat.com/show_bug.cgi?id=2152763
we're really intending to test the client side here, not the
server end, so let's work around this problem on the server end
by installing a grub2 scratch build that's the package from just
before the bad change, but with the release and epoch bumped,
from a side repo.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This turns out to be overcomplicated. We don't need the special
handling for updates any more, because a few months after we
implemented it, we had to make sure the affected update tests had
an empty START_AFTER_TEST anyway, or else openQA would refuse to
schedule them. So we can just rely on the START_AFTER_TEST
condition for those now. We also don't need the additional
INSTALL_NO_USER condition; the only case where it's actually used
is for install_arm_image_deployment_upload on Workstation, and
that test does not have START_AFTER_TEST set, so the other
condition catches it for welcome screen handling purposes. There
is no need to nest the IMAGE_DEPLOY conditional inside a check
for the desktop and the INSTALL_NO_USER var either; we don't
test any other desktop on ARM, and the IMAGE_DEPLOY var is only
set for that one install_arm_image_deployment_upload test.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We recently started using the buildroot repo for Rawhide update
tests, but weren't including it in the image build tests. This
should include it in all the image build tests.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We currently snapshot after every run of _console_wait_login or
_graphical_wait_login, which means we snapshot *twice* on most
update tests as those modules get run twice. However, we almost
never use those snapshots. Snapshotting takes quite some time,
and hits the disk pretty hard, so we should avoid it unless it
is really needed.
We only have a few modules that are not fatal (and so might use
the snapshots), and most of those don't run after one of these
tests, or run after a later module that's also a milestone. Best
I can tell, only two test suites really need to use a snapshot
from a login test: server_cockpit_updates and modularity_tests.
To handle these and potential future cases, we'll add a new
module that does nothing, but is marked 'milestone', so it will
take a snapshot, and load that test after the login test if the
var LOGIN_SNAPSHOT is set, and set that var for those two suites.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We started using this in real composes a year or two back, so
openQA should do the same. It drops the nesting of an ext4 fs
image inside a squashfs image, just using a single squashfs
image instead. This results in smaller images - missing this
is why the images built by openQA were coming out larger than
the real ones.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
* Scarborough provided quite a messy map that resulted
in frequent needle failure. Changing the location
for something better to make it more reliable.
* The zoom test could have failed with a low resolution
image. Adding some timeout to the needle give more
time to load the proper image.
The check_screen function checks for the existing tag
but it only waits 1 second by default. In this time,
Abrt will not even start so we need to prolong
the check_screen timeout to make sure the application
has started (or at least give it enough time to try).
Instead of just redirecting it to a log file, let's tee it, so
simple errors can be read off a screenshot without bothering to
download the file. Also, let's timestamp it (via `ts` from
moreutils) so we can see which bits of it take a long time...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is like the existing tests that build network install and
live images then install them, only for Silverblue. First we
build an ostree, using the standard configuration for the release
and subvariant but with the 'advisory' and 'workarounds' repos
included, so it will contain current stable packages plus the
packages from the update and any workarounds. Then we build an
ostree installer image with the ostree embedded, again including
advisory and workarounds repos in the installer build config so
packages from them will be included in the installer environment.
The image is uploaded, which completes the _ostree_build test.
Then an install_default_update_ostree test runs, which does a
standard install and boot from the installer image.
We do make a change that affects other tests, too. We now run
_advisory_post on live image install tests, as well as this new
ostree install image install test. It was skipped before because
of an exception that's really only needed for the netinst image
install test. In that test, packages from the update won't be
included in the installed system, so we can't run _advisory_post
on it. But for ostree and live image build/install tests, the
installed system *should* include packages from the update, so
we should check and make sure that it does.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There's one point in the tests where we may log into cockpit for
the second time in one run (it depends how a package update
process goes). When this happens, we don't get prompted again
for admin access, so we need to *not* expect that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This reverts commit 3208d15725 and
the two follow-ups. I'm hoping
https://bugzilla.redhat.com/show_bug.cgi?id=2133829 is now
resolved; this was intended to help with that (though I'm not
sure it ever really did), and so we can hopefully ditch it, which
simplifies this code.
This reverts the last few commits which worked around a focus bug
in GTK. This bug is now (I hope) fixed, so I'm dropping the
workarounds so the tests will confirm whether it's fixed.
We have a big problem with Rawhide KDE update tests getting OOM
killed during this phase. Stopping the desktop before we install
updates should save some RAM and help avoid this.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Boy this seems slow in Rawhide currently. This has the effect
of being more defensive around the services page load.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We've seen some failures of the weather test at the start of
weather_report, where the test expects to be at the hourly view,
and instead it seems to be at a sort of broken state:
https://openqa.fedoraproject.org/tests/1505080#step/weather_report/3
I'm guessing this may be because currently aaa_setup clicks the
city name then is immediately complete, so it will immediately
snapshot. I guess this can result in things being stuck in a kind
of intermediate state on snapshot restore. So, to try and avoid
this, let's assert that we reach the hourly view after clicking
the city name, then wait_still_screen for a few seconds to make
sure things are settled down, before we complete and snapshot.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We seem to be hitting very long loads on the Services and Logs
pages of Cockpit in recent Rawhide testing especially. As I don't
have time to deeply debug this at the moment, let's just give it
longer (but make it a soft failure when it takes longer than
expected).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Sometimes the windows are displayed in the reversed order, which
prevents the checks to find the needles and the test fail
even if it should pass. This change should address this case.
We use variables to track test state across modules, sometimes.
As this is all internal to the test logic I didn't bother always
making these variables upper-case, but os-autoinst now treats
lower-case variables as a fatal error, so we have to change.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The change to read-only sysroot for IoT in F37 causes problems
with this rebase test. It's not supported to rebase from an RO
release (37 or 38) to a non-RO release (36). So we need to make
sure we don't try and do that. This uses some quick hack logic,
but it should be OK and sufficiently specific not to break
anything even if we forget to remove it in future.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It seems we still have trouble with this turned on :( About 60%
of tests fail with the client unable to resolve names.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The cluster of bugs for F37/Rawhide should all be resolved now,
and I'm hoping the old upgrade bug is no longer relevant.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This issue appeared when we started testing Rawhide updates, but
I only noticed it today. When testing Rawhide updates after
Branch point, the upgrade tests upgrade from Branched to Rawhide.
On Branched, updates-testing is enabled by default. We only
disable it when we reach `upgrade_run`, but by that point we've
already done a `dnf -y update` in `upgrade_preinstall` and
potentially installed other packages in steps between
`upgrade_preinstall` and `upgrade_run`. That can cause problems,
like today all FreeIPA upgrade tests on Rawhide are failing
because there's a newer freeipa in updates-testing for F37 than
is in the current Rawhide compose.
Solve this by disabling updates-testing before we do the update
in `upgrade_preinstall`. To avoid excessive code duplication,
factor out the repo disabling code.
We'll do this twice on upgrade tests now, but it shouldn't be a
problem.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Also stop re-doing get_var("DESKTOP") because that's dumb. This
should only, at worst, make things slower if unexpected things
happen - it shouldn't cause failures that wouldn't happen anyway.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This will make them slower, but lately type_safely is just not
reliable, particularly in the new_file test, it's constantly
typoing.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Rawhide live image builds are still taking an awful long time
and often failing. I will look more into why later, but for now,
let's bump the timeouts even more just to try and get through
the job backlog.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I'd prefer not to have to do this, but having the tests fail on
every compose and Rawhide update test is just too distracting
to live with.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
