With Rawhide updates, we quite often run into a situation where
a test runs after a *later* version of the package has already
gone stable. This even happens for stable releases too, though
less often. The current shell-based check just always fails on
this case, but it's usually OK, and manually marking every case
like this with an "it's OK!" comment gets tiring. Instead, let's
use a smarter Python script to do the check. We compare the EVR
of all installed update packages with the EVR of the package
from the update. If it's the same, fine. If the installed package
is lower-versioned, that's always an error, and we fail. If the
installed package is higher-versioned, we check whether the
update already went stable. If it did, then we soft fail, because
probably nothing can go wrong at this point (this is the usual
Rawhide case). If the update did not yet go stable, we still
hard fail, because something can go wrong in this case: if the
update *now* goes stable, the older version from the update may
be tagged over the newer version the test got (presumably from
current stable).
If anything goes wrong with the Bodhi check, or the test is
running on a task not an advisory, we treat both cases as fatal.
The script also gives easier-to-understand output than the old
approach, which should be a bonus.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Typing a partial binary name no longer seems to work. Typing the
full binary name works, but differently from before; seems best
to do a partial entry name search so we launch the actual entry,
not the executable directly.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I've seen a few cases of this test failing because there was
some running operation when it tries to do `rpm-ostree install`.
I think this is GNOME Software checking for updates or something,
I've seen it on my own Silverblue install too. Let's just throw
some `rpm-ostree cancel`s at it and hope that helps.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
"Additional repositories" is now hidden behind a dropdown we
have to open first. This will make the test fail on anything
older than Fedora-Rawhide-20230121.n.0, but I don't think we
run this test anywhere that would be a problem.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This PR adds a small test suite to test the Characters applications.
It displays several different groups of characters and then tries
to copy one of the characters and place it into a text editor.
GNOME Software no longer has a welcome screen in any current
Fedora (it was dropped between 35 and 36), but in Rawhide it now
has a popup that prompts you to enable third-party repos which
we need to get rid of, so just convert the welcome screen check
to handle that, and drop all the welcome screen needles.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It seems ending the test right after we create the mutex can
cause the client not to catch it, sometimes. So let's sleep for
a few seconds after creating it to make sure it does.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
These tests weren't doing it, I guess it's just an oversight;
this is probably why they often fail on slow repos.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This makes the two rpm-ostree tests written for IoT - overlaying
and rebasing - work across all rpm-ostree-based flavors we
currently test (IoT, CoreOS and Silverblue) and runs them on
all those flavors.
This requires some other changes. For the Workstation ostree
installer update tests, we have install_default_update_ostree
upload a disk image and run these tests on that image. That means
install_default_update_ostree cannot use a scratch disk (as if
we boot it with two disks but only upload one, the subsequent
tests fail to boot, looking for the missing second disk), but its
specified disk size should be large enough for all updates.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This test fails on Rawhide (because sddm-on-wayland) but we just
got an F37 respin, where it passes but needs some needle updates.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
All the deleted ones haven't been matched for five months. Drop
match level to 90 on the remaining ones, we got a 96 match for
one of them in today's respin test.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I don't know why we wind up with so many slightly different
matches on the login screen. It's weird.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
What's supposed to happen here is the `do_bootloader` invocation
a few lines back boots to the installer, then here, we wait for
the install to complete and the system to reboot, and match the
bootloader again. However, on PXE installs, the bootloader screen
can hang around for quite a long time here, and if it does, we
can match it again before the installer starts up, and move on
too early. Hence the sleep.
It seems on current Rawhide 20 seconds isn't long enough - we're
still matching the installer bootloader after the sleep, see
e.g. https://openqa.stg.fedoraproject.org/tests/2431660#step/_boot_to_anaconda/3
This is causing the test to almost always fail (it'll only pass
if the install+reboot takes less than five minutes). Let's bump
it to 60 seconds and hope that's long enough.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Same conditions as used in main.pm to load the tests in the
normal flow. It makes no sense to do this on non-update tests,
or on the non-matching support server case.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Sadly, dropping this sleep caused the test to start failing
again at least on F36, so we still need it - update the note.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Thankfully this all calmed down a bit so we can simplify it a
lot. Clean things up a bit at the same time; escaping nested
single quotes is a lot clearer than concatening blocks with
different quote marks.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The bug seems to have gone away, at least I don't see that this
soft failure has been hit much for the last two months.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Adam Williamson
Lukas Ruzicka