osbuild is struggling with the pre-release warnings ATM:
https://pagure.io/fedora-iot/issue/57https://github.com/osbuild/images/issues/515
Until that mess is cleaned up we can't really make sensible
assertions for osbuild images, so let's make the check a soft
failure for now (now we know about the bugs, we want to let
the rest of the tests run and not block them on this).
Note for IoT the behaviour has never really been correct (IoT
images never get pre-release warnings), but the logic in this
check matches the wrong behaviour (IoT composes always have
RC- labels even when they're clearly not RCs), so the test
didn't fail. While fixing this in osbuild we might try to get
'correct' behaviour for IoT, and then we'd need to tweak the
logic here.
While we're at it, tweak the implementation a bit; without this
tweak, implementing this 'soft fail if osbuild' behaviour is more
awkward and ugly.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This PR partly solves the issue #301 when it adds the navigation
testing for Gnome. It uses the keyboard combinations to cycle through
running applications and checks that applications could be switched
accordingly. It also tests that you can switch between workspaces
and that you can move an applications to another workspace.
The worker containers aren’t using systemd-resolved so
get_host_dns() can’t find the dns server causing tests that need
it to fail. Add an alternative place to look for the dns server address
that doesn’t rely on systemd-resolved.
Signed-off-by: Deborah Brouwer <deborah.brouwer@collabora.com>
This is actually a very useful to wait a little bit before
we try to operate with the started application, especially
if we want to send keys.
Having this wait time here can save us doing this every time,
we start an application.
The mistypes on KDE are bugging me. Let's see if this - mainly
the wait_screen_change, the other just seems like a logical
tweak - helps at all.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
gvfs from the GNOME 46 megaupdate needs it. It's been pushed
stable but not included in a compose, yet.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It's queued for stable but blocked by the freeze, and
FEDORA-2024-9c9174f3dd requires it. Let's just make it a
workaround to clear the other update's tests.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Just checking for the package update list is wrong in a couple of
ways:
1. It goes wrong for child tests that inherit the main disk from
the parent test, but not the second disk where the actual update
repo lives. They'll have the package list, but not the repo. So
we should test for both the package list and the repo dir.
2. It goes wrong for side tag tests, because we don't write a
package list (or create an update repo) on that path. So on that
path let's check whether we already created the repo definition
we use on that path.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
https://bugzilla.redhat.com/show_bug.cgi?id=2268505 made it
clear that this is a bit of a hole. We don't test installing the
Silverblue image we build on UEFI, only on BIOS. Add this as a
separate test so we don't uselessly upload a disk image we won't
use for any follow-on tests.
This also adds an anaconda build that fixes a bug in this path
as a workaround for F40, so the test won't fail on all F40
updates.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This should mean all F40 update tests get the new background,
and we can once again make that test fail if it doesn't see it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The Java 21 update broke FreeIPA and was waived stable. This
update rebuilds dogtag-pki and should fix it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I missed reverting the g-i-s page skip config when deferring
the webUI Change to F40. Now I changed the conditional in the
openQA test to expect F40 to behave like F39 and earlier, it's
failing because it's seeing too many pages. This should resolve
it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
jlebon and jmarrero figured out the fix for this, while we wait
for the packager to merge it their preferred way, let's use
scratch builds to work around it so every update doesn't get a
failed test.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Another case where an update was submitted after its deps had
gone stable, but before they made a compose. Maybe we really need
to set up a buildroot repo for tests of updates to branched-before
-bodhi-activation too. I'll think about it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Now the fedora-kickstarts PR to remove anaconda-webui was merged
we need this for tests to pass. Remove after next compose.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The new gnome-shell from FEDORA-2024-f543e8595c requires this.
We don't use a buildroot repo for Branched even in the short
period where it acts like Rawhide (no karma or time requirements)
so let's just use a workaround.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
F38 is the oldest thing we test, and it has this flow. So let's
just drop the check here and always hit tab twice.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We should wipe it both before and after running repo_setup to
really get rid of it. Ideally we should stop doing this at all,
but for right now the repo isn't there for Rawhide post-F40
branching yet, so let's keep this until that's done.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We don't need this anaconda scratch build any more (the official
build has the fix now), but we *do* need the new lorax update I
just built to fix installer image builds with branched F40. See
https://github.com/weldr/lorax/pull/1379
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This adds a scratch build with my proposed patch:
https://github.com/rhinstaller/anaconda/pull/5460
to get the tests to start passing again, so we don't have the
flood of red which makes it hard to spot other problems.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is a surprisingly large change as we want to go back to
the console we were previously on after doing it. To do that we
need to know what console we were on, and to know *that*, we need
to port everything that currently uses (ctrl-)alt-fX to switch
consoles to use select_console instead.
This is primarily intended to make running setup_repos.py faster
when it has to download a lot of packages (as typing in hundreds
of package names is quite slow). But it actually makes the whole
thing faster, even when only downloading one or two packages.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This uses a Python script which implements concurrent downloads
(via asyncio) to download workaround and update packages and
configure the repos. This should speed up the process for large
multi-package updates.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This effectively reverts 97618193 - but had to be done manually
and adjusted to maintain support for testing side tags and for
testing multiple tasks, since those features were added since
the update ISO change.
The 'scheduler injects ISOs of packages into the tests' approach
was intended to speed things up, especially for large updates,
and it did, but it had a few drawbacks. It means restarting
older tests from the web UI doesn't work as the ISOs get garbage
collected (you have to re-schedule in this case). And it has the
rather large problem that you can now only schedule tests from
the openQA server (or at least a machine with the openQA asset
share mounted), because the package download and ISO creation
just happen wherever the scheduler is running and assume that
the openQA asset share that will be used by the tests is at
/var/lib/openqa/share in that filesystem.
That's too big of a drawback to continue with this approach, IMO,
so this reverts back to the old way of doing things, with a bit
of refactoring to clean up the flow a little, and with support
for testing side tags and multiple tasks maintained.
As a follow-up I'm going to see if I can replace
_download_packages with a much more efficient downloader script
to mitigate the time this process takes on each test, especially
for large updates.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The issue is not marked as fixed, but per the needle cleanup we
have not hit this workaround condition for a long time. I checked
past failures and it doesn't look like we're still hitting the
problem but the needle stopped matching, or anything.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Seems the bug might just be that plymouth got more sensitive to
fast typing, so instead of waiting, let's try slow typing.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
New Plymouth seems to have a bug where it shows the decryption
prompt briefly then shows a spinner and refreshes it, throwing
away any already-typed input. This is breaking our tests quite
often (any time os-autoinst is "lucky" enough to spot the first
brief appearance of the prompt and start typing). To work around
it, after we first see the prompt, wait for the screen to settle
and re-assert the needle before typing. This should reliably
wait out the refresh cycle.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Plasma 6's color chooser seems to have dropped the nice "basic
colors", so choosing black got harder. Let's try using the HTML
color input box thingy instead, and typing #000000, the HTML
color code for black.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This PR tries to respond to issue #294.
On Silverblue, this will try:
* flatpak install
* flatpak remote-add
* flatpak list
* flatpak remotes
* flatpak remove
* flatpak update
and also it tests that a flatpak can be built.
Currently, the installation via WebUI is mostly pushing the Next button
which seems to be ok for the production which is based in the US.
This PR makes openQA to select languages when the G-I-S runs
before Anaconda. The particular language is selected based on
the LANGUAGE variable.
The latest version of g-i-s grew a "Previous" button on the
final page, and hitting tab once now activates that, not the
Try button. shift-tab *should* get us to Try on both old and
new versions.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
webUI has been deferred to F40, so we need to expect the old UI
flow on F39 now. This should cover everything, I hope.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We're reverting webUI for Fedora 39.
https://bodhi.fedoraproject.org/updates/FEDORA-2023-73c4f1a802
is the update that implements this; adapt the tests to handle it
(by expecting the old flow when testing that update, and editing
the kickstart to drop anaconda-webui when building the live
image).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We now have a fix for the bug on the new webUI flow where the
language and keyboard screens weren't skipped on the first boot
after install, as intended. language is never *really* skipped -
it just turns into welcome - but keyboard is now being skipped,
which messes up the logic here.
For a short time we need to handle both paths, to get the new
anaconda builds through and new composes built. In a couple of
days we can simplify this to just always assume keyboard will
be skipped on the first boot on Workstation live installs on
F39+.
Also drop handling of auth_required in g-i-s - I'm pretty sure
that bug got fixed years ago - and wait_still_screen for three
seconds on each page, to let animated transitions settle.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We just landed the webUI stuff for F39, so now we need these
conditionals to kick in for F39+, not F40+.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is tailored to the initial deployment of webUI in
Workstation live images only; we may need to tweak flows and
approaches as webUI goes further.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Per https://bodhi.fedoraproject.org/updates/FEDORA-2023-5fd964c1bf#comment-3149533
we kinda need to do this to allow this update through, so long as
we're not going to have dnf obsolete dnf5 or anything like that.
It's a bit unfortunate but I don't see an alternative.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We're seeing a lot of tests fail on 404s when trying to access
the koji-rawhide repo (the repo for the Rawhide build tag, which
we use to get packages tagged since the last compose. nirik is
trying to figure this out from the server end, but for now at
least, let's mark the repo as skip_if_unavailable. This should
mean that if we hit a 404, the test will continue, it just won't
have access to the packages from that repo. Occasionally this
will cause a problem - a false failure or false pass - but this
still seems better than every test that hits it failing. The
false pass case is the most concerning, but I would hope in that
case some other tests from the same update would fail, making it
not an issue.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Per nirik, 'repos/rawhide' is just a symlink to 'repos/fXX-build'
and this could possibly be part of our 404 problem. So let's
try using fXX-build directly instead of the symlink.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This repo gets regenerated a lot, so we should be very aggressive
about metadata expiry. Hopefully this will forestall most of
the cases where we get a 404 trying to access this repo's
metadata files.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We're still getting failures from pagure.io even with the retry
stuff. nirik asked us to add this to help figure out what the
heck is going on there.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This tweaks all pagure.io downloads to be retried a few times,
since we seem to be getting failures quite often. We use curl
for this as it has nice options for it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This requires a change in the package we use for base_update_cli
because pandoc-common is not in ELN.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We drop the line for the update ISO from /etc/fstab before
uploading the image after the cockpit_default test, but we don't
make sure it's set up again before Cockpit tries to use it, in
the subsequent Cockpit tests. I don't know why this didn't fail
on stg before, but it sure as hell is failing in prod...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I'm attempting a new approach to the update and workaround repos.
Instead of having each update test recreate them for itself -
which is slow and wastes bandwidth - the dispatcher will create
an ISO at test schedule time and pass it as ISO_2. Then the test
just mounts the ISO. This makes the necessary adjustments on the
test side.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Drop the e2fsprogs scratch build workaround we were using for
https://github.com/fedora-silverblue/issue-tracker/issues/470 -
with the new 'use a custom ref and rebase to the official ref'
thing I implemented for update ostree tests, it shouldn't be
necessary any longer.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I think this is new behaviour in rpm 4.19, or else we would have
run into this before when testing an s390utils update. But it's
easy enough to handle.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
F36 is now EOL, and from F37 onwards, grub is the bootloader in
any situation where it actually matters to do_bootloader (which
is only when we're editing parameters). We do still use syslinux
in the PXE tests on x86_64 BIOS, but we don't edit the parameters
in that case.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
https://github.com/storaged-project/udisks/issues/1102 - udisks2
seems to have a bug where it leaves filesystems mounted at a
"temporary" mount point after creating them. We need to work
around this when it happens or else we'll frequently get test
failures.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This adds a new script - cleanup-needles.py - to use for cleaning
up old needles. It has to be used in conjunction with a database
query; the comment at the top explains how to do that query and
export the needed information. It produces a git commit with
needles that haven't matched since a certain date (specified in
the sql query) removed, subject to a 'keeplist' of needles we
keep even if they seem to be old.
I also enhanced check-needles.py to check for cases where tests
seem to be trying to match a tag we have no needles for. This
was necessary to find cases where the cleanup script was too
aggressive (i.e. the things that wound up in the 'keeplist'),
but it also turned out to find quite a lot of cases where the
code really *was* looking for a needle that had gone in a
previous cleanup or that never existed; the commits before this
one clean up a lot of those cases.
The code to decide which string literals are needle tags is
pretty dumb and hacky and needs some manual cueing sometimes -
that's what the `# testtag` changes in this commit are for.
Making it smarter would probably require this script to get a
lot more complicated and either incorporate or become a
tokenizer, which I don't really want to do.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It looks like neither of these has been a problem for some time.
The notification needle has not matched for a year. The akonadi
needle doesn't exist any more - it was cleaned up in the 2021
needle cleanup, meaning it hadn't matched for weeks in 2021. I
checked the last several months of KDE app start/stop tests and
don't see any case where there was a stray notification that we
missed. So I think we can just ditch this whole mechanism for
now; if we have problems with these notifications again in future
we can put it back.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
For several releases now, the 'new user mode' of g-i-s is just
gone: https://gitlab.gnome.org/GNOME/gnome-initial-setup/-/issues/12
so this whole path where we used to be able to set up Japanese
input methods on first boot after install doesn't work any more.
We had to set up a whole different route to set the input method
via control center instead (which lives in _graphical_input).
This block is never reached any more, and the needles for it were
cleaned up in 2021.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Since 022865ab we do not use start_with_launcher on KDE any more.
The needle for it has since got lost in an unused needle
cleanup. Let's just drop this branch for now; we can add it back
if we ever need it again.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The needle that backs this workaround was dropped in the 2021
needle cleanup, so it's never worked since then. I checked, and
no aarch64 tests seem to be failing on this any more.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Upstream https://github.com/os-autoinst/openQA/pull/4973 requires
us to poke things here a bit. This only works with the newer
os-autoinst and openQA (there may be a way to conditionalize it
to work with both, but I can't be bothered figuring it out, let's
just update).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There's a fairly longstanding issue in GDM where switching layout
just doesn't work sometimes:
https://gitlab.gnome.org/GNOME/gnome-shell/-/issues/6066#note_1707051
there doesn't seem to be any progress on getting this fixed, and
it's annoying constantly restarting tests that fail on it. So
this just makes us try three times to switch before giving up.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
In install_repository_hd_variation , we have two disks attached,
so when we reach the INSTALLATION DESTINATION screen we expect
we have to select the correct target disk. However, as of the
most recent anaconda build in Rawhide, anaconda realizes we're
using a file from the other disk as our install repo, and
'protects' it - i.e. it does not show it on the INSTALLATION
DESTINATION screen, meaning only one disk is shown there, and
when only one disk is shown, it's pre-selected. So when we click,
we actually *un*select it, and the test fails.
Fixing this is a bit awkward; I wanted to add a new variable,
ANACONDA_PROTECTED_DISKS or something, and subtract that from
the NUMDISKS count; but we can't really do that as the enhanced
protection isn't in F38, and we can't easily set variables
differently on different releases (we'd have to set them in the
scheduler code, not just put them in the templates). So we just
code in a doofy condition for this instead. Maybe when F38 is
EOL we can change to the variable approach.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
