Per https://github.com/containers/podman/pull/19302 and
https://github.com/containers/podman/issues/19299 , upstream
have kindly set things up so we can easily run relevant parts of
the upstream test suite as an integration test in openQA. This
should help us catch if changes in other components break key
features of podman.
This only runs any tests with podman 4.6.1 or higher, but with
earlier versions it just does nothing and exits 0, so that's
fine. 4.6.1 is in F39 and Rawhide already, will land for F37
and F38 shortly.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
dnf seems to have some odd trouble with these updates. It
really wants to use the older 2.rc2 builds, even though there
doesn't seem to be any actual *problem* with the 3.rc2 builds.
For the samba server test, passing `--best` to dnf seems to be
enough to make it use the .3.rc2 builds. For the FreeIPA tests,
we have to do a second pass with `--best` after the initial
install.
It's weird that we have to do this, but to get these updates
through - because there doesn't really seem to be a problem
here - let's do it. They will replace the 2.rc2 builds in the
main F39 and Rawhide trees once they land in 'stable' so the
problem shouldn't persist.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
mock still thinks the releasever for Rawhide is 39, which causes
it to use the wrong GPG keys and not be able to install packages.
This overrides that setting in our mock config file, until
mock-core-configs is updated in the distro.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The numbered config file won't always exist right after branch
(there is no fedora-40-x86_64 now, for e.g.) But the named one
always does. This additional variable is a small price to pay for
making the test more robust.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We know this is broken and we don't want it to fail on every
update, so we need to work around the problem for now.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
abbra told me where I was going wrong with the 'expected' target
of the getent command ("AD/" is not a magic string, it's just
"(netbiosname)/", and our netbios name is "SAMDOM"...) so this
fixes that too, trying to avoid hard-coding stuff.
For the kickstart test, it seems like it's a timing issue. We
added this 'install sssd-tools and enable debugging' step to try
and debug it, and instead it fixed it. So...let's just stick
with this, for now, because it's useful to have this debugging
anyway. If the problem starts happening again, we can fiddle
about with it more.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
As 'real' upgrades (using releases/development/rawhide on the
mirrors) do not hit this bug because it has a stale Modular
tree, it makes sense to work around the bug in testing so we can
see if upgrades are broken in any *other* way.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We're seeing a lot of tests fail on 404s when trying to access
the koji-rawhide repo (the repo for the Rawhide build tag, which
we use to get packages tagged since the last compose. nirik is
trying to figure this out from the server end, but for now at
least, let's mark the repo as skip_if_unavailable. This should
mean that if we hit a 404, the test will continue, it just won't
have access to the packages from that repo. Occasionally this
will cause a problem - a false failure or false pass - but this
still seems better than every test that hits it failing. The
false pass case is the most concerning, but I would hope in that
case some other tests from the same update would fail, making it
not an issue.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Per nirik, 'repos/rawhide' is just a symlink to 'repos/fXX-build'
and this could possibly be part of our 404 problem. So let's
try using fXX-build directly instead of the symlink.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This adds a Samba AD server test, and client enrolment tests via
sssd, Cockpit and kickstart. Requires the matching createhdds
commit to add the kickstart to the disk_ks image.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The change to the curl commands to force HTTP/1.1 seems to have
stopped them failing, so let's try doing it for the git clones
too and see if that avoids the problem till we can work out
what is causing it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Modularity will be retired in Fedora 39 and the modular repository
removed. Also, the upcoming version of DNF that is default in
Rawhide, does not support modularity, so the tests are currently
failing anyway.
This tweaks all pagure.io downloads to be retried a few times,
since we seem to be getting failures quite often. We use curl
for this as it has nice options for it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The Flatpak build doesn't have the spellcheck issue at the
moment, and it may be fixed soon in the RPM build. Trying to
'fix' the issue on the flatpak build actually makes the test
fail. So, let's only do the fix if we actually have a misspelled
word.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The tests after it assume new_file ran - they rely on the file
it creates - so it should be considered fatal.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Seems in Rawhide the menu is loading without dividers briefly,
we match there, then the dividers load in and make the menu
longer, so when we click, we hit a different entry in the menu.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The editor started to show spell-checking that would require a lot
of new needles to be created. Theredore, we set the language to
English to stop showing the spelling mistakes in aaa_setup.pm
Also, the application started to have problems with getting correct
focus, so we want to click into the text before the status gets
recorded.
Change the comment on why we put /var/lib/mock on the third hard
disk: we probably can cut it down to two, now, but I don't want
to do it right now.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
D'oh. This is the first time we actually tried to use the new
workarounds ISO thing for real, I forgot to update some paths.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This requires a change in the package we use for base_update_cli
because pandoc-common is not in ELN.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The shifted characters here frequently get mistyped. Let's use
type_safely. If this isn't enough we can try very_safely.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We drop the line for the update ISO from /etc/fstab before
uploading the image after the cockpit_default test, but we don't
make sure it's set up again before Cockpit tries to use it, in
the subsequent Cockpit tests. I don't know why this didn't fail
on stg before, but it sure as hell is failing in prod...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I'm attempting a new approach to the update and workaround repos.
Instead of having each update test recreate them for itself -
which is slow and wastes bandwidth - the dispatcher will create
an ISO at test schedule time and pass it as ISO_2. Then the test
just mounts the ISO. This makes the necessary adjustments on the
test side.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
With the new 2G max EFI system partition size, we were trying to
stuff 12G of Fedora into a 10G disk. That wasn't going to work.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I don't think we need an alternative needle for ppc - the
current 'boot_inactive' needle should work fine on ppc. Let's
just always use that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
As of yesterday's Rawhide, the modular repos are not installed by
default, so of course all the modular tests fail. So, install
the repos before running the tests.
This isn't conditionalized on release version as I don't think
we ever run this test on anything other than Rawhide and
Branched.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This checks we actually deployed the 'fedora-openqa' ref as we
intended to (if not, the rebase test probably won't work
properly or won't test what we want it to).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We have had several problems where rebasing from one release to
another just doesn't work, and we have to work around it somehow.
Sometimes it's difficult or impossible to do. All we want to do
here is check the rebase mechanism itself; we don't actually want
to assert that you can rebase to any specific other release.
For update tests, we should be able to use a non-standard ref
name for the ostree we build, embed into the installer image,
and install. That should mean we can then rebase to the standard
ref name for the same release, which should be much safer than
trying to rebase to a different release. We can't do this for
the compose tests, but at least for update tests I'm hoping this
makes things more robust.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It's not in the images any more. As aleasto pointed out, we're
actually being sent to Discover to install it, and matching on
*that* screen, which isn't what we intend.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
util-linux-user subpackage was removed but comps wasn't updated.
I've fixed comps now, but that won't "kick in" until after the
next Rawhide compose; we need to workaround the issue until then.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
If install errors out, currently we still wait like an hour for
an "install_done" screen that will never come, before we give
up. Since we have a needle for the "unknown error has occurred"
screen, we may as well use it here - if we see that screen, we
can just die immediately. This may go stale if we forget to
update the needle, but it's only one line, so meh.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This adds a new script - cleanup-needles.py - to use for cleaning
up old needles. It has to be used in conjunction with a database
query; the comment at the top explains how to do that query and
export the needed information. It produces a git commit with
needles that haven't matched since a certain date (specified in
the sql query) removed, subject to a 'keeplist' of needles we
keep even if they seem to be old.
I also enhanced check-needles.py to check for cases where tests
seem to be trying to match a tag we have no needles for. This
was necessary to find cases where the cleanup script was too
aggressive (i.e. the things that wound up in the 'keeplist'),
but it also turned out to find quite a lot of cases where the
code really *was* looking for a needle that had gone in a
previous cleanup or that never existed; the commits before this
one clean up a lot of those cases.
The code to decide which string literals are needle tags is
pretty dumb and hacky and needs some manual cueing sometimes -
that's what the `# testtag` changes in this commit are for.
Making it smarter would probably require this script to get a
lot more complicated and either incorporate or become a
tokenizer, which I don't really want to do.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We never hit this path without a system menu button any more,
due to changes in KDE over time. It hasn't been hit for two years.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Now both GNOME and KDE do offline updates on all supported
releases, we never see an 'update done' screen any more. This
branch is left over from when the KDE offline update branch was
still conditional on release number.
If we ever implement this test on a desktop that doesn't do
offline updates, we can put this back easily enough.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There's no need to do all this 'check whether it's selected and
click it if not' stuff (for three different mount points). Just
always click it. If it's already selected, clicking it again
doesn't hurt (one of these stanzas even clicks it *even if it's
selected*!)
If we need to cover both cases, we just need two needles with
the same tag, we don't need separate code paths. In each case,
though, we actually haven't matched one of the needles for ages
(the most recent was part_boot_selected, but now we're using
GPT by default, we won't hit that any more as it'll be the BIOS
boot partition that's selected by default), so delete the needles
we aren't matching any more. If we *do* hit any case where we
need to handle the 'other' state, we can just add the alternative
needle with the same tag.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This has not been hit for a year (on stg; three years on prod).
I *think* it would only be hit if we ran the test on an Everything
image, but as the test is now specifically associated with the
Server install DVD, that doesn't seem likely to happen.
If we somehow *do* hit ext4 pre-selected again, this can still
be handled simply by adding an alternate
anaconda_blivet_part_fs_ext4 needle which matches on ext4 already
being selected; that avoids the need to keep an alternate code
path around.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This branch is very fragile, because the test won't fail if we
miss the match on the security needle. So in practice, we are
never going to notice when the needle goes stale, and we'll just
wind up never triggering this branch and always going down the
other path. That's the current situation: the security_install
needle last matched more than a year ago at least. Let's just
admit the truth here and drop the branch entirely.
Also update the cockpit_updates_restart_ignore needle. This is
in a similar case - we don't really notice when it goes stale,
as the test completes, it just takes a bit longer - but since
this one is quite easy to find, let's just update it instead of
dropping it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This was resolved upstream and we're no longer hitting this bug
in tests on F38, Rawhide or even F37 respins, so we should no
longer need this workaround.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It looks like neither of these has been a problem for some time.
The notification needle has not matched for a year. The akonadi
needle doesn't exist any more - it was cleaned up in the 2021
needle cleanup, meaning it hadn't matched for weeks in 2021. I
checked the last several months of KDE app start/stop tests and
don't see any case where there was a stray notification that we
missed. So I think we can just ditch this whole mechanism for
now; if we have problems with these notifications again in future
we can put it back.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Several tests still had the old 'apps_run_access' name which we
changed some time ago, so these safety checks weren't working.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This check_screen always fails, because the needle doesn't exist
and never has - the commit that added the check_screen didn't
add a matching needle. In every run of the test I've checked from
the last two months, the initially-selected filesystem is always
xfs anyway. Let's just drop the check_screen conditional and
always expect we have to set the correct filesystem.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The 'universal' flavor has been kinda pointless for some time
now. It dates back to the earliest days of openQA, before Pungi
4 was a thing, when composes were very different; we only built
a boot.iso and some live images nightly for Rawhide, these
weren't even formally grouped as a 'compose' at all (fedfind had
to invent the concept). The TCs/RCs had DVD installer images
(not *Server* DVD, at the time, just a universal DVD installer).
We wanted to run some tests on the DVD image if it was available,
but we still wanted to run them for the nightlies, so we invented
a whole mechanism for that - this 'universal' flavor, with some
complicated logic in fedora_openqa which schedules universal on
the 'best available' image it can find in the compose.
All this is functionally obsolete now. All composes we test are
now run through Pungi (except the live respins, but they aren't
relevant here). In current config, the Server DVD is non-failable
on x86_64 and aarch64, which means it will *always be there* -
if it fails to build, the compose itself fails, so we won't test
it. It's failable for ppc64le, but we don't care that much about
ppc64le; I'm fine with these tests just not running if the Server
DVD happens to fail in a ppc64le compose.
As a cherry on top, some of the 'universal' tests aren't really
universal anyway, they fail if you run them on a netinst (off
the top of my head, all the NFS install tests are like this, as
we use the ISO to populate the NFS share on the server end).
So let's just move all the tests that actually need an installer
image to the Server-dvd-iso flavor. Left over in the 'universal'
flavor are upgrade tests, which don't need an ISO at all - they
boot from hard disk images and run an upgrade using repos. We
can change the scheduler logic to be more simple for these, and
just always schedule them, with no ISO attached. We could even
rename this flavor 'upgrade', but it might not be worth it.
One slight complication is that the split happened to be helping
us avoid too many tests in a single support_server cluster; we
have a cluster of five support_server tests on Server-dvd-iso
and five support_server tests on universal. I try to avoid the
clusters getting too big as you need as many worker instances on
at least one worker host as your largest cluster; if you don't
have that many, the cluster's tests simply never get scheduled.
Requiring folks to have at least ten worker instances on one
host to run these tests is a bit of a big ask. So, to handle
that, we create a support_server_2 and have the former universal
tests use that one instead, so we'll have two separate clusters
on Server-dvd-iso now.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Upstream https://github.com/os-autoinst/openQA/pull/4973 requires
us to poke things here a bit. This only works with the newer
os-autoinst and openQA (there may be a way to conditionalize it
to work with both, but I can't be bothered figuring it out, let's
just update).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This keeps failing on the accessibility section, and looking at
the screenshots I realized why. When you press 'down', GNOME
doesn't just 'snap' to the new view, it does a smooth downward
scroll. We're often matching *while it's scrolling*, so the
needle match is right at the bottom of the screen. But then the
animation continues, so when we get to the click action (even
though we use click_lastmatch it's not *instant* in openQA),
the thing we're trying to click (the "Accessibility" section
title) is a bit further up the screen, and the click 'misses'.
So, we need to wait out the scroll then re-assert and click.
This unfortunately will make the test take about 30 seconds
longer, but I don't see another way to do it. We could maybe
shave the wait_still_screen to one second...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Similar to the dedicated tests for these apps, the app can appear
for a split second before the access request, so we match on the
app and don't realize we need to click through the access
request. Handle this the same way we do in the dedicated tests.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Another message changed format a bit, and all the messages are
now showing up in syslog instead of packaging.log, so handle
all possibilities here. I had to split the first check into two
commands because I can't seem to make it work if I try and do it
all in one command with bracket groups :/
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We actually get a softfail because we're expecting it; now it's
been fixed not to show up, drop the code that expects it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We're constantly seeing this test fail on an odd problem where
text editor starts *behind* characters. To handle this, check
if we see text editor and if not, hit alt-tab.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It always launches in basic mode anyway, and sometimes this key
press doesn't work right and leaves a stray 'b' in the entry
field, which messes things up when we get to the calculation
tests.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This group was only added to comps today, so it's not in the
comps in the rawhide repo yet (will be after next compose). The
Koji repo doesn't have normal comps so it's not there either.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Try and make sure maximize actually works - wait for still screen
after hitting Done and before trying to maximize.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There's a failure mode we hit quite often where, when we run
the text editor after copying the character, it pops up *behind*
the character page. So let's close the page first. When we click
the 'Copy Character' button a notification that the character was
copied is briefly displayed and if we hit esc while it's visible
we dismiss *that* not the character page, so hit esc twice to be
safe. If we miss dismissing the notification, the 'extra' press
is safe, it doesn't quit the app.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
A change to how anaconda handles NFS repos changed the log
messages we get when we use one. We may need further changes for
using NFS as a base repo when this change hits Rawhide nightly.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It looks like the desktop_notifications postinstall test on KDE
on F38 is failing currently because the notification is being
shown during the install_default_upload test that precedes it,
so KDE decides not to show it again yet. So, unset the setting
that stores the timestamp of the last time the notification was
shown. This is similar to a thing we already do for GNOME above.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is needed to force the rebase on current CoreOS (because it
has zincati registered by default). It should be harmless in all
other cases.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Reasoning:
1. pandoc is not in critpath so will not itself be tested
2. pandoc is widely used and actively maintained
3. package is noarch
4. package has minimal deps
Hopefully this will work for everything. For some reason, the
"use python3-blivet for pykickstart tests" fails mysteriously
sometimes, see e.g.
https://openqa.stg.fedoraproject.org/tests/2672282
Signed-off-by: Adam Williamson <awilliam@redhat.com>
See e.g. https://openqa.fedoraproject.org/tests/1829593
sometimes we see the app UI briefly before the access prompt
appears. Handle that case by waiting a few seconds and doing
the match again.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
