These two are for a couple of FreeIPA bugs that showed up today
and were worked out with pemensik and mreynolds.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There's a race issue with just treating it as a next button: it's
not in the same place as a next button. Sometimes in the g-i-s
code we actually get ahead of ourselves and click early, which
isn't really a problem when the buttons are all in the same
place, but if we click "Start Setup" in the middle of transition
to the Privacy screen - as in
https://openqa.fedoraproject.org/tests/745034#step/_graphical_wait_login/4
- the click effectively gets lost. So let's make it its own tag
and have the initial assert look for it too. That way we won't
match on it again in the main loop over "@nexts".
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This sets us up to test the release-blocking aarch64 disk images
(Minimal, Server and Workstation). It also allows for testing
armhfp disk images on aarch64 worker hosts (though my testing of
that isn't going too well so far), and fixes the initial-setup
handling for a change upstream ('use password' is now the default
so we don't need to choose it). We rewire disk image deployment
test loading to work through the generic loader code rather than
using ENTRYPOINT, as it allows us to more gracefully handle
graphical (Workstation) vs. console (Server, Minimal), moving
the code for handling console initial-setup to a helper function
just like the code for gnome-initial-setup and having _console_
wait_login call it when appropriate. We also tweak desktop_vt a
bit because now we need to switch from a console running as test
to a desktop, which breaks the assumption that the highest
numbered session of user test is the desktop...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
FreeIPA upgrade test is failing because of
https://bugzilla.redhat.com/show_bug.cgi?id=1886205. The test
failing every time is not useful as we know what the issue is,
so add the update as a workaround to avoid it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Add FEDORA-2020-27f80050a2 as a workaround because without it the
KDE desktop background test fails due to the bug in -6.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We were using it to checkout a git version of python-fedora to
work around a bug, long ago, but we don't do that any more so
we don't need it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
openQA sometimes winds up testing an update that doesn't have
any packages for x86_64 (or aarch64). The most common case is
s390utils, which is on the critpath but only has packages for
s390x. I would ideally like to skip scheduling entirely if the
update has no packages for the arch we're scheduling on, but
sadly that involves using the Koji API which is XML-RPC and I
don't really want to deal with that again. This deals with it
at the test level instead, by checking the error message if
`koji download-build` fails and carrying on if it's the "no
packages for this arch" error. That means if the update has no
packages at all for our arch we're not really testing anything,
but that's better than a bunch of false failures, I guess.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Kernel updates for F31 and F32 went stable so they can come out.
mock 2.6 fixes the bug that occurs when /etc/resolv.conf is a
dangling symlink - this breaks the live_build test.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Kernel 5.8.8 broke the installer by changing return codes for
partition operations, these updates are listed as fixing it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The F32 FreeIPA update that changed the web UI has gone stable
now, so remove it. A pki-core update has just come out that fixes
F32 -> F33 FreeIPA server upgrade test: add this as a workaround
for F33 so that test stops failing on every update.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Previously we were relying on `rpm -q` always outputting the
right package last. We saw some test failures on recent kernel
updates, e.g. https://openqa.fedoraproject.org/tests/658768 ,
which indicate this isn't always the case; there the 'right'
package was second of three for kernel, third of three for
kernel-core and first of three for kernel-modules. So we need to
make it more robust. This uses an additional call:
`rpm -q $pkg --last | head -1` to find the most recent package,
if there are more than one; this should always be the right one,
I hope. Note we cannot just add `--last` to the `rpm -q --qf...`
call because the output when you do that is weird; you get the
output you'd get if you just called `rpm -q --last` first, and
*then* the query-formatted output afterwards (though with the
modified order as expected). There doesn't seem to be any way to
get only the latter.
I also tweaked the log uploading so we always upload the working
logs even when the test passes; it can't hurt anything and it is
sometimes useful to have them.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The FreeIPA UI change that the previous commit adapted to is in
4.8.9. That's stable for Rawhide and F33 already, but still in
testing for F32, and won't go to F31. So we need to make the
change conditional on release number, and we also add the update
to workarounds for F32 so we don't have to do something awkward
while we wait for it to go stable.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
In g-i-s 3.37.91, the first screen has a 'Start Setup' button
rather than a 'Next' button. Easiest thing for us to do here is
just to add a new needle which has the 'next_button' tag even
though it's clearly not a 'Next' button, because then the code
still works :) So do that, but give the file a suggestive name
and explain the situation in a code comment.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It's failing about one in six tries currently, with Bodhi 5.5 on
the server end: https://github.com/fedora-infra/bodhi/issues/4105
this should work around that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The ones that were in there are stable now, plus downloading them
is hitting a bug in Bodhi and breaking tests.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is a bit complex to automate, because we cannot really use
the production Zezere server (provision.fedoraproject.org) as
the test case shows, as we'd have to solve authentication and
we also don't really want to constantly keep registering new
hosts to it that are going to disappear and never be seen again.
So, instead we'll do it by setting up our *own* Zezere, and
provisioning our IoT system in that. We run two tests. The
'ignition' test is the actual IoT 'device'; all it really does
is boot up, sit around, and wait to be provisioned. The 'server'
test first sets up a Zezere server, then logs into it, adds an
ssh key, claims the IoT device, provisions it, and connects to
it to create a special file which tells the 'ignition' test
everything worked and it can close out.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The annoying submenus in the overview app list now scroll right
not down :/ have to adapt this function for that. Had to move
get_release_number earlier because perl ordering.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
See https://pagure.io/background-logo-extension/issue/26 - in
current Rawhide, the search box in the overview is not active
when the overview is opened, so you can't just open the
overview and type, you have to click it first.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This should avoid the bug happening in upgrade tests (I already
built the fix into the base disk image, which should avoid it
happening in other tests).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
See https://gitlab.gnome.org/GNOME/gnome-shell/-/issues/2878 .
GNOME 3.37.2 seems to have a bug with submenus in the app menu;
the first time you open one you can't scroll through it using
the keyboard. On every open after the first it works fine. This
is a quick and dirty workaround - when we're dealing with a
submenu, open it then close it then open it again.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I started out trying to fix os-release for the recent change to
add "Prerelease" tags to the VERSION and PRETTY_NAME fields, then
things spiralled. It got me thinking about the awkward DEVELOPMENT
variable we use, so I decided to get rid of it and refactor the
few things that use it. I refactored the anaconda prerelease tag
check, and wrote a new giant comment that gives details about
exactly how anaconda decides whether to show those tags, to give
context to our choices about when to expect them. This check now
uses a new LABEL variable the scheduler now sets. I also wound up
creating new UP1REL and UP2REL vars to define the 'source' release
for upgrade tests, separate from CURRREL and PREVREL, which are
now never lies - they really are the current stable and previous
stable release, even for update upgrade tests.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The old code waited after launching the terminal, the new code
doesn't, which led to a 'g' being swallowed in the first command
in https://openqa.fedoraproject.org/tests/592759 .
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This adds a new test that implementsQA:Testcase_desktop_login
on both GNOME and KDE.
While working on this, we realized that the "desktop_clean"
needles were really "app menu" needles, and for KDE, this was
a duplication with the new "system menu" needles, because on KDE
the app menu and the system menu are the same. So I (Adam)
started to de-duplicate that, but also realized that "app menu
button" is a much more accurate name for these needles, so I was
renaming the old desktop_clean needles to app_menu_button. That
led me to the realization that "check_desktop_clean" is itself a
dumb name, because we don't (at least, any more, way back in the
mists of time we may have done) do anything to check that the
desktop is "clean" - we're really just asserting that we're at a
desktop *at all*. While thinking *that* through, I *also* realized
that the whole "open the overview and look for the app grid icon"
workaround it did is no longer necessary, because GNOME doesn't
use a translucent top bar any more. That went away in GNOME 3.32,
which is in Fedora 30, our oldest supported release.
So I threw that away, renamed the function "check_desktop",
cleaned up all the needle naming and tagging, and also added an
app menu needle for GNOME in Japanese because we were missing
one (the Japanese tests have been using the "app grid icon"
workaround the whole time).
FreeIPA F31 -> F32 upgrade test is currently failing because
a new pki-core hit F31 stable but not F32 stable yet. It can't go
backwards on upgrade, that breaks stuff. The F32 update has been
pushed stable but just hasn't made mirrors yet as the last F32
nightly compose failed, so let's add it to the workarounds for
now.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Except one that's pushed stable but hasn't made repos yet (as the
last F32 nightly compose failed).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is pending stable, but looks like the update push won't
happen for a few hours, so I'm adding it as a workaround so we
can re-run the tests and get them to pass.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Otherwise the new cockpit main screen needle match fails, because
'machine' is not on the top part of the screen.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This block is kinda weird, but I don't want to fiddle with it
any more right now. I don't know why we do this check_screen
exactly like this.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
...because it comes out as 'cleqr'. Note, this may be fragile if
we start doing more stuff post-install, but for now I think it's
safe, I don't *think* we should ever hit this after running
`loadkeys us`.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It seems like the situation where we need to pull an update from
updates-testing into all update tests to work around some known
issue is going to keep happening. So instead of constantly
adding and then entirely removing bespoke lines for each specific
workaround, let's have a permanent mechanism for doing this: a
hash with release numbers as keys, and arrayrefs of update IDs
as values, and a block to call `bodhi updates download` on the
appropriate array for the release under test. This way, to add
or remove a workaround you just update the hash. If we're at a
point where *no* workarounds are needed the %workarounds hash
can be made entirely empty (it must exist, though) and the code
will be a clean no-op.
The actual workaround here pulls in Lmod updates I just sent out
to work around this issue in one of the KDE update tests:
https://openqa.fedoraproject.org/tests/497160#step/base_update_cli/11
there's some code in Lmod that gets sourced in bash profiles
which breaks openQA's `validate_script_output` by blurping two
lines of informational output into the output of the script.
The update backports a change from upstream Lmod master that
sends that informational output to stderr instead of stdout.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The jss updates all went stable already. Now we have a problem
with SELinux, upower and container-selinux (we need a newer
selinux-policy to avoid upower failures in the services_start
test, but the first attempt to fix it caused the desktop_updates
test to start failing because container-selinux needed adapting
to changes in selinux-policy...let's just pull in the updates
with the latest versions of both to be safe), and one with NSS
that causes Firefox to give false certificate errors sometimes
(this is particularly affecting the FreeIPA browser test). As
usual these should be dropped once the updates go stable.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
On Rawhide Cloud_Base boots, there's some SSH key and network
information printed above the 'login:' prompt, so we can't
expect empty space there. Also tweak console_login() to clear
the screen after logging in, so the login prompt is cleared and
doesn't confuse things on subsequent runs (like it did first
time we tried this). And add a new user logged in needle, as it
seems after we clear the screen the tilde appears in a slightly
different position and the existing needle doesn't match.
https://openqa.fedoraproject.org/tests/489003#step/_console_wait_login/7
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This adds a test for QA:Testcase_Anaconda_User_Interface_VNC -
the VNC install test case. It's implemented as a server/client
pair, with the server booting from the Server DVD image with
`inst.vnc` and the client booting from the desktop base disk
image, setting up networking, then running Boxes to connect to
the server and run the install.
There are various little tweaks to test loading and logic to
handle this, mostly pretty clear. We also move the workaround
for 'spurious auth prompt appears on desktop after you switch
away to a VT and back' out of the desktop update test and into
the `desktop_vt` helper function, since now this test can hit
it as well. We enhance _graphical_wait_login to handle the boot
loader if needed (it has never needed to until now).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This handles a case where KDE shows a notification saying
'PIM Maintenance (Finished)', like this:
https://openqa.fedoraproject.org/tests/477345#step/desktop_notifications/34
we need to click it away for the desktop_notification test to
pass. It also clarifies the difference between this notification
and the eternal 'akonadi_migration_agent is doing something'
popup in the needle names and comments. It also replaces the
'check_screen then assert_and_click if found' pattern in several
notifications-related places with the better 'check_screen then
click_lastmatch if found' pattern now available upstream.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This PR adds the following:
* moves out the presetting procedures, so that two long terminal tests do not have
to run twice
* add methods for application to register when successfully started
* adds a test that checks if all required applications have registered
* server-cockpit-updates tests that Cockpit can be used to update the system.
* server-cockpit-autoupdate tests that users can use dnf-automatic for system
updates.
* cockpit functions were removed from utils.pm and put into an extra library
for cockpit - cockpit.pm which all cockpit tests are now using.
Review cockpit.pm
Review autoupdate test.
Review the update test.
Fix typo in cockpit.pm
Add sleep.
Add missing command.
Delete an unused needle.
This goes with a corresponding fedora_openqa commit that passes
ADVISORY_NVRS when scheduling update jobs. This is to address
https://pagure.io/fedora-qa/fedora_openqa/issue/78 so we can
publish spec-compliant and correct CI Messages: we will decide
what NVRs are in the update at scheduling time and always get
and test those NVRs, rather than the test downloading whatever
is in the update when it is run.
Some consequences: manual restarting or cloning of an update
test scheduled before this change lands will result in failure;
to do this you'll have to add the ADVISORY_NVRS value manually
with clone_job.pl, or simply reschedule the update with
fedora-openqa.
Manual restarting or cloning of an update test scheduled with
this change will always test the same NVRs. If the update has
changed and you want to re-test with the new packages in the
update, you must change ADVISORY_NVRS manually with clone_job,
or reschedule with fedora-openqa.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
reported in autoinst-log.txt
===
"my" variable $files masks earlier declaration in same scope at /var/lib/openqa/share/tests/fedora/lib/utils.pm line 352.
"my" variable $beta masks earlier declaration in same scope at /var/lib/openqa/share/tests/fedora/lib/utils.pm line 995.
===
Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
We need another needle for French installs, and we need to use
ISO_URL for checking the ISO file name if ISO isn't set.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
On F31 update tests, desktop_vt is broken because the 'ps -C'
command is showing 'tty?' as the tty on which Xwayland/Xorg is
running. Let's try using loginctl as a workaround for this.
This ordering of commands should ensure the ps -C output takes
precedence when it's correct.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
A couple of tests failed in F31/Rawhide lately because quitting
Firefox (running on a bare X server) took more than 10 seconds
to get back to a console...not sure why, but it's not something
we ought to waste a lot of time on, let's just bump the timeout.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We do this in quite a few places and need to do it in another,
so let's just have a function for it. It takes a file glob
so we can have it run on a different one for _live_build.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There are some situations in which ISO_URL is not set (e.g. a
manual POST to re-run tests after download has been done). If
there is no ISO_URL just skip the checksum check here.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
A broken lorax got into F29 stable and is causing live image
builds to fail in all F29 update tests. Work around that by
downloading a newer fixed package in _repo_setup_updates.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Current tests sometimes seem to fail because X takes quite a
while to come up for some reason. I can't figure out why, and
this isn't properly a failure of the test (starting the X server
is an implementation detail), so let's just wait a bit longer
for now.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There are three places where we basically want to click away
pop-up update notifications and the buggy akonadi_migration_agent
notification if it's there, in KDE tests. Let's share this code
between them, and also let's record soft failures for the buggy
cases in the desktop_notifications test.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
* Add test to check module defaults.
* Add whitelist download.
* Fix install test to include selected profile to be on the safe side.
* Add test into templates.
type_very_safely commonly gets stuck waiting on a still screen
at the end because when you're typing, there's often a flashing
cursor, and the default similarity for wait_still_screen is 47
which is tight enough that a flashing cursor usually fails it.
So back it off to 45.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We're not downloading all the packages from updates that contain
more than one builds ATM, which makes the test invalid and has
caused some false fails (and may even have caused false passes,
though I can't tell yet). Install a fixed Bodhi as a workaround
for this.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
So, turns out new os-autoinst does *not* still accept the old
argument style for assert_and_click...and old os-autoinst
doesn't accept the new one. This adds a wrapper that handles
both, so our tests can work with old and new os-autoinst. We can
drop this once both deployments are on newer os-autoinst.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
In the new os-autoinst I just sent to staging, the old style
doesn't work any more, breaks all tests. This style should also
work with the older os-autoinst on stable.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
'use password' now seems to be default when creating a user in
text mode, so we need to handle that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
A BIND update broke a test, then went stable, so it's failing on
all updates right now. This subsequent update fixes the bug, so
let's pull it in here until it's pushed stable.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Because we use check_screen not assert_screen here, the match
can actually fail, but the match_has_tag conditional can pass
on the *previous* match, if that happened to also be matching
on a console tag. We don't want that. Let's just make these
into assert_screens to avoid it; I don't think there's any path
where we're actually expecting this to work if those
check_screens didn't match.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Add png files as a background for the needles.
Rewrite the test handling methods to make them ready for KDE
Add the ABRT startstop test.
Make changes and corrections to the abrt test.
Add test for Firewall.
Add test for DNF dragora.
Add wait time for dnfdragora test.
Correct syntax.
Add Language test.
Make some changes to the DNF dragora test.
Add Users test case.
Add needles for DBUS viewer test.
Add Dbusviewer test.
Add Mahjong test and needles for games.
Add Minesweeper tests.
Add Patience test.
Add test for Document Viewer.
add test for gwenview
Add test for koulourpaint.
Add test for Kruler
Add test for Kcolorchooser
Reneedle failing needle.
Add ktorrent tests.
Add tests for CPT editors.
Add test for Krfb
Fix names for those files.
Add test for Kget
Add Akregator test.
Add test for Konversation.
Make Konversation end really.
Add tests for Kmail
Add test for PIM exporter.
Add test for KTnef and Krdc.
Fix problems after test runs.
Make more tries.
Fix needle to be found better.
Fix more errors.
Add test for Falkon.
Add tests for browsers.
Add support for closing tabs into Firefox test.
Add tests for K3B
move needles to correct directory
Add Kaddressbook test.
Add Kontakt text.
Add test for korganizer.
Add menu office needle and correct konqueror needle.
Add test for calligra stage.
Add test for Calligra.
Add test for network connections.
Modify needle for kaddressbook to prevent failing.
Add test for system settings and fix others.
Add test for FMW.
Add test for Dolphin
Add test for Infocenter.
Add test for kparted.
Fix a wrong needle.
Test relnotes.
Fix some errors in tests.
Add test for Discover.
Add test for Ksysguard.
Add tests for Konsole.
Add tests for KDE wallet.
Add tests for several utilities.
Add Krusader test.
Finish utility tests.
Fix some errors.
Fix needle for spectacle.
Add wait time to let Dragora wait for network.
This tweaks the update live and installer image build tests to
store the update repo on a separate disk. This should hopefully
avoid the tests failing due to insufficient disk space when the
update is huge (e.g. KDE or GNOME megaupdates).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The initial implementation here has a problem if we spot the
'successful' screen briefly, then the system reboots normally,
reaches the bootloader and proceeds past it all within 10
seconds; in this case we'll never actually spot the bootloader
and do our stuff. This tweak should continue through the code
block immediately if the bootloader shows up during the ten
seconds, otherwise check again for the 'successful' screen and
reboot.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The f30 tag is currently a lot fresher than what's in the repos,
because we haven't had a compose for five days. This causes
problems in some F30 update tests because the update works fine
with what's in the `f30` tag but not what's on the mirrors. To
deal with this until we get a successful compose, let's just
enable the f30-build repo in the update test process...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We want to get accurate results from the rest of the upgrade
test and we know about this bug, so let's make it a soft fail
so we can see how the test is functioning otherwise.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I've seen a few tests fail lately because Firefox suddenly
opened a tab with some "Privacy Study" in it and switched to it.
Per https://bugzilla.mozilla.org/show_bug.cgi?id=1529626 , this
should be the way to disable these...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The previous attempt at this (in a20ea59) was wrong, because it
meant we didn't create the update repo at all, and we do need
the repo to exist in the support_server test every time (as the
point of the support server is to serve it out via NFS). We just
don't want to *use it on the support_server system itself* if
the releases don't match. So rejig things to do that properly.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We quite often want to run the update tests on a Koji task (not
a Bodhi update) for some reason - usually to test a potential
fix for an issue, or at a maintainer's request to test a change
before it is merged upstream and officially sent out as an
update. Up till now I've always hacked up utils.pm on the
staging server by hand to do this, which is horrible. Together
with a commit to fedora_openqa, this should allow us to do it in
a nice, sane way via the CLI. It's mostly just tweaking the
"updates" repo setup in utils.pm as you'd expect, but there's a
bit of subtlety to it because of the installer tests that use
%ADVISORY% as a variable substitution in the disk image name;
you can't do something like `%ADVISORY or KOJITASK%`, sadly, so
I had to have almost-redundant variables ADVISORY, KOJITASK and
ADVISORY_OR_TASK (we could kinda just live with ADVISORY_OR_TASK
except I didn't want to drop ADVISORY as it's an unnecessary
change from previous behavior).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The packages from this update seem to be breaking F28 update
tests for some reason; a later update has gone stable anyway, so
this is no longer needed.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
A bug showed up in Rawhide where, when you run startx in a tty,
when you exit that X session, the tty quits and returns to the
login prompt. This is a slightly sloppy workaround for that
problem.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
That clever-clever 'check the packages from the update were
installed' thing from yesterday breaks on kernel updates, as
they're installonly; after the update, the new version of the
package is installed, but the *old* version is too, and the way
I implemented the check, it treats that as a failure. Let's try
and handle this a somewhat-clever way (if this fails, I'm just
going to grep out lines with 'kernel' in them, as a *dumb* way).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
If an update test fails before reaching advisory_post, we don't
generate the 'what update packages were installed' and 'were
any update packages *not* installed when they should have been'
logs, but these may well be useful for diagnosing the failure -
so let's also do the same stuff there. Only let's not do it all
twice.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We hit an interesting case in update testing recently:
https://bodhi.fedoraproject.org/updates/FEDORA-2018-115068f60e
An earlier version of that update failed testing. When we dug
into it a bit, we found that the test was failing because an
earlier version of the `pki-server` package was installed than
the version that was in the update; when asked (as part of
FreeIPA deployment) to install it, dnf had noticed that there
were dependency issues with the version of the package from the
update, but it happened to be able to install the version from
the frozen 'stable' repo...so it just went ahead and did that.
In this case, the 'missed' package resulted in a test failure,
but it'd actually be possible for this to happen and the test
to complete; we really ought to notice when this happens, and
treat it as a test failure.
So what this attempts to do is: at the end of all update tests,
check for all installed packages with the same name as a package
from the update, and compare their full NEVR to the one of the
package from the update. If a package with the same name as one
of the update packages is installed, but does not appear to be
the *same NEVR*, we fail, and upload the lists of packages for
manual investigation as to what the heck's going on.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
That whole creaky edifice of conditionals that figured out how
many times to press 'down' was a mess I always hated, and I just
found out that the fix for BLS wasn't complete - I'd assumed in
writing it that systems weren't being migrated to BLS on upgrade
to F30, but actually they are. This makes that design very hard
as we'd have had to find a way to change the number of 'down'
presses part-way through update tests, and all the ways I can
think of to do that would've made this even sillier.
Happily I managed to come up with what looks like a much simpler
approach: just go from the bottom. It seems that in every setup
I can think of to check - all three arches, BLS, no BLS, pre-
install, post-install - the linux line is two lines up from the
bottom of the config stanza (the last line is blank, and the
last line but one is the initramfs line). So we can just press
down 50 times (to make damn sure we're at the bottom) then press
up twice and we should be in the right place, no matter the arch,
the release, or if BLS is in use or not. Whew.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This bug is breaking all update FreeIPA tests; until the updates
go stable, let's pull them in to update tests so the results
are useful.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The one we were using before doesn't seem to exist any more in
Rawhide. /etc/os-release should be fine.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Now the BLS stuff is enabled in Rawhide, we need to press 'down'
a different number of times to reach the 'linux' line when
editing the boot params (I really, really wish there was a
better way to do this :<). It gets tricky as there are all sorts
of cases here (support_server tests use a CURRREL disk image,
and then there's upgrade tests)...I think this covers things for
now.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Since a recent sssd update, console login during FreeIPA tests
is taking unusually long. We don't want this to fail all the
tests, so let's extend the timeout, but with a soft fail.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Somehow, recently, FreeIPA tests are running into Firefox not
quitting because it's showing a warning about closing multiple
tabs. (I think we didn't *get* multiple tabs before but now we
do, for some reason). So let's work around this by clicking
"Close tabs" if the warning appears.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
For some reason, in recent tests, switching to a console after
live install completes is taking a long time, and tests are
failing because we 'only' allow 10 seconds for the login prompt
to appear. This seems to indicate some kind of performance bug,
but we don't really want all liveinst tests to fail on in, this
is not primarily a performance testing framework. So let's
tweak the root_console / console_login bits a bit to allow a
configurable timeout for the login prompt to appear, and use
that to wait 30 secs instead of 10 in this case.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Looking at this, it's a bit weird: the updated packages are
actually included in the upgrade process, but we still run
_advisory_update, which does basically nothing...then reboots.
That's kinda silly and makes the tests a bit flaky, let's fix
it. I don't think there's actually any problem with doing the
upload of updatepkgs.txt in _repo_setup_updates, becase that
already guards against being run more than once, it just bails
very early if it's already been run.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There seems to be a bug in Rawhide lately where, when our tests
want to install a bare X and run Firefox on it, this takes an
unusually long time to start up, with SELinux in enforcing mode.
With SELinux in permissive mode it starts as fast as usual. This
isn't a hard failure and we don't want it to block all later
tests, so let's handle it and treat it as a soft fail.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
OK, we now need to work around this goddamn grub bug in *three*
places, so let's stop copying the loop around and factor it out
instead. The third place is encrypted installs, as they wait
for the decryption prompt on boot.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Per Neal Gompa boot will proceed if we just page through the
error(?) messages displayed when #1618928 happens, so let's do
that to let the tests get further and see what else is broken.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It seems that for some reason the localized layout gets loaded
on the installer VTs by this point in time, so we need to load
'us' again for this complex command to work.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Upstream is gonna change the default from 30 to 0, it seems:
https://github.com/os-autoinst/os-autoinst/pull/965
so let's go ahead and change these two cases where we have no
explicit timeout to have one.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The reason we have all this horrible code to use the commented-
out baseurl lines in the repo files instead of the metalinks
that are usually used is a timing issue with the metalink
system. As a protection against stale mirrors, the metalink
system sends the package manager a list of mirrors *and a list
of recent checksums for the repo metadata*. The package manager
goes out and gets the metadata from the first mirror on the
list, then checksums it; if the checksum isn't on the list of
checksums it got from mirrormanager, it assumes that means the
mirror is stale, and tries the next on the list instead.
The problem is that MM's list of checksums is currently only
updated once an hour (by a cron job). So we kept running into
a problem where, when a test ran just after one of the repos
had been regenerated, the infra mirror it's supposed to use
would be rejected because the checksum wasn't on the list - but
not because the mirror was stale, but because it was too fresh,
it had got the new packages and metadata but mirrormanager's
list of checksums hadn't been updated to include the checksum
for the latest metadata.
All this baseurl munging code was getting ridiculous, though,
what with the tests getting more complicated and errors showing
up in the actual repo files and stuff. It occurred to me that
instead of using the baseurl we can just use the 'mirrorlist'
system instead of 'metalink'. mirrorlist is the dumber, older
system which just provides the package manager a list of mirrors
and nothing else - the whole stale-mirror-detection-checksum
thing does not happen with mirrorlists, the package manager just
tries all the mirrors in order and uses the first that works.
And happily, it's very easy to convert the metalink URLs into
mirrorlist URLs, and it saves all that faffing around trying to
fix up baseurls.
Also, adjust upgrade_boot to do the s/metalink/mirrorlist/
substitution, so upgrade tests don't run into the timing issue
in the steps before the main repo_setup run is done by
upgrade_run, and adjust repo_setup_compose to sub this line out
later.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Now F28 went stable, we're not disabling updates on upgrade any
more, and this bug got exposed: the location of the updates and
updates-testing repos actually changed between F27 and F28, so
the `baseurl` line from fedora-repos in F27 isn't correct for
F28. When doing an upgrade from < 28 to > 27, we need to correct
the URL when we're done installing stuff from the old release
repos but before we start trying to pull stuff from the new
release repos.
This repo munging crap is really getting fragile, it'd be great
if we could get that metadata timing issue resolved so we could
reliably use mirrormanager...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This adds the FreeIPA server and client upgrade tests to a new
updates-server-upgrade flavor which fedora_openqa will schedule
for updates. This way, we can test whether updates break
FreeIPA upgrades, which is a request the FreeIPA team made to
me. This has been deployed on staging for the last week or so
and appears to work fine.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Since gnome-initial-setup-3.28.0-5.fc28 , the g-i-s screens
that are supposed to be suppressed as part of
https://fedoraproject.org/wiki/Changes/ReduceInitialSetupRedundancy
are now suppressed on FAW installs as well as traditional ones.
So adjust the logic accordingly.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Trying to keep track of what these magic numbers mean is really
getting messy, so let's do it a bit more explicitly, using the
page names g-i-s uses internally, and lots of comments. This
should make it clearer and more maintainable when stuff changes.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
aarch64 managed to hit the problem this 'magic timeout' tries
to avoid, so let's extend it :(
e.g. https://openqa.stg.fedoraproject.org/tests/267174
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I believe this should do all the right repo modifications for
add-on Modularity (i.e. F28+ Server installs, for now).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Seems aarch64 needs 12 'down' key presses like ppc64, not 13
like x86_64. Tweak how this is done a bit; the ternary wasn't
elegant any more with the aarch64 change, so just get rid of it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This works around RHBZ #1552814, and it's not incorrect really
because the repo is always empty for Branched. I didn't do it
before because we might theoretically start using the repo for
Branched at some point in the future, and if we did that we'd
probably want it enabled for this test. But to get F28 update
tests working, let's just turn it off for now.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There's a bug causing the 'getting started' screen to crash.
This doesn't really make the system unusable, so treating it
as a soft failure seems appropriate, especially as this will
unblock all the post-install tests on Workstation.
Modular composes don't include these packages, but we need them
to run the web UI tests for FreeIPA and Cockpit. This is the
most reasonable hack I can come up with for now: just use a
non-modular fedora repo to source these packages when doing
Modular compose testing.
If we ever reach an all-Modular future, these packages should
be available in Modular composes I guess, but for now they are
not.
and call save_screenshot to visually check
for debug purpose only
Also change for PowerPC the number of down key to 12
(rather than 12)
Seems to be mandatory since 20170327.
Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
to avoid upgrade_server test to fail with:
"Repository fedora-source has no mirror or baseurl set."
Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
* New OFW variable to identify Open Firmware (used by PowerPC)
* Few needles changes for PowerPC support
* as requested do not change the timers value below for PowerPC
tests/install_source_graphical.pm (300 to 600)
tests/_boot_to_anaconda.pm (300 to 1200)
This will be handled by TIMEOUT_SCALE in templates
Signed-off-by: Guy Menanteau <menantea@linux.vnet.ibm.com>
bodhi-client doesn't depend on the 'koji' package but does need
it to do 'bodhi updates download', which we want to do. So we
must explicitly install it here.
There's a bug in current Rawhide causing sourcing of /etc/bashrc
to fail when logging in as a regular user. This results in the
bash prompt looking different, which is currently a hard fail,
and causes most tests to die. It's better to treat this as a
soft fail so the rest of the test can run. So add a needle to
spot this case, and a little finish function the console login
function calls whenever it's successfully logged in, to check
whether it got the no-profile prompt and register a soft fail.
Well, that OCR needle isn't working out so great, as it seems
to match when it shouldn't:
https://openqa.fedoraproject.org/tests/119217#step/_graphical_wait_login/5
So let's try another approach. Ditch the OCR needle and have a
function for checking we're at a clean desktop. It does the
normal needle match, but if we're on GNOME, it also tries
hitting alt+f1 and seeing if we're at the overview; if so, it
hits alt+f1 again (to go back to the desktop) and returns.
Summary:
This adds a new test suite, run for Workstation and KDE live
images, which does not create a user during install. It then
expects initial-setup (KDE) or gnome-initial-setup (Workstation)
to appear after install, creates a user, and proceeds with
normal boot.
Note the ARM image test already covers the initial-setup text
mode, and the ARM minimal image is the only case where that
actually matters (it's not included in Server).
Test Plan:
Run the new tests, check they work. Run all old
tests, check the changes didn't break them.
Reviewers: jsedlak, jskladan
Reviewed By: jsedlak
Subscribers: tflink
Differential Revision: https://phab.qa.fedoraproject.org/D1185
This repo is causing problems for Branched update tests. The
repo is not available for 26 at all yet. This shouldn't be a
problem as the repo is disabled by default, but it seems that
some things - at least realmd, as used in the FreeIPA enrolment
tests - still try to update the repo's metadata when installing
packages, and fail because it 404s.
Since none of our tests actually needs this repo AFAIK, let's
just delete it in repo_setup.
Branched update tests are all failing because the baseurl in
fedora.repo is incorrect for Branched. This is a rather hacky
fix for this problem. It relies on the scheduler setting the
DEVELOPMENT variable when the update is for Branched (I named
the variable DEVELOPMENT rather than BRANCHED to be more
future-proof).
Alternative options I rejected were:
i) stick with MM links
ii) do something 'clever' to retrieve the URLs from MM
Rejected i) because the timing problem where the infra repo gets
updated before MM has the updated repodata checksums is just too
much of a problem; whenever that happens, dnf will refuse to use
the metadata from the infra repo and go pull it from an external
mirror, which can wind up timing out.
Rejected ii) because it seemed too fancy and not really any more
robust than just doing this and adapting it if Things Change In
Future (TM).
Summary:
This adds some logging related to the update testing workflow,
so we have some idea what we actually tested. We log precisely
which packages were actually downloaded from the update - this
is important as updates can be edited and when examining results
we'll want to know which packages actually got used. We also
add a new module which runs at the end of postinstall and tries
to figure out which packages from the update were installed in
the course of the test. This still isn't a guarantee the test
actually *tested them* in any way, but it at least means they
got installed successfully and didn't interfere with the test.
Test Plan:
Run the update test workflow, check the logs get
uploaded and seem accurate (sometimes some RPM garbage messages
wind up in the package log, I'm not too worried about that at
present). Run the compose test workflow and check it didn't
break.
Reviewers: jsedlak
Reviewed By: jsedlak
Subscribers: tflink
Differential Revision: https://phab.qa.fedoraproject.org/D1149
Summary:
This adds an entirely new workflow for testing distribution
updates. The `ADVISORY` variable is introduced: when set,
`main.pm` will load an early post-install test that sets up
a repository containing the packages from the specified update,
runs `dnf -y update`, and reboots. A new templates file is
added, `templates-updates`, which adds two new flavors called
`updates-server` and `updates-workstation`, each containing
job templates for appropriate post-install tests. Scheduler is
expected to post `ADVISORY=(update ID) HDD_1=(base image)
FLAVOR=updates-(server|workstation)`, where (base image) is one
of the stable release base disk images produced by `createhdds`
and usually used for upgrade testing. This will result in the
appropriate job templates being loaded.
We rejig postinstall test loading and static network config a
bit so that this works for both the 'compose' and 'updates' test
flows: we have to ensure we bring up networking for the tap
tests before we try and install the updates, but still allow
later adjustment of the configuration. We take advantage of the
openQA feature that was added a few months back to run the same
module multiple times, so the `_advisory_update` module can
reboot after installing the updates and the modules that take
care of bootloader, encryption and login get run again. This
looks slightly wacky in the web UI, though - it doesn't show the
later runs of each module.
We also use the recently added feature to specify `+HDD_1` in
the test suites which use a disk image uploaded by an earlier
post-install test, so the test suite value will take priority
over the value POSTed by the scheduler for those tests, and we
will use the uploaded disk image (and not the clean base image
POSTed by the scheduler) for those tests.
My intent here is to enhance the scheduler, adding a consumer
which listens out for critpath updates, and runs this test flow
for each one, then reports the results to ResultsDB where Bodhi
could query and display them. We could also add a list of other
packages to have one or both sets of update tests run on it, I
guess.
Test Plan:
Try a post something like:
HDD_1=disk_f25_server_3_x86_64.img DISTRI=fedora VERSION=25
FLAVOR=updates-server ARCH=x86_64 BUILD=FEDORA-2017-376ae2b92c
ADVISORY=FEDORA-2017-376ae2b92c CURRREL=25 PREVREL=24
Pick an appropriate `ADVISORY` (ideally, one containing some
packages which might actually be involved in the tests), and
matching `FLAVOR` and `HDD_1`. The appropriate tests should run,
a repo with the update packages should be created and enabled
(and dnf update run), and the tests should work properly. Also
test a regular compose run to make sure I didn't break anything.
Reviewers: jskladan, jsedlak
Reviewed By: jsedlak
Subscribers: tflink
Differential Revision: https://phab.qa.fedoraproject.org/D1143
I accidentally left the `my $self = shift` lines in these when
changing them from methods into functions, so they don't work
right at all. Whoops. Sorry.
Summary:
This adds a couple of new exporter modules, renames main_common
to utils (this is a better name: openSUSE's main_common is
functions used in main.pm, utils is what they call their module
full of miscellaneous commonly-used functions), and moves a
bunch of utility functions that were previously needlessly
implemented as instance methods in base classes into the
exporter modules. That means we can get rid of all the annoying
$self-> syntax for calling them.
We get rid of `fedorabase` entirely, as it's no longer useful
for anything. Other base classes keep the 'standard' methods
(like `post_fail_hook`) and methods which actually need to be
methods (like `root_console`, whose behaviour is different in
anacondatest and installedtest).
Test Plan:
Do a full test suite run and check everything lines
up. There should be no functional differences from before at all,
this is just a re-org.
Reviewers: jskladan, garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames
Reviewed By: garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames
Subscribers: tflink
Differential Revision: https://phab.qa.fedoraproject.org/D1080
