It seems Rawhide auto-signing is working fine now: openQA claims
this needle has 'never' been matched (which really means 'not
for a long time'), and I can't find any test fail in the last
year which looks like it landed on this 'authenticate' screen
but the needle failed to match, or anything like that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
https://openqa.stg.fedoraproject.org/tests/424393 is a failure
where the 'Download' [updates] button was already visible when
we went to the tab. We already checked whether an 'apply' button
is visible and skipped the 'refresh' click if so, but because
the 'download' button is a new thing, we weren't skipping the
'refresh' click if 'download' was already visible.
So in this case, even though we could already see 'download', we
went ahead and clicked 'refresh'...then *immediately* started
looking for 'download'. It seems that Software did not refresh
and remove the 'Download' button *immediately* when we pressed
'refresh' - it left the 'Download' button visible briefly, and
*in this brief window*, we clicked it. *Then* Software kinda
'noticed' we'd clicked 'Update', and it seems it just sort of
throws away our click on 'Download' at that point and does the
refresh.
So at that point, the test thinks it's clicked 'Download' and
expects to see 'Apply', but actually the 'Download' click got
more or less thrown away, so the test fails, sitting at the
'Download' button.
To solve this, let's just extend the existing check to skip the
'refresh' click if 'download' *or* 'apply' are already visible.
There is a sort of possibility here that we could wind up
downloading and installing some updates that existed and were
noticed *before* we did our python3-kickstart trick, but not
install the python3-kickstart update, and cause the test to fail
because of that, but that doesn't seem to have happened before
when we were seeing the 'update' button, so I think I'm not
going to borrow trouble. If it happens, we'll deal with it I
guess.
The comment talks only about KDE, but clearly it can be the case
that an automatic check makes the button visible on GNOME too,
so let's rewrite the comment too.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The fix for this bug was sent to all releases now, so we should
not need the workaround any more. Let's kill it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is breaking the memory_check tests. I just reproduced it
manually and the UI *does* come back to life if you wait some
time; let's see if we can work around the bug this way.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
If an update test fails before reaching advisory_post, we don't
generate the 'what update packages were installed' and 'were
any update packages *not* installed when they should have been'
logs, but these may well be useful for diagnosing the failure -
so let's also do the same stuff there. Only let's not do it all
twice.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We hit an interesting case in update testing recently:
https://bodhi.fedoraproject.org/updates/FEDORA-2018-115068f60e
An earlier version of that update failed testing. When we dug
into it a bit, we found that the test was failing because an
earlier version of the `pki-server` package was installed than
the version that was in the update; when asked (as part of
FreeIPA deployment) to install it, dnf had noticed that there
were dependency issues with the version of the package from the
update, but it happened to be able to install the version from
the frozen 'stable' repo...so it just went ahead and did that.
In this case, the 'missed' package resulted in a test failure,
but it'd actually be possible for this to happen and the test
to complete; we really ought to notice when this happens, and
treat it as a test failure.
So what this attempts to do is: at the end of all update tests,
check for all installed packages with the same name as a package
from the update, and compare their full NEVR to the one of the
package from the update. If a package with the same name as one
of the update packages is installed, but does not appear to be
the *same NEVR*, we fail, and upload the lists of packages for
manual investigation as to what the heck's going on.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
From local experimentation, it still actually produces the
output, even though it prints the message about the order being
wrong and exits 1.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Weirdly, occasionally some update tests seem to fail because
the 'comm' util we use to produce the list of packages from the
update that were actually tested during the job doesn't think
one of the input files is in alphabetical order, even though we
sort them both when they're produced. I don't know if this is
possibly due to the definition of 'alphabetical order' changing
as part of the update, or what. But we really shouldn't *fail*
the test when this happens, as it's not part of the functional
test, we're just producing convenience data. So, let's handle
the command failing, and if it happens, upload the input files
so we can maybe figure out why it's unhappy...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The previous commit would lead to the 'workaround' getting hit
incorrectly, and might have had some other issues...tweak it a
bit.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
GNOME Software 3.30.5 split the offline update process into two
separate 'download' and 'apply' phases. So we need to handle
clicking 'download' before 'apply', if that happens.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Somehow, recently, FreeIPA tests are running into Firefox not
quitting because it's showing a warning about closing multiple
tabs. (I think we didn't *get* multiple tabs before but now we
do, for some reason). So let's work around this by clicking
"Close tabs" if the warning appears.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Lately, we can't be sure the desktop will be on tty1 after we
do 'systemctl isolate graphical.target'. For recent Workstation
lives it actually shows up on tty2.
We could be 'clever' and switch to tty2 on F29+ Workstation
lives...but actually it seems like if we just don't do anything,
systemd switches us to the correct tty. So let's rely on that,
at least as long as it's working.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
At least one test (desktop_notifications_postinstall) boots from
the disk image uploaded by install_default_upload, and needs to
access the grub menu. On F29+ Workstation this is failing,
because the grub menu is now hidden by default, so when the test
boots, it never sees the bootloader screen, and fails.
I considered trying to teach it to hold down shift or hit f8 or
esc at the right time, but that seems like it might be hard. So
instead let's just try to disable the hidden menu when we're
about to upload the installed system image. This is kinda going
against the 'preserve natural system behaviour' principle we try
to use for openQA, but I think it's OK as we do have other tests
that will exercise the 'hidden boot menu' stuff to some extent.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
On F27 we don't get a 'Software is up to date' screen because
there's an upgrade available. Let's work with the refresh button
instead.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We're not seeing *exactly* #1314991 any more, but we're seeing
something that looks quite similar: the first attempt to find
updates just doesn't find any. No error message, no updates. I
have reported a bug for this and am investigating it, in the
meantime, let's restore the workaround, elaborated a bit, and
looking for the 'Software is up to date' screen instead of the
error message.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I rather suspect the *bug* is still basically present and it's
why this test often fails, but we no longer seem to see the
*error message* which lets us detect the bug happening. This
needle has not been hit by any test for six months. So let's
remove the workaround as it adds complexity.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
For some reason, in recent tests, switching to a console after
live install completes is taking a long time, and tests are
failing because we 'only' allow 10 seconds for the login prompt
to appear. This seems to indicate some kind of performance bug,
but we don't really want all liveinst tests to fail on in, this
is not primarily a performance testing framework. So let's
tweak the root_console / console_login bits a bit to allow a
configurable timeout for the login prompt to appear, and use
that to wait 30 secs instead of 10 in this case.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The FreeIPA upgrade test didn't actually check that FreeIPA is
actually running after the upgrade and reboot, it just kinda
assumed it is. Let's add a check to the start of the 'check'
test module that makes sure ipa.service actually comes up to
'active' state. This'll make it clearer when tests are failing
because FreeIPA didn't come up right after the upgrade. The
check will run on non-upgrade tests too, but that's fine.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This adds a new test intended to just check boot chain things
for updates. It doesn't run any test modules besides the stock
update ones, but sets a variable, ADVISORY_BOOT_TEST, which
causes _advisory_update to do some additional stuff after
installing the updates but before rebooting: it forces regen
of the initramfs and bootloader config, and reinstalls the
bootloader on BIOS (not UEFI as it's not relevant). If the
following boot fails, we probably have a bug somewhere.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There's this annoying problem where the screen sometimes goes
messed up after ipa-server-uninstall. 'clear' doesn't seem to
really work to fix it up either. Let's try flipping between
ttys. I don't like this much as it's already a pain trying to
work out / remember what tty we might possibly be on at any
given time, but I think we're always on either 1 or 3 here, so
let's do ctrl-alt-f1 ctrl-alt-f3 to ensure at least one change
and wind up on tty3...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Looking at this, it's a bit weird: the updated packages are
actually included in the upgrade process, but we still run
_advisory_update, which does basically nothing...then reboots.
That's kinda silly and makes the tests a bit flaky, let's fix
it. I don't think there's actually any problem with doing the
upload of updatepkgs.txt in _repo_setup_updates, becase that
already guards against being run more than once, it just bails
very early if it's already been run.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I'm going to figure out if it's a bug that it takes so long, but
for now let's just bump the timeout.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
RHBZ#1625572 is for gnome-initial-setup running in 'first login'
mode after it's already run in 'user creation' mode (which isn't
meant to happen). This works around that so the subsequent tests
can run. We don't soft-fail because meh.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
If USER_LOGIN is false we can just return; when we reach the
login screen. We don't need a huge conditional when we don't do
anything *after* it, in the false case.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It transpires that decommissioning wipes some stuff, like the
dirsrv logs. Obviously we want these included in the logs we
upload for reference purposes, so let's upload earlier.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This adds a set of jobs to test FreeIPA replication. We deploy
a server, deploy a replica of that server, then enrol a client
against the replica and run the client tests.
At first I was planning to add the replica testing into the
main set of FreeIPA tests, but the test ordering/blocking (via
mutexes and barriers and what-have-you) just turns into a big
nightmare that way. This way seems rather simpler to deal with.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We'd really like to know if FreeIPA is working aside from this
crasher bug, so let's workaround it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We kind of want to know if FreeIPA is working aside from this
known bug, so let's treat it as a soft failure and work around
it. But only for Rawhide, not for F27/F28 updates tests.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
https://bugzilla.redhat.com/show_bug.cgi?id=1600823 shows a
case where systemd throws a service that would usually have been
started out of the boot process *entirely* in order to resolve a
dependency loop. This means the service won't show up as failed,
it will just be inactive when it should be active. This still
should constitute a failure of this test, so let's add a check
for the log message that indicates this situation.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
As for the domain controller role, stop using rolekit on F29+,
as it's going away. Continue using it on <F29.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
...because by this point in the upgrade test, the system is
upgraded, and rolekit won't be there on F29+.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Rolekit is going away. At least for the F29 cycle, though, we
still want to test basically the same functionality. This ports
the 'domain controller role' test to use ipa-server-install
directly rather than rolectl.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It seems that for some reason the localized layout gets loaded
on the installer VTs by this point in time, so we need to load
'us' again for this complex command to work.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We need this as part of the fix for #1593028, at least until
the kernel package is changed to no longer have
CONFIG_CMDLINE="console=ttyAMA0" in the config for aarch64
builds. Fully fixing the bug also requires some change to the
kernel or dracut or something.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The way this currently works, the test unconditionally waits 60
seconds for the "Timbuktu screen" (the warning dialog shown on
pre-release images) to appear when anaconda is starting up, even
if it's testing an image where it doesn't show up. Now we test
Atomic nightlies and live respins and stuff this happens quite a
lot, so let's avoid it. This way if the hub appears during those
60 seconds we'll spot it right away and continue, otherwise we
behave the same as before.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Upstream is gonna change the default from 30 to 0, it seems:
https://github.com/os-autoinst/os-autoinst/pull/965
so let's go ahead and change these two cases where we have no
explicit timeout to have one.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The reason we have all this horrible code to use the commented-
out baseurl lines in the repo files instead of the metalinks
that are usually used is a timing issue with the metalink
system. As a protection against stale mirrors, the metalink
system sends the package manager a list of mirrors *and a list
of recent checksums for the repo metadata*. The package manager
goes out and gets the metadata from the first mirror on the
list, then checksums it; if the checksum isn't on the list of
checksums it got from mirrormanager, it assumes that means the
mirror is stale, and tries the next on the list instead.
The problem is that MM's list of checksums is currently only
updated once an hour (by a cron job). So we kept running into
a problem where, when a test ran just after one of the repos
had been regenerated, the infra mirror it's supposed to use
would be rejected because the checksum wasn't on the list - but
not because the mirror was stale, but because it was too fresh,
it had got the new packages and metadata but mirrormanager's
list of checksums hadn't been updated to include the checksum
for the latest metadata.
All this baseurl munging code was getting ridiculous, though,
what with the tests getting more complicated and errors showing
up in the actual repo files and stuff. It occurred to me that
instead of using the baseurl we can just use the 'mirrorlist'
system instead of 'metalink'. mirrorlist is the dumber, older
system which just provides the package manager a list of mirrors
and nothing else - the whole stale-mirror-detection-checksum
thing does not happen with mirrorlists, the package manager just
tries all the mirrors in order and uses the first that works.
And happily, it's very easy to convert the metalink URLs into
mirrorlist URLs, and it saves all that faffing around trying to
fix up baseurls.
Also, adjust upgrade_boot to do the s/metalink/mirrorlist/
substitution, so upgrade tests don't run into the timing issue
in the steps before the main repo_setup run is done by
upgrade_run, and adjust repo_setup_compose to sub this line out
later.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Sometimes rebooting during upgrade tests seems to take longer
than these timeouts allow, so let's bump them a bit.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The text changed from 'added repo' to 'enabled repo' in Rawhide
after F28, so let's handle both at least till F28 is EOL.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We've been seeing an odd case lately where the language select
screen is not foregrounded when it appears (so all text is
grey). It happens very occasionally on x86_64, but a lot on
ppc64. To work around this, let's add a needle that matches the
inactive screen, and click on the screen when it appears just
to make sure it's active.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We do the 'desktop update' test for KDE via the notification
icon thingy, and it behaves differently depending on whether it
has already detected there are updates or not. The test only
works at present in the case where it *hasn't* - it expects the
notification icon to be in the extended panel and it expects to
see a 'refresh' button, neither of which is the case if it's
already noticed there are updates to install.
We should also force PackageKit to update its list of available
updates after we set up our 'special' update, otherwise on this
path KDE will only install the updates it found *before* we did
our stuff, and the test will fail as our special update won't be
there.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
In F28 tests, the notification 'counter' thing that we rely on
to check there's only *one* notification seems to suddenly
disappear...right around 10 minutes after the desktop starts up,
which is just how long our test idles for to catch crashes that
happen a little after boot. That causes test fails. Let's try
just cutting the wait down to 8 minutes to see if that helps.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
KDE in F28+ seems to show a network connection notification on
live boot, for some reason. Just dismiss it to help the test
pass.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This test expects to pick up from freeipa_webui, but that test
is not fatal (i.e. it can fail and we still carry on to this
one). We should probably make them independent, but for now,
just check if 'test3' (one of the users freeipa_webui creates
and that this test requires) actually exists, at the start. If
not, we can just die right away.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This reduces duplication, but it also means that if the FreeIPA
web UI module fails, the password change module will pick up
from a point where Firefox is set up and won't fail in a bogus
way because it isn't.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This mouse placement is in the middle of where the 'install
addon' popover appears in Firefox, and that seems like it
sometimes causes the popover to immediately disappear in KDE.
This is pretty corner-case-y so I don't wanna report it as a
bug, let's just tweak the cursor hiding location and see if it
solves the problem.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
...as somehow a Workstation live install currently has the
desktop on tty3, I have no idea why (g-i-s not quitting right?)
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is the best workaround I can think of for RHBZ #1553807 -
just check (in the 60 second 'move the mouse' loop) if anaconda
is still running, based on whether its icon is in the top bar
(on Workstation live installs only, obviously).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Seems with the long period of not doing anything and possibly
with very aggressive timeouts in Fedora 28, Workstation live
wants to blank the screen while we're installing. Stop it.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Previous approach wouldn't work for tests that run after the
install test...let's just set a password from a chroot after
install completes. Don't really like this as it changes the
'real' install process a bit, but it's the least invasive short
term fix at least. We can maybe do something more sudo-y later
with a bit more thought.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Workstation live installs for F28+ drop the user creation and
root password panes from anaconda, so we need to not try and
use them any more. But we still want the old behaviour for F27.
I'm hoping this approach will work, if not, we'll find out soon
enough. This removes the install_no_user test for F28+ as it
will no longer differ from the install_default test.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This adds a check that the default package set selection is
actually correct, where possible and appropriate, as part of
the `_software_selection` test. We do this by examining the
`packaging.log` log file and checking which environment group
was selected.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is currently broken, but openQA doesn't notice; we really
should. We could also check the default in other cases, but I
think that's less clear-cut, as it's kind of an anaconda design
choice, it's not mandated in Fedora requirements anywhere.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There's a problem with using `--releasever=rawhide` for upgrade
tests ATM - see #1531356 . To avoid this, we'll try using the
real Rawhide release number (which I'm adapting the scheduler
code to discover and pass in as `RAWREL`).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There's a bug causing the 'getting started' screen to crash.
This doesn't really make the system unusable, so treating it
as a soft failure seems appropriate, especially as this will
unblock all the post-install tests on Workstation.
It takes an unusally long time for Modular images to get from
language selection to the 'timbuktu screen', so give 'em a bit
more time. See bug report for more info.
Modular composes don't include these packages, but we need them
to run the web UI tests for FreeIPA and Cockpit. This is the
most reasonable hack I can come up with for now: just use a
non-modular fedora repo to source these packages when doing
Modular compose testing.
If we ever reach an all-Modular future, these packages should
be available in Modular composes I guess, but for now they are
not.
Since April there's been some kind of issue in the F26 base
image which means gnome-initial-setup doesn't run on the first
user login (as it should). The F25 base image is fine. I've
not yet had the time to look into this.
I put a workaround in place to prevent this problem causing
false fails of update tests that boot from the F26 base image,
but didn't apply the same workaround to upgrade tests, which
is why upgrade tests from F26 Workstation always fail - they
expect g-i-s to run on first login (which happens after the
upgrade, in upgrade tests) and it doesn't. So let's just extend
the workaround to apply to upgrade tests too, for now, until we
can figure out why this happens.
The default action on the reboot confirmation dialog changed
from Reboot to Cancel, so when we hit enter, we just cancel the
reboot. Tweak this to hit tab on F27+ (but not <F26, so update
tests continue to work too).
The font Firefox uses when we don't ensure dejavu is installed
seems to bounce around a bit, so let's ensure the dejavu fonts
are there before we start Firefox. Also update a needle for
this.
We can deal with this annoying bug by looking out for the error
we see when it happens, hitting the 'refresh' button again, and
resetting the loop counter to 1 (requires changing the loop to
a C-style loop).
previously required on f25 host with qemu 2.7.1-6
it is not needed anymore on f26 with qemu 2.9.0-5
This reverts commit 0eb15266117aae47f663297f5f332d480d8549b9.
This reverts commit 8b2977f1d618316ded61420df4fc7d2afd07cbf4.
The initial commit was required for PowerPC
until qemu 2.7.1-6 (in f25) not required anymore
since qemu 2.9.0-5 (in f26)
by direct grep of mount command
because nfs mounting not traced in ananconda or packaging log.
Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
This is required because anaconda is still checking for it
even if not mandatory. Already tracked by bug
https://bugzilla.redhat.com/show_bug.cgi?id=1172791
Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
because started qcow2 may be obsolete for update repo.
Note: despite deprecated "update" alias,
continue to use it rather than "upgrade" command.
Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
PowerPC arches have the empty disk automatically
mounted on the second position in anaconda (vdb).
Thus, trig installation on second disk.
Change disk checking to point on correct disk.
Warning: this is a workaround specific correction
addressing a specific case.
This will have to be improved/changed with a more
generic code as suggested by Adam Williamson in
https://pagure.io/fedora-qa/os-autoinst-distri-fedora/pull-request/1#comment-31858
proposal for a next commit :)
Signed-off-by: Guy Menanteau <menantea@linux.vnet.ibm.com>
Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>
* New OFW variable to identify Open Firmware (used by PowerPC)
* Few needles changes for PowerPC support
* as requested do not change the timers value below for PowerPC
tests/install_source_graphical.pm (300 to 600)
tests/_boot_to_anaconda.pm (300 to 1200)
This will be handled by TIMEOUT_SCALE in templates
Signed-off-by: Guy Menanteau <menantea@linux.vnet.ibm.com>
Previously we just got a screenshot of some AVCs or coredumps,
which told us something was wrong but didn't really help debug
it. So, let's upload the output of the commands and then also
use the post-fail hook to upload the system logs, which should
give us much more info to work with.
We often want to see the logs from the FreeIPA deployment test
even if that test passes - to look for some detail that doesn't
cause a test to fail, for instance, or if one of the *client*
tests failed for a reason that involves the server. So, let's
do that.
That other one didn't help, so let's try this - try and spot if
the spoke is in the unexpected state (the needle should only
match if the spoke is done processing and still in warning
state, it shouldn't match while the needle is still thinking)
and click through it again if so.
Well, that OCR needle isn't working out so great, as it seems
to match when it shouldn't:
https://openqa.fedoraproject.org/tests/119217#step/_graphical_wait_login/5
So let's try another approach. Ditch the OCR needle and have a
function for checking we're at a clean desktop. It does the
normal needle match, but if we're on GNOME, it also tries
hitting alt+f1 and seeing if we're at the overview; if so, it
hits alt+f1 again (to go back to the desktop) and returns.
IIRC disk_guided_empty is the only storage test that clicks
through INSTALLATION DESTINATION *really* fast, so let's try
adding a 2-second sleep to it to see if it works around the
'sometimes spoke shows as incomplete' bug that cropped up in
F26 and hasn't been fixed yet and tends to cause failures.
Summary:
As we're getting the Workstation dvd-ostree (OStree installer
image) built for Rawhide now, let's try testing it.
Test Plan:
Run the tests on a Rawhide compose that works and
has the image (e.g. 20170615.n.0). Check that new tests work
as expected and old tests are not adversely affected. A
corresponding diff for fedora_openqa will be coming to take
care of scheduling. Note that the tests will often soft fail
for now; this is intentional due to RHBZ#1193590, the bash
prompt for root is incorrect on ostree installs, so I have
added a needle that matches the incorrect prompt but which is
flagged as a workaround needle (so causing the test result to
be a soft fail).
Reviewers: jsedlak, jskladan
Reviewed By: jsedlak
Subscribers: tflink
Differential Revision: https://phab.qa.fedoraproject.org/D1211
Summary:
This just adds the FreeIPA web UI and password change
test modules to the FreeIPA upgrade test (client end). It's
useful to check out these features too. We don't need to
separate these into separate jobs, as we're not trying to
fill out different matrix checkboxes here, we just want to
know whether everything works.
Test Plan:
Run the test, see that the modules work properly.
I was actually expecting this to fail given the issues with
the upgrade on the server end, but it seems to pass.
Reviewers: jsedlak, jskladan
Reviewed By: jsedlak
Subscribers: tflink
Differential Revision: https://phab.qa.fedoraproject.org/D1207
Summary:
This adds an upgrade variant of the FreeIPA tests, with only
the simplest client enrolment (sssd) for now. The server test
starts from the N-1 release and deploys the domain controller
role. The client test similarly starts from the N-1 release
and, when the server is deployed, enrols as a domain client.
Then the server upgrades itself, while the client waits (as the
server is its name server). Then the client upgrades itself,
while the server does some self-checks. The server then waits
for the client to do its checks before decommissioning itself,
as usual. So, summary: *deployment* of both server and client
occurs on N-1, then both are upgraded, then the actual *checks*
occur on N.
In my testing, this all more or less works, except the role
decommission step fails. This failure seems to be a genuine one
so far as I can tell; I intend to file a bug for it soon.
Test Plan:
Run the new tests, check they work. Run the existing
FreeIPA tests (both the compose and the update variants), check
they both behave the same.
Reviewers: jsedlak, jskladan
Reviewed By: jsedlak
Subscribers: tflink
Differential Revision: https://phab.qa.fedoraproject.org/D1204
Previously this module changed test1's password, so that it
would still be able to work even if the webui test module
failed (so test3/test4 didn't get created). But this means
that, for about 30 seconds, test1's password is 'loremipsum'
not 'batterystaple', and if one of the *other* client test
jobs happens to hit a point where it has to auth as test1
during the 30 seconds test1's password is different, it will
fail. This looks to be what happened to the join_kickstart
test the last few days - it failed because it tried to login
as test1 during the password change window.
By using the test3 user instead (which is only used by the
join_cockpit test, currently) we avoid this problem, at the
cost that the password_change module will always fail if the
webui module fails.
With the latest F26 base images, it seems like g-i-s fails to
run at first login. This is clearly some kind of bug somewhere
and I'll investigate it, but it shouldn't be causing the update
tests to fail - we can still validly run the tests with g-i-s
not running. So for now, adjust the _graphical_wait_login test
to tolerate this behaviour when running update tests.
Summary:
Loading the same module more than once *kinda* works, but it
shows up all kinds of funky in the openQA web interface. There's
a POO for this:
https://progress.opensuse.org/issues/10514
But it doesn't seem like it's going to be resolved immediately,
so in the mean time maybe we should avoid doing it so we don't
have to deal with the weirdness it produces in the web UI. So
here's a kinda icky hack that uses symlinks and stuff to load
multiple instances of 'the same' test module.
Test Plan:
Run an update test, look at how it looks in the web
UI and confirm it's a lot clearer and less buggy. Check there
aren't any bugs in the loading approach. This is deployed on stg
so you can look at it there.
Reviewers: jsedlak, jskladan
Reviewed By: jsedlak
Subscribers: tflink
Differential Revision: https://phab.qa.fedoraproject.org/D1186
It's not really a good idea to have the comments that explain
the test_flags in *every* test, because they can go stale and
then we either have to live with them being old or update them
all. Like, now. So let's just take 'em all out. There's always
a reference in the openQA and os-autoinst docs, and those get
updated faster.
More importantly, add the new `ignore_failure` flag to relevant
tests - all the tests that don't have the 'important' or
'fatal' flag at present. Upstream killed the 'important' flag
(making all tests 'important' by default), I got it replaced
with the 'ignore_failure' flag, we now need to explicitly mark
all modules we want the 'ignore_failure' behaviour for.
The way this was set up before, if `anaconda_main_hub` matched
immediately but some spoke was still in a 'processing' state,
it only had 30 seconds (default `assert_and_click` timeout) to
complete and allow the 'Begin Installation' button to appear.
It seems unnecessary to match on *both* needles, really, so
let's just give 300 seconds for the `begin_installation` needle
to appear. It's not going to appear on any other screen.
This problem caused a couple of spurious failures today -
https://openqa.fedoraproject.org/tests/77839 and
https://openqa.fedoraproject.org/tests/77858 - because they
took a bit too long for the INSTALLATION DESTINATION spoke to
clear.
Summary:
This adds a new test suite, run for Workstation and KDE live
images, which does not create a user during install. It then
expects initial-setup (KDE) or gnome-initial-setup (Workstation)
to appear after install, creates a user, and proceeds with
normal boot.
Note the ARM image test already covers the initial-setup text
mode, and the ARM minimal image is the only case where that
actually matters (it's not included in Server).
Test Plan:
Run the new tests, check they work. Run all old
tests, check the changes didn't break them.
Reviewers: jsedlak, jskladan
Reviewed By: jsedlak
Subscribers: tflink
Differential Revision: https://phab.qa.fedoraproject.org/D1185
