Committing without review as this causes failures...try to make
sure we only run the AVC test when it makes sense, and fix
running it on the French install test.
Summary:
This has all console tests check for AVCs (with ausearch) and
crashes (with coredumpctl) at post-install stage. It's non-
fatal as this doesn't really mean the test failed, but we want
to spot when there are unexpected AVCs or crashes.
Test Plan:
Run some console tests, check it works right. I only
tested with one test, since so many are broken on Rawhide ATM
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1066
Committing without review as this is pretty trivial. Running
top *after* all the other collect_data tasks leads to some
meaningless fluctuations in the output; I intended to look for
activity caused by stuff running 'as usual', not activity from
the other collect_data tasks.
Summary:
I've been wanting to do this for a while; I think it'll let us
check for some significant changes between composes. This should
cause runs of a few test cases to collect and upload info on:
* installed packages
* free memory
* disk space
* active services
* 1 minute of CPU usage info (via top)
immediately after install and initial login. In some cases this
will be useful / interesting simply to look at directly, but
we can also have check-compose analyze the data and include
significant changes in its reports.
Test Plan:
Run affected tests, make sure the data collection
works.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1046
Summary:
GNOME's update notification criteria are pretty braindead: it
fires the update check timer once on login then once every hour
thereafter, but only actually checks for and notifies of updates
once a day if it's after 6am(?!?!?!). So we have to do a bunch
of fiddling around to ensure we reliably get a notification.
Move the clock to 6am if it's earlier than that, and reset the
'last update check' timer to 48 hours ago, then log in to GNOME
after that.
Note: I thought this still wasn't fully reliable, but I've looked
into all the recent failures of either test on staging and
there's only one which was really 'no update notification came
up', and the logs clearly indicate PK did run an update check,
so I don't think that was a test bug (I think something went
wrong with the update check). The other failures are all 'GNOME
did something wacky', plus one case where the needle didn't quite
match because I think the match area is slightly too tall; I'll
fix that in a second.
Test Plan:
Run the tests on both KDE and GNOME and check they
work properly now (assuming nothing unrelated breaks, like KDE
crashing...)
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1039
We're not really *testing* shutdown here, we're just shutting
down to make sure the uploaded disk image is clean. So we don't
really mind if shutdown takes a while. It often seems to take
longer than 1 minute on KDE installs and cause a soft fail, so
let's bump the timeout to 3 minutes.
Summary:
os-autoinst implements `script_run` itself now, we aren't
required to implement it ourselves any more. os-autoinst's
implementation is better than ours, as it allows for verifying
the script actually ran (via the redirect-output-to-serial-
console trick).
So this drops our implementation so we'll just use the upstream
one. Where I judged we don't want to bother with the 'check
the command actually ran' feature I've adjusted our direct
`script_run` calls to pass a wait time of 0, which skips the
'wait for command to run' stuff entirely and just does a simple
'type the string and hit enter'.
Because of how the inheritance works, our `assert_script_run`
calls already used the os-autoinst `script_run`, rather than
the one from our distribution.
This should prevent `prepare_test_packages` sometimes going
wrong right after removing the python3-kickstart package, as
we'll properly wait for that removal to complete now (before
we weren't, we'd just start typing the next command while it
was still running, which could result in lost keypresses).
Test Plan:
Check all tests still run OK (I've tried this on
staging and it seems fine).
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1034
Summary:
Since we started using `type_very_safely` for typing the root
password, we starting hitting a race issue. If we complete the
root password spoke so slowly that the software deployment
process completes in the meantime, anaconda will wait until we
complete the spoke then immediately flip to the 'post-install
configuration' step, at which point access to the USER CREATION
spoke is blocked.
We don't hit this case on regular RPM installs or live installs
as the deployment phase still takes a while for both of those,
but we are sometimes hitting it for the Atomic ostree install
image, as the software deployment phase is pretty fast there.
We *could* just not bother creating a user and testing we can
log in as a user for that test, but I don't like that approach,
we *should* be testing that user creation and login works OK
for ostree installs. So instead, let's just type the root
password a bit less safely for ostree installs; this will be
more vulnerable to typing errors but hopefully will avoid the
race problem.
Test Plan:
Run a few Atomic installs, see if they hit the race.
Might need to run other tests at the same time, and you may not
be able to hit it, this is obviously dependent on the I/O of
the worker host...
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1022
Summary:
Since we can match on multiple needles, we can drop the loop
from console_login and instead do it this way, which is simpler
and should work better on ARM (the timeouts will scale and
allow ARM to be slow here). Also move it to main_common as
there's no logical reason for it to be a class method.
Also remove the `check` arg. `check` was only set to 0 by two
tests, _console_shutdown and anacondatest's _post_fail_hook.
For _console_shutdown, I think I just wanted to give it the
best possible chance of succeeding. But we're really not going
to lose anything significant by checking, the only case where
check=>0 would've helped is if the 'good' needle had stopped
matching, and all sorts of other tests will fail in that case.
anacondatest was only using it to save a screenshot of whatever
was on the tty if it didn't reach a root console, which doesn't
seem that useful, and we'll get screenshots from check_screen
and assert_screen anyway.
Test Plan:
Run all tests, check they behave as expected and
none inappropriately fails on console login.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1016
As with the text install test, there's an additional choice on
the 'Time settings' path compared to whatever image garret
developed the test on; right after 'Time settings' you have to
pick 'Set timezone' or 'Configure NTP servers'. So adjust the
test to handle this, just like we did there.
Summary:
I started out wanting to fix an issue I noticed today where
graphical upgrade tests were failing because they didn't wait
for the graphical login screen properly; the test was sitting
at the 'full Fedora logo' state of plymouth for a long time,
so the current boot_to_login_screen's wait_still_screen was
triggered by it and the function wound up failing on the
assert_screen, because it was still some time before the real
login screen appeared.
So I tweaked the boot_to_login_screen implementation to work
slightly differently (look for a login screen match, *then* -
if we're dealing with a graphical login - wait_still_screen
to defeat the 'old GPU buffer showing login screen' problem
and assert the login screen again). But while working on it,
I figured we really should consolidate all the various places
that handle the bootloader -> login, we were doing it quite
differently in all sorts of different places. And as part of
that, I converted the base tests to use POSTINSTALL (and thus
go through the shared _wait_login tests) instead of handling
boot themselves. As part of *that*, I tweaked main.pm to not
require all POSTINSTALL tests have the _postinstall suffix on
their names, as it really doesn't make sense, and renamed the
tests.
Test Plan: Run all tests, see if they work.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1015
I have a better fix for this coming, but it's a big change that
requires review, so for now, do this. The tests are actually
failing because the 30 second wait_still_screen is too short(!)
- it's triggering while the test is sitting at the 'full Fedora
logo' bootsplash screen then doing a 30 second assert_screen
graphical_login, which is failing because it actually takes
more than 60 seconds to get from the 'full F' screen to the
login screen. So let's just make the wait_still_screen 45
seconds for now.
use 'ps' output for Xorg and Xwayland. We'd need some new
openQA var to get this right by 'guessing', as it's vt1 for
Workstation when running live - so long as autologin worked -
but vt2 after install. We'd need a var or some other thing to
detect which case we're running in. LIVE doesn't do it, it's
set even when running a post-install test from a live image.
So instead let's just do it a bit more cleverly. This also
gives us a bit of insurance against changes in GDM, SDDM etc.
behaviour, so long as Xwayland or Xorg is running (and we can
add additional processes to the list, like gnome-shell, if
needed/appropriate). We assume the *final* listed process -
i.e. the most recently-started one - will be the desktop;
this covers gdm's behaviour of starting up on vt1 then running
the user session on vt2. We can make this function more complex
and add args if we ever get to the point where we have multi-
user tests running or anything (e.g. allow to pass a username
and only look for that user's processes).
Landing without review as this broke the live variant of the
test on Workstation in production (kinda not sure why it worked
in testing, or I didn't notice that it failed, but never mind).
I've tested it on staging.
When a package is unsigned, KDE will prompt for authentication.
Let's handle this. But count it as a soft fail, because
puiterwijk claims that Rawhide packages will be autosigned
from next week, so this *should* not happen and would indicate
an unsigned package in the repos. We make the KDE 'update
complete' needle area smaller because the wider area includes
some transparency and so will only match if the update applet
is open; this area will match whether it's open (no auth case)
or not open (auth case - the applet seems to disappear after
you provide the password in the auth prompt).
Pushing without review as the test is in production so I want
to make sure it works correctly.
(Also, hey, check out that array match for assert_screen and
that match_has_tag! This is gonna make some things so much
easier...thanks upstream)
Summary:
this more or less covers desktop_error_checks and desktop_
update_notification, though it can't really distinguish
between them easily. All we know is that if both the live and
postinstall versions of this test pass, both of those tests
pass. Any fails will have to be investigated manually.
Test Plan:
Run the tests for both KDE and Workstation, see
what happens. Workstation will fail for F25 and Rawhide at
present, due to SELinux/abrt notifications.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1004
Summary:
Very similar to the CLI update test, but using the desktops'
update applications. This is based off the CLI update test
branch as it uses the shared functions that branch introduced.
We do not use the fake update packages, as they don't really
do anything useful for these tests; for dnf they can help us
distinguish between issues with the dnf mechanism and issues
with the repos, but we can't really tell that in the graphical
case. So we only use the python3-kickstart package here.
Test Plan:
Run the test on both KDE and GNOME and ensure it
performs as intended. I've been testing it on staging, so you
can see it there.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1010
Summary:
this uses a couple of test repos with fake packages to test the
basic dnf mechanisms are working, then messes around with the
python3-kickstart package a bit to try and test the default repo
configuration is working, keys are in place and so on. We use
python3-kickstart because we should be able to rely on the copy
of that package in the 'stable' repo being installable (or else
the compose would have failed), but it shouldn't be vital to
the operation of the system.
Test Plan: Run the test, see if it works.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1006
Summary:
by waiting for the bootloader in _boot_to_anaconda rather than
_console_wait_login, we can ensure that we use the anaconda
post-fail hook and thus get logs uploaded when a kickstart
install fails.
Test Plan:
Run a kickstart install test that fails and check
anaconda logs get uploaded. Then run one that works and make
sure it...still works.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1005
I guess this was developed on F24, or an older F25? There is
an additional choice showing up in production: before the tz
choice, it asks if you want to set the tz or configure NTP. So
we have to pick tz.
Summary:
the main thing this does is try and type slower in X - this
should cover nearly everywhere we type anything in X, and make
it type slower. We also add a bit more safety checking to some
old tests which didn't have it (mainly _do_install_and_reboot)
- wait_still_screen after typing to make sure all the keypresses
were registered before continuing.
This is an attempt to mitigate the problems we've seen where
the wrong text gets typed into the wrong places and the tests
break.
This branch is live on staging atm. It still has *some* issues,
but I do think it's an improvement.
Test Plan:
run the tests (probably several times), compare to
runs without the change, see if it's better or worse...
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D993
Summary:
Pretty straightforward tests which deploy the database
server role and exercise it a bit.
Test Plan: Run the tests, check they work properly.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D991
In F25+, Firefox seems to do kerberos auth automatically, so if
we go to the FreeIPA admin URL while kerberos-authed as test4,
we are logged in right away as test4 - neat! But not what we
wanted. So let's kdestroy.
when we run firefox in a bare X session, by default we get an
800x600 firefox in a 1024x768 X server with some dead black
space to the right and bottom of the screen. Now it turns out
that if the mouse is in the dead space, Firefox will not get
any keystrokes we send.
This didn't used to be a problem, but I made it into one with
this os-autoinst change:
https://github.com/os-autoinst/os-autoinst/pull/559
that makes os-autoinst move the cursor to 1023,767 after each
`assert_and_click`, instead of 0x0 as it did before, unless the
cursor has previously been explicitly place somewhere. So in
this case it gets moved to the dead space, and Firefox stops
responding to keypresses after the first `assert_and_click`.
We could equally well fix this by setting the cursor to 0,0
after running Firefox, but I like this more as it makes sure
we won't run into the same problem some other way, and makes
the videos and screenshots look nicer.
This fixes the realmd_join_cockpit test that's been failing
ever since I installed an os-autoinst with that fix. Committing
without review as it's a straightforward fix and I want the
test working again...
I assumed the 'compose location' sent by fedmsg was https, but
in fact it's http (you get redirected to https when you access
it). Could just change the default back to 4, but why not make
it properly robust. Sending without review so this doesn't go
wrong all weekend.
Summary:
we have a long-standing problem with all the tests that hit
the repositories. The tests are triggered as soon as a compose
completes. At this point in time, the compose is not synced to
the mirrors, where the default 'fedora' repo definition looks;
the sync happens after the compose completes, and there is also
a metadata sync step that must happen after *that* before any
operation that uses the 'fedora' repository definition will
actually use the packages from the new compose. Thus all net
install tests and tests that installed packages have been
effectively testing the previous compose, not the current one.
We have some thoughts about how to fix this 'properly' (such
that the openQA tests wouldn't have to do anything special,
but their 'fedora' repository would somehow reflect the compose
under test), but none of them is in place right now or likely
to happen in the short term, so in the mean time this should
deal with most of the issues. With this change, everything but
the default_install tests for the netinst images should use
the compose-under-test's Everything tree instead of the 'fedora'
repository, and thus should install and test the correct
packages.
This relies on a corresponding change to openqa_fedora_tools
to set the LOCATION openQA setting (which is simply the base
location of the compose under test).
Test Plan:
Do a full test run, check (as far as you can) tests run sensibly
and use appropriate repositories.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D989
Summary:
the dictionary error bug was fixed some time back, so drop this
workaround for it.
Test Plan:
Run all tests for F25 and Rawhide and verify they don't need
this workaround any longer.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D988
Summary: Pretty simple!
Test Plan:
Check the upgrade tests work the same as before the
change.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D979
in the recent commit to always use nogpgcheck I inadvertently
broke the upgrade tests, by dropping the `--releasever` from
the `dnf system-upgrade download` command. So fix that.
Summary:
Except when running on the pre-upgrade release in the upgrade
tests (where GPG check should always be OK).
Currently we always need to use --nogpgcheck on Rawhide, and we
must also use it on Branched prior to the Bodhi activation
point. At present we don't really have any simple way to know
when the Bodhi activation point has kicked in. We could assume
that it's safe to do GPG checking for 'candidate' (not nightly)
composes, but even that isn't 100% safe and isn't really the
*right* thing to do. So I think for now it's best to just always
use --nogpgcheck , until we come up with a decent way to check
for Bodhi enablement, or releng figures things out so we can
rely on packages being signed in Rawhide and in Branched before
Bodhi enablement.
Test Plan:
Check the tests all still run, make sure I didn't
miss any dnf calls.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D964
Summary:
pretty simple stuff here. The distinction between 'firefox' and
'browser' is that the 'browser' needles I expect would also be
correct for other default browsers, while the 'firefox' needles
are specific to Firefox. We need '-kde' variants of some Firefox
needles where interface text is included, because the font is
Cantarell in GNOME but whatever the default 'sans' font is in
KDE - I suppose we should really use -thatfontsname rather than
-kde, but I can't think what it's called...
I couldn't do the 'log in to FAS' bit of the test since we don't
really have a sane way to provide a password while not exposing
it to the public.
Test Plan:
Run the test, check it works - for both KDE and
Workstation.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D938
Summary:
again, added as a non-fatal module for realmd_join_cockpit as
it's convenient to do it here. Also abstract a couple of ipa
bits into a new exporter package in the style of SUSE's
mm_network, rather than using ill-fitting class inheritance as
we have before - we should probably convert our existing class
based stuff to work this way.
Also a few minor tweaks and clean-ups of the other tests:
The path in console_login() where we detect login of a regular
user when we want root or vice versa and log out was actually
broken because it would 'wait' for the result of the 'exit'
command, which obviously doesn't work (as it relies on running
another command afterwards, and we're no longer at a shell).
This commit no longer actually uses that path, but I spotted
the bug with an earlier version of this which did, and we may
as well keep the fix.
/var/log/lastlog is an apparently-extremely-large sparse file.
A couple of times it seemed to cause tar to run very slowly
while creating the /var/log archive for upload on failure. It's
no use for diagnosing bugs, so we may as well exclude it from
the archive.
I caught cockpit webUI login failing one time when testing the
test, so threw in a wait_still_screen before starting to type
the URL, as we have for the FreeIPA webUI.
I also caught a timing issue with the openQA webUI policy add
step; the test flips from the Users screen to the HBAC screen
then clicks the 'add' button, but there's actually an identical
'add' button on *both* screens, so it could wind up trying to
click the one on the Users screen instead, if the web UI took
a few milliseconds to switch. So we throw in a needle match to
make sure we're actually on the HBAC screen before clicking the
button.
We make the freeipa_webui test a 'milestone' so that if the
new test fails, restoring to the last-known-good milestone
doesn't take so long; it actually seems like openQA can get
confused and try to cancel the test if restoring the milestone
takes a *really* long time, and wind up with a zombie qemu
process, which isn't good. This seems to avoid that happening.
Test Plan:
In the simple case, just run all the FreeIPA-related
tests on Fedora 24 (as Rawhide is broken) and make sure they all
work properly. To get a bit more advanced you can throw in an
`assert_script_run 'false'` in either of the non-fatal tests to
break it and make sure things go properly when that happens (the
last milestone should be restored - which should be right after
freeipa_webui, sitting at tty1 - and run properly; things are
set up so each test starts with root logged in on tty1).
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D935
Summary:
we can test this quite easily by placing the standard openQA
updates image in the NFS repo used for the NFS repo install
tests. We just have to copy the contents of the ISO (instead of
directly exporting the ISO loop mount as an NFS share) so we
can add this extra file.
At first I planned to combine this with the NFS repo variation
test, but when you use a remote stage2 like this it changes repo
setup such that the packaging.log line we look for to verify
the remote repo was used does not show up, and there's enough
fuzziness in how anaconda-dracut fudges inst.repo and
inst.stage2 that it's probably a good idea to test them
separately anyhow.
Test Plan:
Run the new test and the other NFS tests, make sure
this one works and the others don't break.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D929
Summary:
This requires us to handle decryption each time we reboot in
the upgrade process, so factor that little block out into the
base class so we don't have to keep pasting it. It's also a
bit tricky to integrate into the 'catch a boot loop' code we
have to deal with #1349721, but I think this should work. There
is a matching openqa_fedora_tools diff to generate the disk
image.
Test Plan:
Run the tests, check that they work, run the other
upgrade and encrypted install tests and check they still work
properly too.
Reviewers: garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D922
Summary:
try to catch a boot loop after `dnf system-upgrade reboot`, if
we do, set the test to soft_fail and pass enforcing=0 to work
around it.
Test Plan:
Run the upgrade_foo tests and see that they now soft
fail instead of hard failing (unless there are any other issues).
Run the upgrade_2_foo tests and make sure they still pass (i.e.
we don't erroneously soft fail them).
Reviewers: garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D912
Summary:
as a new, non-fatal test step in the cockpit enrolment test,
because it kinda fits in there; we have an enrolled system with
a web browser *right there*. This will require making the wiki
reporting stuff slightly cleverer so we can say 'report a pass
for this wiki test instance if this test step passed', but that
should be possible. Making this non-fatal means the rest of the
cockpit enrolment test will go ahead even if the freeipa web UI
fails.
The 'check if we can log in' stuff is identical to freeipa_
client_postinstall except with different user names, so we could
potentially factor that out somehow, but I couldn't think of a
super clean way to do it so for now it's just copied.
Note this diff is on top of the freeipa-realmd branch which
is for D894, it's not on top of develop.
Test Plan:
Run the modified test and see if it works. No other
tests are modified, so they should be OK.
Reviewers: garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D895
Summary:
This is a pretty straightforward IPA joining test. Since I
figured out how to set up a DHCP server for support_server,
let's do the same for the domain controller so we can simplify
these enrolment tests a bit.
We also extend the timeout on installing haveged on the server
a bit (2 minutes is a bit low when it hits a slow metadata
download), and drop an unnecessary clone_host_file from the
cockpit join test (it was only there for testing and the next
operation immediately overwrites it).
Test Plan:
Do a full server DVD test run, check the new test
works and none of the others broke.
Reviewers: garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D894
