Summary:
the main thing this does is try and type slower in X - this
should cover nearly everywhere we type anything in X, and make
it type slower. We also add a bit more safety checking to some
old tests which didn't have it (mainly _do_install_and_reboot)
- wait_still_screen after typing to make sure all the keypresses
were registered before continuing.
This is an attempt to mitigate the problems we've seen where
the wrong text gets typed into the wrong places and the tests
break.
This branch is live on staging atm. It still has *some* issues,
but I do think it's an improvement.
Test Plan:
run the tests (probably several times), compare to
runs without the change, see if it's better or worse...
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D993
Summary:
Pretty straightforward tests which deploy the database
server role and exercise it a bit.
Test Plan: Run the tests, check they work properly.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D991
In F25+, Firefox seems to do kerberos auth automatically, so if
we go to the FreeIPA admin URL while kerberos-authed as test4,
we are logged in right away as test4 - neat! But not what we
wanted. So let's kdestroy.
when we run firefox in a bare X session, by default we get an
800x600 firefox in a 1024x768 X server with some dead black
space to the right and bottom of the screen. Now it turns out
that if the mouse is in the dead space, Firefox will not get
any keystrokes we send.
This didn't used to be a problem, but I made it into one with
this os-autoinst change:
https://github.com/os-autoinst/os-autoinst/pull/559
that makes os-autoinst move the cursor to 1023,767 after each
`assert_and_click`, instead of 0x0 as it did before, unless the
cursor has previously been explicitly place somewhere. So in
this case it gets moved to the dead space, and Firefox stops
responding to keypresses after the first `assert_and_click`.
We could equally well fix this by setting the cursor to 0,0
after running Firefox, but I like this more as it makes sure
we won't run into the same problem some other way, and makes
the videos and screenshots look nicer.
This fixes the realmd_join_cockpit test that's been failing
ever since I installed an os-autoinst with that fix. Committing
without review as it's a straightforward fix and I want the
test working again...
I assumed the 'compose location' sent by fedmsg was https, but
in fact it's http (you get redirected to https when you access
it). Could just change the default back to 4, but why not make
it properly robust. Sending without review so this doesn't go
wrong all weekend.
Summary:
we have a long-standing problem with all the tests that hit
the repositories. The tests are triggered as soon as a compose
completes. At this point in time, the compose is not synced to
the mirrors, where the default 'fedora' repo definition looks;
the sync happens after the compose completes, and there is also
a metadata sync step that must happen after *that* before any
operation that uses the 'fedora' repository definition will
actually use the packages from the new compose. Thus all net
install tests and tests that installed packages have been
effectively testing the previous compose, not the current one.
We have some thoughts about how to fix this 'properly' (such
that the openQA tests wouldn't have to do anything special,
but their 'fedora' repository would somehow reflect the compose
under test), but none of them is in place right now or likely
to happen in the short term, so in the mean time this should
deal with most of the issues. With this change, everything but
the default_install tests for the netinst images should use
the compose-under-test's Everything tree instead of the 'fedora'
repository, and thus should install and test the correct
packages.
This relies on a corresponding change to openqa_fedora_tools
to set the LOCATION openQA setting (which is simply the base
location of the compose under test).
Test Plan:
Do a full test run, check (as far as you can) tests run sensibly
and use appropriate repositories.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D989
Summary:
the dictionary error bug was fixed some time back, so drop this
workaround for it.
Test Plan:
Run all tests for F25 and Rawhide and verify they don't need
this workaround any longer.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D988
Summary: Pretty simple!
Test Plan:
Check the upgrade tests work the same as before the
change.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D979
in the recent commit to always use nogpgcheck I inadvertently
broke the upgrade tests, by dropping the `--releasever` from
the `dnf system-upgrade download` command. So fix that.
Summary:
Except when running on the pre-upgrade release in the upgrade
tests (where GPG check should always be OK).
Currently we always need to use --nogpgcheck on Rawhide, and we
must also use it on Branched prior to the Bodhi activation
point. At present we don't really have any simple way to know
when the Bodhi activation point has kicked in. We could assume
that it's safe to do GPG checking for 'candidate' (not nightly)
composes, but even that isn't 100% safe and isn't really the
*right* thing to do. So I think for now it's best to just always
use --nogpgcheck , until we come up with a decent way to check
for Bodhi enablement, or releng figures things out so we can
rely on packages being signed in Rawhide and in Branched before
Bodhi enablement.
Test Plan:
Check the tests all still run, make sure I didn't
miss any dnf calls.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D964
Summary:
pretty simple stuff here. The distinction between 'firefox' and
'browser' is that the 'browser' needles I expect would also be
correct for other default browsers, while the 'firefox' needles
are specific to Firefox. We need '-kde' variants of some Firefox
needles where interface text is included, because the font is
Cantarell in GNOME but whatever the default 'sans' font is in
KDE - I suppose we should really use -thatfontsname rather than
-kde, but I can't think what it's called...
I couldn't do the 'log in to FAS' bit of the test since we don't
really have a sane way to provide a password while not exposing
it to the public.
Test Plan:
Run the test, check it works - for both KDE and
Workstation.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D938
Summary:
again, added as a non-fatal module for realmd_join_cockpit as
it's convenient to do it here. Also abstract a couple of ipa
bits into a new exporter package in the style of SUSE's
mm_network, rather than using ill-fitting class inheritance as
we have before - we should probably convert our existing class
based stuff to work this way.
Also a few minor tweaks and clean-ups of the other tests:
The path in console_login() where we detect login of a regular
user when we want root or vice versa and log out was actually
broken because it would 'wait' for the result of the 'exit'
command, which obviously doesn't work (as it relies on running
another command afterwards, and we're no longer at a shell).
This commit no longer actually uses that path, but I spotted
the bug with an earlier version of this which did, and we may
as well keep the fix.
/var/log/lastlog is an apparently-extremely-large sparse file.
A couple of times it seemed to cause tar to run very slowly
while creating the /var/log archive for upload on failure. It's
no use for diagnosing bugs, so we may as well exclude it from
the archive.
I caught cockpit webUI login failing one time when testing the
test, so threw in a wait_still_screen before starting to type
the URL, as we have for the FreeIPA webUI.
I also caught a timing issue with the openQA webUI policy add
step; the test flips from the Users screen to the HBAC screen
then clicks the 'add' button, but there's actually an identical
'add' button on *both* screens, so it could wind up trying to
click the one on the Users screen instead, if the web UI took
a few milliseconds to switch. So we throw in a needle match to
make sure we're actually on the HBAC screen before clicking the
button.
We make the freeipa_webui test a 'milestone' so that if the
new test fails, restoring to the last-known-good milestone
doesn't take so long; it actually seems like openQA can get
confused and try to cancel the test if restoring the milestone
takes a *really* long time, and wind up with a zombie qemu
process, which isn't good. This seems to avoid that happening.
Test Plan:
In the simple case, just run all the FreeIPA-related
tests on Fedora 24 (as Rawhide is broken) and make sure they all
work properly. To get a bit more advanced you can throw in an
`assert_script_run 'false'` in either of the non-fatal tests to
break it and make sure things go properly when that happens (the
last milestone should be restored - which should be right after
freeipa_webui, sitting at tty1 - and run properly; things are
set up so each test starts with root logged in on tty1).
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D935
Summary:
we can test this quite easily by placing the standard openQA
updates image in the NFS repo used for the NFS repo install
tests. We just have to copy the contents of the ISO (instead of
directly exporting the ISO loop mount as an NFS share) so we
can add this extra file.
At first I planned to combine this with the NFS repo variation
test, but when you use a remote stage2 like this it changes repo
setup such that the packaging.log line we look for to verify
the remote repo was used does not show up, and there's enough
fuzziness in how anaconda-dracut fudges inst.repo and
inst.stage2 that it's probably a good idea to test them
separately anyhow.
Test Plan:
Run the new test and the other NFS tests, make sure
this one works and the others don't break.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D929
Summary:
This requires us to handle decryption each time we reboot in
the upgrade process, so factor that little block out into the
base class so we don't have to keep pasting it. It's also a
bit tricky to integrate into the 'catch a boot loop' code we
have to deal with #1349721, but I think this should work. There
is a matching openqa_fedora_tools diff to generate the disk
image.
Test Plan:
Run the tests, check that they work, run the other
upgrade and encrypted install tests and check they still work
properly too.
Reviewers: garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D922
Summary:
try to catch a boot loop after `dnf system-upgrade reboot`, if
we do, set the test to soft_fail and pass enforcing=0 to work
around it.
Test Plan:
Run the upgrade_foo tests and see that they now soft
fail instead of hard failing (unless there are any other issues).
Run the upgrade_2_foo tests and make sure they still pass (i.e.
we don't erroneously soft fail them).
Reviewers: garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D912
Summary:
as a new, non-fatal test step in the cockpit enrolment test,
because it kinda fits in there; we have an enrolled system with
a web browser *right there*. This will require making the wiki
reporting stuff slightly cleverer so we can say 'report a pass
for this wiki test instance if this test step passed', but that
should be possible. Making this non-fatal means the rest of the
cockpit enrolment test will go ahead even if the freeipa web UI
fails.
The 'check if we can log in' stuff is identical to freeipa_
client_postinstall except with different user names, so we could
potentially factor that out somehow, but I couldn't think of a
super clean way to do it so for now it's just copied.
Note this diff is on top of the freeipa-realmd branch which
is for D894, it's not on top of develop.
Test Plan:
Run the modified test and see if it works. No other
tests are modified, so they should be OK.
Reviewers: garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D895
Summary:
This is a pretty straightforward IPA joining test. Since I
figured out how to set up a DHCP server for support_server,
let's do the same for the domain controller so we can simplify
these enrolment tests a bit.
We also extend the timeout on installing haveged on the server
a bit (2 minutes is a bit low when it hits a slow metadata
download), and drop an unnecessary clone_host_file from the
cockpit join test (it was only there for testing and the next
operation immediately overwrites it).
Test Plan:
Do a full server DVD test run, check the new test
works and none of the others broke.
Reviewers: garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D894
Summary:
Set up the support server to provide DHCP/DNS functionality and
an NFS server, providing a kickstart. Add a kickstart test just
like the other root-user-crypted-net kickstart tests except it
gets the kickstart from the support server via NFS. Also add NFS
repository tests and a second support server for Server-dvd-iso
flavor: this test must run on that flavor to ensure that packages
are actually available. The support server just mounts the
attached 'DVD' and exports it via NFS.
Note we don't need to do anything clever to avoid IP conflicts
between the two support servers, because os-autoinst-openvswitch
ensures each worker group is on its own VLAN.
As part of adding the NFS repo tests, I did a bit of cleanup,
moving little things we were repeating a lot into anacondatest,
and sharing the 'check if the repo was used' logic between all
the tests (by making it into a test step that's loaded for all
of them). I also simplified the 'was repo used' checks a bit,
it seems silly to run a 'grep' command inside the VM then have
os-autoinst do a grep on the output (which is effectively what
we were doing before), instead we'll just use a single grep
within the VM, and clean up the messy quoting/escaping a bit.
Test Plan:
Run all tests - at least all repository tests - and
check they work (make sure the tests are actually still sane,
not just that they pass). I've done runs of all the repo tests
and they look good to me, but please double-check. I'm currently
re-running the whole 24-20160609.n.0 test on staging with these
changes.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D888
Summary:
this is following a SUSE model for tests where we need a server
end but don't want setting up the server to constitute a real
test in itself, we want it to be stable. The 'support_server'
test just boots a pre-built (by createhdds) disk image, sets up
networking, and runs the iSCSI server.
To run the iSCSI test we need to handle networking config in
anaconda (or we would need to set the support server up as a
DHCP server, which may be worth considering), so this adds that.
We also need to be able to specify the target device for a
volume in custom partitioning, so this adds that too.
Test Plan:
Build the necessary support server disk image (use
D883), then run the test and make sure it works. Also make sure
all other tests continue to work.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D884
Summary:
This requires a few other changes:
* turn clone_host_resolv into clone_host_file, letting you clone
any given host file (cloning /etc/hosts seems to make both
server deployment and client enrolment faster/more reliable)
* allow loading of multiple POSTINSTALL tests (so we can share
the freeipa_client_postinstall test). Note this is compatible,
existing uses will work fine
* move initial password change for the IPA test users into the
server deployment test (so the client tests don't conflict over
doing that)
* add GRUB_POSTINSTALL, for specifying boot parameters for boot of
the installed system, and make it work by tweaking _console_wait
_login (doesn't work for _graphical_wait_login yet, as I didn't
need that)
* make the static networking config for tap tests into a library
function so the tests can share it
* handle ABRT problem dirs showing up in /var/spool/abrt as well
as /var/tmp/abrt (because the enrol attempt hits #1330766 and
the crash report shows up in /var/spool/abrt, don't ask me why
the difference, I just work here)
* specify the DNS servers from the worker host's resolv.conf as
the forwarders for the FreeIPA server when deploying it; if we
don't do this, rolekit defaults to using the root servers as
forwarders(!) and thus we get the public, not phx2-appropriate,
results for e.g. mirrors.fedoraproject.org, some of which the
workers can't reach, so PackageKit package install always fails
(boy, was it fun figuring THAT mess out)
Even after all that, the test still doesn't actually pass, but
I'm reasonably confident this is because it's hitting actual bugs,
not because it's broken. It runs into #1330766 nearly every time
(I think I saw *one* time the enrolment actually succeeded), and
seems to run into a subsequent bug I hadn't seen before when
trying to work around that by trying the join again (see
https://bugzilla.redhat.com/show_bug.cgi?id=1330766#c37 ).
Test Plan:
Run the test, see what happens. If you're really lucky,
it'll actually pass. But you'll probably run into #1330766#c37,
I'm mostly posting for comment. You'll need a tap-capable openQA
instance to test this.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D880
Summary:
This adds tests for the Server_cockpit_default and cockpit_basic
test cases. Some notes: I was initially thinking of combining
these into a single test with multiple test modules and coming
up with a system for doing wiki reporting based on individual
test module status, but because we'll also want to do a cockpit
FreeIPA enrol test, I decided against it. We don't really want
to combine all three because then we would skip the cockpit
tests whenever FreeIPA server deployment failed, which isn't
ideal. So since we'll need a separate FreeIPA enrolment test
anyway it doesn't really make sense to go to the trouble of
designing a system for loading multiple postinstall tests (though
I have an idea for that!) and a per-module wiki reporting system.
This was the most minimal and hopefully reliable method for
running Cockpit from a stock Server install that I could think
of. An alternative approach would be to have, say, the most
recent stable Workstation live as a 'stock' asset and have two
tests, one which runs a stock Server install and just waits and
another which boots the live image and accesses the cockpit
running on the other box, but that seems a bit over-complex. It
is not possible to have dependencies between tests for different
ISOs, in case you were wondering about having a Workstation live
test which runs parallel with a Server DVD test, we can't do
that. One funny thing is the font that winds up getting used for
the desktop, but I don't *think* that should be a problem.
Picking needles was a bit tricky; any improvement suggestions
are welcome. I'm hoping it turns out to be safe to rely on some
dbus log messages being present; I think logging into Cockpit
triggers activation of the realmd dbus interface, so there
*should* always be some messages related to that. An alternative
would just be to match on a sliver of the dark grey table header
and the light grey row beneath it and assume that'll always be
the first message (whatever the message is), but then we have to
find some area of the message details screen which is always
present for any message, and it just seems a tad more likely to
result in false passes. Similary I'm making an assumption that
auditd is always going to show up on the first page of the
Services screen and the details screen will always show that
'loaded...enabled' text.
Test Plan:
Run the tests and see if they work! See
https://openqa.stg.fedoraproject.org/tests/21373 and
https://openqa.stg.fedoraproject.org/tests/21371 for my tests.
Reviewers: garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D874
Summary:
The test here is a bit ugly, but it should work. Better ideas
welcome =)
Test Plan:
Run the test, check it works (and maybe hack it up
a bit and check it fails properly too, it worked first time for
me which is always suspicious)
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D870
this stuff is only appropriate when a switched keyboard layout
is in play. Left unconditional like this, it breaks the normal
encrypted install test case.
Summary:
We'll often be able to reach a root console and upload logs if
this test fails, and we want to do that so we can figure out
what's going on.
Test Plan:
Have the test fail and check that it manages to upload
logs.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D856
Summary:
Requires new needles and test suite and job template, plus a
few tweaks to handle 'switched' keyboard layouts (so we use the
switched layout in the username and password).
Test Plan:
Run the test and see that it...fails. But that's OK!
It's a genuine bug: RHBZ #1333998 . At least make sure it gets
to that point and no other tests have broken and all the needles
look sane.
Reviewers: garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D846
ARM actually doesn't have "install" test, but in install matrix,
there is test whether ARM disk boots into initial_setup. HDD is saved
after this test for Base tests.
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D853
the updates-testing here was never meant to be permanent, it
was only added when the plugin was very new and we knew the
version in stable was busted. The test cases do not say to use
u-t, so let's not.
We've seen this test fail several times recently because of very
slow metadata downloads at this point, so let's give it longer
to run.
This should be a better fix for the problem of the install_done
needle matching in the sidebar gradient while the button is
transitioning. We just assert_screen first, then once we hit it,
we wait_still_screen for 3 seconds, then assert_and_click the
button.
the first param to assert_and_click is *the button number*, not
the wait time. So as soon as I set that for anything it was not
gonna work. So going back to the original approach but fixing
that, hope this works.
Summary:
I really just want to add the desktop_terminal test, but I think
this refactor is in order now. It splits up loading of the
various test phases (much as SUSE do it) and allows us to run
the post-install tests without the install tests, for e.g. I
tweaked things to allow the upgrade tests to use the existing
_wait_login tests for final login and combine the two upgrade
postinstall tests into one simple one.
This comes with a bit of a behaviour change to make graphical
wait login behave the same as console wait login: it will log
in unless USER_LOGIN is set to 'false'. Previously it only
logged in if both USER_LOGIN and USER_PASSWORD were set, which
I don't think ever happened in a graphical test, so we never
actually did a graphical login. The intent here is we should do
a login on the default_install tests. That's going a bit beyond
the test case, but it seems like a reasonable thing to test. We
can set USER_LOGIN to false if we don't want to do it.
Test Plan:
Do a full test run, make sure the new tests work and
no old tests break.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D839
Summary:
These require openQA tap networking to allow the server and
client boxes to communicate, and require masquerading (NAT) so
the server at least can reach a repository (dnf/rolekit really,
really do not want to work without a repo connection).
They use the 'parallel' test support to have the server deploy
run first while the client enrol test waits at the grub menu
until the server is done before it goes ahead.
This is all deployed and working on stg. The really tricky bit
was getting all the openvswitch and firewall config right in
ansible.
We *could* do the server deploy test as a follow-on from the
default install test to save the install, but then we'd have to
teach it to change the hostname and set up static networking
post-install. I'm not sure if it's worth doing that.
This requires the corresponding openqa_fedora_tools commit that
adds the hard disks (containing the kickstarts - it's possible
to get them from remote during install, but we have to set up
name resolution or hard code the IP of the server).
Test Plan:
Deploy this and the openqa_fedora_tools commit,
generate the disks, configure the networking (good luck! See
the docs in openqa_fedora_tools) and see if you can run the
tests. If you're using Docker, uh...sorry. You somehow need to
set things up so the workers can use tap interfaces that can
talk to each other and are NATed to the outside world. Have fun.
I can talk you through it on IRC...
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D831
Summary:
Rawhide currently seems to have a bug in spell check dictionary
load, which causes the test to fail as it requires another Done
click. So add a workaround needle that handles this case.
Test Plan:
Apply the patch, run some tests, see if they work. I
did a test run on staging:
https://openqa.stg.fedoraproject.org/tests/13331
Reviewers: garretraziel, jskladan
Reviewed By: jskladan
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D815
Summary:
BOOT_UPDATES_IMG_URL is a pretty misleading name - it used to
be the actual URL, but now it's simply a boolean that decides
whether we look for the effect of the openQA updates image or
not. TEST_UPDATES seems clearer.
GRUBADD does the same thing as GRUB, on top of it. The point of
this is so we can add an option to the scheduler CLI that lets
you say 'run the normal tests, but with this updates image' -
so we can easily (albeit manually triggered) check the impact
of some anaconda change that needs testing. It should never be
set in the templates or the tests, it's there strictly for the
scheduler (whether that's fedora_openqa_schedule or literally a
person calling `client isos post`) to use as a kind of override.
The tests that test updates image loading will probably fail
when doing this, but all other tests should work as intended,
including ones that specify GRUB, becase the extra params will
just get added on top. That's why I invented a new var instead
of just letting the scheduler override GRUB's value when POST
ing.
Test Plan:
Check the rename didn't break anything (updates tests
still work). Run tests with GRUBADD param, make sure value is
correctly appended to cmdline both when GRUB is also specified
and when it is not.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D801
Summary:
per details in T759, the 'unipony' updates image we use to test
the updates image features doesn't work with latest anaconda (f24
and Rawhide). I've built a new updates image which uses a neat
anaconda feature that allows you to override CSS with a file in
a special location; it sets the background for disk capacity
texts on the INSTALLATION DESTINATION spoke to be pink. This
lets us use a simple needle that just looks for a pink blob on
that spoke, on the basis that it's unlikely there'll ever be a
pink blob there for any other reason, so if there is one, the
updates image worked. There will be an accompanying tools diff
to change the updates disk image to use the new updates image.
Test Plan:
Do a test run and check the updates image tests pass
and no other tests are broken. You'll need to pull in the tools
diff and re-generate the updates disk image to check that test,
the scsi_updates_img test should work with just this diff.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D799
Summary:
these together test QA:Testcase_kickstart_firewall from the
Server matrix. I'll have to come up with some kinda way to
handle reporting that, might be tricky.
Couple of tweaks to overall test flow: tests can now specify
a POSTINSTALL variable which will load a post-install test
following a naming convention, and tests can specify USER_LOGIN
as 'false' to disable the 'log in as a user' step entirely. We
could easily adjust the kickstarts to create a user so the test
could log in as one, but it seems like an unnecessary step and
I liked the idea of allowing the user login to be skipped.
Test Plan:
Schedule 'universal' tests, check the new tests run
and pass or fail as they should, check no other test is broken
by the logic flow changes.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D792
Summary:
I believe the failures in the Server DVD chained Base tests are
happening because the VM is not cleanly shut down before the disk
image is uploaded. This adds a shutdown step to all tests that
upload a disk image (so, for now, just default_install). To keep
things simple it just runs 'shutdown' from a root console, rather
than using graphical desktop shutdown methods, as the aim is only
to make the disk state clean, not to test shutdown exactly.
I've tested this on staging; a Server DVD test run with this
change produced a full set of passed tests, as opposed to all
the Base tests failing because the system didn't boot properly.
Workstation and KDE tests seem to work fine also.
For the record, SUSE does much the same thing as this commit.
Test Plan:
Do a full test run and make sure everything that worked
before still does. Check that all default_install tests have a
_console_shutdown step added, and it works, and all chained tests
work (or fail for some unrelated reason, but make sure this
doesn't break them).
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D787
with Pungi 4, the public repos are product-y, we need to add
/Everything/ to the path between the release and the arch.
Again pushing without review to get the tests working.
With the arrival of Pungi 4, the scheduler is no longer using
fedfind-provided BUILD and FLAVOR values, but ones derived from
Pungi properties. BUILD is now simply the Pungi compose_id.
FLAVOR is produced by joining the Pungi variant, type, and
format with '-' characters as the separators.
Pungi, unfortunately, does not treat 'Rawhide' as a release, it
synthesizes a release number for Rawhide composes and places
that in the compose ID. To cope with that, for now, the
scheduler will set RAWHIDE to '1' if the compose is a Rawhide
one. As we have to adapt all places where we parse the release
in any case, this commit consolidates them into a fedorabase
subroutine.
For the one place where we also used to parse the 'milestone'
from fedfind, there is a placeholder get_milestone subroutine
which currently returns an empty string, as I don't yet have a
good handle on how to draw the kinds of distinctions fedfind
mapped to 'milestone' from Pungi metadata.
Summary:
With the previous change to the server_(sata)_multi test, we
need to adjust the post-install test to use vdb not sdb, the
disks are virtIO now (not PATA as they were with 4.2)
Test Plan:
Check the server_multi test actually completes
properly now
Reviewers: garretraziel, jskladan
Reviewed By: jskladan
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D732
Summary:
D637 / ec6b3ff4 switched from using needle matches to using
validate_script_output when we want to run a console command
and check the result. validate_script_output is kinda over-
powered when all you want to do is check the command succeeded
(returned 0), though. testapi provides assert_script_run for
doing exactly that - it runs a script and fails if the script
fails (returns anything but 0). This gives us cleaner code and
is slightly more robust; validate_script_output uses the mini
web server on the worker, which I've occasionally seen crap
out, so it seems good to avoid using it when possible. assert_
script_run doesn't need it.
Test Plan: Check all (affected) tests still work properly.
Reviewers: jskladan, garretraziel
Reviewed By: jskladan, garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D714
Summary:
Not much to say, pretty much just implements the test case using
some commands I dug up that give us handy 0/1 exit statuses.
The assert_script_run function (from testapi) simply runs a
command/script and passes or fails based on the exit status;
we use a handy bash-ism when we *want* the exit status to be 1.
Test Plan: Run the test and check that it passes (properly).
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D713
I messed up the last commit and mistakenly included this test,
which I've been working on and is not yet reviewed. The commit
was only supposed to add base_services_start. I'll send a new
diff for service_manipulation.
Summary:
pretty simple, just make sure no services failed to start. We
may run into the rngd issue here, not sure, let's land it and
see!
Test Plan:
I guess run the test and see what happens? I haven't
actually tested this myself yet, so, yeah.
Reviewers: garretraziel, jskladan
Reviewed By: garretraziel, jskladan
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D710
Summary:
so here's our first attempt to use the 'carry on from a previous
test' stuff! This adds a base_selinux test that uses a disk
image from a previous default_install run, and adds jobtemplates
to run base_selinux for appropriate products: generic_boot
(for nightly tests), server_dvd, and workstation_live. Note that
you'll want to either update to the newest openQA package I just
built in COPR or create /var/lib/openqa/share/factory/tmp owned
by geekotest; openQA tries to use that directory as MOJO_TMPDIR
but in 4.2, if the directory doesn't exist, it doesn't create it,
and we wind up with the default MOJO_TMPDIR which is /tmp; when
the disk image is uploaded it creates a huge temp file in /tmp
and may well exhaust the available space as it's a tmpfs. I've
backported a recent upstream commit that tries to create the
directory if it doesn't exist, in 4.2-10.
It seems like openQA is smart enough to figure out the
dependencies correctly, so the 'base_selinux' test for each
product depends on the 'default_install' test for the same
product (not any of the other default_install runs) and will
use the hard disk image it produces.
Test Plan:
Do a full test run and make sure base_selinux tests
appear for appropriate products, depend on the correct default_
install test, the default_install test uploads the hard disk
image correctly, and the base_selinux test runs correctly. And
of course that nothing else broke in the process...
Reviewers: jskladan, garretraziel
Reviewed By: jskladan
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D699
Summary:
Instead of sitting there waiting 6000 seconds twice, when DNF
explicitly tells us it failed, just die.
This is why we haven't been getting proper compose check reports
lately; the upgrade tests are failing, waiting 6000 seconds to
time out, then being cloned and tried again, waiting another 6000
seconds. This is just barely going beyond check-compose's 8 hour
wait limit, as it's some time before the upgrade tests even get
started (they're low in the priority list). We're still going to
have that problem if the tests fail any other way, but this at
least catches that case.
Test Plan:
Run the upgrade tests and see that they fail quicker
(assuming the dependency problems they're dying on aren't fixed).
Maybe also do a 22-23 upgrade test and check it still succeeds
properly.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D650
Summary:
When Rawhide's using a debug kernel, the install is pretty slow,
and often times out. This gives us a 33% longer timeout for the
install process when running on Rawhide. This will slow Rawhide
runs down a bit when there are genuine failures at this point,
but it seems kind of unavoidable.
Test Plan: Do a Rawhide run and see if we get fewer false fails.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D643
Instead of removing the ` QA:Testcase_Anaconda_user_creation` from all
the testsuites, make OpenQA login in console (as user) each time.
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D636
Summary:
Updating the stable release prior to doing the update can take
a long time if the image hasn't been updated for a while, and
the upgrade download process itself can take a long time. If
the screen blanks out in either case, either the following
needle match may fail (if we're waiting for a needle) or 'still
screen' may be detected early (if we're waiting for a still
screen), so let's disable screen blanking to avoid it.
Test Plan: Run the upgrade tests and see if they work.
Reviewers: garretraziel, jskladan
Reviewed By: jskladan
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D628
Summary:
For Rawhide, the text in os-release-fedora is 'rawhide' not
'Rawhide', so lc the $VERSION value when checking. For other
releases it's digits, so the lc won't change anything - lc('23')
is '23'.
Test Plan: Run the Rawhide upgrade tests, they should succeed.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D620
Summary:
T617 makes some good points about the language tags; this is my
suggestion for an improvement. It requires a bit of cleverness
in unregister_prefix_tags(), but the upshot is that you don't
need to know to set any special tags when creating needles, a
needle with no language-related tags will be considered as valid
for all languages. You have to explicitly add LANGUAGE- tag(s)
to a needle for the language filtering to 'kick in' in any way.
If a needle has at least one LANGUAGE- tag, it will be filtered
unless it has the appropriate tag for the job's specified
language (default is still 'english').
With this approach, only needles which we specifically want to
*only* match their tagged language(s) need the tags, so we can
drop all those -ALL tags.
We're using LANGUAGE- instead of ENV-LANGUAGE- now because the
ENV- tag names denote tags that are treated slightly specially
by openQA, and this is not one. We cannot cleanly use
ENV-INSTLANG because openQA has a hardwired default of 'en_US'
for that.
Test Plan:
Check both English and French tests still work as
intended.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D589
Summary:
this handles Non-English European Language Install. Basically
it's a bunch of new screenshots for existing tag names, plus
a bit of configurability in _boot_to_anaconda and tweaking some
existing needles to do non-text matches. The weird 'half-the-
icon' needles are for cases where there may or may not be a
warning triangle but we want to click it either way (saves
duplicating the needle).
This also sets up a convention for tagging what languages a
needle is appropriate for. If it's specifically appropriate for
one or more languages, a tag ENV-LANGUAGE-(LANGUAGE) should be
applied for each language, where (LANGUAGE) is the install
language in upper-case ('LANGUAGE' variable, which should also
be the string that will be typed into the language selection
screen). If the needle ought to be used for *all* languages -
i.e. it's not a text match, or any text in the match is known
not to be translated - the tag ENV-INSTLANG-ALL should be
applied.
To back this, main.pm now unregisters all needles that are not
tagged with either ENV-LANGUAGE-ALL or the tag for the language
actually being used (if the LANGUAGE var is not set, we assume
english). The point of this is to check the install is actually
translated; if we allow all needles to match, the test would
pass even if no translations appeared at all.
Test Plan:
Run all tests and make sure you get the expected
results. You can schedule a run against 23 Beta TC1 to see the
French test fails 'correctly' when translations are missing.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D577
Summary:
This is a first cut which more or less works for now. Issues:
1) We're not really testing the BUILD, here. All the test does
is try and upgrade to the specified VERSION - so it'll be using
the latest 'stable' for the given VERSION at the time the test
runs. This isn't really that terrible, but especially for TC/RC
validation, we might want to make things a bit more elaborate
and set up the repo for the actual BUILD (and disable the main
repos).
2) We'd actually need --nogpgcheck for non-Rawhide, at one
specific point in the release cycle - after Branching but
before Bodhi activation (which is when we can be sure all
packages are signed). This won't matter until 24 branches, and
maybe releng will have it fixed by then...if not, I'll tweak
it.
3) We don't really test that the upgrade actually *happened*
for desktop, at the moment - the only thing in the old test
that really checked that was where we checked for the fedup
boot menu entry, but that has no analog in dnf. What we should
probably do is check that GUI login works, then switch to a
console and check /etc/fedora-release just as the minimal test
does.
Test Plan:
Run the tests. Note that creating the desktop disk
image doesn't work ATM, so I can't verify the desktop test
works, but the minimal one seems to (with D565). There'll be
a matching diff for openqa_fedora_tools to update the test
case names there.
Reviewers: jskladan, garretraziel
Reviewed By: jskladan, garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D567
I've seen no_swap fail several times with the 'Click for details
or press Done again to continue.' message showing (e.g. job
967 on BOS). I think this is just because sometimes we try and
click Done too fast, so introduce a 1-second sleep between Done
clicks to try and solve that.
Summary:
since we did this live at Flock today, I figured I'd tidy it
up and submit it. This is an 'optional' test, but some people
do run this way so it'd be nice to have it. This adds another
little helper method in anacondatest.pm, for deleting partitions,
which works much like the others added in previous commits.
Test Plan: Schedule a test run, see if the test runs and works.
Reviewers: jskladan, garretraziel
Reviewed By: jskladan, garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D503
Summary:
This adds three new custom storage tests and some needles to
support them, and tweaks the custom storage methods a bit to
address some things that cropped up in writing the tests. A
new method is added for changing the filesystem, as that's
a distinct operation from changing the device type.
This also restores the previous behaviour of select_disks()
where it handled selecting custom partitioning when needed.
Turns out it's pretty common to use regex'es in perl! Who'd'a
thought.
A corresponding commit to add the tests to openqa_fedora_tools
is coming.
There's no post-install step for the tests yet; I'll try and
write those up and add them soon.
Test Plan:
Do a full run, including the new tests, on Alpha RC2 and check
all are scheduled correctly and run correctly. The LVM thinp
test is expected to fail as it catches a genuine bug.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D490
Summary:
This contains several tweaks to storage handling. It adds a
method for disk selection which all the storage tests can
share. It sets up a more extensible approach for main.pm to
run the storage tests, instead of an ever-growing forest of
'else' clauses. Finally it sets up a couple of methods for
changing partitioning schemes on the custom part screen and
uses one of them in the software RAID test; the other will
be used for other custom storage tests.
This kills the two_disks needle. I could keep it and work
it into select_disks, but it doesn't fit naturally and I
really just don't see the point of the needle. The only thing
we lose is we don't check that anaconda actually sees two
disks in the 'attach two disks, only install to one' test
(that's server_sata_multi), but the other multi-disk tests
will serve to catch that case failing for some reason.
What I actually intended to do was add some more tests for
different custom part storage types, but it seemed a good
idea to do some of this cleanup so that can be implemented
efficiently. I'll have followups for that.
Test Plan:
Run all tests and ensure they work exactly as
before (not just that they still pass, but that the correct
test steps are actually scheduled in each case.)
Reviewers: garretraziel, jskladan
Reviewed By: garretraziel, jskladan
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D475
Summary:
I was having a weird failure that I finally figured out: when
software source config runs then storage config, the 'begin_
installation' needle can match while the 'slide down from the
top' animation is still playing, and by the time os-autoinst
positions the cursor to click where the button was when the
match happened, it's moved down and we wind up clicking outside
the button area. So, wait a sec before clicking to avoid this.
Test Plan: Run the server_sata_multi test.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D464
Summary:
Root console in anaconda got broken by RHBZ #1222413 - no
shell on tty2. Decided to clean up console use in general as
part of fixing it.
This creates a class 'fedorabase' and has 'anacondalog' and
'fedoralog' both inherit from it. boot_to_login_screen is
moved there (as it seems appropriate) and it has a new
method, console_login, which basically handles 'get me a
shell on a console': if we're already at one it returns,
if not it'll type the user name and the password *if
necessary* (sometimes it's not) and return once it sees a
prompt. It takes a hash of named parameters for user,
password and 'check', which is whether it should die if it
fails to reach a console or not (some users don't want it
to).
anacondalog and fedoralog both get 'root_console' methods
which do something appropriate and then call
console_login; both have a hash of named parameters,
anacondalog's version only bothers with 'check', while
fedoralog's also accepts 'tty' to pick the tty to use.
This also adjusts all things which try to get to a console
prompt to use either root_console or console_login as
appropriate.
It also tweaks the needle tags a bit, drops some unneeded
needles, and adds a new 'user console prompt' needle; we
really just need two versions of the root prompt needle
and two of the user prompt needle (one for <F23, one for
F23+ - the console font changed in F23, and the @ character
at least doesn't match between the two). I think we still
need the <F23 case for upgrade tests, for now.
Test Plan:
Do a full test run and see that more tests
succeed. I've done a run on happyassassin with a hack to
workaround the SELinux issue for interactive installs,
and the results look good. I also fiddled about a bit to
test some different cases, like forcing a failure in a
live test to test post_fail_hook (and hence root_console)
in that scenario, and forcing failures after some console
commands had been run to check that it DTRT when we've
already reached a console, etc.
Reviewers: jskladan, garretraziel
Reviewed By: jskladan, garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D462
This requires adding products, flavors and needles and test
cases, and tweaking some existing ones to handle the
slightly different behaviour of live images in shared tests.
To handle the different main hub screens in live and non-live,
a less stringent needle is added which is unregistered for
non-live tests, so they don't match on it before they've
finished updating repository metadata.
There are a few small bugfix tweaks in this too, like some
delays in user creation to try and avoid intermittent failures
there.
A new root_logged_in needle is also included, to handle a new
console font in Rawhide - that has nothing strictly to do with
live testing, it just happened to come up while working on
this.
Adam Williamson
Jan Sedlák
Josef Skladanka
Petr Schindler
Garret Raziel