As suggested by @kparal, this adds a test that specifies an
additional repository using a metalink. The repository contains
a single package, 'testpackage', that supplements glibc (so it
should always get installed). The test runs an install then
checks that testpackage got installed.
We also deduplicate a pair of needles which were matching on the
same anaconda UI feature (an "add" button) and use that same
needle in this test.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This PR fixes issue #188. It adds a test suite to test basic
functionality of Evince and brings the following features:
* test scripts for various Evince functions.
* needles to support the Evince test scripts
* new template variables `TESTPATH` and `POSTINSTALL_LOAD_ALL` (see
below)
* new logic in `main.py` (see below)
The new variables and the new logic make it easier to create test
suites for post-installation tests. If TESTPATH is used, OpenQA
will take all tests mentioned in POSTINSTALL from that specified
TESTPATH. If both TESTPATH and POSTINSTALL_LOAD_ALL are used, then
OpenQA will run all tests it can find at the TESTPATH location.
If POSTINSTALL and POSTINSTALL_LOAD_ALL are set simultaneously,
then only POSTINSTALL will be taken into account and OpenQA will
only load tests mentioned there.
This PR adds a new test that automates the above mentioned test case.
It starts the installation in text mode using the `install_text` test
case, which it interrupts using the Anaconda crash trigger.
When the crash happens, it goes through the process of reporting
the bug to Bugzilla, checks that Bugzilla sends a positive
confirmation of the action, but also performs some REST API
calls to do a proper check and then it closes the bug to clean up.
Setting HDD_1 to %HDD_2% is broken in recent openQA:
https://github.com/os-autoinst/openQA/pull/3309#issuecomment-721906935
I count this as a bug, but we can solve it like this, I think -
we don't actually need to set this test up so carefully to only
have the disk image as HDD_1 and no HDD_2, we can actually just
let the disk image be HDD_2 and have an empty disk as HDD_1 and
the test still works, qemu will boot from the second disk and we
can upload it and everything's fine. So let's just go with that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This sets us up to test the release-blocking aarch64 disk images
(Minimal, Server and Workstation). It also allows for testing
armhfp disk images on aarch64 worker hosts (though my testing of
that isn't going too well so far), and fixes the initial-setup
handling for a change upstream ('use password' is now the default
so we don't need to choose it). We rewire disk image deployment
test loading to work through the generic loader code rather than
using ENTRYPOINT, as it allows us to more gracefully handle
graphical (Workstation) vs. console (Server, Minimal), moving
the code for handling console initial-setup to a helper function
just like the code for gnome-initial-setup and having _console_
wait_login call it when appropriate. We also tweak desktop_vt a
bit because now we need to switch from a console running as test
to a desktop, which breaks the assumption that the highest
numbered session of user test is the desktop...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This adds a test suite for fifloader (renamed fifloader.py for
test sanity). It adds JSON Schema form schemas for both FIF and
upstream openQA template data, and has fifloader (optionally,
but by default) validate both input and output data against the
schemas. It also adds a tox.ini configured to run the fifloader
tests, use fifloader to validate the template files, and do diff
coverage and lint checks. It also adjusts the Zuul config to run
tox instead of the test job.
There are also some pylint cleanups, since the new tests run
pylint.
fifcheck, fifconverter and tojson.pm are removed, as they were
mainly only needed for one-time conversion of the old format
templates; now they are in the git history we can always recover
them if we need them.
Along with all this I updated the README a bit to explain some
of it (and explain FIF better), and to explicitly state that this
repo is GPLv2+ licensed, and added GPL headers to some of the
files.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The French install/boot test fails on aarch64 due to a bit of an
ordering issue:
https://openqa.stg.fedoraproject.org/tests/665124
we run `uefi_postinstall`, which does `loadkeys us`, before we
run `_console_login`, which still expects the French layout, so
it breaks. The safest way to solve this I think is to add a new
variable that lets us skip `uefi_postinstall` - I don't want to
change the ordering so we load `uefi_postinstall` after we read
`POSTINSTALL` in case that breaks things somewhere else.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This adds a test for QA:Testcase_Anaconda_User_Interface_VNC -
the VNC install test case. It's implemented as a server/client
pair, with the server booting from the Server DVD image with
`inst.vnc` and the client booting from the desktop base disk
image, setting up networking, then running Boxes to connect to
the server and run the install.
There are various little tweaks to test loading and logic to
handle this, mostly pretty clear. We also move the workaround
for 'spurious auth prompt appears on desktop after you switch
away to a VT and back' out of the desktop update test and into
the `desktop_vt` helper function, since now this test can hit
it as well. We enhance _graphical_wait_login to handle the boot
loader if needed (it has never needed to until now).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This PR adds the following:
* moves out the presetting procedures, so that two long terminal tests do not have
to run twice
* add methods for application to register when successfully started
* adds a test that checks if all required applications have registered
The root and user spokes got removed from the Silverblue
installer, so handle that the same way as Workstation live.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Add png files as a background for the needles.
Rewrite the test handling methods to make them ready for KDE
Add the ABRT startstop test.
Make changes and corrections to the abrt test.
Add test for Firewall.
Add test for DNF dragora.
Add wait time for dnfdragora test.
Correct syntax.
Add Language test.
Make some changes to the DNF dragora test.
Add Users test case.
Add needles for DBUS viewer test.
Add Dbusviewer test.
Add Mahjong test and needles for games.
Add Minesweeper tests.
Add Patience test.
Add test for Document Viewer.
add test for gwenview
Add test for koulourpaint.
Add test for Kruler
Add test for Kcolorchooser
Reneedle failing needle.
Add ktorrent tests.
Add tests for CPT editors.
Add test for Krfb
Fix names for those files.
Add test for Kget
Add Akregator test.
Add test for Konversation.
Make Konversation end really.
Add tests for Kmail
Add test for PIM exporter.
Add test for KTnef and Krdc.
Fix problems after test runs.
Make more tries.
Fix needle to be found better.
Fix more errors.
Add test for Falkon.
Add tests for browsers.
Add support for closing tabs into Firefox test.
Add tests for K3B
move needles to correct directory
Add Kaddressbook test.
Add Kontakt text.
Add test for korganizer.
Add menu office needle and correct konqueror needle.
Add test for calligra stage.
Add test for Calligra.
Add test for network connections.
Modify needle for kaddressbook to prevent failing.
Add test for system settings and fix others.
Add test for FMW.
Add test for Dolphin
Add test for Infocenter.
Add test for kparted.
Fix a wrong needle.
Test relnotes.
Fix some errors in tests.
Add test for Discover.
Add test for Ksysguard.
Add tests for Konsole.
Add tests for KDE wallet.
Add tests for several utilities.
Add Krusader test.
Finish utility tests.
Fix some errors.
Fix needle for spectacle.
Add wait time to let Dragora wait for network.
The previous attempt at this (in a20ea59) was wrong, because it
meant we didn't create the update repo at all, and we do need
the repo to exist in the support_server test every time (as the
point of the support server is to serve it out via NFS). We just
don't want to *use it on the support_server system itself* if
the releases don't match. So rejig things to do that properly.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
I caught a few failed support_server update tests which failed
because packages from the update weren't installed; they weren't
installed because the update was for F28, but the support server
runs on an F29 disk image. Obviously we should skip the 'install
packages from the update' step in the support_server test unless
the versions match.
Arguably we could *never* install updates in support_server, but
I think it's actually valuable to know if the update breaks
support_server if we *are* on the right version.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We quite often want to run the update tests on a Koji task (not
a Bodhi update) for some reason - usually to test a potential
fix for an issue, or at a maintainer's request to test a change
before it is merged upstream and officially sent out as an
update. Up till now I've always hacked up utils.pm on the
staging server by hand to do this, which is horrible. Together
with a commit to fedora_openqa, this should allow us to do it in
a nice, sane way via the CLI. It's mostly just tweaking the
"updates" repo setup in utils.pm as you'd expect, but there's a
bit of subtlety to it because of the installer tests that use
%ADVISORY% as a variable substitution in the disk image name;
you can't do something like `%ADVISORY or KOJITASK%`, sadly, so
I had to have almost-redundant variables ADVISORY, KOJITASK and
ADVISORY_OR_TASK (we could kinda just live with ADVISORY_OR_TASK
except I didn't want to drop ADVISORY as it's an unnecessary
change from previous behavior).
Signed-off-by: Adam Williamson <awilliam@redhat.com>
In https://bugzilla.redhat.com/show_bug.cgi?id=1669256 it became
obvious that there's a missing feature in the new installer test
for updates: the update is both used in the image build process
and built into the installer environment itself, but it is not
actually included in the installed package set. This can be a
problem if the update has a bug that manifests *only* at install
time if it is in the install transaction (which is exactly the
case there), because the test will not catch this, and nor will
any other test.
So this commit makes `support_server` set up the update repo and
serve it out via NFS when it's run in an update context, and
makes the actual update install test run parallel with it and
use that repository. This way the install should include the
package(s) from the update. (It also of course means the test
fails if an update breaks NFS or something like that, but hey,
we want to know that!)
A parallel commit for fedora_openqa is necessary to add the
required CURRREL setting for the updates-installer flavor.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This adds a test which builds a netinst image potentially with
the package(s) from the update, and uploads that image. It also
adds a test which runs a default install using that image. This
is intended to check whether the update breaks the creation or
use of install images; particularly this will let us test
anaconda etc. updates. We also update the minimal disk image
name, as we have to make it bigger to accommodate this test,
and making it bigger changes its name - the actual change to
the disk image itself is in createhdds. We also have to redo a
bunch of installer needles for F28 fonts, after I removed them
a month or so back...
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Looking at this, it's a bit weird: the updated packages are
actually included in the upgrade process, but we still run
_advisory_update, which does basically nothing...then reboots.
That's kinda silly and makes the tests a bit flaky, let's fix
it. I don't think there's actually any problem with doing the
upload of updatepkgs.txt in _repo_setup_updates, becase that
already guards against being run more than once, it just bails
very early if it's already been run.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Previous approach wouldn't work for tests that run after the
install test...let's just set a password from a chroot after
install completes. Don't really like this as it changes the
'real' install process a bit, but it's the least invasive short
term fix at least. We can maybe do something more sudo-y later
with a bit more thought.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It's really INSTALL_NO_USER, not USER_LOGIN='false'. Also, we
need to make root_console work with no root password, sigh.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Sigh, 'overloaded' product templates like that don't quite work.
So, let's try doing it a different way, in main.pm.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Current Modular composes don't include any graphical desktop
bits, so they're never going to want to do this. (This makes
sense for e.g. non-English install tests; we still want to
make sure Russian install works on Modular composes, we just
have to skip the post-install graphical input testing).
Summary:
As we're getting the Workstation dvd-ostree (OStree installer
image) built for Rawhide now, let's try testing it.
Test Plan:
Run the tests on a Rawhide compose that works and
has the image (e.g. 20170615.n.0). Check that new tests work
as expected and old tests are not adversely affected. A
corresponding diff for fedora_openqa will be coming to take
care of scheduling. Note that the tests will often soft fail
for now; this is intentional due to RHBZ#1193590, the bash
prompt for root is incorrect on ostree installs, so I have
added a needle that matches the incorrect prompt but which is
flagged as a workaround needle (so causing the test result to
be a soft fail).
Reviewers: jsedlak, jskladan
Reviewed By: jsedlak
Subscribers: tflink
Differential Revision: https://phab.qa.fedoraproject.org/D1211
Summary:
This adds an upgrade variant of the FreeIPA tests, with only
the simplest client enrolment (sssd) for now. The server test
starts from the N-1 release and deploys the domain controller
role. The client test similarly starts from the N-1 release
and, when the server is deployed, enrols as a domain client.
Then the server upgrades itself, while the client waits (as the
server is its name server). Then the client upgrades itself,
while the server does some self-checks. The server then waits
for the client to do its checks before decommissioning itself,
as usual. So, summary: *deployment* of both server and client
occurs on N-1, then both are upgraded, then the actual *checks*
occur on N.
In my testing, this all more or less works, except the role
decommission step fails. This failure seems to be a genuine one
so far as I can tell; I intend to file a bug for it soon.
Test Plan:
Run the new tests, check they work. Run the existing
FreeIPA tests (both the compose and the update variants), check
they both behave the same.
Reviewers: jsedlak, jskladan
Reviewed By: jsedlak
Subscribers: tflink
Differential Revision: https://phab.qa.fedoraproject.org/D1204
Summary:
Loading the same module more than once *kinda* works, but it
shows up all kinds of funky in the openQA web interface. There's
a POO for this:
https://progress.opensuse.org/issues/10514
But it doesn't seem like it's going to be resolved immediately,
so in the mean time maybe we should avoid doing it so we don't
have to deal with the weirdness it produces in the web UI. So
here's a kinda icky hack that uses symlinks and stuff to load
multiple instances of 'the same' test module.
Test Plan:
Run an update test, look at how it looks in the web
UI and confirm it's a lot clearer and less buggy. Check there
aren't any bugs in the loading approach. This is deployed on stg
so you can look at it there.
Reviewers: jsedlak, jskladan
Reviewed By: jsedlak
Subscribers: tflink
Differential Revision: https://phab.qa.fedoraproject.org/D1186
Summary:
This adds some logging related to the update testing workflow,
so we have some idea what we actually tested. We log precisely
which packages were actually downloaded from the update - this
is important as updates can be edited and when examining results
we'll want to know which packages actually got used. We also
add a new module which runs at the end of postinstall and tries
to figure out which packages from the update were installed in
the course of the test. This still isn't a guarantee the test
actually *tested them* in any way, but it at least means they
got installed successfully and didn't interfere with the test.
Test Plan:
Run the update test workflow, check the logs get
uploaded and seem accurate (sometimes some RPM garbage messages
wind up in the package log, I'm not too worried about that at
present). Run the compose test workflow and check it didn't
break.
Reviewers: jsedlak
Reviewed By: jsedlak
Subscribers: tflink
Differential Revision: https://phab.qa.fedoraproject.org/D1149
Summary:
This adds an entirely new workflow for testing distribution
updates. The `ADVISORY` variable is introduced: when set,
`main.pm` will load an early post-install test that sets up
a repository containing the packages from the specified update,
runs `dnf -y update`, and reboots. A new templates file is
added, `templates-updates`, which adds two new flavors called
`updates-server` and `updates-workstation`, each containing
job templates for appropriate post-install tests. Scheduler is
expected to post `ADVISORY=(update ID) HDD_1=(base image)
FLAVOR=updates-(server|workstation)`, where (base image) is one
of the stable release base disk images produced by `createhdds`
and usually used for upgrade testing. This will result in the
appropriate job templates being loaded.
We rejig postinstall test loading and static network config a
bit so that this works for both the 'compose' and 'updates' test
flows: we have to ensure we bring up networking for the tap
tests before we try and install the updates, but still allow
later adjustment of the configuration. We take advantage of the
openQA feature that was added a few months back to run the same
module multiple times, so the `_advisory_update` module can
reboot after installing the updates and the modules that take
care of bootloader, encryption and login get run again. This
looks slightly wacky in the web UI, though - it doesn't show the
later runs of each module.
We also use the recently added feature to specify `+HDD_1` in
the test suites which use a disk image uploaded by an earlier
post-install test, so the test suite value will take priority
over the value POSTed by the scheduler for those tests, and we
will use the uploaded disk image (and not the clean base image
POSTed by the scheduler) for those tests.
My intent here is to enhance the scheduler, adding a consumer
which listens out for critpath updates, and runs this test flow
for each one, then reports the results to ResultsDB where Bodhi
could query and display them. We could also add a list of other
packages to have one or both sets of update tests run on it, I
guess.
Test Plan:
Try a post something like:
HDD_1=disk_f25_server_3_x86_64.img DISTRI=fedora VERSION=25
FLAVOR=updates-server ARCH=x86_64 BUILD=FEDORA-2017-376ae2b92c
ADVISORY=FEDORA-2017-376ae2b92c CURRREL=25 PREVREL=24
Pick an appropriate `ADVISORY` (ideally, one containing some
packages which might actually be involved in the tests), and
matching `FLAVOR` and `HDD_1`. The appropriate tests should run,
a repo with the update packages should be created and enabled
(and dnf update run), and the tests should work properly. Also
test a regular compose run to make sure I didn't break anything.
Reviewers: jskladan, jsedlak
Reviewed By: jsedlak
Subscribers: tflink
Differential Revision: https://phab.qa.fedoraproject.org/D1143
Summary:
This adds a new test, memory_check, which just does a default
package set install with `inst.debug` parameter then uploads
the memory usage file (`/tmp/memory.dat`) at the end. We can
have check-compose use the data to analyze changes in memory
usage over time.
Test Plan:
Fire off the Workstation network install image tests
and make sure the memory usage test runs and works on all three
machines. This is live on staging already.
Reviewers: jskladan, garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames
Reviewed By: garretraziel_but_actually_jsedlak_who_uses_stupid_nicknames
Subscribers: tflink
Differential Revision: https://phab.qa.fedoraproject.org/D1082
Summary:
Include some basic testing of Japanese input, and split the
input testing (including Russian) into a separate module, since
it's not really part of 'login' testing.
Test Plan:
Run the test, and the Russian and French tests too to
make sure they didn't break. Tested on staging. Note the Japanese
test soft fails, intentionally, at present, as I discovered a bug
while working on it:
https://bugzilla.gnome.org/show_bug.cgi?id=776189
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1072
Committing without review as this causes failures...try to make
sure we only run the AVC test when it makes sense, and fix
running it on the French install test.
Summary:
This has all console tests check for AVCs (with ausearch) and
crashes (with coredumpctl) at post-install stage. It's non-
fatal as this doesn't really mean the test failed, but we want
to spot when there are unexpected AVCs or crashes.
Test Plan:
Run some console tests, check it works right. I only
tested with one test, since so many are broken on Rawhide ATM
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1066
Summary:
I started out wanting to fix an issue I noticed today where
graphical upgrade tests were failing because they didn't wait
for the graphical login screen properly; the test was sitting
at the 'full Fedora logo' state of plymouth for a long time,
so the current boot_to_login_screen's wait_still_screen was
triggered by it and the function wound up failing on the
assert_screen, because it was still some time before the real
login screen appeared.
So I tweaked the boot_to_login_screen implementation to work
slightly differently (look for a login screen match, *then* -
if we're dealing with a graphical login - wait_still_screen
to defeat the 'old GPU buffer showing login screen' problem
and assert the login screen again). But while working on it,
I figured we really should consolidate all the various places
that handle the bootloader -> login, we were doing it quite
differently in all sorts of different places. And as part of
that, I converted the base tests to use POSTINSTALL (and thus
go through the shared _wait_login tests) instead of handling
boot themselves. As part of *that*, I tweaked main.pm to not
require all POSTINSTALL tests have the _postinstall suffix on
their names, as it really doesn't make sense, and renamed the
tests.
Test Plan: Run all tests, see if they work.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D1015
This adds possiblity to specify more than one test in ENTRYPOINT
variable - test names should be separated with spaces and loading is done
as with POSTINSTALL.
Add _console_shutdown test to ARM image deployment.
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D973
Summary:
Set up the support server to provide DHCP/DNS functionality and
an NFS server, providing a kickstart. Add a kickstart test just
like the other root-user-crypted-net kickstart tests except it
gets the kickstart from the support server via NFS. Also add NFS
repository tests and a second support server for Server-dvd-iso
flavor: this test must run on that flavor to ensure that packages
are actually available. The support server just mounts the
attached 'DVD' and exports it via NFS.
Note we don't need to do anything clever to avoid IP conflicts
between the two support servers, because os-autoinst-openvswitch
ensures each worker group is on its own VLAN.
As part of adding the NFS repo tests, I did a bit of cleanup,
moving little things we were repeating a lot into anacondatest,
and sharing the 'check if the repo was used' logic between all
the tests (by making it into a test step that's loaded for all
of them). I also simplified the 'was repo used' checks a bit,
it seems silly to run a 'grep' command inside the VM then have
os-autoinst do a grep on the output (which is effectively what
we were doing before), instead we'll just use a single grep
within the VM, and clean up the messy quoting/escaping a bit.
Test Plan:
Run all tests - at least all repository tests - and
check they work (make sure the tests are actually still sane,
not just that they pass). I've done runs of all the repo tests
and they look good to me, but please double-check. I'm currently
re-running the whole 24-20160609.n.0 test on staging with these
changes.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D888
Summary:
this is following a SUSE model for tests where we need a server
end but don't want setting up the server to constitute a real
test in itself, we want it to be stable. The 'support_server'
test just boots a pre-built (by createhdds) disk image, sets up
networking, and runs the iSCSI server.
To run the iSCSI test we need to handle networking config in
anaconda (or we would need to set the support server up as a
DHCP server, which may be worth considering), so this adds that.
We also need to be able to specify the target device for a
volume in custom partitioning, so this adds that too.
Test Plan:
Build the necessary support server disk image (use
D883), then run the test and make sure it works. Also make sure
all other tests continue to work.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D884
Summary:
This requires a few other changes:
* turn clone_host_resolv into clone_host_file, letting you clone
any given host file (cloning /etc/hosts seems to make both
server deployment and client enrolment faster/more reliable)
* allow loading of multiple POSTINSTALL tests (so we can share
the freeipa_client_postinstall test). Note this is compatible,
existing uses will work fine
* move initial password change for the IPA test users into the
server deployment test (so the client tests don't conflict over
doing that)
* add GRUB_POSTINSTALL, for specifying boot parameters for boot of
the installed system, and make it work by tweaking _console_wait
_login (doesn't work for _graphical_wait_login yet, as I didn't
need that)
* make the static networking config for tap tests into a library
function so the tests can share it
* handle ABRT problem dirs showing up in /var/spool/abrt as well
as /var/tmp/abrt (because the enrol attempt hits #1330766 and
the crash report shows up in /var/spool/abrt, don't ask me why
the difference, I just work here)
* specify the DNS servers from the worker host's resolv.conf as
the forwarders for the FreeIPA server when deploying it; if we
don't do this, rolekit defaults to using the root servers as
forwarders(!) and thus we get the public, not phx2-appropriate,
results for e.g. mirrors.fedoraproject.org, some of which the
workers can't reach, so PackageKit package install always fails
(boy, was it fun figuring THAT mess out)
Even after all that, the test still doesn't actually pass, but
I'm reasonably confident this is because it's hitting actual bugs,
not because it's broken. It runs into #1330766 nearly every time
(I think I saw *one* time the enrolment actually succeeded), and
seems to run into a subsequent bug I hadn't seen before when
trying to work around that by trying the join again (see
https://bugzilla.redhat.com/show_bug.cgi?id=1330766#c37 ).
Test Plan:
Run the test, see what happens. If you're really lucky,
it'll actually pass. But you'll probably run into #1330766#c37,
I'm mostly posting for comment. You'll need a tap-capable openQA
instance to test this.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D880
Summary:
I really just want to add the desktop_terminal test, but I think
this refactor is in order now. It splits up loading of the
various test phases (much as SUSE do it) and allows us to run
the post-install tests without the install tests, for e.g. I
tweaked things to allow the upgrade tests to use the existing
_wait_login tests for final login and combine the two upgrade
postinstall tests into one simple one.
This comes with a bit of a behaviour change to make graphical
wait login behave the same as console wait login: it will log
in unless USER_LOGIN is set to 'false'. Previously it only
logged in if both USER_LOGIN and USER_PASSWORD were set, which
I don't think ever happened in a graphical test, so we never
actually did a graphical login. The intent here is we should do
a login on the default_install tests. That's going a bit beyond
the test case, but it seems like a reasonable thing to test. We
can set USER_LOGIN to false if we don't want to do it.
Test Plan:
Do a full test run, make sure the new tests work and
no old tests break.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D839
Summary:
these together test QA:Testcase_kickstart_firewall from the
Server matrix. I'll have to come up with some kinda way to
handle reporting that, might be tricky.
Couple of tweaks to overall test flow: tests can now specify
a POSTINSTALL variable which will load a post-install test
following a naming convention, and tests can specify USER_LOGIN
as 'false' to disable the 'log in as a user' step entirely. We
could easily adjust the kickstarts to create a user so the test
could log in as one, but it seems like an unnecessary step and
I liked the idea of allowing the user login to be skipped.
Test Plan:
Schedule 'universal' tests, check the new tests run
and pass or fail as they should, check no other test is broken
by the logic flow changes.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D792
Summary:
I believe the failures in the Server DVD chained Base tests are
happening because the VM is not cleanly shut down before the disk
image is uploaded. This adds a shutdown step to all tests that
upload a disk image (so, for now, just default_install). To keep
things simple it just runs 'shutdown' from a root console, rather
than using graphical desktop shutdown methods, as the aim is only
to make the disk state clean, not to test shutdown exactly.
I've tested this on staging; a Server DVD test run with this
change produced a full set of passed tests, as opposed to all
the Base tests failing because the system didn't boot properly.
Workstation and KDE tests seem to work fine also.
For the record, SUSE does much the same thing as this commit.
Test Plan:
Do a full test run and make sure everything that worked
before still does. Check that all default_install tests have a
_console_shutdown step added, and it works, and all chained tests
work (or fail for some unrelated reason, but make sure this
doesn't break them).
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D787
Let's use the same principle with DESKTOP tag as we use with
LANGUAGE tag. Where there are any GNOME-only (or KDE-only) needles, tag
them with `DESKTOP-gnome` and delete then when different `DESKTOP` value
is specified. This will help with `QA:Testcase_Package_Sets_KDE_Package_Install`
testcase - we will be able to use almost the same code and check that KDE
got installed properly.
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D670
Summary:
T617 makes some good points about the language tags; this is my
suggestion for an improvement. It requires a bit of cleverness
in unregister_prefix_tags(), but the upshot is that you don't
need to know to set any special tags when creating needles, a
needle with no language-related tags will be considered as valid
for all languages. You have to explicitly add LANGUAGE- tag(s)
to a needle for the language filtering to 'kick in' in any way.
If a needle has at least one LANGUAGE- tag, it will be filtered
unless it has the appropriate tag for the job's specified
language (default is still 'english').
With this approach, only needles which we specifically want to
*only* match their tagged language(s) need the tags, so we can
drop all those -ALL tags.
We're using LANGUAGE- instead of ENV-LANGUAGE- now because the
ENV- tag names denote tags that are treated slightly specially
by openQA, and this is not one. We cannot cleanly use
ENV-INSTLANG because openQA has a hardwired default of 'en_US'
for that.
Test Plan:
Check both English and French tests still work as
intended.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D589
