perl2json converts the openQA templates file to JSON (so we
can parse it with Python). kstest-converter does the meat of
the work of translating kickstart-tests to openQA tests: its
'kickstarts' subcommand produces .ks files from the .ks.in
files (you need to then make them available via HTTP somehow),
its 'templates' subcommand produces openQA 'test suites' and
'job templates' and merges them into an existing templates
file. Then you can load the templates file and fire off the
tests.
Summary:
per details in T759, the 'unipony' updates image we use to test
the updates image features doesn't work with latest anaconda (f24
and Rawhide). I've built a new updates image which uses a neat
anaconda feature that allows you to override CSS with a file in
a special location; it sets the background for disk capacity
texts on the INSTALLATION DESTINATION spoke to be pink. This
lets us use a simple needle that just looks for a pink blob on
that spoke, on the basis that it's unlikely there'll ever be a
pink blob there for any other reason, so if there is one, the
updates image worked. There will be an accompanying tools diff
to change the updates disk image to use the new updates image.
Test Plan:
Do a test run and check the updates image tests pass
and no other tests are broken. You'll need to pull in the tools
diff and re-generate the updates disk image to check that test,
the scsi_updates_img test should work with just this diff.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D799
Some needle changes to account for latest Rawhide. There seems
to be a bug between anaconda and GTK+ 3.20 causing text in the
disk selection widget not to be styled correctly: this commit
adds needles for that which are tagged as 'workaround', meaning
openQA will complete the tests but mark them as requiring a
workaround to work ('soft fail'). The bug is #1322036. There is
also an intentional change to the volume list in custom part;
the mount point names are now in dark grey rather than black (in
fact this has always been intended, but it's been broken for a
long time, according to davidshea). I cleaned up the part select
needles at the same time; the new ones have the non-variant
names, as davidshea says the same change is coming to F24 and
the original needles haven't matched for months. The F23 Atomic
install test doesn't hit any of these needles, so we don't need
to keep the original needles around for that. The most recent
variants are kept, as they'll be needed for F24 tests until the
new anaconda build goes stable for F24. The 'freetype262'
variant of part_select_swap hasn't matched for two months and is
dropped.
Summary:
this should avoid unnecessary disk uploads and hopefully help
further reduce the incidence of weird failures in the chained
tests.
With this change we should only upload disk images for the cases
where we're actually going to run the chained tests: we won't
upload disk images for default_install runs on images we don't
run the chained tests for, or for the UEFI job for images we
*do* run the chained tests for.
We only actually need to run the current chained tests
for Server DVD, Workstation live and KDE live x86_64; there's
no need to run them for Everything boot, so we drop that.
Test Plan:
Do a full test run and make sure all tests run
properly and we now only upload disk images where we really need
to.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D794
Summary:
these together test QA:Testcase_kickstart_firewall from the
Server matrix. I'll have to come up with some kinda way to
handle reporting that, might be tricky.
Couple of tweaks to overall test flow: tests can now specify
a POSTINSTALL variable which will load a post-install test
following a naming convention, and tests can specify USER_LOGIN
as 'false' to disable the 'log in as a user' step entirely. We
could easily adjust the kickstarts to create a user so the test
could log in as one, but it seems like an unnecessary step and
I liked the idea of allowing the user login to be skipped.
Test Plan:
Schedule 'universal' tests, check the new tests run
and pass or fail as they should, check no other test is broken
by the logic flow changes.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D792
Summary:
We named a bunch of the tests 'server_foo' back when we were
starting out and didn't really know what we were doing. They're
all really installation tests, not 'server' tests. I actually
want to start adding some Server tests now, so it seems like a
good time to fix that mess. This standardizes on 'install_' as
a prefix for installation tests, converting all the 'server_'
tests and fixing up a couple of odd cases to use 'install_'.
Test Plan:
Apply along with the matching commit for tools, do
a full test run and report test, and make sure all tests run
with the new names and correct ResTups are generated for wiki
submission.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D790
Summary:
I believe the failures in the Server DVD chained Base tests are
happening because the VM is not cleanly shut down before the disk
image is uploaded. This adds a shutdown step to all tests that
upload a disk image (so, for now, just default_install). To keep
things simple it just runs 'shutdown' from a root console, rather
than using graphical desktop shutdown methods, as the aim is only
to make the disk state clean, not to test shutdown exactly.
I've tested this on staging; a Server DVD test run with this
change produced a full set of passed tests, as opposed to all
the Base tests failing because the system didn't boot properly.
Workstation and KDE tests seem to work fine also.
For the record, SUSE does much the same thing as this commit.
Test Plan:
Do a full test run and make sure everything that worked
before still does. Check that all default_install tests have a
_console_shutdown step added, and it works, and all chained tests
work (or fail for some unrelated reason, but make sure this
doesn't break them).
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D787
Summary:
First off, this revises the anaconda crash handling needles a
bit. We ditch gtk3195 and update anaconda_error to reflect
current F24/Rawhide. We keep the old anaconda_error around for
now as anaconda_error-23, to handle crashes in the F23 two-week
Atomic nightlies. We also add an 'early' variant, which is for
when (I think) the installer crashes very early, before it's
loaded in GTK+ settings; when that happens, the dialog uses a
different font. The screenshot comes from a recent Rawhide test
that crashed.
We also restore the anaconda `post_fail_hook` code to click
the Report button when a crash happens. This was erroneously
removed in D637. Before the Report button is clicked, the
`anaconda-tb` file exists but the libreport stuff in `/var/tmp`
does not. By removing this, we lost the libreport bits from
the uploaded files, which makes it harder to report crashes. So
let's add it back.
Finally we fix the actual tarring and uploading of `/var/tmp`;
also in D637 this got broken because it was being tarred up in
whatever directory the commands happened to be running in, but
we were still trying to upload it from `/var/tmp`.
https://openqa.stg.fedoraproject.org/tests/8444 was run with
these changes, and has `/var/tmp` correctly uploaded.
Test Plan:
Run some test that crashes, make sure the crash
handling all works correctly.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D768
GTK+ 3.19.10(?) has changed things in current F24/Rawhide;
weirdly, done-freetype262 seems to match sometimes but not
always. Add a new needle from a current screenshot but keep
freetype262 around for now just in case. Drop some variants
that should never be needed any more, rename the original
needle to -23 to mark that we're only keeping it around for
the F23 two-week Atomic tests. For now keep french-cantarell20
as the 'official' French needle, we'll see if it hits any
failures when the tests are working a bit better.
with Pungi 4, the public repos are product-y, we need to add
/Everything/ to the path between the release and the arch.
Again pushing without review to get the tests working.
With the arrival of Pungi 4, the scheduler is no longer using
fedfind-provided BUILD and FLAVOR values, but ones derived from
Pungi properties. BUILD is now simply the Pungi compose_id.
FLAVOR is produced by joining the Pungi variant, type, and
format with '-' characters as the separators.
Pungi, unfortunately, does not treat 'Rawhide' as a release, it
synthesizes a release number for Rawhide composes and places
that in the compose ID. To cope with that, for now, the
scheduler will set RAWHIDE to '1' if the compose is a Rawhide
one. As we have to adapt all places where we parse the release
in any case, this commit consolidates them into a fedorabase
subroutine.
For the one place where we also used to parse the 'milestone'
from fedfind, there is a placeholder get_milestone subroutine
which currently returns an empty string, as I don't yet have a
good handle on how to draw the kinds of distinctions fedfind
mapped to 'milestone' from Pungi metadata.
Summary:
With the previous change to the server_(sata)_multi test, we
need to adjust the post-install test to use vdb not sdb, the
disks are virtIO now (not PATA as they were with 4.2)
Test Plan:
Check the server_multi test actually completes
properly now
Reviewers: garretraziel, jskladan
Reviewed By: jskladan
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D732
Summary:
per T691, this has never actually tested SATA. With os-autoinst
4.2 it actually tests PATA, with os-autoinst 4.3 it breaks.
We can't find a way to make it do what we want with os-autoinst
4.3, so we're dropping the HDDMODEL bit entirely for now. We
will file a ticket upstream to see if this can be solved. We
keep the test itself because it's also the only test that hits
'guided_multi'; I'll post a matching diff for tools to change
the wiki reporting config to match.
Test Plan:
Schedule tests with os-autoinst 4.3 and see if this
one runs properly now.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D730
mostly cantarell changes. GTK+ changes are bugs, so put 'bug'
in the name to remind us of this. There's one non-typical
change, to main_hub_installation_source; I removed the match
on the text label of the spoke entirely instead of updating it,
in line with other similar needles, it's unnecessary and only
serves to make the needle more fragile.
Summary:
we have a KDE column for the 'base' tests, so we should run
them on the kde_live flavor.
Test Plan:
Schedule a full test run, ensure all three base
tests are scheduled and run correctly for KDE live.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D720
Summary:
D637 / ec6b3ff4 switched from using needle matches to using
validate_script_output when we want to run a console command
and check the result. validate_script_output is kinda over-
powered when all you want to do is check the command succeeded
(returned 0), though. testapi provides assert_script_run for
doing exactly that - it runs a script and fails if the script
fails (returns anything but 0). This gives us cleaner code and
is slightly more robust; validate_script_output uses the mini
web server on the worker, which I've occasionally seen crap
out, so it seems good to avoid using it when possible. assert_
script_run doesn't need it.
Test Plan: Check all (affected) tests still work properly.
Reviewers: jskladan, garretraziel
Reviewed By: jskladan, garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D714
Summary:
Not much to say, pretty much just implements the test case using
some commands I dug up that give us handy 0/1 exit statuses.
The assert_script_run function (from testapi) simply runs a
command/script and passes or fails based on the exit status;
we use a handy bash-ism when we *want* the exit status to be 1.
Test Plan: Run the test and check that it passes (properly).
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D713
I messed up the last commit and mistakenly included this test,
which I've been working on and is not yet reviewed. The commit
was only supposed to add base_services_start. I'll send a new
diff for service_manipulation.
Summary:
pretty simple, just make sure no services failed to start. We
may run into the rngd issue here, not sure, let's land it and
see!
Test Plan:
I guess run the test and see what happens? I haven't
actually tested this myself yet, so, yeah.
Reviewers: garretraziel, jskladan
Reviewed By: garretraziel, jskladan
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D710
Summary:
Along with the matching change to fedora-openqa-schedule to pass
these variables in when scheduling jobs, this avoids hardcoding
the release numbers for the upgrade tests (which means someone
has to remember to edit them every release). The new createhdds
similarly uses get_current_release() to decide what releases it
needs images for, so all this should hook up and work magically
without any human intervention required.
For clarity, the effect of the '_upgrade_' tests is "run an
upgrade from the 'current' Fedora release to whatever release
is being tested", and the effect of the '_upgrade_2_' tests is
"run an upgrade from the 'previous' Fedora release to whatever
release is being tested".
Test Plan:
Apply with D702, schedule upgrade tests, and make
sure the correct hard disk image filenames are used.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D703
This adds missing default_install tests to templates - we've missed
KDE live and Server DVD on UEFI for some reason and Workstation netinst was
missing. After this, we should have all non-optional tests from "Default
boot and install" section covered.
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D693
Summary:
so here's our first attempt to use the 'carry on from a previous
test' stuff! This adds a base_selinux test that uses a disk
image from a previous default_install run, and adds jobtemplates
to run base_selinux for appropriate products: generic_boot
(for nightly tests), server_dvd, and workstation_live. Note that
you'll want to either update to the newest openQA package I just
built in COPR or create /var/lib/openqa/share/factory/tmp owned
by geekotest; openQA tries to use that directory as MOJO_TMPDIR
but in 4.2, if the directory doesn't exist, it doesn't create it,
and we wind up with the default MOJO_TMPDIR which is /tmp; when
the disk image is uploaded it creates a huge temp file in /tmp
and may well exhaust the available space as it's a tmpfs. I've
backported a recent upstream commit that tries to create the
directory if it doesn't exist, in 4.2-10.
It seems like openQA is smart enough to figure out the
dependencies correctly, so the 'base_selinux' test for each
product depends on the 'default_install' test for the same
product (not any of the other default_install runs) and will
use the hard disk image it produces.
Test Plan:
Do a full test run and make sure base_selinux tests
appear for appropriate products, depend on the correct default_
install test, the default_install test uploads the hard disk
image correctly, and the base_selinux test runs correctly. And
of course that nothing else broke in the process...
Reviewers: jskladan, garretraziel
Reviewed By: jskladan
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D699
3.19.5 seems to change the colors of buttons slightly and also
the rendering of checkboxes/radio buttons, they're now placed
somewhat differently relative to their labels.
Summary:
this is following a couple of upstream changes I noticeed while
playing with dump_templates. Upstream has completely got rid of
the 'variables' keys from all dicts, and when you run dump_
templates the output does not contain any 'variables' keys, so
it seems reasonable to ditch these entirely (I think it was some
old thing that got subsumed into 'settings'). Also, when you
run dump_templates the 'prio' values come out in JobTemplates,
not in TestSuites; if you look at load_templates there's a bit
marked "we have to migrate the prio from the TestSuite to the
JobTemplate" which does exactly what it sounds like, so it
seems a good idea to move these instead of relying on that to
always be there.
While I had to re-do all the priorities anyway I tried to clean
them up a bit. The idea is roughly this:
10 - 19: most-critical sanity tests (i.e. default_install)
20 - 29: Alpha tests
30 - 39: Beta tests
40 - 49: Final tests
50+ : Optional tests
within each group I ordered 64-bit first, UEFI second, and
32-bit last (usually just as x0 / x1 / x2).
Test Plan:
Check that the file is valid and loads correctly,
and that everything works more or less the same except the
order of tests run is a bit different.
Reviewers: jskladan, garretraziel
Reviewed By: jskladan, garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D692
Summary:
Apparently it'll be something like anaconda.core.(PID). This
should result in that getting compressed and uploaded only if it
exists.
Test Plan:
Try and hit the Mystery Crasher and see if we get the
damn core file this time.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D688
Summary:
When anaconda manages to actually crash the python interpreter,
there should be a /tmp/anaconda.core containing the core dump
(per clumens, see calls to 'gcore' in isys.c). Let's upload it.
This might help us track down the mysterious occasional crashes
openQA seems to trigger (RHBZ #1289704)
Test Plan:
Try and trigger a python crash and see if the
file got uploaded. Of course, I did an entire freaking run on
staging and for *ONCE* not one test hit the mysterious crash,
thanks Murphy. I did at least check that this doesn't break
a 'normal' failure, if the file isn't there things don't
explode.
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D686
garretraziel took these with F23, I think. Theoretically we
could just have these ones and ditch his, as we're not running
KDE install tests on F23, but we might want to discuss that kind
of question more generally, so for now let's just add the
variant.
We may need more variants for the post-install matches, but the
test doesn't actually get that far ATM because of dependency
issues in the KDE package set, so I'll deal with that once we
can actually do a successful KDE install.
