With Rawhide updates, we quite often run into a situation where
a test runs after a *later* version of the package has already
gone stable. This even happens for stable releases too, though
less often. The current shell-based check just always fails on
this case, but it's usually OK, and manually marking every case
like this with an "it's OK!" comment gets tiring. Instead, let's
use a smarter Python script to do the check. We compare the EVR
of all installed update packages with the EVR of the package
from the update. If it's the same, fine. If the installed package
is lower-versioned, that's always an error, and we fail. If the
installed package is higher-versioned, we check whether the
update already went stable. If it did, then we soft fail, because
probably nothing can go wrong at this point (this is the usual
Rawhide case). If the update did not yet go stable, we still
hard fail, because something can go wrong in this case: if the
update *now* goes stable, the older version from the update may
be tagged over the newer version the test got (presumably from
current stable).
If anything goes wrong with the Bodhi check, or the test is
running on a task not an advisory, we treat both cases as fatal.
The script also gives easier-to-understand output than the old
approach, which should be a bonus.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Same conditions as used in main.pm to load the tests in the
normal flow. It makes no sense to do this on non-update tests,
or on the non-matching support server case.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is safer if the advisory stuff was done on a previous test
run. Hilariously, this exposed a dumb mistake I made years ago
in installedtest.pm and never noticed before: the calls to
advisory_* at the bottom of that file are meant to be in the
post_fail_hook, but they weren't, which meant they got called
by the scheduler. This didn't cause any failures because the
first line caused them to return immediately based on a get_var
call (which it's OK to do in the scheduler), but changing it
to a script_run call (which it's *not* OK to do in scheduling)
caused all the tests to blow up immediately and confused me
*a lot* until I spotted this!
Signed-off-by: Adam Williamson <awilliam@redhat.com>
When I enabled _advisory_post for live and ostree install tests,
the point was to check that updated packages were included in
the install media and used during installation. We shouldn't run
a system update in _repo_setup_updates on this path because it
will hide the problem if the updated packages weren't included.
The INSTALL variable is for this purpose - it was previously
used to skip _advisory_post on the same path. At the same time
let's remove some stray settings of this var on non-update tests
as it serves no purpose there.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Per discussion at https://pagure.io/fedora-ci/general/issue/376
it really feels like this is the right thing to do. There are no
buildroot overrides for Rawhide, so we don't have to worry about
cross-pollution. The buildroot repo only contains builds that
have been tagged stable since the most recent Rawhide compose,
and thus will go into the next one. It makes sense to test later
updates against these. This avoids issues like:
https://openqa.fedoraproject.org/tests/overview?distri=fedora&version=38&build=Update-FEDORA-2022-30a952e331&groupid=2
where the tests of an update failed because it requires another
update which had been submitted and tagged stable previously, but
after the last compose.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Also use get_var("TEST") for installer_build - no point trying
to upload these logs for the other tests in the same flavor,
they won't be there.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is like the existing tests that build network install and
live images then install them, only for Silverblue. First we
build an ostree, using the standard configuration for the release
and subvariant but with the 'advisory' and 'workarounds' repos
included, so it will contain current stable packages plus the
packages from the update and any workarounds. Then we build an
ostree installer image with the ostree embedded, again including
advisory and workarounds repos in the installer build config so
packages from them will be included in the installer environment.
The image is uploaded, which completes the _ostree_build test.
Then an install_default_update_ostree test runs, which does a
standard install and boot from the installer image.
We do make a change that affects other tests, too. We now run
_advisory_post on live image install tests, as well as this new
ostree install image install test. It was skipped before because
of an exception that's really only needed for the netinst image
install test. In that test, packages from the update won't be
included in the installed system, so we can't run _advisory_post
on it. But for ostree and live image build/install tests, the
installed system *should* include packages from the update, so
we should check and make sure that it does.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There's no need to run this twice (which can happen on some
paths), so if the first file already exists, just bail. Also,
don't bother uploading the config files any more - that was just
for debug while I was making this stuff work, now it works, and
this saves some time.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There's one point in the tests where we may log into cockpit for
the second time in one run (it depends how a package update
process goes). When this happens, we don't get prompted again
for admin access, so we need to *not* expect that.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
It seems to time out a lot on lab but not on prod, for some
reason. Let's just give it a little longer.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
From 280, the cockpit package disabled logging in as root by
default. We could wipe that config file, but it seems better to
respect the default config and log in as the admin user 'test'
instead of as root.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This is simpler if we just always lowercase $iso, plus it saves
us when somebody (*cough*) messes up the casing.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
mock 3.2 does not allow running as root. This breaks the live
image build tests. It's being reverted upstream, this scratch
build has the reversion backported.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
desktop-backgrounds update went stable, but we also need an
f37-backgrounds update which was only just submitted to fix the
problem.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
podman tests are currently failing for all Rawhide updates. Not
sure what triggered this, but this update seems to fix it, though
it's failing its own test suite so not yet being pushed stable.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
We use variables to track test state across modules, sometimes.
As this is all internal to the test logic I didn't bother always
making these variables upper-case, but os-autoinst now treats
lower-case variables as a fatal error, so we have to change.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This issue appeared when we started testing Rawhide updates, but
I only noticed it today. When testing Rawhide updates after
Branch point, the upgrade tests upgrade from Branched to Rawhide.
On Branched, updates-testing is enabled by default. We only
disable it when we reach `upgrade_run`, but by that point we've
already done a `dnf -y update` in `upgrade_preinstall` and
potentially installed other packages in steps between
`upgrade_preinstall` and `upgrade_run`. That can cause problems,
like today all FreeIPA upgrade tests on Rawhide are failing
because there's a newer freeipa in updates-testing for F37 than
is in the current Rawhide compose.
Solve this by disabling updates-testing before we do the update
in `upgrade_preinstall`. To avoid excessive code duplication,
factor out the repo disabling code.
We'll do this twice on upgrade tests now, but it shouldn't be a
problem.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
When we run Firefox directly on X lately, we often hit a bug
where X just suddenly exits in the middle of doing stuff in
Firefox. I'm not sure if this is a bug in X or in Firefox (if
Firefox crashed, X would immediately exit). Let's see if this
helps get any info on what's going on with Firefox.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
samba-client-libs with an soname bump made today's Rawhide, a
freeipa rebuilt against it did not, so we need this in overrides
or all FreeIPA tests fail on all Rawhide updates.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Editing the files under /ostree/deploy doesn't work on Rawhide
any more because it's been made read-only. Thanks to Timothée
Ravier for pointing out this way of doing it, which is better
anyway and avoids awful quotation issues in the code.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
A samba update was pushed for F37 without rebuilds of sssd and
freeipa (which needed to be rebuilt against it). Tests of any
of those updates on its own will fail, so including them all as
workarounds so the tests for each should work. I'll make sure
they all get pushed stable.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
The file conflict between older webkit2gtk4.1-jsc and new
javascriptcoregtk4.1 is causing all Workstation update tests to
fail on all Rawhide updates. This new version of webkitgtk adds
obsoletes to javascriptcoregtk4.1 which we hope should fix it.
Adding the x86_64 build directly as other arches aren't done
yet, and we only run the Workstation update tests on x86_64
anyway.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
