tests: add testsuite and rebuild-wget tests, update others

This commit is contained in:
Miloš Prchlík 2023-05-30 14:53:51 +00:00 committed by nick clifton
parent b18e2741e0
commit 0cfbdb1f11
12 changed files with 316 additions and 135 deletions

View File

@ -10,6 +10,6 @@ product_versions:
- rhel-9
decision_context: osci_compose_gate
rules:
- !PassingTestCaseRule {test_case_name: osci.brew-build.tier0.functional}
- !PassingTestCaseRule {test_case_name: baseos-ci.brew-build.tier1.functional}
- !PassingTestCaseRule {test_case_name: osci.brew-build.rebuild.validation}
- !PassingTestCaseRule {test_case_name: baseos-ci.brew-build.gate-build-fast-lane.functional}
- !PassingTestCaseRule {test_case_name: baseos-ci.brew-build.gate-build-slow-lane.functional}

29
plans/build-gating.fmf Normal file
View File

@ -0,0 +1,29 @@
#
# Build/PR gating tests for binutils
#
summary: binutils tests for build/PR gating
adjust:
- because: "Plan to be ran when executed locally, or executed by CI system to gate a build or PR."
when: >-
trigger is defined
and trigger != commit
and trigger != build
enabled: false
- because: "CRB and Buildroot repositories must be enabled for various BuildRequires"
when: distro == rhel-9
prepare+:
- how: shell
script: dnf config-manager --set-enabled beaker-CRB
# - how: shell
# script: dnf config-manager --set-enabled beaker-Buildroot
discover:
how: fmf
filter: "tag:gate-build"
execute:
how: tmt

View File

@ -1,5 +0,0 @@
summary: CI Gating Plan
discover:
how: fmf
execute:
how: tmt

View File

@ -1,2 +0,0 @@
The test's Makefiles are not used in Fedora CI infrastructure. But are kept here
for backward compatibility with traditional beakerlib test harness in RHEL.

View File

@ -1,67 +0,0 @@
# Copyright (c) 2009, 2012 Red Hat, Inc. All rights reserved.
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
# Rebuild kernel, install it, reboot, and check if we're running the correct
# kernel. Tailored specificaly for binutils buildroot testing process.
#
# Author: Milos Prchlik <mprchlik@redhat.com>
#
# Based on gcc/Sanity/rebuild-kernel by:
# Author: Michal Nowak <mnowak@redhat.com>
# Author: Marek Polacek <polacek@redhat.com>
export TEST=/tools/binutils/Sanity/rebuild-kernel-and-reboot
export TESTVERSION=1.0
BUILT_FILES=
FILES=$(METADATA) runtest.sh Makefile PURPOSE
.PHONY: all install download clean
run: $(FILES) build
./runtest.sh
build: $(BUILT_FILES)
chmod a+x ./runtest.sh
clean:
rm -f *~ $(BUILT_FILES)
include /usr/share/rhts/lib/rhts-make.include
$(METADATA): Makefile
@echo "Owner: Milos Prchlik <mprchlik@redhat.com>" > $(METADATA)
@echo "Name: $(TEST)" >> $(METADATA)
@echo "Path: $(TEST_DIR)" >> $(METADATA)
@echo "TestVersion: $(TESTVERSION)" >> $(METADATA)
@echo "Description: Rebuild kernel, install it, reboot, and check if we're running the correct kernel" >> $(METADATA)
@echo "Type: Sanity" >> $(METADATA)
@echo "TestTime: 20h" >> $(METADATA)
@echo "RunFor: binutils" >> $(METADATA)
@echo "Requires: binutils" >> $(METADATA)
@echo "Requires: gcc" >> $(METADATA)
@echo "Requires: yum-utils rng-tools" >> $(METADATA)
@echo "Requires: rpm-build" >> $(METADATA)
@echo "Requires: newt-devel python-devel perl-ExtUtils-Embed unifdef elfutils-libelf-devel" >> $(METADATA)
@echo "Requires: elfutils-devel pciutils-devel" >> $(METADATA)
@echo "Requires: wget hmaccalc binutils-devel glibc-static texinfo gdb ecj gcc-java ppl-devel cloog-ppl-devel graphviz gmp-devel mpfr-devel" >> $(METADATA)
@echo "Requires: xmlto asciidoc net-tools pesign" >> $(METADATA)
@echo "License: GPLv3+" >> $(METADATA)
rhts-lint $(METADATA)

View File

@ -1,20 +0,0 @@
Rebuild kernel, install it, reboot, and check if we're running the correct
kernel. Tailored specificaly for binutils buildroot testing process.
Author: Milos Prchlik <mprchlik@redhat.com>
Based on gcc/Sanity/rebuild-kernel by:
Author: Michal Nowak <mnowak@redhat.com>
Author: Marek Polacek <polacek@redhat.com>
Using MACHINE_SET__STRONG host-filter for this test, by setting the "hardware"
section in TCMS properly. That has several effects:
- kernel rebuild is very resoruce-intensive task, and having more powerful
boxes for it is simply good,
- this task will get its own boxes, not clobbered by additional kernel packages
that are usually installed by other tasks in the same run. E.g.kernel-debuginfo,
when installed, will conflict with freshly build kernel packages. This should
workaround such situations,
- this tasks reboots its boxes - should such reboot break something, don't ruin
the whole run by it, right?

View File

@ -1,17 +1,9 @@
summary: Rebuild kernel, install it, reboot, and check if we're running the correct
kernel
summary: Check whether newly built kernel boots correctly
description: |
Rebuild kernel, install it, reboot, and check if we're running the correct
kernel. Tailored specificaly for binutils buildroot testing process.
Author: Milos Prchlik <mprchlik@redhat.com>
Based on gcc/Sanity/rebuild-kernel by:
Author: Michal Nowak <mnowak@redhat.com>
Author: Marek Polacek <polacek@redhat.com>
Using MACHINE_SET__STRONG host-filter for this test, by setting the "hardware"
section in TCMS properly. That has several effects:
A strong machine is recommended:
- kernel rebuild is very resoruce-intensive task, and having more powerful
boxes for it is simply good,
@ -21,13 +13,19 @@ description: |
workaround such situations,
- this tasks reboots its boxes - should such reboot break something, don't ruin
the whole run by it, right?
Based on gcc/Sanity/rebuild-kernel by:
Author: Michal Nowak <mnowak@redhat.com>
Author: Marek Polacek <polacek@redhat.com>
contact:
- Milos Prchlik <mprchlik@redhat.com>
component:
- binutils
tag:
- gate-build
test: ./runtest.sh
framework: beakerlib
recommend:
require:
- binutils
- gcc
- yum-utils
@ -47,9 +45,6 @@ recommend:
- texinfo
- gdb
- ecj
- gcc-java
- ppl-devel
- cloog-ppl-devel
- graphviz
- gmp-devel
- mpfr-devel
@ -58,5 +53,15 @@ recommend:
- net-tools
- pesign
duration: 20h
extra-summary: /tools/binutils/Sanity/rebuild-kernel-and-reboot
extra-task: /tools/binutils/Sanity/rebuild-kernel-and-reboot
#adjust:
# - because: "C9S-specific requirements"
# when: distro == c9s
# require+:
# - ppl-devel
# - cloog-ppl-devel
# - gcc-java
# Dropping TCMS integration, these links should not be needed anymore.
# extra-summary: /tools/binutils/Sanity/rebuild-kernel-and-reboot
# extra-task: /tools/binutils/Sanity/rebuild-kernel-and-reboot

View File

@ -37,11 +37,11 @@ case "$JUST_BUILD" in
;;
esac
LD="${LD:-$(which ld)}"
GCC="${GCC:-$(which gcc)}"
LD="${LD:-/usr/bin/ld}"
GCC="${GCC:-/usr/bin/gcc}"
PACKAGE="${PACKAGE:-$(rpm --qf '%{name}\n' -qf $(which $LD) | head -1)}"
GCC_PACKAGE="${GCC_PACKAGE:-$(rpm --qf '%{name}\n' -qf $(which $GCC) | head -1)}"
PACKAGE="${PACKAGE:-$(rpm --qf '%{name}\n' -qf "$(which "$LD")" | head -1)}"
GCC_PACKAGE="${GCC_PACKAGE:-$(rpm --qf '%{name}\n' -qf "$(which "$GCC")" | head -1)}"
PACKAGES="${PACKAGES:-$PACKAGE}"
@ -61,11 +61,12 @@ unset ARCH
rlJournalStart
rlPhaseStartSetup
rlLogInfo "PACKAGES=$PACKAGES"
rlLogInfo "KERNEL=$KERNEL"
rlLogInfo "REQUIRES=$REQUIRES"
rlLogInfo "JUST_BUILD=$JUST_BUILD"
rlLogInfo "COLLECTIONS=$COLLECTIONS"
rlLogInfo "SKIP_COLLECTION_METAPACKAGE_CHECK=$SKIP_COLLECTION_METAPACKAGE_CHECK"
rlLogInfo "LD=$LD"
rlLogInfo "GCC=$GCC"
rlLogInfo "KERNEL=$KERNEL"
rlLogInfo "JUST_BUILD=$JUST_BUILD"
# We'll use a lot of disk space (tens of GBs at least). The root FS
# often is the most beefy one in CI environemnts. Let's use it but also
@ -78,23 +79,8 @@ rlJournalStart
rlLogInfo " $line"
done
# We optionally need to skip checking for the presence of the metapackage
# because that would pull in all the dependent toolset subrpms. We do not
# always want that, especially in CI.
_COLLECTIONS="$COLLECTIONS"
if ! test -z $SKIP_COLLECTION_METAPACKAGE_CHECK; then
for c in $SKIP_COLLECTION_METAPACKAGE_CHECK; do
rlLogInfo "ignoring metapackage check for collection $c"
export COLLECTIONS=$(shopt -s extglob && echo ${COLLECTIONS//$c/})
done
fi
rlLogInfo "(without skipped) COLLECTIONS=$COLLECTIONS"
rlAssertRpm --all
export COLLECTIONS="$_COLLECTIONS"
rlRun "AFTER_REBOOT=no"
if [ "$REBOOTCOUNT" != "0" ]; then
@ -164,7 +150,7 @@ rlJournalStart
if [ "$(arch)" = "s390x" ]; then
rlRun "grubby --info=ALL"
rlLogInfo "Default kernel is $(grubby --default-kernel), index $(grubby --default-index)"
rlRun "KERNEL_FILE=$(ls -1 /boot/vmlinuz-*${RPM_BUILD_ID}.$(arch))"
rlRun "KERNEL_FILE=$(ls -1 /boot/vmlinuz-*${RPM_BUILD_ID}."$(arch)")"
rlRun "ls -al $KERNEL_FILE"
rlRun "grubby --set-default $KERNEL_FILE"
rlLogInfo "Default kernel is $(grubby --default-kernel), index $(grubby --default-index)"

View File

@ -0,0 +1,20 @@
summary: Rebuild wget
description: ""
contact:
- Milos Prchlik <mprchlik@redhat.com>
component:
- binutils
tag:
- gate-build
test: ./runtest.sh
framework: beakerlib
require:
- binutils
- wget
- rpm-build
- gcc
duration: 1h
# Dropping TCMS integration, these links should not be needed anymore.
# extra-summary: /tools/binutils/Sanity/rebuild-wget
# extra-task: /tools/binutils/Sanity/rebuild-wget

View File

@ -0,0 +1,70 @@
#!/bin/bash
# vim: dict+=/usr/share/beakerlib/dictionary.vim cpt=.,w,b,u,t,i,k
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#
# runtest.sh of /tools/binutils/Sanity/rebuild-wget
# Description: Rebuild wget
# Author: Milos Prchlik <mprchlik@redhat.com>
#
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#
# Copyright (c) 2015 Red Hat, Inc.
#
# This copyrighted material is made available to anyone wishing
# to use, modify, copy, or redistribute it subject to the terms
# and conditions of the GNU General Public License version 2.
#
# This program is distributed in the hope that it will be
# useful, but WITHOUT ANY WARRANTY; without even the implied
# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
# PURPOSE. See the GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public
# License along with this program; if not, write to the Free
# Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
# Boston, MA 02110-1301, USA.
#
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Include Beaker environment
. /usr/share/beakerlib/beakerlib.sh || exit 1
LD="${LD:-/usr/bin/ld}"
GCC="${GCC:-/usr/bin/gcc}"
BINUTILS_PACKAGE="$(rpm --qf '%{name}\n' -qf "$(which "$LD")" | head -1)"
GCC_PACKAGE="$(rpm --qf '%{name}\n' -qf "$(which "$GCC")" | head -1)"
PACKAGES="${PACKAGES:-$BINUTILS_PACKAGE}"
REQUIRES="${REQUIRES:-$GCC_PACKAGE wget}"
rlJournalStart
rlPhaseStartSetup
rlLogInfo "PACKAGES=$PACKAGES"
rlLogInfo "REQUIRES=$REQUIRES"
rlLogInfo "COLLECTIONS=$COLLECTIONS"
rlLogInfo "LD=$LD"
rlLogInfo "GCC=$GCC"
rlAssertRpm --all
rlRun "TmpDir=\$(mktemp -d)" 0 "Creating tmp directory"
rlRun "pushd $TmpDir"
rlFetchSrcForInstalled "wget"
rlRun "SRPM=`find . -name 'wget-*.src.rpm'`"
rlRun "rpm -ivh --define='_topdir $TmpDir' $SRPM"
rlRun "SPECFILE=`find $TmpDir/ -name '*.spec'`"
rlRun "dnf builddep -y $SPECFILE"
rlPhaseEnd
rlPhaseStartTest
rlRun "rpmbuild --define='_topdir $TmpDir' -bb --clean $SPECFILE"
rlPhaseEnd
rlPhaseStartCleanup
rlRun "popd"
rlRun "rm -r $TmpDir" 0 "Removing tmp directory"
rlPhaseEnd
rlJournalPrintText
rlJournalEnd

View File

@ -0,0 +1,29 @@
summary: The test rebuilds binutils.src.rpm and runs the suite.
description: ''
contact:
- Milos Prchlik <mprchlik@redhat.com>
component:
- binutils
tag:
- gate-build
test: ./runtest.sh
framework: beakerlib
require:
- binutils
- texinfo
- glibc-static
- dejagnu
- bison
- flex
- zlib-static
- sharutils
- yum-utils
- libstdc++-static
- rpm-build
- gcc-c++
- annobin
duration: 3h
# Dropping TCMS integration, these links should not be needed anymore.
# extra-summary: /tools/binutils/Sanity/testsuite
# extra-task: /tools/binutils/Sanity/testsuite

136
tests/Sanity/testsuite/runtest.sh Executable file
View File

@ -0,0 +1,136 @@
#!/bin/bash
# vim: dict+=/usr/share/beakerlib/dictionary.vim cpt=.,w,b,u,t,i,k
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#
# runtest.sh of /tools/binutils/Sanity/testsuite
#
# Description: The test rebuilds binutils.src.rpm and runs the suite.
# The test is based on /tools/binutils/testsuite, but it had to be
# rewritten.
#
# Author: Michael Petlan <mpetlan@redhat.com>
#
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#
# Copyright (c) 2015 Red Hat, Inc.
#
# This copyrighted material is made available to anyone wishing
# to use, modify, copy, or redistribute it subject to the terms
# and conditions of the GNU General Public License version 2.
#
# This program is distributed in the hope that it will be
# useful, but WITHOUT ANY WARRANTY; without even the implied
# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
# PURPOSE. See the GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public
# License along with this program; if not, write to the Free
# Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
# Boston, MA 02110-1301, USA.
#
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Include Beaker environment
. /usr/share/beakerlib/beakerlib.sh || exit 1
LD="${LD:-/usr/bin/ld}"
GCC="${GCC:-/usr/bin/gcc}"
PACKAGE="${PACKAGE:-$(rpm --qf '%{name}\n' -qf "$(which "$LD")" | head -1)}"
GCC_PACKAGE="${GCC_PACKAGE:-$(rpm --qf '%{name}\n' -qf "$(which "$GCC")" | head -1)}"
PACKAGES="${PACKAGES:-$PACKAGE}"
REQUIRES="${REQUIRES:-$GCC_PACKAGE}"
rlJournalStart
rlPhaseStartSetup
ARCH="$(rlGetPrimaryArch)"
if [ "$ARCH" = "x86_64" ]; then
BUILD_ARCHS="aarch64 ppc64le s390x x86_64"
else
BUILD_ARCHS="$ARCH"
fi
rlLogInfo "PACKAGES=$PACKAGES"
rlLogInfo "REQUIRES=$REQUIRES"
rlLogInfo "COLLECTIONS=$COLLECTIONS"
rlLogInfo "LD=$LD"
rlLogInfo "GCC=$GCC"
rlLogInfo "ARCH=$ARCH"
rlLogInfo "BUILD_ARCHS=$BUILD_ARCHS"
rlAssertRpm --all
# temp dir
rlRun "TESTDIR=$PWD"
rlRun "TmpDir=\$(mktemp -d)" 0 "Creating tmp directory"
rlRun "LOGDIR=$TmpDir/LOGS"
rlRun "BUILDDIR=$(rpm --define='_topdir $TmpDir' --eval=%_builddir)"
rlRun "CURRENT_BUILD=${BUILDDIR}/binutils-$(rpmquery "$PACKAGE" --queryformat=%{VERSION})"
rlRun "mkdir $LOGDIR"
rlRun "pushd $TmpDir"
# fetch'n'build the source
rlFetchSrcForInstalled "$PACKAGE"
rlRun "yum-builddep -y *.src.rpm"
rlRun "SRPM=$(find . -name 'binutils-*.src.rpm')"
rlRun "rpm -ivh --define='_topdir $TmpDir' $SRPM"
rlRun "SPECFILE=$(find "$TmpDir/" -name '*.spec')"
rlRun "dnf builddep -y $SPECFILE"
rlPhaseEnd
rlPhaseStartSetup "Rebuild binutils"
rlRun "rpmbuild -bc --define='_topdir $TmpDir' $SPECFILE"
for build_arch in $BUILD_ARCHS; do
rlRun "cp $CURRENT_BUILD/build-${build_arch}-redhat-linux/binutils/binutils.log $LOGDIR/binutils-$build_arch.log"
rlRun "cp $CURRENT_BUILD/build-${build_arch}-redhat-linux/binutils/binutils.sum $LOGDIR/binutils-$build_arch.sum"
if [ "$build_arch" = "$ARCH" ]; then
rlRun "cp $CURRENT_BUILD/build-${build_arch}-redhat-linux/ld/ld.log $LOGDIR/ld-$build_arch.log"
rlRun "cp $CURRENT_BUILD/build-${build_arch}-redhat-linux/ld/ld.sum $LOGDIR/ld-$build_arch.sum"
fi
rlRun "cp $CURRENT_BUILD/build-${build_arch}-redhat-linux/gas/testsuite/gas.log $LOGDIR/gas-$build_arch.log"
rlRun "cp $CURRENT_BUILD/build-${build_arch}-redhat-linux/gas/testsuite/gas.sum $LOGDIR/gas-$build_arch.sum"
done
rlPhaseEnd
for TOOL in binutils ld gas; do
for build_arch in $BUILD_ARCHS; do
if [[ "$TOOL" == "ld" && "$build_arch" != "$ARCH" ]]; then
continue
fi
tool_label="$TOOL / $build_arch"
tool_log="$LOGDIR/$TOOL-$build_arch"
rlPhaseStartTest "$tool_label"
rlLogInfo "$tool_label Summary"
rlLogInfo "$(grep -A 50 '=== .* Summary ===' "$tool_log.sum")"
rlRun "awk \"/=== $TOOL tests ===/,/=== $TOOL Summary ===/\" $tool_log.sum > $tool_log.tests" 0,1 "Save results of all tests"
rlRun "grep -E '^FAIL: ' $tool_log.tests | sort > $tool_log.failed" 0,1 "Save failures"
rlRun "grep '# of unexpected failures' $tool_log.sum" 0,1 "Checking number of unexpected failures"
rlLogInfo "$(sort < "$tool_log.failed")"
rlLogInfo "$(grep -E '^[A-Z]+:' "$tool_log.tests" | sort)"
rlPhaseEnd
done
done
rlPhaseStartCleanup
rlRun "tar czf $TmpDir/logs.tgz $LOGDIR/*.sum $LOGDIR/*.log $LOGDIR/*.tests $LOGDIR/*.failed"
rlRun "tar czf $TmpDir/buildroot.tgz $CURRENT_BUILD/"
rlFileSubmit logs.tar.gz
rlFileSubmit buildroot.tgz
rlRun "popd" # $TmpDir
rlRun "rm -r $TmpDir" 0 "Removing tmp directory"
rlPhaseEnd
rlJournalPrintText
rlJournalEnd