selftest: Add basic test framework
Now, by execute `make test-run` in tests/, kexec-tools sanity tests
will be performed using VMs. There are currently 3 test cases, for local
kdump, nfs kdump and ssh kdump.
For each test VM, the selftest framework will create a snapshot layer,
do setup as required by the test case, this ensure each test runs in a
clean VM.
This framework will install a custom systemd service that starts when
system have finished booting, and the service will do basic routine
(fetch and set boot counter, etc..), then call the test case which is
installed in /kexec-kdump-test/test.sh in VM.
Each VM will have two serial consoles, one for ordinary console usage,
one for the test communication and log. The test script will watch the
second test console to know the test status.
The test cases are located in tests/scripts/testcases, documents about
the test cases structure will be provided in following commits.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
2020-07-31 07:34:40 +00:00
|
|
|
#!/bin/bash
|
|
|
|
|
2021-06-21 01:18:11 +00:00
|
|
|
_kill_if_valid_pid() {
|
|
|
|
local _pid="$1"
|
|
|
|
if ps -p "$_pid" > /dev/null
|
|
|
|
then
|
|
|
|
kill "$_pid"
|
|
|
|
fi
|
|
|
|
}
|
|
|
|
|
|
|
|
_recursive_kill() {
|
|
|
|
local _pid="$1"
|
|
|
|
local _children _child
|
|
|
|
|
|
|
|
_children=$(pgrep -P "$_pid")
|
|
|
|
if [ -n "$_children" ]; then
|
|
|
|
for _child in $_children
|
|
|
|
do
|
|
|
|
_recursive_kill "$_child"
|
|
|
|
_kill_if_valid_pid "$_child"
|
|
|
|
done
|
|
|
|
fi
|
|
|
|
_kill_if_valid_pid "$_pid"
|
|
|
|
}
|
|
|
|
|
selftest: Add basic test framework
Now, by execute `make test-run` in tests/, kexec-tools sanity tests
will be performed using VMs. There are currently 3 test cases, for local
kdump, nfs kdump and ssh kdump.
For each test VM, the selftest framework will create a snapshot layer,
do setup as required by the test case, this ensure each test runs in a
clean VM.
This framework will install a custom systemd service that starts when
system have finished booting, and the service will do basic routine
(fetch and set boot counter, etc..), then call the test case which is
installed in /kexec-kdump-test/test.sh in VM.
Each VM will have two serial consoles, one for ordinary console usage,
one for the test communication and log. The test script will watch the
second test console to know the test status.
The test cases are located in tests/scripts/testcases, documents about
the test cases structure will be provided in following commits.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
2020-07-31 07:34:40 +00:00
|
|
|
_kill_all_jobs() {
|
|
|
|
local _jobs=$(jobs -r -p)
|
2021-06-21 01:18:11 +00:00
|
|
|
local _job
|
selftest: Add basic test framework
Now, by execute `make test-run` in tests/, kexec-tools sanity tests
will be performed using VMs. There are currently 3 test cases, for local
kdump, nfs kdump and ssh kdump.
For each test VM, the selftest framework will create a snapshot layer,
do setup as required by the test case, this ensure each test runs in a
clean VM.
This framework will install a custom systemd service that starts when
system have finished booting, and the service will do basic routine
(fetch and set boot counter, etc..), then call the test case which is
installed in /kexec-kdump-test/test.sh in VM.
Each VM will have two serial consoles, one for ordinary console usage,
one for the test communication and log. The test script will watch the
second test console to know the test status.
The test cases are located in tests/scripts/testcases, documents about
the test cases structure will be provided in following commits.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
2020-07-31 07:34:40 +00:00
|
|
|
|
2021-06-21 01:18:11 +00:00
|
|
|
if [ -n "$_jobs" ]; then
|
|
|
|
for _job in $_jobs
|
|
|
|
do
|
|
|
|
_recursive_kill "$_job"
|
|
|
|
done
|
|
|
|
fi
|
selftest: Add basic test framework
Now, by execute `make test-run` in tests/, kexec-tools sanity tests
will be performed using VMs. There are currently 3 test cases, for local
kdump, nfs kdump and ssh kdump.
For each test VM, the selftest framework will create a snapshot layer,
do setup as required by the test case, this ensure each test runs in a
clean VM.
This framework will install a custom systemd service that starts when
system have finished booting, and the service will do basic routine
(fetch and set boot counter, etc..), then call the test case which is
installed in /kexec-kdump-test/test.sh in VM.
Each VM will have two serial consoles, one for ordinary console usage,
one for the test communication and log. The test script will watch the
second test console to know the test status.
The test cases are located in tests/scripts/testcases, documents about
the test cases structure will be provided in following commits.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
2020-07-31 07:34:40 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
trap '
|
|
|
|
ret=$?;
|
|
|
|
_kill_all_jobs
|
|
|
|
exit $ret;
|
|
|
|
' EXIT
|
|
|
|
|
|
|
|
trap 'exit 1;' SIGINT
|
|
|
|
|
|
|
|
BASEDIR=$(realpath $(dirname "$0"))
|
|
|
|
. $BASEDIR/test-lib.sh
|
|
|
|
TESTCASEDIR="$BASEDIR/testcases"
|
|
|
|
|
|
|
|
console=0
|
|
|
|
testcases=""
|
|
|
|
|
|
|
|
while [ $# -gt 0 ]; do
|
|
|
|
case $1 in
|
|
|
|
'')
|
|
|
|
break
|
|
|
|
;;
|
|
|
|
--console )
|
|
|
|
console=1
|
|
|
|
;;
|
|
|
|
-*)
|
|
|
|
echo "Invalid option $1"
|
|
|
|
;;
|
|
|
|
*)
|
|
|
|
testcases+=" $1"
|
|
|
|
;;
|
|
|
|
esac
|
|
|
|
shift;
|
|
|
|
done
|
|
|
|
|
|
|
|
if [ -z "$testcases" ]; then
|
|
|
|
echo "==== Starting all tests: ===="
|
|
|
|
testcases=$(ls -1 $TESTCASEDIR)
|
|
|
|
else
|
|
|
|
echo "==== Starting specified tests: ===="
|
|
|
|
fi
|
|
|
|
echo ${testcases##*/}
|
|
|
|
echo
|
|
|
|
|
|
|
|
declare -A results
|
|
|
|
ret=0
|
|
|
|
|
|
|
|
for test_case in $testcases; do
|
|
|
|
echo "======== Running Test Case $test_case ========"
|
|
|
|
results[$test_case]="<Test Skipped>"
|
|
|
|
|
|
|
|
testdir=$TESTCASEDIR/$test_case
|
|
|
|
script_num=$(ls -1 $testdir | wc -l)
|
|
|
|
scripts=$(ls -r -1 $testdir | tr '\n' ' ')
|
|
|
|
test_outputs=""
|
|
|
|
read main_script aux_script <<< "$scripts"
|
|
|
|
|
|
|
|
if [ -z "$main_script" ]; then
|
|
|
|
echo "ERROR: Empty testcase dir $testdir"
|
|
|
|
continue
|
|
|
|
fi
|
|
|
|
|
|
|
|
for script in $scripts; do
|
|
|
|
echo "---- Building image for: $script ----"
|
|
|
|
echo "-------- Output image is: $(get_test_image $testdir/$script)"
|
|
|
|
echo "-------- Building log is: $(get_test_image $testdir/$script).log"
|
|
|
|
|
|
|
|
mkdir -p $(dirname $(get_test_image $testdir/$script))
|
|
|
|
build_test_image $testdir/$script &> $(get_test_image $testdir/$script).log
|
|
|
|
|
|
|
|
if [ $? -ne 0 ]; then
|
|
|
|
echo "Failing building image!"
|
|
|
|
continue 2
|
|
|
|
fi
|
|
|
|
done
|
|
|
|
|
|
|
|
for script in $aux_script; do
|
|
|
|
echo "---- Starting VM: $script ----"
|
|
|
|
|
|
|
|
script="$testdir/$script"
|
|
|
|
echo "-------- Qemu cmdline: $(get_test_qemu_cmd_file $script)"
|
|
|
|
echo "-------- Console log: $(get_test_console_file $script)"
|
|
|
|
echo "-------- Test log: $(get_test_output_file $script)"
|
|
|
|
test_outputs+="$(get_test_output_file $script) "
|
2020-07-31 07:34:42 +00:00
|
|
|
|
selftest: Add basic test framework
Now, by execute `make test-run` in tests/, kexec-tools sanity tests
will be performed using VMs. There are currently 3 test cases, for local
kdump, nfs kdump and ssh kdump.
For each test VM, the selftest framework will create a snapshot layer,
do setup as required by the test case, this ensure each test runs in a
clean VM.
This framework will install a custom systemd service that starts when
system have finished booting, and the service will do basic routine
(fetch and set boot counter, etc..), then call the test case which is
installed in /kexec-kdump-test/test.sh in VM.
Each VM will have two serial consoles, one for ordinary console usage,
one for the test communication and log. The test script will watch the
second test console to know the test status.
The test cases are located in tests/scripts/testcases, documents about
the test cases structure will be provided in following commits.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
2020-07-31 07:34:40 +00:00
|
|
|
rm -f $(get_test_console_file $script)
|
|
|
|
rm -f $(get_test_output_file $script)
|
|
|
|
|
|
|
|
$(run_test_sync $script > $(get_test_console_file $script)) &
|
|
|
|
|
2020-11-18 15:33:06 +00:00
|
|
|
sleep 5
|
selftest: Add basic test framework
Now, by execute `make test-run` in tests/, kexec-tools sanity tests
will be performed using VMs. There are currently 3 test cases, for local
kdump, nfs kdump and ssh kdump.
For each test VM, the selftest framework will create a snapshot layer,
do setup as required by the test case, this ensure each test runs in a
clean VM.
This framework will install a custom systemd service that starts when
system have finished booting, and the service will do basic routine
(fetch and set boot counter, etc..), then call the test case which is
installed in /kexec-kdump-test/test.sh in VM.
Each VM will have two serial consoles, one for ordinary console usage,
one for the test communication and log. The test script will watch the
second test console to know the test status.
The test cases are located in tests/scripts/testcases, documents about
the test cases structure will be provided in following commits.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
2020-07-31 07:34:40 +00:00
|
|
|
done
|
|
|
|
|
|
|
|
script="$main_script"
|
|
|
|
echo "---- Starting test VM: $(basename $script) ----"
|
|
|
|
script="$testdir/$script"
|
|
|
|
|
|
|
|
echo "-------- Qemu cmdline: $(get_test_qemu_cmd_file $script)"
|
|
|
|
echo "-------- Console log: $(get_test_console_file $script)"
|
|
|
|
echo "-------- Test log: $(get_test_output_file $script)"
|
|
|
|
test_outputs+="$(get_test_output_file $script) "
|
2020-07-31 07:34:42 +00:00
|
|
|
|
selftest: Add basic test framework
Now, by execute `make test-run` in tests/, kexec-tools sanity tests
will be performed using VMs. There are currently 3 test cases, for local
kdump, nfs kdump and ssh kdump.
For each test VM, the selftest framework will create a snapshot layer,
do setup as required by the test case, this ensure each test runs in a
clean VM.
This framework will install a custom systemd service that starts when
system have finished booting, and the service will do basic routine
(fetch and set boot counter, etc..), then call the test case which is
installed in /kexec-kdump-test/test.sh in VM.
Each VM will have two serial consoles, one for ordinary console usage,
one for the test communication and log. The test script will watch the
second test console to know the test status.
The test cases are located in tests/scripts/testcases, documents about
the test cases structure will be provided in following commits.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
2020-07-31 07:34:40 +00:00
|
|
|
rm -f $(get_test_console_file $script)
|
|
|
|
rm -f $(get_test_output_file $script)
|
|
|
|
|
|
|
|
if [ $console -eq 1 ]; then
|
|
|
|
run_test_sync $script | tee $(get_test_console_file $script)
|
selftest: run-test.sh: wait for subprocess instead of kill it
When run tests with 2 VMs, for example nfs/ssh kdump tests, client VM will do the
crash and dump, server VM will do vmcore saving and if-vmcore-exists
check.
Previously, when client VM finishes running, run-test.sh will kill the lead background
process, and then check if server VM has outputted "TEST PASSED" or "TEST FAILED" string.
However it didn't wait for server VM to finish. As a result, the server VM's final
outputs are not collected and checked, leaving the test result as "TEST RESULT NOT FOUND"
sometimes.
For example, the following is the pstree status of $(jobs -p) before it
gets killed. We can see the server VM is still running:
run-test.sh,172455 /root/kexec-tools/tests/scripts/run-test.sh --console nfs-early-kdump
└─run-test.sh,172457 /root/kexec-tools/tests/scripts/run-test.sh --console...
└─timeout,172480 --foreground 10m /root/kexec-tools/tests/scripts/run-qemu...
└─qemu-system-x86,172481 -enable-kvm -cpu host -nodefaults...
├─{qemu-system-x86},172489
├─{qemu-system-x86},172492
├─{qemu-system-x86},172493
├─{qemu-system-x86},172628
└─{qemu-system-x86},172629
In this patch, we will wait for $(jobs -p) to finish, in order to get
the complete output of test results.
Signed-off-by: Tao Liu <ltao@redhat.com>
Acked-by: Coiby Xu <coxu@redhat.com>
2021-12-30 03:26:42 +00:00
|
|
|
[ -n "$(jobs -p)" ] && wait $(jobs -p)
|
selftest: Add basic test framework
Now, by execute `make test-run` in tests/, kexec-tools sanity tests
will be performed using VMs. There are currently 3 test cases, for local
kdump, nfs kdump and ssh kdump.
For each test VM, the selftest framework will create a snapshot layer,
do setup as required by the test case, this ensure each test runs in a
clean VM.
This framework will install a custom systemd service that starts when
system have finished booting, and the service will do basic routine
(fetch and set boot counter, etc..), then call the test case which is
installed in /kexec-kdump-test/test.sh in VM.
Each VM will have two serial consoles, one for ordinary console usage,
one for the test communication and log. The test script will watch the
second test console to know the test status.
The test cases are located in tests/scripts/testcases, documents about
the test cases structure will be provided in following commits.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
2020-07-31 07:34:40 +00:00
|
|
|
else
|
|
|
|
$(run_test_sync $script > $(get_test_console_file $script)) &
|
|
|
|
watch_test_outputs $test_outputs
|
|
|
|
fi
|
|
|
|
|
|
|
|
res="$(gather_test_result $test_outputs)"
|
2020-07-31 07:34:42 +00:00
|
|
|
|
selftest: Add basic test framework
Now, by execute `make test-run` in tests/, kexec-tools sanity tests
will be performed using VMs. There are currently 3 test cases, for local
kdump, nfs kdump and ssh kdump.
For each test VM, the selftest framework will create a snapshot layer,
do setup as required by the test case, this ensure each test runs in a
clean VM.
This framework will install a custom systemd service that starts when
system have finished booting, and the service will do basic routine
(fetch and set boot counter, etc..), then call the test case which is
installed in /kexec-kdump-test/test.sh in VM.
Each VM will have two serial consoles, one for ordinary console usage,
one for the test communication and log. The test script will watch the
second test console to know the test status.
The test cases are located in tests/scripts/testcases, documents about
the test cases structure will be provided in following commits.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
2020-07-31 07:34:40 +00:00
|
|
|
[ $? -ne 0 ] && ret=$(expr $ret + 1)
|
|
|
|
results[$test_case]="$res"
|
2021-06-21 01:18:11 +00:00
|
|
|
_kill_all_jobs
|
selftest: Add basic test framework
Now, by execute `make test-run` in tests/, kexec-tools sanity tests
will be performed using VMs. There are currently 3 test cases, for local
kdump, nfs kdump and ssh kdump.
For each test VM, the selftest framework will create a snapshot layer,
do setup as required by the test case, this ensure each test runs in a
clean VM.
This framework will install a custom systemd service that starts when
system have finished booting, and the service will do basic routine
(fetch and set boot counter, etc..), then call the test case which is
installed in /kexec-kdump-test/test.sh in VM.
Each VM will have two serial consoles, one for ordinary console usage,
one for the test communication and log. The test script will watch the
second test console to know the test status.
The test cases are located in tests/scripts/testcases, documents about
the test cases structure will be provided in following commits.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
2020-07-31 07:34:40 +00:00
|
|
|
echo -e "-------- Test finished: $test_case $res --------"
|
2020-07-31 07:34:42 +00:00
|
|
|
for script in $scripts; do
|
|
|
|
script="$testdir/$script"
|
|
|
|
output="$(get_test_output_file $script) "
|
|
|
|
image="$(get_test_image $script)"
|
|
|
|
vmcore="$(sed -n 's/^VMCORE: \(\S*\).*/\1/p' $output)"
|
|
|
|
kernel="$(sed -n 's/^KERNEL VERSION: \(\S*\).*/\1/p' $output)"
|
|
|
|
if [ -n "$vmcore" ]; then
|
|
|
|
echo "You can retrive the verify the vmcore file using following command:"
|
|
|
|
echo "./scripts/copy-from-image.sh \\"
|
|
|
|
echo " $image \\"
|
|
|
|
echo " $vmcore ./"
|
|
|
|
echo "Kernel package verion is: $kernel"
|
|
|
|
fi
|
|
|
|
done
|
selftest: Add basic test framework
Now, by execute `make test-run` in tests/, kexec-tools sanity tests
will be performed using VMs. There are currently 3 test cases, for local
kdump, nfs kdump and ssh kdump.
For each test VM, the selftest framework will create a snapshot layer,
do setup as required by the test case, this ensure each test runs in a
clean VM.
This framework will install a custom systemd service that starts when
system have finished booting, and the service will do basic routine
(fetch and set boot counter, etc..), then call the test case which is
installed in /kexec-kdump-test/test.sh in VM.
Each VM will have two serial consoles, one for ordinary console usage,
one for the test communication and log. The test script will watch the
second test console to know the test status.
The test cases are located in tests/scripts/testcases, documents about
the test cases structure will be provided in following commits.
Signed-off-by: Kairui Song <kasong@redhat.com>
Acked-by: Dave Young <dyoung@redhat.com>
2020-07-31 07:34:40 +00:00
|
|
|
done
|
|
|
|
|
|
|
|
echo "======== Test results ========"
|
|
|
|
for i in ${!results[@]}; do
|
|
|
|
echo "----------------"
|
|
|
|
echo -e "$i:\t\t${results[$i]}"
|
|
|
|
done
|
|
|
|
|
|
|
|
exit $ret
|