Hash :
eaac33fe
Author :
Date :
2011-07-01T11:12:31
parallel-tests: simplify testsuite summary Prefer a more deterministic, "tabular" format for the testsuite summary, always listing the numbers of passed, failed, xfailed, xpassed, skipped and errored tests, even when these numbers are zero. This simplify the logic of testsuite summary creation, makes it more easily machine-parseable, and will probably allow for easier addition of new kinds of test results in the future. * lib/am/check.am (am__tty_colors_dummy): New make variable, to reduce code duplication. Extracted from previous versions of $(am__tty_colors), and extended by defining two new variables `$mgn' and `$brg'. [%?COLOR%, %!?COLOR%] (am__tty_colors): Use that new variable. (am__text_box): Delete, is not needed anymore. ($(TEST_SUITE_LOG)): Rewrite associated rules to implement the new testsuite summary format. * NEWS: Update. * tests/check10.test: Don't run with the parallel-tests harness too, that makes no sense anymore. * tests/color.test: Update and adjust. * tests/color2.test: Likewise. * tests/parallel-tests.test: Likewise. * tests/parallel-tests3.test: Likewise. * tests/parallel-tests6.test: Likewise. * tests/parallel-tests9.test: Likewise. * tests/parallel-tests-unreadable-log.test: Likewise. * tests/parallel-tests-empty-testlogs.test: Likewise. * tests/parallel-tests-log-override-recheck.test: Likewise. * tests/parallel-tests-no-spurious-summary.test: Likewise. * tests/test-driver-custom-multitest.test: Likewise. * tests/test-driver-end-test-results.test: Likewise. * tests/parallel-tests-no-color-in-log.test: New test. * tests/testsuite-summary-color.test: Likewise. * tests/testsuite-summary-count.test: Likewise. * tests/testsuite-summary-count-many.test: Likewise. * tests/testsuite-summary-reference-log.test: Likewise. * tests/testsuite-summary-checks.sh: New auxiliary script, used by the new tests above. * tests/extract-testsuite-summary: Likewise. * tests/trivial-test-driver: Optimize for speed when there are lots of of tests. * tests/Makefile.am (EXTRA_DIST): Distribute them. (testsuite-summary-color.log, testsuite-summary-count.log): Depend on them. (testsuite-summary-count-many.log): Depend on the auxiliary scripts 'trivial-test-driver' and 'extract-testsuite-summary'. (TESTS): Update.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106
#! /bin/sh
# Copyright (C) 2011 Free Software Foundation, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2, or (at your option)
# any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# Check the testsuite summary with the parallel-tests harness. This
# script is meant to be sourced by other test script, so that it can
# be used to check different scenarios (colorized and non-colorized
# testsuite output, packages with and without bug-report addresses,
# testsuites in subdirectories, ...)
parallel_tests=yes
. ./defs || Exit 1
case $use_colors in
yes)
AM_COLOR_TESTS=always; export AM_COLOR_TESTS
TERM=ansi; export TERM
am_opts='parallel-tests color-tests'
;;
no)
am_opts='parallel-tests'
;;
*)
fatal_ "invalid use_colors='$use_colors'";;
esac
cat > configure.in <<END
AC_INIT([GNU AutoFoo], [7.1], [bug-automake@gnu.org])
AM_INIT_AUTOMAKE([$am_opts])
AC_CONFIG_FILES([Makefile])
AC_OUTPUT
END
cat > Makefile.am <<'END'
TEST_EXTENSIONS = .t
T_LOG_COMPILER = $(SHELL) $(srcdir)/log-compiler
## Will be defined at runtime.
TESTS =
$(TESTS):
END
cat > log-compiler <<'END'
#!/bin/sh
case "$1" in
pass*|*/pass*|xpass*|*/xpass*) exit 0;;
fail*|*/fail*|xfail*|*/xfail*) exit 1;;
skip*|*/skip*) exit 77;;
error*|/error*) exit 99;;
copy*|*/copy*) cat "$1";;
*) exit 99;;
esac
END
# Quite complexish, but allow the tests in client scripts to be written
# in a "data-driven fashion".
do_check ()
{
cat > summary.exp
expect_failure=false
xfail_tests=''
tests="TESTS='$*'"
for t in $*; do
case $t in fail*|xpass*|error*) expect_failure=:;; esac
case $t in xfail*|xpass*) xfail_tests="$xfail_tests $t";; esac
done
test -z "$xfail_tests" || xfail_tests="XFAIL_TESTS='$xfail_tests'"
st=0
eval "env $tests $xfail_tests \$MAKE -e check > stdout || st=\$?"
cat stdout
if $expect_failure; then
test $st -gt 0 || Exit 1
else
test $st -eq 0 || Exit 1
fi
$PERL -w "$testsrcdir"/extract-testsuite-summary stdout > summary.got \
|| fatal_ "cannot extract testsuite summary"
cat summary.exp
cat summary.got
if test $use_colors = yes; then
# Use cmp, not diff, because the files might contain binary data.
compare=cmp
else
compare=diff
fi
$compare summary.exp summary.got || Exit 1
}
br='============================================================================'
$ACLOCAL
$AUTOCONF
$AUTOMAKE --add-missing
: