Hash :
0c81b43f
Author :
Date :
2011-07-28T12:25:22
test harness: use new `.trs' files to hold test metadata With this change, the test harness will keep test metadata in dedicated `.trs' files, instead of having them embedded into the `.log' files. This allows for easier forward-compatibility and extension of test metadata, and for more flexibility in the format of the `.log' files. Note that this change makes the `:end-metadata:' field obsolete. * doc/automake.texi (Parallel Test Harness, Log files generation and test results recording): Document the new APIs and behaviour; some related minor rewordings and fixlets. * NEWS: Update. * automake.in (handle_tests): When bringing in the content of `check2.am', substitute %BASE% with the basename of the `.log' file being created by a rule. Add the generated `.trs' files to the list of files to be cleaned by "make mostlyclean". * lib/am/check.am (am__test_driver_flags): Rename ... (am__common_driver_flags): ... to this, and remove the flags `--test-name' and `--log-file' from it: they are now define in the proper rules in `check2.am'. (am__TEST_BASES): New internal variable, holding the names of the tests, with any registered extension removed. (am__stealth_MAKE): New internal variable, can be used instead of $(MAKE) in recipes requiring a recursive call to make, but which are not intended to be executed by "make -n". (.log.trs): New suffix rule, to recover from deletion of `.trs' files. ($(TEST_SUITE_LOG)): Almost completely rewritten to follow the new API of "test logs in `.log' files, test metadata in `.trs' files". It goes to some length to work correctly in face of unreadable or missing `.log' and `.trs' files, and to error out with proper error messages when this is not possible. [%?PARALLEL_TESTS%] (check-TESTS): Also remove relevant "stale" `.trs' files (in addition to `.log files) before remaking the $(TEST_SUITE_LOG). (recheck, recheck-html): Look for the `:recheck:' field in the `.trs' files, not in the `.log' files. * lib/am/check2.am (?GENERIC?%EXT%.log, ?!GENERIC?%OBJ%): Adjust the call to the test driver, in particularly passing the new option `--trs-file'. [%am__EXEEXT%] (?GENERIC?%EXT%$(EXEEXT).log): Likewise. * lib/tap-driver ($USAGE): Adjust the help screen. (Getopt::Long::GetOptions): Handle the `--trs-file' option, through the use of ... ($trs_file): ... this new global variable. (finish): Write metadata for the test run to `$trs_file' rather then to `$log_file', through the use of ... (write_test_results): ... this new function. * lib/test-driver (print_usage): Update the help screen. (Option parsing): Handle the `--trs-file' option, through the use of ... ($resfile): ... this new global variable. (Main code): Write metadata for the test run to `$trsfile' rather than to `$logfile'. Minor related adjustments to comments. * tests/.gitignore: Ignore `*.trs' files. * tests/parallel-tests-unreadable-log.test: Moved ... * tests/parallel-tests-unreadable.test: ... to this, and extended to also check the semantics for unreadable `.trs' files. * tests/test-driver-end-metadata.test: Deleted as obsolete. * tests/test-driver-metadata-no-leading-space.test: Likewise. * tests/test-driver-global-log.test: Renamed ... * tests/test-metadata-global-log.test: ... to this, and modified as to verify the new APIs and semantics. * tests/test-driver-recheck: Renamed ... * tests/test-metadata-recheck.test: ... to this, and modified likewise. * tests/parallel-tests-once.test: New test. * tests/parallel-tests-make-n.test: Likewise. * test-metadata-results.test: Likewise. * test-missing.test: Likewise. * test-missing2.test: Likewise. * test-trs-basic.test: Likewise. * test-trs-recover.test: Likewise. * test-trs-recover2.test: Likewise. * tests/Makefile.am (TESTS): Update.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166
#! /bin/sh
# Copyright (C) 2011 Free Software Foundation, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2, or (at your option)
# any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# Custom test drivers: "abstract" XFAIL_TESTS support.
parallel_tests=yes
. ./defs || Exit 1
cat >> configure.in <<'END'
AC_SUBST([nihil], [])
AC_SUBST([ac_xfail_tests], ['x5.test x6$(test_suffix)'])
AC_CONFIG_FILES([sub1/Makefile sub2/Makefile])
AC_OUTPUT
END
cat > Makefile.am <<'END'
SUBDIRS = . sub1 sub2
TEST_LOG_DRIVER = $(srcdir)/td
TESTS = pass.test xfail.test
XFAIL_TESTS = xfail.test
END
mkdir sub1 sub2
cat > sub1/Makefile.am <<END
empty =
TEST_LOG_DRIVER = \$(top_srcdir)/td
# XFAIL_TESTS should gracefully handle TAB characters, and multiple
# whitespaces.
XFAIL_TESTS =\$(empty)${tab}x1.test x2.test${tab}x3.test${tab}\
x4.test ${tab} x5.test x6.test${tab}\$(empty)
TESTS = pass.test x1.test x2.test x3.test x4.test x5.test x6.test
END
cat > sub2/Makefile.am <<'END'
AUTOMAKE_OPTIONS = -Wno-portability-recursive
TEST_LOG_DRIVER = $(srcdir)/../td
# XFAIL_TESTS should gracefully AC_SUBST @substitution@ and
# make variables indirections.
an_xfail_test = x1.test
test_suffix = .test
v0 = x3.test
v1 = v
v2 = 0
XFAIL_TESTS = $(an_xfail_test) x2.test @nihil@ x3${test_suffix}
XFAIL_TESTS += $($(v1)$(v2)) x4.test @ac_xfail_tests@
TESTS = pass.test x1.test x2.test x3.test x4.test x5.test x6.test
END
cat > pass.test <<'END'
#!/bin/sh
exit 0
END
cat > xfail.test <<'END'
#!/bin/sh
exit 1
END
chmod a+x pass.test xfail.test
cp pass.test sub1/pass.test
cp pass.test sub2/pass.test
for i in 1 2 3 4 5 6; do
cp xfail.test sub1/x$i.test
cp xfail.test sub2/x$i.test
done
cat > td <<'END'
#! /bin/sh
set -e; set -u
test_name=INVALID
log_file=/dev/null
trs_file=/dev/null
expect_failure=no
while test $# -gt 0; do
case $1 in
--test-name) test_name=$2; shift;;
--expect-failure) expect_failure=$2; shift;;
--log-file) log_file=$2; shift;;
--trs-file) trs_file=$2; shift;;
# Ignored.
--color-tests) shift;;
--enable-hard-errors) shift;;
# Explicitly terminate option list.
--) shift; break;;
# Shouldn't happen
*) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
esac
shift
done
st=0
"$@" || st=$?
case $st,$expect_failure in
0,no)
echo "PASS: $test_name" | tee "$log_file"
echo ":test-result: PASS" > "$trs_file"
;;
1,no)
echo "FAIL: $test_name" | tee "$log_file"
echo ":test-result: FAIL" > "$trs_file"
;;
0,yes)
echo "XPASS: $test_name" | tee "$log_file"
echo ":test-result: XPASS" > "$trs_file"
;;
1,yes)
echo "XFAIL: $test_name" | tee "$log_file"
echo ":test-result: XFAIL" > "$trs_file"
;;
*)
echo "INTERNAL ERROR" >&2
exit 99
;;
esac
END
chmod a+x td
$ACLOCAL
$AUTOCONF
$AUTOMAKE
./configure
$MAKE check >stdout || { cat stdout; Exit 1; }
cat stdout
test `grep -c '^PASS:' stdout` -eq 3
test `grep -c '^XFAIL:' stdout` -eq 13
for dir in sub1 sub2; do
cd $dir
cp pass.test x1.test
cp x2.test pass.test
$MAKE check >stdout && { cat stdout; Exit 1; }
cat stdout
test "`cat pass.trs`" = ":test-result: FAIL"
test "`cat x1.trs`" = ":test-result: XPASS"
test "`cat x2.trs`" = ":test-result: XFAIL"
grep '^FAIL: pass\.test$' stdout
grep '^XPASS: x1\.test$' stdout
grep '^XFAIL: x2\.test$' stdout
count_test_results total=7 pass=0 xpass=1 fail=1 xfail=5 skip=0 error=0
cd ..
done
: