Hash :
0c81b43f
Author :
Date :
2011-07-28T12:25:22
test harness: use new `.trs' files to hold test metadata With this change, the test harness will keep test metadata in dedicated `.trs' files, instead of having them embedded into the `.log' files. This allows for easier forward-compatibility and extension of test metadata, and for more flexibility in the format of the `.log' files. Note that this change makes the `:end-metadata:' field obsolete. * doc/automake.texi (Parallel Test Harness, Log files generation and test results recording): Document the new APIs and behaviour; some related minor rewordings and fixlets. * NEWS: Update. * automake.in (handle_tests): When bringing in the content of `check2.am', substitute %BASE% with the basename of the `.log' file being created by a rule. Add the generated `.trs' files to the list of files to be cleaned by "make mostlyclean". * lib/am/check.am (am__test_driver_flags): Rename ... (am__common_driver_flags): ... to this, and remove the flags `--test-name' and `--log-file' from it: they are now define in the proper rules in `check2.am'. (am__TEST_BASES): New internal variable, holding the names of the tests, with any registered extension removed. (am__stealth_MAKE): New internal variable, can be used instead of $(MAKE) in recipes requiring a recursive call to make, but which are not intended to be executed by "make -n". (.log.trs): New suffix rule, to recover from deletion of `.trs' files. ($(TEST_SUITE_LOG)): Almost completely rewritten to follow the new API of "test logs in `.log' files, test metadata in `.trs' files". It goes to some length to work correctly in face of unreadable or missing `.log' and `.trs' files, and to error out with proper error messages when this is not possible. [%?PARALLEL_TESTS%] (check-TESTS): Also remove relevant "stale" `.trs' files (in addition to `.log files) before remaking the $(TEST_SUITE_LOG). (recheck, recheck-html): Look for the `:recheck:' field in the `.trs' files, not in the `.log' files. * lib/am/check2.am (?GENERIC?%EXT%.log, ?!GENERIC?%OBJ%): Adjust the call to the test driver, in particularly passing the new option `--trs-file'. [%am__EXEEXT%] (?GENERIC?%EXT%$(EXEEXT).log): Likewise. * lib/tap-driver ($USAGE): Adjust the help screen. (Getopt::Long::GetOptions): Handle the `--trs-file' option, through the use of ... ($trs_file): ... this new global variable. (finish): Write metadata for the test run to `$trs_file' rather then to `$log_file', through the use of ... (write_test_results): ... this new function. * lib/test-driver (print_usage): Update the help screen. (Option parsing): Handle the `--trs-file' option, through the use of ... ($resfile): ... this new global variable. (Main code): Write metadata for the test run to `$trsfile' rather than to `$logfile'. Minor related adjustments to comments. * tests/.gitignore: Ignore `*.trs' files. * tests/parallel-tests-unreadable-log.test: Moved ... * tests/parallel-tests-unreadable.test: ... to this, and extended to also check the semantics for unreadable `.trs' files. * tests/test-driver-end-metadata.test: Deleted as obsolete. * tests/test-driver-metadata-no-leading-space.test: Likewise. * tests/test-driver-global-log.test: Renamed ... * tests/test-metadata-global-log.test: ... to this, and modified as to verify the new APIs and semantics. * tests/test-driver-recheck: Renamed ... * tests/test-metadata-recheck.test: ... to this, and modified likewise. * tests/parallel-tests-once.test: New test. * tests/parallel-tests-make-n.test: Likewise. * test-metadata-results.test: Likewise. * test-missing.test: Likewise. * test-missing2.test: Likewise. * test-trs-basic.test: Likewise. * test-trs-recover.test: Likewise. * test-trs-recover2.test: Likewise. * tests/Makefile.am (TESTS): Update.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133
#! /bin/sh
# Copyright (C) 2011 Free Software Foundation, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2, or (at your option)
# any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# Check parallel harness features:
# - recovery from unreadable `.trs' files, in various scenarios
parallel_tests=yes
. ./defs || Exit 1
cat >> configure.in <<END
AC_OUTPUT
END
cat > Makefile.am << 'END'
TESTS = foo.test bar.test
TEST_LOG_COMPILER = true
END
: > foo.test
: > bar.test
TEST_STATUS=0; export TEST_STATUS
$ACLOCAL
$AUTOCONF
$AUTOMAKE -a
./configure
: > t
chmod a-r t
test ! -r t || Exit 77
rm -f t
: Create the required log files.
$MAKE check
test -f foo.trs
test -f bar.trs
: Recreate with a sweeping "make check".
chmod a-r bar.trs
$MAKE check
test -f foo.trs
test -r foo.trs
test -f bar.trs
test -r bar.trs
: Again, this time with one .trs file missing and the other
: one unreadable.
rm -f foo.trs
chmod a-r bar.trs
$MAKE check
test -f foo.trs
test -r foo.trs
test -f bar.trs
test -r bar.trs
: Again, but using "make recheck" this time.
rm -f foo.trs
chmod a-r bar.trs
$MAKE recheck >stdout || { cat stdout; Exit 1; }
cat stdout
test -f foo.trs
test -r foo.trs
test -f bar.trs
test -r bar.trs
grep '^PASS: foo\.test' stdout
grep '^PASS: bar\.test' stdout
: More complex interactions with "make recheck" are OK.
chmod a-r bar.log bar.trs
$MAKE recheck >stdout || { cat stdout; Exit 1; }
cat stdout
test -f bar.trs
test -r bar.trs
grep '^PASS: bar\.test' stdout
grep 'foo\.test' stdout && Exit 1
count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0
: Recreate by remaking the global test log.
chmod a-r foo.trs
rm -f test-suite.log
$MAKE test-suite.log >stdout || { cat stdout; Exit 1; }
cat stdout
test -f foo.trs
test -r foo.trs
grep '^PASS: foo\.test' stdout
grep 'bar\.test' stdout && Exit 1
# Also test that have only run before should be counted in the final
# testsuite summary.
grep '^# TOTAL: *2$' stdout
: Setup for the next check.
: > baz.test
sed 's/^TESTS =.*/& baz.test/' Makefile > t
diff t Makefile && Exit 99
mv -f t Makefile
$MAKE check
test -f foo.trs
test -f bar.trs
test -f baz.trs
: Interactions with lazy test reruns are OK.
chmod a-r foo.trs
$sleep
touch stamp
$sleep
touch bar.test
RECHECK_LOGS= $MAKE -e check >stdout || { cat stdout; Exit 1; }
cat stdout
test -r foo.trs
is_newest bar.trs bar.test
grep '^PASS: foo\.test' stdout
grep '^PASS: bar\.test' stdout
grep 'baz\.test' stdout && Exit 1
# Also test that have only run before should be counted in the final
# testsuite summary.
grep '^# TOTAL: *3$' stdout
: