check.am

Show log
Commit

Hash : 0c81b43f
Author :
Date : 2011-07-28T12:25:22

test harness: use new `.trs' files to hold test metadata

With this change, the test harness will keep test metadata in
dedicated `.trs' files, instead of having them embedded into the
`.log' files.  This allows for easier forward-compatibility and
extension of test metadata, and for more flexibility in the
format of the `.log' files.  Note that this change makes the
`:end-metadata:' field obsolete.

* doc/automake.texi (Parallel Test Harness, Log files generation
and test results recording): Document the new APIs and behaviour;
some related minor rewordings and fixlets.
* NEWS: Update.
* automake.in (handle_tests): When bringing in the content of
`check2.am', substitute %BASE% with the basename of the `.log'
file being created by a rule.  Add the generated `.trs' files
to the list of files to be cleaned by "make mostlyclean".
* lib/am/check.am (am__test_driver_flags): Rename ...
(am__common_driver_flags): ... to this, and remove the flags
`--test-name' and `--log-file' from it: they are now define in
the proper rules in `check2.am'.
(am__TEST_BASES): New internal variable, holding the names of
the tests, with any registered extension removed.
(am__stealth_MAKE): New internal variable, can be used instead of
$(MAKE) in recipes requiring a recursive call to make, but which
are not intended to be executed by "make -n".
(.log.trs): New suffix rule, to recover from deletion of `.trs'
files.
($(TEST_SUITE_LOG)): Almost completely rewritten to follow the
new API of "test logs in `.log' files, test metadata in `.trs'
files".  It goes to some length to work correctly in face of
unreadable or missing `.log' and `.trs' files, and to error out
with proper error messages when this is not possible.
[%?PARALLEL_TESTS%] (check-TESTS): Also remove relevant "stale"
`.trs' files (in addition to `.log files) before remaking the
$(TEST_SUITE_LOG).
(recheck, recheck-html): Look for the `:recheck:' field in the
`.trs' files, not in the `.log' files.
* lib/am/check2.am (?GENERIC?%EXT%.log, ?!GENERIC?%OBJ%): Adjust
the call to the test driver, in particularly passing the new
option `--trs-file'.
[%am__EXEEXT%] (?GENERIC?%EXT%$(EXEEXT).log): Likewise.
* lib/tap-driver ($USAGE): Adjust the help screen.
(Getopt::Long::GetOptions): Handle the `--trs-file' option,
through the use of ...
($trs_file): ... this new global variable.
(finish): Write metadata for the test run to `$trs_file' rather
then to `$log_file', through the use of ...
(write_test_results): ... this new function.
* lib/test-driver (print_usage): Update the help screen.
(Option parsing): Handle the `--trs-file' option, through the
use of ...
($resfile): ... this new global variable.
(Main code): Write metadata for the test run to `$trsfile' rather
than to `$logfile'.
Minor related adjustments to comments.
* tests/.gitignore: Ignore `*.trs' files.
* tests/parallel-tests-unreadable-log.test: Moved ...
* tests/parallel-tests-unreadable.test: ... to this, and extended
to also check the semantics for unreadable `.trs' files.
* tests/test-driver-end-metadata.test: Deleted as obsolete.
* tests/test-driver-metadata-no-leading-space.test: Likewise.
* tests/test-driver-global-log.test: Renamed ...
* tests/test-metadata-global-log.test: ... to this, and modified
as to verify the new APIs and semantics.
* tests/test-driver-recheck: Renamed ...
* tests/test-metadata-recheck.test: ... to this, and modified
likewise.
* tests/parallel-tests-once.test: New test.
* tests/parallel-tests-make-n.test: Likewise.
* test-metadata-results.test: Likewise.
* test-missing.test: Likewise.
* test-missing2.test: Likewise.
* test-trs-basic.test: Likewise.
* test-trs-recover.test: Likewise.
* test-trs-recover2.test: Likewise.
* tests/Makefile.am (TESTS): Update.

Download

lib/am/check.am

## automake - create Makefile.in from Makefile.am
## Copyright (C) 2001, 2003, 2006, 2007, 2008, 2009, 2010, 2011 Free
## Software Foundation, Inc.

## This program is free software; you can redistribute it and/or modify
## it under the terms of the GNU General Public License as published by
## the Free Software Foundation; either version 2, or (at your option)
## any later version.

## This program is distributed in the hope that it will be useful,
## but WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
## GNU General Public License for more details.

## You should have received a copy of the GNU General Public License
## along with this program.  If not, see <http://www.gnu.org/licenses/>.

am__tty_colors_dummy = \
  mgn= red= grn= lgn= blu= brg= std=; \
  am__color_tests=no
if %?COLOR%
# If stdout is a non-dumb tty, use colors.  If test -t is not supported,
# then this fails; a conservative approach.  Of course do not redirect
# stdout here, just stderr.
am__tty_colors = \
$(am__tty_colors_dummy); \
test "X$(AM_COLOR_TESTS)" != Xno \
&& test "X$$TERM" != Xdumb \
&& { test "X$(AM_COLOR_TESTS)" = Xalways || test -t 1 2>/dev/null; } \
&& { \
  am__color_tests=yes; \
  red='[0;31m'; \
  grn='[0;32m'; \
  lgn='[1;32m'; \
  blu='[1;34m'; \
  mgn='[0;35m'; \
  brg='[1m'; \
  std='[m'; \
}
else !%?COLOR%
am__tty_colors = $(am__tty_colors_dummy)
endif !%?COLOR%

.PHONY: check-TESTS

if %?PARALLEL_TESTS%

include inst-vars.am

## New parallel test driver.
##
## This code is adapted from check.mk which was originally
## written at EPITA/LRDE, further developed at Gostai, then made
## its way from GNU coreutils to end up, largely rewritten, in
## Automake.
##
## It provides special support for "unit tests", that is to say,
## tests that (once run) no longer need to be re-compiled and
## re-run at each "make check", unless their sources changed.  To
## enable unit-test supports, set RECHECK_LOGS to empty.  In such a
## setting, that heavily relies on correct dependencies, its users may
## prefer to define EXTRA_PROGRAMS instead of check_PROGRAMS, because
## it allows intertwined compilation and execution of the tests.
## Sometimes this helps catching errors earlier (you don't have to
## wait for all the tests to be compiled).
##
## Define TEST_SUITE_LOG to be the name of the global log to create.
## Define TEST_LOGS to the set of logs to include in it.  It defaults
## to $(TESTS), with `.test' and `@EXEEXT@' removed, and `'.log'
## appended.
##
## In addition to the magic "exit 77 means SKIP" feature (which was
## imported from automake), there is a magic "exit 99 means ERROR" feature
## which is useful if you need to issue a hard error no matter whether the
## test is XFAIL or not.  You can disable this feature by setting the
## variable DISABLE_HARD_ERRORS to a nonempty value.

# Restructured Text title.
am__rst_title = sed 's/.*/   &   /;h;s/./=/g;p;x;p;g;p;s/.*//'

# Solaris 10 'make', and several other traditional 'make' implementations,
# pass "-e" to $(SHELL), and POSIX 2008 even requires this.  Work around it
# by disabling -e (using the XSI extension "set +e") if it's set.
am__sh_e_setup = case $$- in *e*) set +e;; esac

# Default flags passed to test drivers.
am__common_driver_flags = \
  --color-tests "$$am__color_tests" \
  --enable-hard-errors "$$am__enable_hard_errors" \
  --expect-failure "$$am__expect_failure"

# To be inserted before the command running the test.  Creates the
# directory for the log if needed.  Stores in $dir the directory
# containing $f, in $tst the test, in $log the log.  Executes the
# developer- defined test setup AM_TESTS_ENVIRONMENT (if any), and
# passes TESTS_ENVIRONMENT.  Set up options for the wrapper that
# will run the test scripts (or their associated LOG_COMPILER, if
# thy have one).
am__check_pre =						\
$(am__sh_e_setup);					\
$(am__vpath_adj_setup) $(am__vpath_adj)			\
$(am__tty_colors);					\
srcdir=$(srcdir); export srcdir;			\
am__odir=`echo "./$@" | sed 's|/[^/]*$$||'`;		\
test "x$$am__odir" = x. || $(MKDIR_P) "$$am__odir" || exit $$?;	\
if test -f "./$$f"; then dir=./;			\
elif test -f "$$f"; then dir=;				\
else dir="$(srcdir)/"; fi;				\
tst=$$dir$$f; log='$@'; 				\
if test -n '$(DISABLE_HARD_ERRORS)'; then		\
  am__enable_hard_errors=no; 				\
else							\
  am__enable_hard_errors=yes; 				\
fi; 							\
## The use of $dir below is required to account for VPATH
## rewriting done by Sun make.
case " $(XFAIL_TESTS) " in				\
  *[\ \	]$$f[\ \	]* | *[\ \	]$$dir$$f[\ \	]*) \
    am__expect_failure=yes;;				\
  *)							\
    am__expect_failure=no;;				\
esac; 							\
$(AM_TESTS_ENVIRONMENT) $(TESTS_ENVIRONMENT)

# The names of the tests, with any registered extension removed.  Or
# equivalently, the names of the test logs, with the `.log' extension
# renoved.  This honours runtime overriding of TESTS and TEST_LOGS.
am__TEST_BASES = $(TEST_LOGS:.log=)

# This can be used instead of $(MAKE) in recipes requiring a recursive call
# to make, but which are not intended to be executed by "make -n".  See the
# GNU make manual for more details.
am__stealth_MAKE = $(MAKE)

# Recover from deleted `.trs' file; this should ensure that
# "rm -f foo.log; make foo.trs" re-run `foo.test', and re-create
# both `foo.log' and `foo.trs'.
.log.trs:
	rm -f $< $@ && $(am__stealth_MAKE) $(AM_MAKEFLAGS) $<

$(TEST_SUITE_LOG): $(TEST_LOGS)
	@$(am__sh_e_setup); $(am__tty_colors); \
	bases='$(am__TEST_BASES)'; \
	ws='[ 	]'; \
## We need to ensures that all the required `.trs' and `.log' files will
## be present and readable.  The direct dependencies of $(TEST_SUITE_LOG)
## only ensure that all the `.log' files exists; they don't ensure that
## the `.log' files are readable, and worse, they don't ensure that the
## `.trs' files even exist.
	redo_bases=`for b in $$bases; do \
	              test -f $$b.trs && test -r $$b.trs \
	                && test -f $$b.log && test -r $$b.log \
		        || echo $$b; \
	            done`; \
	if test -n "$$redo_bases"; then \
## Uh-oh, either some `.log' files were unreadable, or some `.trs' files
## were missing (or unreadable).  We need to re-run the corresponding
## tests in order to re-create them.
	  redo_logs=`for i in $$redo_bases; do echo $$i.log; done`; \
	  redo_results=`for i in $$redo_bases; do echo $$i.trs; done`; \
	  rm -f $$redo_logs && rm -f $$redo_results \
	    && $(am__stealth_MAKE) $(AM_MAKEFLAGS) $$redo_logs; \
	fi; \
## Sanity check: each unreadable or non-existent test result file should
## has been properly remade at this point, as should the corresponding log
## file.
	st=0; \
	for b in $$redo_bases; do \
	  for e in trs log; do \
	    if test ! -f $$b.$$e || test ! -r $$b.$$e; then \
	      echo "fatal: making $@: failed to create $$b.$$e" >&2; \
	      st=1; \
	    fi; \
	  done; \
	done; \
	test $$st -eq 0 || exit 1; \
## List of test result files.
	results=`for b in $$bases; do echo $$b.trs; done`; \
	test -n "$$results" || results=/dev/null; \
## Prepare data for the test suite summary.  These do not take into account
## unreadable test results, but they'll be appropriately updated later if
## needed.
	all=`  grep "^$$ws*:test-result:"           $$results | wc -l`; \
	pass=` grep "^$$ws*:test-result:$$ws*PASS"  $$results | wc -l`; \
	fail=` grep "^$$ws*:test-result:$$ws*FAIL"  $$results | wc -l`; \
	skip=` grep "^$$ws*:test-result:$$ws*SKIP"  $$results | wc -l`; \
	xfail=`grep "^$$ws*:test-result:$$ws*XFAIL" $$results | wc -l`; \
	xpass=`grep "^$$ws*:test-result:$$ws*XPASS" $$results | wc -l`; \
	error=`grep "^$$ws*:test-result:$$ws*ERROR" $$results | wc -l`; \
## Whether the testsuite was successful or not.
	if test `expr $$fail + $$xpass + $$error` -eq 0; then \
	  success=true; \
	else \
	  success=false; \
	fi; \
## Make $br a line of exactly 76 `=' characters, that will be used to
## enclose the testsuite summary report when displayed on the console.
	br='==================='; br=$$br$$br$$br$$br; \
## When writing the test summary to the console, we want to color a line
## reporting the count of some result *only* if at least one test
## experienced such a result.  This function is handy in this regard.
	result_count () \
	{ \
	    if test x"$$1" = x"--maybe-color"; then \
	      maybe_colorize=yes; \
	    elif test x"$$1" = x"--no-color"; then \
	      maybe_colorize=no; \
	    else \
	      echo "$@: invalid 'result_count' usage" >&2; exit 4; \
	    fi; \
	    shift; \
	    desc=$$1 count=$$2; \
	    if test $$maybe_colorize = yes && test $$count -gt 0; then \
	      color_start=$$3 color_end=$$std; \
	    else \
	      color_start= color_end=; \
	    fi; \
	    echo "$${color_start}# $$desc $$count$${color_end}"; \
	}; \
## A shell function that creates the testsuite summary.  We need it
## because we have to create *two* summaries, one for test-suite.log,
## and a possibly-colorized one for console output.
	create_testsuite_report () \
	{ \
	  result_count $$1 "TOTAL:" $$all   "$$brg"; \
	  result_count $$1 "PASS: " $$pass  "$$grn"; \
	  result_count $$1 "SKIP: " $$skip  "$$blu"; \
	  result_count $$1 "XFAIL:" $$xfail "$$lgn"; \
	  result_count $$1 "FAIL: " $$fail  "$$red"; \
	  result_count $$1 "XPASS:" $$xpass "$$red"; \
	  result_count $$1 "ERROR:" $$error "$$mgn"; \
	}; \
## Write "global" testsuite log.
	{								\
	  echo "$(PACKAGE_STRING): $(subdir)/$(TEST_SUITE_LOG)" |	\
	    $(am__rst_title);						\
	  create_testsuite_report --no-color;				\
	  echo;								\
	  echo ".. contents:: :depth: 2";				\
	  echo;								\
	  for b in $$bases; do						\
## FIXME: one fork per test -- this is horrendously inefficient!
	    if grep "^$$ws*:copy-in-global-log:$$ws*no$$ws*$$" $$b.trs 	\
	         >/dev/null; then :; 					\
## If we cannot read the .log file of a test ...
	    elif test ! -r $$b.log; then				\
## ... print an error message to the console ...
	      echo "ERROR: cannot read $$b.log" >&2;			\
## ... and a warning in the `test-suite.log' file.
	      echo && echo "WARNING: could not read $$b.log!";		\
	    else							\
	      echo && cat $$b.log;					\
	    fi;								\
	  done;								\
	} >$(TEST_SUITE_LOG).tmp;					\
	mv $(TEST_SUITE_LOG).tmp $(TEST_SUITE_LOG);			\
## Emit the test summary on the console.
	if $$success; then						\
	  col="$$grn";							\
	 else								\
	  col="$$red";							\
	  test x"$$VERBOSE" = x || cat $(TEST_SUITE_LOG);		\
	fi;								\
## Multi line coloring is problematic with "less -R", so we really need
## to color each line individually.
	echo "$${col}$$br$${std}"; 					\
	echo "$${col}Testsuite summary for $(PACKAGE_STRING)$${std}";	\
	echo "$${col}$$br$${std}"; 					\
## This is expected to go to the console, so it might have to be colorized.
	create_testsuite_report --maybe-color;				\
	echo "$$col$$br$$std";						\
	if $$success; then :; else					\
	  echo "$${col}See $(subdir)/$(TEST_SUITE_LOG)$${std}";		\
	  if test -n "$(PACKAGE_BUGREPORT)"; then			\
	    echo "$${col}Please report to $(PACKAGE_BUGREPORT)$${std}";	\
	  fi;								\
	  echo "$$col$$br$$std";					\
	fi;								\
## Be sure to exit with the proper exit status.
	$$success

RECHECK_LOGS = $(TEST_LOGS)

# Run all the tests.
check-TESTS:
## Expand $(RECHECK_LOGS) only once, to avoid exceeding line length limits.
	@list='$(RECHECK_LOGS)'; test -z "$$list" || rm -f $$list
	@list='$(RECHECK_LOGS:.log=.trs)'; test -z "$$list" || rm -f $$list
## We always have to remove TEST_SUITE_LOG, to ensure its rule is run
## in any case even in lazy mode: otherwise, if no test needs rerunning,
## or a prior run plus reruns all happen within the same timestamp
## (can happen with a prior `make TESTS=<subset>'),
## then we get no log output.
## OTOH, this means that, in the rule for `$(TEST_SUITE_LOG)', we
## cannot use `$?' to compute the set of lazily rerun tests, lest
## we rely on .PHONY to work portably.
##
## Trailing whitespace in `TESTS = foo.test $(empty)' causes GNU make
## 3.80 to erroneously expand $(TESTS_LOGS) to `foo.log .log'.
## Work around this bug.
	@test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG)
	@list='$(TEST_LOGS)';						\
	list=`for f in $$list; do					\
	  test .log = $$f || echo $$f;					\
	done | tr '\012\015' '  '`;					\
	$(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_LOG) TEST_LOGS="$$list"

AM_RECURSIVE_TARGETS += check

## -------------- ##
## Produce HTML.  ##
## -------------- ##

.log.html:
	@list='$(RST2HTML) $$RST2HTML rst2html rst2html.py';		\
	for r2h in $$list; do						\
	  if ($$r2h --version) >/dev/null 2>&1; then			\
	    R2H=$$r2h;							\
	  fi;								\
	done;								\
	if test -z "$$R2H"; then					\
	  echo >&2 "cannot find rst2html, cannot create $@";		\
	  exit 2;							\
	fi;								\
	$$R2H $< >$@.tmp
	@mv $@.tmp $@

# Be sure to run check first, and then to convert the result.
# Beware of concurrent executions.  Run "check" not "check-TESTS", as
# check-SCRIPTS and other dependencies are rebuilt by the former only.
# And expect check to fail.
check-html:
	@if $(MAKE) $(AM_MAKEFLAGS) check; then			\
	  rv=0; else rv=$$?;					\
	fi;							\
	$(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_HTML) || exit 4;	\
	exit $$rv

.PHONY: check-html
.MAKE: check-html

AM_RECURSIVE_TARGETS += check-html

## -------------------- ##
## Rechecking failures. ##
## -------------------- ##

## Rerun all tests that experienced an error or an unexpected failure.
recheck recheck-html:
	@ws='[ 	]';							\
	target=`echo $@ | sed 's,^re,,'`;				\
	bases='$(am__TEST_BASES)';					\
	list=`for b in $$bases; do 					\
## Skip tests that haven't been run, but recover gracefully from deleted
## `.trs' files.
	        test ! -f $$b.trs && test ! -f $$b.log && continue;	\
## FIXME: one fork per test -- this is horrendously inefficient!
	        grep "^$$ws*:recheck:$$ws*no$$ws*$$" $$b.trs 		\
		  >/dev/null 2>&1 || echo $$b.log; 			\
	      done | tr '\012\015' '  '`;				\
## This apparently useless munging helps to avoid a nasty bug (a
## segmentation fault!) on Solaris XPG4 make.
	list=`echo "$$list" | sed 's/ *$$//'`;				\
	$(MAKE) $(AM_MAKEFLAGS) $$target AM_MAKEFLAGS='$(AM_MAKEFLAGS) TEST_LOGS="'"$$list"'"'

.PHONY: recheck recheck-html
.MAKE: recheck recheck-html

AM_RECURSIVE_TARGETS += recheck recheck-html

else !%?PARALLEL_TESTS%

check-TESTS: $(TESTS)
	@failed=0; all=0; xfail=0; xpass=0; skip=0; \
	srcdir=$(srcdir); export srcdir; \
## Make sure Solaris VPATH-expands all members of this list, even
## the first and the last one; thus the spaces around $(TESTS)
	list=' $(TESTS) '; \
	$(am__tty_colors); \
	if test -n "$$list"; then \
	  for tst in $$list; do \
	    if test -f ./$$tst; then dir=./; \
## Note: Solaris 2.7 seems to expand TESTS using VPATH.  That's
## why we also try `dir='
	    elif test -f $$tst; then dir=; \
	    else dir="$(srcdir)/"; fi; \
	    if $(TESTS_ENVIRONMENT) $${dir}$$tst $(AM_TESTS_FD_REDIRECT); then \
## Success
	      all=`expr $$all + 1`; \
	      case " $(XFAIL_TESTS) " in \
	      *[\ \	]$$tst[\ \	]*) \
		xpass=`expr $$xpass + 1`; \
		failed=`expr $$failed + 1`; \
		col=$$red; res=XPASS; \
	      ;; \
	      *) \
		col=$$grn; res=PASS; \
	      ;; \
	      esac; \
	    elif test $$? -ne 77; then \
## Failure
	      all=`expr $$all + 1`; \
	      case " $(XFAIL_TESTS) " in \
	      *[\ \	]$$tst[\ \	]*) \
		xfail=`expr $$xfail + 1`; \
		col=$$lgn; res=XFAIL; \
	      ;; \
	      *) \
		failed=`expr $$failed + 1`; \
		col=$$red; res=FAIL; \
	      ;; \
	      esac; \
	    else \
## Skipped
	      skip=`expr $$skip + 1`; \
	      col=$$blu; res=SKIP; \
	    fi; \
	    echo "$${col}$$res$${std}: $$tst"; \
	  done; \
## Prepare the banner
	  if test "$$all" -eq 1; then \
	    tests="test"; \
	    All=""; \
	  else \
	    tests="tests"; \
	    All="All "; \
	  fi; \
	  if test "$$failed" -eq 0; then \
	    if test "$$xfail" -eq 0; then \
	      banner="$$All$$all $$tests passed"; \
	    else \
	      if test "$$xfail" -eq 1; then failures=failure; else failures=failures; fi; \
	      banner="$$All$$all $$tests behaved as expected ($$xfail expected $$failures)"; \
	    fi; \
	  else \
	    if test "$$xpass" -eq 0; then \
	      banner="$$failed of $$all $$tests failed"; \
	    else \
	      if test "$$xpass" -eq 1; then passes=pass; else passes=passes; fi; \
	      banner="$$failed of $$all $$tests did not behave as expected ($$xpass unexpected $$passes)"; \
	    fi; \
	  fi; \
## DASHES should contain the largest line of the banner.
	  dashes="$$banner"; \
	  skipped=""; \
	  if test "$$skip" -ne 0; then \
	    if test "$$skip" -eq 1; then \
	      skipped="($$skip test was not run)"; \
	    else \
	      skipped="($$skip tests were not run)"; \
	    fi; \
	    test `echo "$$skipped" | wc -c` -le `echo "$$banner" | wc -c` || \
	      dashes="$$skipped"; \
	  fi; \
	  report=""; \
	  if test "$$failed" -ne 0 && test -n "$(PACKAGE_BUGREPORT)"; then \
	    report="Please report to $(PACKAGE_BUGREPORT)"; \
	    test `echo "$$report" | wc -c` -le `echo "$$banner" | wc -c` || \
	      dashes="$$report"; \
	  fi; \
	  dashes=`echo "$$dashes" | sed s/./=/g`; \
	  if test "$$failed" -eq 0; then \
	    col="$$grn"; \
	  else \
	    col="$$red"; \
	  fi; \
## Multi line coloring is problematic with "less -R", so we really need
## to color each line individually.
	  echo "$${col}$$dashes$${std}"; \
	  echo "$${col}$$banner$${std}"; \
	  test -z "$$skipped" || echo "$${col}$$skipped$${std}"; \
	  test -z "$$report" || echo "$${col}$$report$${std}"; \
	  echo "$${col}$$dashes$${std}"; \
	  test "$$failed" -eq 0; \
	else :; fi

endif !%?PARALLEL_TESTS%

Download

kc3-lang/automake/lib/am/check.am

Commit

lib/am/check.am