test-driver

Show log
Commit

Hash : 0c81b43f
Author :
Date : 2011-07-28T12:25:22

test harness: use new `.trs' files to hold test metadata

With this change, the test harness will keep test metadata in
dedicated `.trs' files, instead of having them embedded into the
`.log' files.  This allows for easier forward-compatibility and
extension of test metadata, and for more flexibility in the
format of the `.log' files.  Note that this change makes the
`:end-metadata:' field obsolete.

* doc/automake.texi (Parallel Test Harness, Log files generation
and test results recording): Document the new APIs and behaviour;
some related minor rewordings and fixlets.
* NEWS: Update.
* automake.in (handle_tests): When bringing in the content of
`check2.am', substitute %BASE% with the basename of the `.log'
file being created by a rule.  Add the generated `.trs' files
to the list of files to be cleaned by "make mostlyclean".
* lib/am/check.am (am__test_driver_flags): Rename ...
(am__common_driver_flags): ... to this, and remove the flags
`--test-name' and `--log-file' from it: they are now define in
the proper rules in `check2.am'.
(am__TEST_BASES): New internal variable, holding the names of
the tests, with any registered extension removed.
(am__stealth_MAKE): New internal variable, can be used instead of
$(MAKE) in recipes requiring a recursive call to make, but which
are not intended to be executed by "make -n".
(.log.trs): New suffix rule, to recover from deletion of `.trs'
files.
($(TEST_SUITE_LOG)): Almost completely rewritten to follow the
new API of "test logs in `.log' files, test metadata in `.trs'
files".  It goes to some length to work correctly in face of
unreadable or missing `.log' and `.trs' files, and to error out
with proper error messages when this is not possible.
[%?PARALLEL_TESTS%] (check-TESTS): Also remove relevant "stale"
`.trs' files (in addition to `.log files) before remaking the
$(TEST_SUITE_LOG).
(recheck, recheck-html): Look for the `:recheck:' field in the
`.trs' files, not in the `.log' files.
* lib/am/check2.am (?GENERIC?%EXT%.log, ?!GENERIC?%OBJ%): Adjust
the call to the test driver, in particularly passing the new
option `--trs-file'.
[%am__EXEEXT%] (?GENERIC?%EXT%$(EXEEXT).log): Likewise.
* lib/tap-driver ($USAGE): Adjust the help screen.
(Getopt::Long::GetOptions): Handle the `--trs-file' option,
through the use of ...
($trs_file): ... this new global variable.
(finish): Write metadata for the test run to `$trs_file' rather
then to `$log_file', through the use of ...
(write_test_results): ... this new function.
* lib/test-driver (print_usage): Update the help screen.
(Option parsing): Handle the `--trs-file' option, through the
use of ...
($resfile): ... this new global variable.
(Main code): Write metadata for the test run to `$trsfile' rather
than to `$logfile'.
Minor related adjustments to comments.
* tests/.gitignore: Ignore `*.trs' files.
* tests/parallel-tests-unreadable-log.test: Moved ...
* tests/parallel-tests-unreadable.test: ... to this, and extended
to also check the semantics for unreadable `.trs' files.
* tests/test-driver-end-metadata.test: Deleted as obsolete.
* tests/test-driver-metadata-no-leading-space.test: Likewise.
* tests/test-driver-global-log.test: Renamed ...
* tests/test-metadata-global-log.test: ... to this, and modified
as to verify the new APIs and semantics.
* tests/test-driver-recheck: Renamed ...
* tests/test-metadata-recheck.test: ... to this, and modified
likewise.
* tests/parallel-tests-once.test: New test.
* tests/parallel-tests-make-n.test: Likewise.
* test-metadata-results.test: Likewise.
* test-missing.test: Likewise.
* test-missing2.test: Likewise.
* test-trs-basic.test: Likewise.
* test-trs-recover.test: Likewise.
* test-trs-recover2.test: Likewise.
* tests/Makefile.am (TESTS): Update.

Download

lib/test-driver

#! /bin/sh
# test-driver - basic driver script for the `parallel-tests' mode.

scriptversion=2011-08-01.21; # UTC

# Copyright (C) 2011 Free Software Foundation, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2, or (at your option)
# any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

# As a special exception to the GNU General Public License, if you
# distribute this file as part of a program that contains a
# configuration script generated by Autoconf, you may include it under
# the same distribution terms that you use for the rest of that program.

# This file is maintained in Automake, please report
# bugs to <bug-automake@gnu.org> or send patches to
# <automake-patches@gnu.org>.

# Make unconditional expansion of undefined variables an error.  This
# helps a lot in preventing typo-related bugs.
set -u

fatal ()
{
  echo "$0: fatal: $*" >&2
  exit 1
}

usage_error ()
{
  echo "$0: $*" >&2
  print_usage >&2
  exit 2
}

print_usage ()
{
  cat <<END
Usage:
  test-driver --test-name=NAME --log-file=PATH --trs-file=PATH
              [--expect-failure={yes|no}] [--color-tests={yes|no}]
              [--enable-hard-errors={yes|no}] [--] TEST-SCRIPT
The \`--test-name' and \`--log-file' options are mandatory.
END
}

# Restructured Text section.
rst_section () { sed 'p;s/./=/g;p;g'; }

# TODO: better error handling in option parsing (in particular, ensure
# TODO: $logfile, $trsfile and $test_name are defined).
test_name= # Used for reporting.
logfile=   # Where to save the output of the test script.
trsfile=   # Where to save the result(s) the test script.
expect_failure=no
color_tests=no
enable_hard_errors=yes
while test $# -gt 0; do
  case $1 in
  --help) print_usage; exit $?;;
  --version) echo "test-driver $scriptversion"; exit $?;;
  --test-name) test_name=$2; shift;;
  --log-file) logfile=$2; shift;;
  --trs-file) trsfile=$2; shift;;
  --color-tests) color_tests=$2; shift;;
  --expect-failure) expect_failure=$2; shift;;
  --enable-hard-errors) enable_hard_errors=$2; shift;;
  --) shift; break;;
  -*) usage_error "invalid option: '$1'";;
  esac
  shift
done

if test $color_tests = yes; then
  red='[0;31m' # Red.
  grn='[0;32m' # Green.
  lgn='[1;32m' # Light green.
  blu='[1;34m' # Blue.
  mgn='[0;35m' # Magenta.
  std='[m'     # No color.
else
  red= grn= lgn= blu= mgn= std=
fi

tmpfile=$logfile-t
do_exit='rm -f $tmpfile; (exit $st); exit $st'
trap "st=129; $do_exit" 1
trap "st=130; $do_exit" 2
trap "st=141; $do_exit" 13
trap "st=143; $do_exit" 15
rm -f $tmpfile

# Test script is run here.
"$@" >$tmpfile 2>&1
estatus=$?
if test $enable_hard_errors = no && test $estatus -eq 99; then
  estatus=1
fi

case $estatus:$expect_failure in
  0:yes) col=$red res=XPASS recheck=yes gcopy=yes;;
  0:*)   col=$grn res=PASS  recheck=no  gcopy=no;;
  77:*)  col=$blu res=SKIP  recheck=no  gcopy=yes;;
  99:*)  col=$mgn res=ERROR recheck=yes gcopy=yes;;
  *:yes) col=$lgn res=XFAIL recheck=no  gcopy=yes;;
  *:*)   col=$red res=FAIL  recheck=yes gcopy=yes;;
esac

# Report outcome to console.
echo "${col}${res}${std}: $test_name"

# Register the test result, and other relevant metadata.
echo ":test-result: $res (exit status: $estatus)" > $trsfile
echo ":recheck: $recheck" >> $trsfile
echo ":copy-in-global-log: $gcopy" >> $trsfile

# Now write log file.
{
  echo "$res: $test_name (exit: $estatus)" | rst_section
  echo
  cat $tmpfile
} > $logfile
rm -f $tmpfile

# Local Variables:
# mode: shell-script
# sh-indentation: 2
# eval: (add-hook 'write-file-hooks 'time-stamp)
# time-stamp-start: "scriptversion="
# time-stamp-format: "%:y-%02m-%02d.%02H"
# time-stamp-time-zone: "UTC"
# time-stamp-end: "; # UTC"
# End:

Download

kc3-lang/automake/lib/test-driver

Commit

lib/test-driver