|
641a5a4b
|
2012-02-16T10:46:23
|
|
maint: run "make update-copyright"
|
|
|
e00d6c98
|
2011-09-10T21:59:09
|
|
tests defs: more namespace-safeness
($using_tap): Renamed ...
($am_using_tap): ... to this.
($parallel_tests): Renamed ...
($am_parallel_tests): ... to this.
($test_prefer_config_shell): Renamed ...
($am_test_prefer_config_shell): ... to this.
($original_AUTOMAKE): Renamed ...
($am_original_AUTOMAKE): ... to this.
($original_ACLOCAL): Renamed ...
($am_original_ACLOCAL): ... to this.
* tests/defs, tests/defs-static.in, tests/gen-testsuite-part,
tests/README, and all callers: Adjusted.
* tests/Makefile.am (AM_TESTS_ENVIRONMENT): Adjust, and rewrite
to make future adjustments easier.
* tests/defs (funcs_file_): Renamed ...
(am_funcs_file): ... to this.
* Makefile.am (sc_tests_obsolete_variables): New maintainer
check.
(syntax_check_rules): Add it.
|
|
|
dffe171f
|
2011-08-03T15:46:02
|
|
Merge branch 'test-protocols' into tap-testsuite-work
|
|
|
4014b502
|
2011-07-26T14:51:20
|
|
test harness: allow more metadata in log files
This change reworks and improves the parallel test harness to use
more specialized reStructuredText fields in the log files (instead
of relying on specially-placed of "magic lines" and more indirect
semantical formatting); the new fields are the following:
- ":recheck:": tell whether the associated test will have to be
re-run by "make recheck";
- ":copy-in-global-log:": tell whether the content of the log
file should be copied in the "global log" `test-suite.log';
- ":end-metadata:", which inhibits the scanning of the rest of
the log file (for what concerns test metadata).
Also, the special `:test-result:' value "END" has been removed,
superseded by the new `:end-metadata:' field.
* doc/automake.texi (Log files generation and test results
recording): Document the new API and semantics. Remove or fix
some obsolete comments.
* lib/am/check.am ($(TEST_SUITE_LOG), recheck, recheck-html):
Adjust comments and code.
* lib/tap-driver (finish): Adjust, with the help of ...
(must_recheck, copy_in_global_log): ... these new functions.
* lib/test-driver (Main code): Adjust, with the help of ...
($recheck, $gcopy): ... these new variables.
* tests/trivial-test-driver: Update to obey the new APIs.
* tests/test-driver-recheck.test: Likewise.
* tests/test-driver-global-log.test: Likewise.
* tests/tap-passthrough.test: Relax the test, avoiding to check
what is written in `test-suite.log'; such check has been moved ...
* tests/tap-global-log.test: ... in this new test, and extended.
* tests/test-driver-metadata-no-leading-space.test: New test.
* tests/test-driver-end-test-results.test: Removed, it checked
the old APIs; superseded by ...
* tests/test-driver-end-metadata.test: ... this new test.
* tests/tap-log.test: Improve syncing with ...
* tests/test-log.test: ... this new test.
* tests/parallel-tests.test: Remove some duplication w.r.t. this
last new test. Updated heading comments.
* tests/Makefile.am (TESTS): Update.
|
|
|
1bb5974d
|
2011-07-04T18:12:07
|
|
test defs: new auxiliary function 'count_test_results'
* tests/defs (count_test_results): New function.
* tests/check11.test: Use it.
* tests/test-driver-custom-multitest.test: Likewise.
* tests/test-driver-custom-multitest-recheck.test: Likewise.
* tests/test-driver-custom-multitest-recheck2.test: Likewise.
* tests/parallel-tests-log-override-recheck.test: Likewise.
* tests/parallel-tests-log-override-recheck.test: Likewise.
* tests/parallel-tests-no-spurious-summary.test: Likewise, and
slightly improve debugging output.
* tests/parallel-tests.test: Make use of `count_test_results'.
Also, make grepping of "make check" output slightly stricter
* tests/parallel-tests9.test: Likewise.
* tests/parallel-tests-log-override-2.test: Likewise, and throw
in a small optimization.
|
|
|
eaac33fe
|
2011-07-01T11:12:31
|
|
parallel-tests: simplify testsuite summary
Prefer a more deterministic, "tabular" format for the testsuite
summary, always listing the numbers of passed, failed, xfailed,
xpassed, skipped and errored tests, even when these numbers are
zero. This simplify the logic of testsuite summary creation,
makes it more easily machine-parseable, and will probably allow
for easier addition of new kinds of test results in the future.
* lib/am/check.am (am__tty_colors_dummy): New make variable, to
reduce code duplication. Extracted from previous versions of
$(am__tty_colors), and extended by defining two new variables
`$mgn' and `$brg'.
[%?COLOR%, %!?COLOR%] (am__tty_colors): Use that new variable.
(am__text_box): Delete, is not needed anymore.
($(TEST_SUITE_LOG)): Rewrite associated rules to implement the
new testsuite summary format.
* NEWS: Update.
* tests/check10.test: Don't run with the parallel-tests harness
too, that makes no sense anymore.
* tests/color.test: Update and adjust.
* tests/color2.test: Likewise.
* tests/parallel-tests.test: Likewise.
* tests/parallel-tests3.test: Likewise.
* tests/parallel-tests6.test: Likewise.
* tests/parallel-tests9.test: Likewise.
* tests/parallel-tests-unreadable-log.test: Likewise.
* tests/parallel-tests-empty-testlogs.test: Likewise.
* tests/parallel-tests-log-override-recheck.test: Likewise.
* tests/parallel-tests-no-spurious-summary.test: Likewise.
* tests/test-driver-custom-multitest.test: Likewise.
* tests/test-driver-end-test-results.test: Likewise.
* tests/parallel-tests-no-color-in-log.test: New test.
* tests/testsuite-summary-color.test: Likewise.
* tests/testsuite-summary-count.test: Likewise.
* tests/testsuite-summary-count-many.test: Likewise.
* tests/testsuite-summary-reference-log.test: Likewise.
* tests/testsuite-summary-checks.sh: New auxiliary script, used
by the new tests above.
* tests/extract-testsuite-summary: Likewise.
* tests/trivial-test-driver: Optimize for speed when there are
lots of of tests.
* tests/Makefile.am (EXTRA_DIST): Distribute them.
(testsuite-summary-color.log, testsuite-summary-count.log): Depend
on them.
(testsuite-summary-count-many.log): Depend on the auxiliary scripts
'trivial-test-driver' and 'extract-testsuite-summary'.
(TESTS): Update.
|
|
|
15400bca
|
2011-06-30T14:55:43
|
|
parallel-tests: new recognized test result 'ERROR'
* lib/am/check.am ($(TEST_SUITE_LOG)): Recognize a new test result
`ERROR'. Use it when encountering unreadable test logs (previously
a simple `FAIL' was used in this situations).
* lib/test-driver: Set the global test result to `ERROR' when the
test exit status is 99. When doing colorized output, color `ERROR'
results in magenta.
* doc/automake.texi (Log files generation and test results
recording): Update by also listing `ERROR' among the list of valid
`:test-results:' arguments.
* NEWS: Update.
* tests/trivial-test-driver: Update.
* tests/parallel-tests.test: Likewise.
* tests/parallel-tests-harderror.test: Likewise.
* tests/parallel-tests-no-spurious-summary.test: Likewise.
* tests/test-driver-global-log.test: Likewise.
* tests/test-driver-recheck.test: Likewise.
* tests/test-driver-custom-multitest-recheck.test: Likewise.
* tests/test-driver-custom-multitest-recheck2.test: Likewise.
* tests/test-driver-custom-multitest.test: Likewise.
* tests/test-driver-custom-no-html.test: Likewise.
* tests/test-driver-end-test-results.test: Likewise.
* tests/color.test: Likewise. Also, make stricter, and also test
from VPATH.
* tests/color2.test: Likewise, and improve syncing with color.test.
* tests/parallel-tests-exit-statuses.test: New test.
* tests/parallel-tests-console-output.test: Likewise.
* tests/Makefile.am (TESTS): Update.
|
|
|
6623ea61
|
2011-06-28T12:01:04
|
|
Merge branch 'master' into testsuite-work
* master:
Revert "docs: parallel-tests is not experimental anymore"
parallel-tests: stricter checks on DISABLE_HARD_ERRORS support
docs: parallel-tests is not experimental anymore
check: document and test $(TEST_SUITE_LOG) overriding
|
|
|
b1109e2f
|
2011-06-28T11:26:15
|
|
Merge branch 'maint'
* maint:
Revert "docs: parallel-tests is not experimental anymore"
parallel-tests: stricter checks on DISABLE_HARD_ERRORS support
docs: parallel-tests is not experimental anymore
check: document and test $(TEST_SUITE_LOG) overriding
|
|
|
ae610ad3
|
2011-05-23T14:26:58
|
|
tests: tweak and improve tests on "Simple Tests" driver
* tests/check.test: Make grepping of generated Makefile.in
slightly stricter. Add trailing `:' command.
* tests/check2.test: Ensure verbose printing of captured make
output. Make grepping of generated Makefile.in and of make
output stricter. Add trailing `:' command.
* tests/check3.test: Likewise. Also, fix minor error in a
comment.
* tests/check4.test: Stricter grepping of make output. Use
`using_gmake' subroutine instead of copy&paste from defs.in
to determine whether $MAKE is GNU make. Add a trailing `:'
command.
* tests/check6.test: Add trailing `:' command.
* tests/check10.test: Prefer redirection `>' over `>>' when
creating new files. Ensure verbose printing of captured make
output.
* tests/check11.test: Likewise.
* tests/check7.test: Likewise. Make grepping of make output
stricter. Prefer "make distcheck" over a simple "make check"
in a VPATH build. Add trailing `:' command.
(a.c): Format following GNU coding standards.
(Makefile.am): Explicitly declare target `print-xfail-tests'
as PHONY. Distribute $(check_SCRIPTS), for distcheck.
* tests/parallel-tests.test: Prefer redirection `>' over `>>'
when creating new files.
* tests/parallel-tests2.test: Likewise.
* tests/parallel-tests9.test: Likewise.
* tests/parallel-tests8.test: Likewise. Make grepping of
automake stderr stricter.
* tests/check8.test: Likewise. Make formatting of created
*.c files more consistent with GNU Coding Standards.
* tests/check-subst-prog.test (configure.in): Break overly
long lines.
(pass-prog.c, xfail-prog.c): Format them more consistently
with GNU Coding Standards.
* tests/parallel-tests3.test: Use a shorter and clearer skip
message. Use the `unindent' subroutine to improve readability
of indented loop bodies. Simplify the hack used to please
maintainer-check.
* tests/parallel-tests5.test: Make formatting of created *.c
files more consistent with GNU Coding Standards. Use the
`unindent' subroutine to improve readability of indented loop
bodies.
* tests/parallel-tests10.test: Add blank line, for clarity.
Prefer trailing `:' over trailing `Exit 0', for consistency
with other tests.
|
|
|
f31a38ae
|
2011-05-22T09:35:17
|
|
parallel-tests: stricter checks on DISABLE_HARD_ERRORS support
* tests/parallel-tests-harderror.test: New test, doing more
in-depth checks on DISABLE_HARD_ERRORS.
* tests/parallel-tests.test: Remove tests on DISABLE_HARD_ERRORS,
now redundant.
* tests/Makefile.am (TESTS): Update.
|
|
1456186d
|
2011-01-16T12:15:19
|
|
Merge branch 'maint'
|
|
|
e5fb6dce
|
2011-01-16T12:12:51
|
|
Fix parallel-tests.test failure with HP-UX make.
* tests/parallel-tests.test: Sleep inside inner tests, so logs
are newer than logs of tests they depend on, for HP-UX make.
Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
|
|
|
79103003
|
2011-01-08T19:40:04
|
|
tests: enable 'errexit' shell flag by default.
* tests/defs: Enable `errexit' shell flag (near the end).
Removed redundant comment about the enabling of shell traces.
* tests/README (Writing test cases): Update, and use nicer
formatting in a couple of places.
* All tests: Adjusted by removing now-redundant calls to
'set -e'.
|
|
|
03449117
|
2010-03-04T20:22:17
|
|
Avoid generation of `tests/defs-p' file
* tests/defs.in: In the generated `configure.in' snippet: call
`AM_INIT_AUTOMAKE' with the `parallel-tests' option if the shell
variable `parallel_tests' is set to `yes'.
* tests/Makefile.am (defs-p): Target removed.
(check_SCRIPTS): Removed `defs-p'.
(clean-local-check): Do not unlink `defs-p' anymore.
($(parallel_tests)): Transformation rules for the test scripts
adjusted.
* tests/gen-parallel-tests: Selection rules for the test
scripts adjusted.
* tests/parallel-tests.test: Set `$parallel_tests' to `yes'
then include `./defs' (rather than simply including `./defs-p').
* tests/parallel-tests2.test: Likewise.
* tests/parallel-tests3.test: Likewise.
* tests/parallel-tests4.test: Likewise.
* tests/parallel-tests5.test: Likewise.
* tests/parallel-tests6.test: Likewise.
* tests/parallel-tests7.test: Likewise.
* tests/parallel-tests8.test: Likewise.
* tests/parallel-tests9.test: Likewise.
* tests/parallel-tests10.test: Likewise.
* tests/README (Section "Writing Test Cases", subsection "Do"):
Adjusted the parts referring to tests checking `parallel-tests'
behaviour. Some other minor related improvements.
* tests/.gitignore (defs-p): Removed.
|
|
|
f923b46e
|
2010-03-04T21:09:13
|
|
Remove redundant unset of variable TESTS from some test scripts.
* tests/color.test: Do not unset the `TESTS' variable, as it's
already unset in the `defs' file.
* tests/check5.test: Likewise.
* tests/check8.test: Likewise.
* tests/check9.test: Likewise.
* tests/check10.test: Likewise.
* tests/check11.test: Likewise.
* tests/parallel-tests.test: Likewise.
* tests/parallel-tests3.test: Likewise.
* tests/parallel-tests4.test: Likewise.
* tests/parallel-tests5.test: Likewise.
* tests/parallel-tests6.test: Likewise.
* tests/parallel-tests7.test: Likewise.
|
|
|
fcf2f560
|
2009-05-17T15:32:40
|
|
Revert Automake license to GPLv2+.
Automake will move to GPLv3+ once the Exception statement has
been rewritten to use the new GPLv3 exception language. This
change does not impact the COPYING file that may be installed
by `automake --add-missing'.
* COPYING: Revert to GPLv2. All uses changed.
* NEWS: Update.
Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
|
|
|
03002293
|
2009-04-10T13:37:31
|
|
parallel-tests: more testsuite exposure.
* tests/parallel-tests.test: Also test overriding TEST_LOGS and
TESTS.
Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
|
|
|
8f33eec4
|
2009-03-28T22:58:34
|
|
parallel-tests: redo lazy checking: recheck and RECHECK_LOGS.
Replace the LAZY_TEST_SUITE API with a simpler yet more powerful
one: RECHECK_LOGS specifies those tests which are to be removed
in any case before testing. Provide a `recheck' convenience
target to set RECHECK_LOGS to all failed and unexpectedly passed
tests. Document several ways to limit the set of tests run.
* lib/am/check.am [PARALLEL_TESTS] (RECHECK_LOGS): New variable,
default to $(TESTS_LOGS).
(check-TESTS): Remove $(RECHECK_LOGS) not $(TEST_LOGS). Drop use
of LAZY_TEST_SUITE.
($(TEST_SUITE_LOG)): Do not output note about lazy rerun, as
LAZY_TEST_SUITE is gone.
(recheck): New target.
(recheck-am, recheck-TESTS): New internal targets.
* doc/automake.texi (Tests): Update @vindex for TESTS and
TEST_LOGS. Replace description of LAZY_TEST_SUITE with a list
of ways the set of tests to be run can be modified. Document
RECHECK_LOGS and the recheck target.
* tests/defs.in: Unset RECHECK_LOGS not LAZY_TEST_SUITE.
* tests/parallel-tests.test: Adjust, replacing LAZY_TEST_SUITE
with corresponding RECHECK_LOGS settings, and add another
RECHECK_LOGS test.
* tests/parallel-tests9.test: New test, test `recheck'.
* tests/Makefile.am: Update.
Suggestion and different implementation by Akim Demaille.
Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
|
|
|
4c7f26e8
|
2009-03-11T20:36:14
|
|
Fix LAZY_TEST_SUITE handling and $(TEST_SUITE_LOG) recreation.
* lib/am/check.am (check-TESTS): Expand `$(TEST_LOGS)' only once
in the rule command, for systems with low command line limits.
Remove $(TEST_SUITE_LOG) even in LAZY_TEST_SUITE mode.
($(TEST_SUITE_LOG)): Always recreate $(TEST_SUITE_LOG). Mention
lazy mode in the summary output.
* tests/parallel-tests.test: Test LAZY_TEST_SUITE semantics.
Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
|
|
|
5a17fc03
|
2009-03-10T21:10:25
|
|
parallel-tests: Fix summary output.
* lib/am/check.am (am__text_box): Fix unportable sed script,
replacing `\n' in the right hand side of an `s' command with
a literal newline. Kudos to Bruno Haible for the newline idea.
* tests/parallel-tests.test: Update test to expose this.
Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
|
|
|
099f454e
|
2009-03-08T14:05:56
|
|
New tests for `parallel-tests'.
* tests/parallel-tests.test: New file, to expose a bunch of
features of `parallel-tests': VERBOSE, clean, TEST_SUITE_LOG,
test dependencies, DISABLE_HARD_ERRORS.
* tests/parallel-tests2.test: New file, test check-html.
* tests/parallel-tests3.test: New file, test concurrency.
* tests/parallel-tests4.test: New file, test suffix rules.
* tests/parallel-tests5.test: New file, demonstrate compile/test
concurrency.
* tests/defs.in: Add a `required' check for rst2html.
* tests/Makefile.am: Update.
Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
|