kc3-lang/automake
Branch :
Log
| Author | Commit | Date | CI | Message |
|---|---|---|---|---|
 |
e55b4ab1 | 2011-08-14 14:46:53 | maintcheck: look for TAP-based tests without TAP plan * Makefile.am (sc_tests_tap_plan): New maintainer check. (syntax_check_rules): Add it. * tests/suffix10.tap: Declare a TAP plan. * tests/add-missing.tap: Likewise. | |
|
38f12a30 | 2011-08-14 14:00:53 | testsuite: avoid confusing SKIP messages for errored tests * tests/defs [$use_tap] (exit trap): Don't print the test plan in case the client script exited with a non-zero status. This avoids the display of redundant "SKIP" messages for tests with lazy plans that error out before any test is run. | |
|
0d8a113d | 2011-08-13 14:29:28 | testsuite: keep more debugging info around in 'add-missing.tap' * tests/add-missing.tap: Use a different directory for each test, so that in case of failure more information is kept around. Since we are at it, remove redundant TAP diagnostic, and define `$am_create_testdir' to "empty" to avoid bringing in unused auxiliary files. | |
|
726540fa | 2011-08-13 12:39:59 | testsuite: more granular count of test results in our TAP library * tests/tap-functions.sh ($tap_bad_count_): Removed, superseded by ... ($tap_fail_count_, $tap_xpass_count_): ... these new variables, which keep more granular counts. ($tap_pass_count_: New variable. * tests/defs (exit trap): Adjust and simplify accordingly. | |
|
836ca564 | 2011-08-13 11:52:26 | testsuite: fixlets and improvements in two long TAP-based tests * tests/depmod.tap: Clean up the subdirectories used by tests that passed, to avoid ending up with a too big test directory. This is especially important since, on each and every system, at least some tests in this script are expected to be skipped (which ones exactly depends on the system), thus causing the test directory not to be removed when the script terminates. * tests/instspc.tap: Likewise (with the difference that tests here are not expected to be skipped, but to xfail). Give more explicit error message in case of setup failure. Skip the proper number of tests when required, to avoid spurious errors due to mismatched plan. Also, really avoid to run the unsupported test cases when the need to skip them is detected. | |
|
a956d986 | 2011-08-09 23:27:56 | Merge branch 'test-protocols' into tap-testsuite-work * test-protocols: test harness: recursive make invocations must use $(AM_MAKEFLAGS) | |
|
02dfe16d | 2011-08-09 23:01:23 | test harness: recursive make invocations must use $(AM_MAKEFLAGS) * lib/am/check.am ($(TEST_SUITE_LOG)): Pass $(AM_MAKEFLAGS) to $(MAKE) when issuing a recursive make call. Problem detected by the `sc_tests_make_without_am_makeflags' maintainer check. | |
|
b3c532b0 | 2011-08-09 22:54:03 | Merge branch 'master' into test-protocols * master: tests: fix various blunders in 'suffix-chain.test' | |
|
0bb7fe0d | 2011-08-09 22:52:06 | Merge branch 'master' into tap-testsuite-work * master: tests: fix various blunders in 'suffix-chain.test' (edited) | |
|
1f9b3058 | 2011-08-09 22:44:55 | tests: fix various blunders in 'suffix-chain.test' * tests/suffix-chain.test: Fix various blunders that were causing this test to fail spuriously: append to `configure.in', not to `configure.ac'; add a `;' character after the `return' statement in an input file; use tabs, not spaces, to indent Makefile recipes; and use more uncommon suffixes that don't trigger built-in recipes with FreeBSD make. | |
|
e61b581a | 2011-08-09 22:30:26 | testsuite: improvements to our shell TAP library and APIs These improvements impact only the Automake's own testsuite. * tests/tap-functions.sh (plan_): Allow new special arguments "unknown" a.k.a. "later" and "lazy" a.k.a. "now". Perform sanity check the given arguments. Adjust incorrect comments. (lazy_plan_): Removed, superseded by the new version of `plan_'. (not): New function, run a command and inverts its exit status. (ok_, not_ok_, skip_): These are just shorthands for common `result_' usages. (xfail_, xpass_, reset_test_count_): Removed, they are not used, nor does a legitimate use for them seem in sight. (not): New function, to run a command inverting its exit status. (command_not_ok_): Removed, superseded by `command_ok_' + `not'. (command_ok_): Extended to accept TAP directives. * tests/defs (trap): Don't use `late_plan_' anymore, use `plan_ "now"' instead. * tests/ac-output-old.tap: Update to take advantage of the new APIs. * tests/self-check-env-sanitize.tap: Likewise. * tests/suffix-chain.tap: Likewise. * tests/suffix10.tap: Likewise. * tests/suffix11.tap: Likewise. * tests/suffix3.tap: Likewise. * tests/suffix8.tap: Likewise. * tests/tap-bad-prog.tap: Likewise. * tests/self-check-exit.tap: Likewise, plus a small improvement in the name of a variable. | |
|
6b5dccd3 | 2011-08-09 19:14:13 | testsuite: convert some `suffix*' tests to TAP This will allow us to explicitly declare some expected failures involving Solaris make as such, instead of partly papering over them with skips and partly letting them fail too loudly. * tests/suffix-chain.test: Rename ... * tests/suffix-chain.tap: ... to this, and convert to the use of TAP. Also, fix some bugs since we are at it, and ... ($required): ... require `cc'. * tests/suffix3.test: Rename ... * tests/suffix3.tap: ... to this, and convert to the use of TAP. Also, since we are at it, throw in some formatting fixes. * tests/suffix8.test: Rename ... * tests/suffix8.tap: ... to this, and convert to the use of TAP. Also, since we are at it, throw in some formatting fixes. * tests/suffix10.test: Rename ... * tests/suffix10.tap: ... to this, and convert to the use of TAP. * tests/suffix11.test: Rename ... * tests/suffix11.tap: ... to this, and convert to the use of TAP. * tests/Makefile.am (TESTS): Update. | |
|
1a758e7d | 2011-08-09 18:13:15 | Merge branch 'test-protocols' into tap-testsuite-work * test-protocols: tests: don't let a known Solaris make bug poison too many tests gitignore: more use of anchors | |
|
18fb6351 | 2011-08-09 18:04:14 | Merge branch 'master' into test-protocols * master: tests: don't let a known Solaris make bug poison too many tests gitignore: more use of anchors | |
|
f501499a | 2011-08-09 17:51:51 | tests: don't let a known Solaris make bug poison too many tests See automake bug#7670 and bug#7824. * tests/defs (make_can_chain_suffix_rules): New function, tell whether the make implementation in use can chain suffix rules automatically or not. * tests/suffix3.test: Use it to avoid issuing calls to make that are unportable to make implementations that are not smart enough to chain suffix rules automatically. * tests/suffix8.test: Use it to avoid issuing calls to make that * tests/suffix10.test: Use it to avoid issuing calls to make that * tests/suffix11.test: Use it to avoid issuing calls to make that * tests/suffix-chain.test: New test, exposes the limitation that we have papered over in the tests above. | |
|
fe1b11eb | 2011-08-09 16:43:08 | gitignore: more use of anchors * .gitignore: Anchor files that are intended to be ignored only if found in the same directory of the `.gitignore' file, not also in its subdirectories. * doc/.gitignore, doc/amhello/.gitignore, lib/Automake/.gitignore, lib/Automake/tests/.gitignore, tests/.gitignore: Likewise. Also, where needed, add new entries that were once implied by the non-anchored entries in the upper-level `.gitignore' files. | |
|
73199a16 | 2011-08-09 15:38:51 | Merge branch 'test-protocols' into tap-testsuite-work (no-op) | |
|
6a50baf3 | 2011-08-09 15:27:30 | Merge branch 'master' into test-protocols * master: testsuite: avoid spurious failure of self checks on Solaris | |
|
e171f615 | 2011-08-09 15:24:03 | testsuite: avoid spurious failure of self checks on Solaris This is just a quick workaround to prevent having too much noise in the testsuite; more proper fixes have been committed into the 'testsuite-work' branch. * tests/self-check-dir.test: Relax a test to avoid unimportant failure on Solaris 10. * tests/self-check-exit.test: Likewise. | |
|
b615fcd5 | 2011-08-09 14:02:10 | self tests: avoid spurious output on console, and related fixlets * tests/self-check-cleanup.test: Set `$stderr_fileno_' to `2' in the self tests, to avoid spurious diagnostic on console. * tests/self-check-exit.tap: Only unset `am_explicit_skips' and `$stderr_fileno_' in the self tests, not at the global level, it is more correct. ($init): New variable, to reduce code duplication. Some refactoring and adjustments to the name of the tests. | |
|
376293a9 | 2011-08-09 12:13:53 | test defs: few more sanity checks * tests/defs: Catch some more (very unlikely) set-up or internal errors, and diagnose them explicitly. This is intended mostly to be a theoretic-only improvement. | |
|
07d9b536 | 2011-08-09 12:00:41 | test defs: really make them "set -e" clean Motivated by a spurious failure of test `instsh3-w.shtst' on Solaris 10. * tests/defs: Make really "set -e" clean, to avoid spurious failures in case this file is sourced by a test script that has already set the `errexit' flag. To be sure we don't regress, move the `set -e' call much earlier, and make the comment telling about the expected "set -e" cleanliness more loud and outstanding. | |
|
389bd6b3 | 2011-08-09 10:49:01 | self tests: register an expected failures with Solaris /bin/sh Solaris 10 /bin/sh erroneously exit with success right away when the following three conditions are met at the same time: 1. the `errexit' flag is active, 2. an exit trap is installed, and 3. a non-existing command is issued. * tests/self-exit.tap: When that bug is detected, issue an XFAIL rather than a FAIL. Since we are at it, improve by avoiding creation of stray temporary files when testing for non-executable commands. | |
|
4c4beef6 | 2011-08-09 10:23:34 | gitignore: update and improve for the testsuite files * tests/.gitignore: Don't ignore temporary partial log files, autogenerated tests and makefile fragments that are not used anymore. Anchor all the remaining patterns. | |
|
98804316 | 2011-08-09 00:38:39 | self tests: some new uses of TAP, few extensions * tests/self-check-me.test: Renamed ... * tests/self-check-me.tap: .. to this, and converted to use TAP. * tests/self-check-exit.test: Renamed ... * tests/self-check-exit.tap: .. to this, and converted to use TAP. Set `$stderr_fileno_' to `2', to avoid spurious messages on the console. Add a new test, and relax another one. * tests/self-check-env-sanitize.test: Renamed ... * tests/self-check-env-sanitize.tap: .. to this, and converted to the use TAP. * tests/Makefile.am (TESTS): Update. | |
|
e3b6255f | 2011-08-08 23:35:28 | test defs: test directory creation more configurable After this change, the individual tests will be able to tell `./defs' whether or not to create a temporary test directory for them, and if yes, whether or not to pre-populate it. * tests/defs: Only create the temporary test directory if the variable `$am_create_testdir' is not set to "no". In this case, only pre-populate it if that variable is not set to `$empty'. * tests/defs-static.in: Check that `$am_create_testdir' is not exported in the environment. * tests/self-check-env-sanitize.test: Update. * tests/self-check-dir.test: Extend, and make deeper and more robust. * tests/README (Section "Writing test cases" subsection "Do"): Update. * tests/autodist.test: Set `$am_create_testdir' instead of removing by hand files created by ./defs. * tests/tap-doc2.test: Likewise. | |
|
c463a964 | 2011-08-08 18:23:40 | testsuite: more uses of the `seq_' function * tests/parallel-tests-exit-statuses.test: Use the `seq_' function from `tests/defs', instead of duplicating its semantics by hand. * tests/testsuite-summary-count.test: Likewise. | |
|
53599ad1 | 2011-08-08 18:10:55 | testsuite: drop redundant environment sanitization * tests/Makefile.am (TESTS_ENVIRONMENT): Don't remove `planned' from the environment. * tests/defs-static.in: Don't complain if `planned' is set in the environment. * tests/self-check-env-sanitize.test: Update. | |
|
ff1b4768 | 2011-08-08 16:41:42 | test defs: fix bug in plain version of `skip_all_' * tests/plain-functions.sh (skip_all_): Call `skip_', not `skip'. | |
|
0b1c914b | 2011-08-08 16:37:36 | test defs: small cleanups and tweakings * tests/defs: Where possible, prefer `framework_failure_' or `fatal_' to `Exit 99'. Try to catch more possible hard errors and setup failures. Do not hide expected error messages. Avoid a couple of forks when copying files from `$(top_srcdir)/lib'. ($signal): Remove, no more needed. | |
|
8be644b8 | 2011-08-08 15:43:53 | testsuite: improve interface for giving early TAP plan Since commit `v1.11-976-g4740f56', an early TAP plan can also be specified after input lines have already been seen, as long as no one of them is a TAP result line. We can now take advantage of this in our own testsuite, allowing the TAP-based scripts to specify their earlier plan after having included `./defs'. In particular, this offers the benefit of allowing those scripts to issue a "skip all" plan (e.g., "1..0 # SKIP reason ...") after the inclusion of `./defs', which wasn't possible with the older interface (introduced in commit `v1.11-1114-g0286308'). * tests/defs: Don't try to call `plan_' if `$planned' is defined, the calling scripts can do that directly now. * tests/ac-output-old.tap: Use `plan_' directly instead of setting `$planned' before including `./defs'. * tests/automake-cmdline.tap: Likewise. * tests/depmod.tap: Likewise. * tests/instspc.tap: Likewise. * tests/tap-bad-prog.tap: Likewise. * tests/add-missing.tap: Don't set `$planned'. | |
|
150be8ee | 2011-08-08 15:23:11 | test defs: fix bug in TAP version of `skip_all_' * tests/tap-functions (skip_all_): Set `$have_tap_plan_' to "yes" when called, so that the exit trap won't write another TAP plan. | |
|
f4896832 | 2011-08-08 15:00:25 | test defs: fix skip directives to work also with TAP-based tests * test/defs: Use `skip_all_', not `skip_', to issue a skip of the whole calling test script. | |
|
0e6a7aeb | 2011-08-08 14:49:19 | Merge branch 'testsuite-work' into tap-testsuite-work (no-op) | |
|
2ad99318 | 2011-08-08 14:49:06 | Merge branch 'test-protocols' into tap-testsuite-work * test-protocols: test defs: more environment cleanup test harness: more minor VPATH-rewrite issues test harness: "make -n check" works correctly with Solaris make | |
|
2cf22db7 | 2011-08-08 14:42:42 | Merge branch 'master' into test-protocols * master: test defs: more environment cleanup | |
|
b1289ad8 | 2011-08-08 14:42:24 | Merge branch 'master' into testsuite-work * master: test defs: more environment cleanup | |
|
f807dd5a | 2011-08-08 14:40:59 | Merge branch 'maint' * maint: test defs: more environment cleanup | |
|
c7c62a4a | 2011-08-08 14:39:16 | test defs: more environment cleanup * tests/defs.in: Also unset variables AM_COLOR_TESTS and AM_TESTS_ENVIRONMENT. | |
|
1e0238ad | 2011-08-08 13:18:50 | test harness: more minor VPATH-rewrite issues * tests/am/check.am (am__set_TESTS_bases, $(TEST_SUITE_LOGS), am--redo-logs, recheck, recheck-html): Cosmetic fixlets to minimize the risk of unwanted VPATH rewrites. (check-TESTS): Likewise, and normalize trailing whitespace since we are at it. Bugs exposed by test cases `check6-p.test' and `check7-p.test'. | |
|
bf15d302 | 2011-08-08 12:50:12 | test harness: "make -n check" works correctly with Solaris make To avoid some recipes being executed with "make -n", we used the trick of issuing recursive make calls not directly with $(MAKE), but with a simple indirection instead: ... am__stealth_MAKE = $(MAKE) ... target: ... $(am__stealth_MAKE) ... ... This hack worked with GNU make, but, unfortunately, not with Solaris make. The present change removes the hack, and uses more proper idioms to avoid unwanted recipe executions with "make -n". Bug exposed by test case `parallel-tests-dry-run.test'. * lib/am/check.am (am__stealth_MAKE): Remove, no more used. (.log.trs): Break recipe in multiple shell invocations. ($(TEST_SUITE_LOG)): Likewise, and move some corner-case parts of the recipe to ... (am--redo-logs): ... this new internal helper .PHONY target. | |
|
b0a7f1ef | 2011-08-08 10:45:08 | Merge branch 'testsuite-work' into tap-testsuite-work (no-op) | |
|
f70e8b59 | 2011-08-08 10:43:32 | Merge branch 'test-protocols' into tap-testsuite-work * test-protocols: tests: fix spurious failure with Solaris make test harness: fixlet in workaround for VPATH rewrite issue | |
|
b9dfd77a | 2011-08-08 10:40:28 | Merge branch 'master' into test-protocols * master: tests: fix spurious failure with Solaris make | |
|
805b1b3a | 2011-08-08 10:30:58 | Merge branch 'master' into testsuite-work * master: tests: fix spurious failure with Solaris make | |
|
17ca784b | 2011-08-08 10:29:25 | Merge branch 'maint' * maint: tests: fix spurious failure with Solaris make | |
|
142dab55 | 2011-08-08 10:25:00 | Merge branch 'am-distcheck-configure-flags' into maint * am-distcheck-configure-flags: tests: fix spurious failure with Solaris make | |
|
d9e4f1ec | 2011-08-08 10:20:47 | tests: fix spurious failure with Solaris make * tests/distcheck-configure-flags-am.test: Avoid using `+=' too liberally with AM_DISTCHECK_CONFIGURE_FLAGS, since the line breaks so introduced, in conjunction with single quotes, might confuse Solaris make. | |
|
232dcc6d | 2011-08-08 09:31:03 | test harness: fixlet in workaround for VPATH rewrite issue Fixes a blunder in previous commit `v1.11-983-gfda3de5'. * lib/am/check.am (am__set_TESTS_bases): Add forgotten end-of-line anchor to sed expression. In comments, add reference to ... * tests/test-trs-basic.test: ... this test, which has been adapted and tightened (and tweaked). | |
|
cc98b9c6 | 2011-08-08 01:43:34 | Merge branch 'test-protocols' into tap-testsuite-work * test-protocols: fix: regenerate Makefiles test harness: work around a VPATH rewrite issue parallel-tests: work around Solaris XPG4 make segfault testsuite: fix weird spurious failure with Solaris /bin/sh testsuite: fix another spurious failure on Solaris make testsuite: fix two spurious failures on Solaris make tap driver: refactor and remove dead code testsuite: fix spurious failure in a test on TAP support tap: plan location is more liberal w.r.t. non-TAP lines testsuite: remove now-passing test from XFAIL_TESTS | |
|
a0cd0d4b | 2011-08-08 01:39:12 | fix: regenerate Makefiles | |
|
fda3de57 | 2011-08-08 01:23:42 | test harness: work around a VPATH rewrite issue * lib/am/check.am (am__TEST_BASES): Removed, it's role taken over by ... (am__set_TESTS_bases): ... these new variable. ($(TEST_SUITE_LOG): Use it, to avoid VPATH rewrite issues. * automake.in (handle_tests): Update the code for the cleanup of the `.trs' file to use `$(TEST_LOGS)' instead of `$(am__TEST_BASES)'. * tests/test-harness-vpath-rewrite.test: New test, better exposing the problem fixed by this change * tests/Makefile.am (TESTS): Add it. * tests/test-trs-basic.test: Update and extend. | |
|
152fcd20 | 2011-08-08 00:16:56 | parallel-tests: work around Solaris XPG4 make segfault Issue exposed by test `posixsubst-tests-p.test', and similar to the problem solved by commit `v1.11-159-ge7aa360'. * lib/am/check.am [%?PARALLEL_TESTS%] (check-TESTS): Trim trailing whitespace from $list, to avoid triggering a nasty bug (potential segfault) on Solaris XPG4 make and Heirloom make. | |
|
0ef3ef52 | 2011-08-07 23:48:02 | testsuite: fix weird spurious failure with Solaris /bin/sh Solaris /bin/sh, when killed with a SIGTERM or SIGINT signal, can apparently end up exiting with exit status 208, instead of leaving the correct wide exit status to the parent. See: <http://dbaspot.com/shell/396118-bourne-shell-exit-code-term.html> Work around this incompatibility. * tap-signal.test: Look for the above Solaris bug. (signal_caught): Adapt to handle it. | |
|
1072f686 | 2011-08-07 23:05:37 | testsuite: fix another spurious failure on Solaris make * tests/parallel-tests-log-override-recheck.test: Filter make output before grepping it, for make implementations that, like Solaris' one, print the whole of the failed recipe on failure. | |
|
b7e96983 | 2011-08-07 22:50:23 | testsuite: fix two spurious failures on Solaris make * tests/tests-environment-and-log-compiler.test: Do not fail if "make -n check" fails, as that is issued only for debugging, its semantics being tested in other test scripts. * tests/test-driver-strip-vpath.test: Likewise. | |
|
42851b79 | 2011-08-07 21:34:56 | tap driver: refactor and remove dead code * lib/tap-driver (get_tap_line): Removed, ... (all callers): ... updated to use $parser->next directly instead. (peek_tap_line, unget_tap_line, @tap_lines): Remove, they're not used anymore. (TAP_PEEKING): Block deleted, all its subroutines and variables having been removed. | |
|
30bef0f3 | 2011-08-07 20:26:34 | testsuite: fix spurious failure in a test on TAP support * tests/tap-passthrough-exit.test: When the `--ignore-exit' flag of the TAP driver is used, don't look for a message reporting the non-zero exit statuses of tests in the log files; such message is not expected to be there anymore. Related simplifications. | |
|
4740f56e | 2011-08-07 20:07:35 | tap: plan location is more liberal w.r.t. non-TAP lines With this change, only lines that are TAP results will matter w.r.t. the position of the TAP plan in the input; for example, this input: this is a non-TAP line # and this a TAP diagnostic line 1..1 ok 1 was considered to be an error, diagnosed with a message "test plan in middle of output"; as effect of the current change, such input is now valid. This is more consistent with the behaviour of the `prove' utility. * lib/tap-driver ($lineno): Removed, no more needed. ($tap_stopped): New global variable. (stringify_test_result): Return "ERROR" if a TAP result is found when `$tap_stopped' is set to true. (handle_tap_test): Diagnose TAP results that comes after a late plan. Add a couple of blank lines, for clarity. (handle_tap_plan): Set `$tap_stopped' to true after a late plan is encountered. Do not complain anymore for extra non-TAP lines preceding or following the plan. Adjust comments. (main): Don't increment $lineno anymore. * tests/tap-plan.test: Extend a bit, and remove stale comment. * tests/tap-color.test: Adjust. * tests/tap-passthrough.test: Likewise. * tests/tap-plan-corner.test: Adjust and extend. * tests/tap-plan-errors.test: Likewise. * tests/tap-plan-middle.test: New test. * tests/tap-plan-corner2.test: Delete, it's obsolete now. * tests/Makefile.am (XFAIL_TESTS): Remove it. (tap_with_common_setup_tests): Likewise, and add `tap-plan-corner.test'. | |
|
e0da0d07 | 2011-08-07 18:17:27 | testsuite: remove now-passing test from XFAIL_TESTS * tests/Makefile.am (XFAIL_TESTS): Remove `tap-signal.test', which is passing since previous commit `v1.11-974-gc7fa872'. | |
|
5f5b225a | 2011-08-07 15:54:26 | test defs: get name of TAP-using tests correctly This change fixes the names of the temporary directories used by the TAP-based tests (e.g., test `foo.tap' now uses `foo.dir' rather than `foo.tap.dir'); more importantly, this make the variable `$destdir' defined in `tests/defs' usable also in the TAP tests. ($me): Also strip the any `.tap' suffix from the name of the test. | |
|
5c1bb4e2 | 2011-08-07 15:20:00 | testsuite: TAP tests properly decide when to remove tempdirs Before this change, the TAP tests in the Automake testsuite were removing the temporary test directory even when they failed or were skipped, thus making debugging more difficult. * tests/tap-functions.sh (incr_tap_count): Removed, superseded by ... (incr_): ... this function, which can increment the value of any variable passed to it. (result_): Updated to use `incr_' instead of the now-removed `incr_tap_count_'. Keep count of failures, xfailures, xpasses, and skips, using ... ($tap_skip_count_, $tap_bad_count, _$tap_xfail_count_): ... these new variables. * tests/defs (trap): Try to use their values to decide whether the temporary directory being used by the test script should be removed or not. Other code reorganizations. And move the code for the removal of the temporary directory out to ... (rm_rf_): ... this new subroutine. (Main code): Use that instead of duplicating the code. | |
|
3f5dcfb0 | 2011-08-06 22:42:24 | testsuite: improve and refactor our custom TAP shell library This change is mostly done in preparation for the next one. * tests/tap-functions.sh (result_with_directive_): Merge into ... (result_): ... this function, which has been greatly extended and improved in various ways. (ok_, not_ok_, skip_, xfail_, xpass_): Adjust. Adjust comments. Remove an extra empty line. * tests/instspc.tap: Use the new improved interface of `result_'. * tests/tap-bad-prog.tap: Likewise. | |
|
11aa5492 | 2011-08-07 11:17:18 | testsuite: update "user interface" description in tests/README * tests/README (Subsection "Interpretation"): Mention the "ERROR" outcome. (Subsection "Getting details from failures"): Don't tell that the test scripts can be run directly, this is not 100% true; anymore; instead ... (Subsection "About the tests"): ... use this new subsection to explain how to run the tests (either simple or TAP-based) by hand. (Subsection "Supported shells"): Extend to refer to TAP tests too. | |
|
0286308c | 2011-08-07 09:42:48 | testsuite: TAP tests can have early plan now * tests/defs: When using TAP, allow the user to specify the number of tests through the `$planned' variable. (trap): Do not print the auto-generated plan if an explicit one has already been printed. * tests/defs-static.in: Complain and bail out if the `$planned' variable is exported in the environment. Reformat the code to facilitate future additions. * tests/Makefile.am (TESTS_ENVIRONMENT): Neutralize `$planned'. * tests/self-check-env-sanitize.test: Update. * tests/ac-output-old.tap: Give an explicit TAP plan, by defining `$planned' just before including `./defs'. * tests/depmode.tap: Likewise. * tests/automake-cmdline.tap: Likewise. * tests/tap-bad-prog.tap: Likewise. * tests/instspc.tap: Likewise, and related adjustments. * tests/add-missing.tap: Make it explicit that we have no a-priori plan, and tell why. | |
|
7fb04769 | 2011-08-07 11:31:13 | testsuite: run autogenerated tests with $(LOG_COMPILER) too This change enhances consistency in the testsuite. * tests/Makefile.am: Some cosmetic reordering, to keep the definition of FOO_LOG_COMPILER near to the place where `.foo' is added to $(TEST_EXTENSIONS). (PTEST_LOG_COMPILER): Run `$(srcdir)/parallel-tests.sh' with `$(LOG_COMPILER)', not with `$(SHELL)'. (SHTST_LOG_COMPILER): Likewise. | |
|
297c8cf7 | 2011-08-07 11:22:11 | testsuite: run TAP and "plain" tests with the same $(LOG_COMPILER) With this change, it will be easier to override, at make time, the shell used to run the tests. * tests/Makefile.am (LOG_COMPILER): Define to `$(SHELL)'. (TEST_LOG_COMPILER): Re-define to `$(LOG_COMPILER)'. (TAP_LOG_COMPILER): Likewise. | |
|
45567bd1 | 2011-08-06 22:29:17 | testsuite: run TAP-based tests explicitly with $(SHELL) * tests/Makefile.am (TAP_LOG_COMPILER): Define to `$(SHELL)'. | |
|
27359395 | 2011-08-06 22:15:50 | Merge branch 'test-protocols' into tap-testsuite-work * test-protocols: tap driver: handle signals received by the tests being run test driver: a preparatory refactoring (2) tap driver: a preparatory refactoring (1) cosmetics: typofix in the name of a function in the TAP driver tap: scripts with a SKIP plan but with exit status != 0 must error | |
|
c7fa8726 | 2011-08-06 21:41:37 | tap driver: handle signals received by the tests being run * lib/tap-driver (get_test_exit_message): Also deal with signals, by using the `wait' method of the TAP::Parser object instead of the `exit' method. This required the use of the standard perl module `POSIX'. * doc/automake.texi (Use TAP with the Automake test harness): Document that `--ignore-exit' has effect also on terminating signals. Add a "synchronizing" comment that references the tests 'tap-exit.test' and 'tap-signal.test'. * tests/tap-signal.test: Extend and adjust. | |
|
9f8d51bd | 2011-08-06 20:54:26 | test driver: a preparatory refactoring (2) * lib/tap-driver (finish): Move code to fetch the message about the exit status of the test being run to ... (get_test_exit_message): ... this new subroutine. | |
|
596ae384 | 2011-08-06 20:48:09 | tap driver: a preparatory refactoring (1) * lib/tap-driver (finish): Do not flush the TAP stream nor fetch the exit status of the TAP producer if running with the `--ignore-exit' option. Obviously, don't try to write such exit status in the test log anymore. | |
|
ff37c9ba | 2011-08-06 20:23:54 | cosmetics: typofix in the name of a function in the TAP driver * lib/tap-driver (testuite_error): Renamed ... (testsuite_error): ... to this, and minor related reformatting. | |
|
265671e4 | 2011-08-06 20:21:18 | tap: scripts with a SKIP plan but with exit status != 0 must error This change has been motivated by Automake's own testsuite. Some TAP tests there were erroring out (due to unexpected, unhandled failures) before having encountered TAP result, so that the simple-minded shell library implementing TAP generation ended up printing a "1..0" trailing test plan; this caused the script to be reported as a SKIP rather than an ERROR -- a nasty false negative. * lib/tap-driver: Add prototypes for each subroutine, to free up the order in which they can be defined and called. (main): Move the code checking for a bad exit status of the TAP producer ... (finish): ... here, and flush the TAP stream to ensure that the parser always obtains the producer's exit status. * tests/tap-skip-whole-badexit.test: New test. * tests/Makefile.am (tap_with_common_setup_tests): Add it. | |
|
0bc42aa6 | 2011-08-06 18:33:24 | Merge branch 'test-protocols' into tap-testsuite-work * test-protocols: tap: fix whitespace munging of diagnostic messages | |
|
bf630c55 | 2011-08-06 17:19:09 | testsuite: refactor and cleanup 'instspc.tap' * tests/instspc.tap (is_in_list): New helper subroutine. (expected_to_fail): Re-implement using it. (define_problematic_string): Likewise. Also, rename the special arguments `build-fail' and 'install-fail' to respectively `builddir-fail' and `destdir-fail', and other related changes. (Test data definition): Adapt. ($instspc_xfail_builds_list): Renamed ... ($builddir_xfails): ... to this. ($instspc_xfail_installs_list): Renamed ... ($destdir_xfails): ... to this. ($instspc_names_list): Renamed ... ($test_names_list): ... to this. ($instspc_test_string): Renamed ... ($test_string): ... to this. Add some explicative and "FIXME" comments. | |
|
2a792eb2 | 2011-08-06 17:42:38 | tap: fix whitespace munging of diagnostic messages * lib/tap-driver (extract_tap_comment): Pass the `g' flag to the substitution operator, to strip also trailing whitespaces. Fixes a failure in test `tap-whitespace-normalization.test'. | |
|
3ea0d417 | 2011-08-06 14:20:08 | testsuite: use TAP for `depmod*' tests, related simplifications * tests/depmod-tests.sh: Delete this complex and multifarious script, moving all it checks it used to perform into ... * tests/depmod.tap: ... this TAP-based test script. And extend it quite a bit, since the new form of the test was papering over some pre-existing failures (this was due to the fact that the values used for the VPATH were shorter in the new test script). * tests/depmod-data.test: Deleted. * tests/Makefile.am ($(srcdir)/depmod-tests.am): Don't include nor generate it anymore. (EXTRA_DIST): Don't distribute depmod-tests.sh anymore. (TEST_EXTENSIONS): Remove '.depmod'. (DEPMOD_LOG_COMPILER): Deleted. (depmod-data.log, $(depmod_tests:.depmod=.log)): Delete dependencies for these files, which are not used anymore. ($(depmod_tests)): Delete this dummy dependency declaration. (TESTS): Remove $(depmod_tests). (handwritten_tests): Remove depmod-data.test, add depmod.tap. * bootstrap: Don't generate tests/depmod-tests.am anymore. | |
|
8b4dfc2d | 2011-08-06 12:35:05 | testsuite: use TAP for `instspc*' tests, related simplifications * tests/instspc-tests.sh: Delete this complex and multifarious script, moving all it checks it used to perform into ... * tests/instspc.tap: ... this TAP-based test script. * tests/instspc-data.test: Deleted. * tests/Makefile.am ($(srcdir)/instspc-tests.am): Don't include nor generate it anymore. (EXTRA_DIST): Don't distribute instspc-tests.sh anymore. (TEST_EXTENSIONS): Remove '.instspc'. (INSTSPC_LOG_COMPILER): Deleted. (XFAIL_TESTS): Remove $(instspc_xfail_tests). (instspc-data.log, $(instspc_tests:.instspc=.log)): Delete dependencies for these files, which are not used anymore. ($(instspc_tests)): Delete this dummy dependency declaration. (TESTS): Remove $(instspc_tests). (handwritten_tests): Remove instspc-data.test, add instspc.tap. * bootstrap: Don't generate tests/instspc-tests.am anymore. | |
|
683a7cff | 2011-08-06 10:51:07 | testsuite: do fd redirections with $(AM_TESTS_FD_REDIRECT) * tests/Makefile.am: The redirections of file descriptors needed by our test scripts is now done using the $(AM_TESTS_FD_REDIRECT) variable (the new blessed way since commit `v1.11-906-gb9e9d54'), not using an hack involving $(AM_TESTS_ENVIRONMENT). This has the further benefit of allowing the use of $(TESTS_ENVIRONMENT) again on part of the users. * tests/Makefile.am (AM_TESTS_ENVIRONMENT): Remove redirection of file descriptors. (AM_TESTS_FD_REDIRECT): Redirect file descriptor 9 to original stderr. Comments adjusted. * tests/plain-functions.sh ($stderr_fileno_): Update comment. | |
|
0cf750d7 | 2011-08-05 22:58:03 | testsuite: use custom TAP diagnostic in our own tests This change allow us to easily and transparently avoid spurious TAP diagnostic in our own testsuite, with the help of the new `--diagnostic-string' option of our TAP test driver. * tests/Makefile.am (TAP_LOG_DRIVER_FLAGS): Also pass option `--diagnostic-string' with value `#%#' (necessarily obtained through some shell indirections). * tests/tap-functions.sh (diag_): Assume that the string denoting TAP diagnostic is stored in the `$diag_string_' variable. Update comments. ($diag_string_): Initialize to "#". * tests/defs ($diag_string_): Re-define to "#%#". (show_): Remove, it's obsolete now. * add-missing.tap: Don't use `show_' anymore to avoid spurious TAP diagnostic; `cat' should suffice now. | |
|
3027d901 | 2011-08-05 21:14:53 | Merge branch 'test-protocols' into tap-testsuite-work * test-protocols: tap: fix typo in TAP driver version message tap docs: we don't support pragma or version directives (yet) tap: new option to change the string designating TAP diagnostic check: add small "synchronization" comment to `test-driver' maintcheck: fix spurious failure w.r.t. use of bare "exit" check: update comments to reflect recent heavy changes check: small cleanups and refactorings in test harness and drivers testsuite: use AM_TESTS_FD_REDIRECT where appropriate docs: some fixlets in documentation on testsuites support test harness: be aware of more metadata, simplify test drivers tap: check that also a trailing TAP plan can hold a skip directive | |
|
a3345817 | 2011-08-05 20:59:03 | tap: fix typo in TAP driver version message * lib/tap-driver (anonymous subroutine printing the script version message): Add missing trailing newline to the version message. | |
|
144628d7 | 2011-08-05 20:52:34 | tap docs: we don't support pragma or version directives (yet) * doc/automake.texi (Incompatibilities with other TAP parsers and drivers): Document that limitation. Normalize an unrelated "TODO" item. | |
|
ccf385b9 | 2011-08-05 19:08:16 | tap: new option to change the string designating TAP diagnostic Here we introduce a new option `--diagnostic-string' in our TAP test driver, that allows the user to specify which string should denote the beginning of a TAP diagnostic line. This change is not gratuitous, nor result if over-engineering: it is motivated by real issues that have emerged during the use of TAP in the Automake's own testsuite (see the commit `v1.11-1082-g9b967c2' "testsuite: yet more use of TAP, and related extensions"). * doc/automake.texi (Use TAP with Automake test harness): Document the new option. (Incompatibilities with other TAP parsers and drivers): Report it as a potential source of incompatibility. * lib/tap-driver ($diag_string): New global variable, defaulting to "#", and whose value can be changed ... (Getopt::Long::GetOptions): ... by the newly recognized option `--diagnostic-string'. (handle_tap_comment): Subroutine removed, some of its simple logic inlined ... (main): ... in here, where now ... (extract_tap_comment): ... this new subroutine is used. ($USAGE): Adjust. * tests/tap-diagnostic.test: Make one check slightly stricter. * tests/tap-diag-custom.test: New test. * tests/Makefile.am (tap_other_tests): Add it. | |
|
cf2e31d3 | 2011-08-05 16:48:32 | check: add small "synchronization" comment to `test-driver' * lib/test-driver: Add comment to the code initializing ANSI color escapes, telling to keep it in sync with the similar initialization in `lib/am/check.am'. | |
|
f5cb0635 | 2011-08-05 14:27:12 | maintcheck: fix spurious failure w.r.t. use of bare "exit" * tests/check12.test: Cosmetic changes to avoid triggering a spurious failure of the `sc_tests_Exit_not_exit' maintainer check. | |
|
a0d4d97a | 2011-08-05 14:16:19 | check: update comments to reflect recent heavy changes * lib/am/check.am [%?PARALLEL_TESTS%]: Update the comments to reflect the recent changes. Remove quite many comments that were merely duplicating excerpts from the Automake manual. | |
|
4839ca92 | 2011-08-05 12:08:20 | check: small cleanups and refactorings in test harness and drivers * lib/tap-driver (yn): New subroutine, converts a boolean value to a "yes" or "no" string. (write_test_results): Use it, and related reformatting. * lib/test-driver (trap): Also remove the `.trs' file on signals, for extra safety. (fatal): Remove this function, it's never used. ($logfile, $trsfile): Renamed ... ($log_file, $trs_file): ... to these, for clarity, and in order to be more consistent with the `tap-driver' script. Improve a couple of comments. * automake.in (handle_tests): Don't define anymore the now-obsolete make macro `$(TEST_LOGS_TMP)', nor add it to the list of files to be removed upon "make mostlyclean". * lib/am/check.am ($(TEST_SUITE_LOG)): New shell function `f_ok', tells whether a path refers to an existing, regular, readable file. Use it throughout. (recheck): Be safer w.r.t. make implementation that run recipes with `errexit' shell flag active. | |
|
5e3ae4e1 | 2011-08-05 10:36:06 | testsuite: use AM_TESTS_FD_REDIRECT where appropriate * parallel-tests-interrupt.test: Use AM_TESTS_FD_REDIRECT, not AM_TESTS_ENVIRONMENT, to define file descriptors' redirection for "make check". Fix botched comment. | |
|
bc5ba174 | 2011-08-05 10:17:01 | docs: some fixlets in documentation on testsuites support * doc/automake.texi (Script-based Testsuites): Rename node ... (Scripts-based Testsuites): ... to this. Break overly long lines in an example (were causing problems with PDF output). (Parallel Test Harness): Don't intend paragraphs that come after examples or bulleted list. (Overview of Custom Test Drivers Support): Fix typo s/the the/the/. (Declaring Custom Test Drivers): When giving an aside, prefer comma to parentheses, as it disrupts the reading flow less. (API for Custom Test Drivers): Remove a sentence that was basically duplicated from the previous subsection. (Command-line arguments for test drivers): Don't intend paragraph coming after an itemized list. Clarify wording and fix grammaros. Add a couple of cross-references to earlier explanations of `.log' and `.trs' files. (Links and external resources): Node renamed ... (Links and external resources on TAP): ... to this, for clarity. Slightly extend and clarify introductory sentence. | |
|
956c58ab | 2011-08-05 09:28:17 | test harness: be aware of more metadata, simplify test drivers * lib/am/check.am ($(TEST_SUITE_LOG)): When producing the global test log, take into account the new metadata field `:global-test-result:, to write nicely formatted RST section titles, instead of leaving this chore to the individual test drivers. (am_rst_section): Re-introduce this variable, as removed in commit `v1.11-870-ga27c9c4'. * lib/test-driver, lib/tap-driver: Write the global test result as metadata in the `.trs' file, not as (part of) a formatted RST subsection title in the `.log' file. Related simplifications; in particular, get rid of temporary files usage. * doc/automake.texi (Log files generation and test results recording): Document the new metadata. * tests/check12.test: Update. * tests/parallel-tests-harderror.test: Likewise. * tests/parallel-tests-interrupt.test: Likewise. * tests/parallel-tests-log-override-2.test: Likewise. * tests/parallel-tests-log-override-recheck.test: Likewise. * tests/parallel-tests-unreadable.test: Likewise. * tests/tap-global-result.test: Likewise. * tests/test-metadata-results.test: Likewise. * tests/test-log.test: Likewise, and another minor unrelated fix. * tests/test-metadata-global-result.test: New test. * tests/Makefile.am (TESTS): Update. | |
|
0e886a67 | 2011-08-04 14:14:02 | testsuite: more correct names for a couple of tests * tests/parallel-tests-ext-driver.test: Rename ... * tests/parallel-tests-log-compiler-1.test: ... to this, and adjust heading comments. * tests/parallel-tests-ext-driver-prog.test: Rename ... * tests/parallel-tests-log-compiler-2.test: ... to this, and adjust heading comments. * tests/Makefile.am (TESTS): Update. | |
|
0bc3abcb | 2011-08-04 13:55:46 | testsuite: fix another spurious failure * tests/parallel-tests-ext-driver.test: Name out custom/dummy log compilers as `foo-compiler' rather `foo-driver'; not only this is less confusing, it also avoids errors due to the dummy log compiler `test-driver' overriding the `test-driver' helper script installed by Automake. * tests/parallel-tests-ext-driver-prog.test: Adjust heading comments. | |
|
e842d5f2 | 2011-08-04 13:44:01 | testsuite: fix a spurious failure * tests/parallel-tests-ext-driver.test: Exporting of environment variables directly from $(LOG_COMPILER) variables is not allowed anymore. Adjust to this. | |
|
9b967c28 | 2011-08-04 11:08:24 | testsuite: yet more use of TAP, and related extensions * tests/defs (show_): New function, display the contents of one or more files on stdout, with bells & whistles (both for cosmetic and practical reasons, the latter aimed at avoiding spurious TAP diagnostic). (AUTOMAKE_run): Extend and adapt to make it compatible with TAP based tests. Since we are at it, make its implementation more namespace-safe, and improve its command-line interface. (AUTOMAKE_fails): Adapt to the new command-line interface of `AUTOMAKE_run'. * tests/add-missing.test: Renamed ... * tests/add-missing.tap: ... to this, and converted to the use of the TAP protocol, and of the new `show_' function. * tests/automake.test: Renamed ... * tests/automake-cmdline.tap: ... to this, and converted to the use of the TAP protocol. * tests/getopt.test: Adapt to the new command-line interface of `AUTOMAKE_run'. * tests/cond44.test: Likewise. * tests/cond45.test: Likewise. * tests/configure.test: Likewise. * tests/license2.test: Likewise. * tests/parallel-am.test: Likewise. * tests/parallel-am3.test: Likewise. * tests/tags2.test: Likewise. * tests/werror3.test: Likewise. * tests/werror4.test: Likewise. * tests/amopts-variable-expansion.test: Likewise. * tests/warnings-win-over-strictness.test: Likewise. * tests/Makefile.am (TESTS): Update. (TAP_LOG_DRIVER_FLAGS): Add `--merge', so that TAP diagnostic is reported in the testsuite progress output. | |
|
445674ff | 2011-08-03 20:22:50 | tap: check that also a trailing TAP plan can hold a skip directive * tests/tap-skip-whole-lastline.test: New test. * tests/Makefile.am (tap_with_common_setup_tests): Add it. | |
|
0bc7d68a | 2011-08-03 18:11:17 | Merge branch 'test-protocols' into tap-testsuite-work * test-protocols: maintcheck: fix more maintainer-check failures maintcheck: fix maintainer-check failures, both real and spurious gitignore: ignore `*.trs' files in lib/Automake/tests testsuite: list another "forgotten" test script in Makefile.am | |
|
9a3268ca | 2011-08-03 17:54:18 | maintcheck: fix more maintainer-check failures * tests/test-driver-custom-no-html.test (no-rst): Use `EOF', not `EoL', as the here-document delimiter. * tests/test-trs-basic.test: Use `cat + here-doc' rather than `echo' when creating the dummy test scripts, to please maintainer-check. * tests/test-trs-recover.test: Use creative quoting where needed, to please maintainer-check. * tests/parallel-tests-no-color-in-log.test: Likewise. * tests/parallel-tests-dry-run.test: Likewise. | |
|
f5c022ed | 2011-08-03 17:24:19 | Merge branch 'master' into test-protocols * master: maintcheck: fix maintainer-check failures, both real and spurious docs: how to use '-I' option in AM_CPPFLAGS for best VPATH support tests: relax a test on amhello examples to cater to Solaris tar test defs: function 'is_newest' now works also with directories docs: add references between the 2 sections on java support Sync auxiliary files from upstream. fix typo in recent ChangeLog entry docs, tests: synchronize examples on silent-rules from config.site tests: fix weakness in 'tests-environment-backcompat.test' tests: portability fixes in tests on amhello examples remake: add test ensuring that slower remakes don't hang remake: fix outdated comment in configure.am docs, tests: synchronize examples from docs to tests coverage: new test on parallel-tests TESTS runtime overriding cosmetics: fix typos in recent ChangeLog entries |