kc3-lang/automake/lib/am/check.am

lib/am/check.am

Branch

Log

Author Commit Date CI Message
Reuben Thomas 60e0e0e1 2025-04-18T14:20:05 doc: fix whitespace typo in comment. * lib/am/check.am: remove a spurious space after a hyphen.
Paul Eggert 61075eab 2025-01-01T14:31:02 maint: make update-copyright
Karl Berry cde0e3bb 2024-06-19T09:41:22 doc: rewrite subsecond NEWS entry, sanity.m4 "make" testing wrong, etc. * m4/sanity.m4 (_AM_FILESYSTEM_TIMESTAMP_RESOLUTION): mention that testing of "make" is wrong here; but $(MAKE) is not available? Do some special pleading that it won't matter in practice; we'll see. https://lists.gnu.org/archive/html/automake/2024-06/msg00051.html (AM_SANITY_CHECK): use $[]* instead of $[*]2 to avoid the confusing apparent (but not) use of shell arrays; the square brackets are parsed off by m4. * lib/am/check.am (output_system_information): en passant, mention that $(AWK) should always be defined since AC_PROG_AWK is part of AM_INIT_AUTOMAKE nowadays. * NEWS: rewrite subsecond NEWS entry:workaround, more references.
Collin Funk 6cb69ad8 2024-06-17T14:44:28 test: Improve output of system information in test-suite.log. Fixes https://bugs.gnu.org/71421. * lib/am/check.am ($(TEST_SUITE_LOG): Use $(AWK) instead of 'awk'. Remove an extra space in uname output. Start the line with "System information", and other sections with "Distribution information". * NEWS: add reference to #71421.
Bruno Haible a2d56238 2024-06-17T10:06:25 automake: allow omitting logs of skipped tests from test-suite.log. * doc/automake.texi (Parallel Test Harness): Describe the contents of test-suite.log. Mention IGNORE_SKIPPED_LOGS. (Command-line arguments for test drivers): Document the --collect-skipped-logs option. * lib/test-driver (print_usage): Document --collect-skipped-logs. (collect_skipped_logs): New variable. (gcopy): Use collect_skipped_logs. * lib/am/check.am (am__common_driver_flags): Add the contents of am__collect_skipped_logs. (am__check_pre): Set am__collect_skipped_logs. * t/parallel-tests-without-skipped-logs.sh: New file, based on t/parallel-tests-no-color-in-log.sh. * t/list-of-tests.mk (handwritten_TESTS): Add it. * NEWS: Mention the change.
Paul Eggert 1d35638b 2024-06-07T08:41:45 maint: spelling and whitespace fixes Most of these spelling fixes are just to comments and documentation. However, some affect tests as follows: * t/cond36.sh (tparse.h): Fix misspelling of dependency. * t/disthook.sh: Fix misspelling of file. * t/help3.sh: Fix misspelling of long option. * t/instdir-ltlib.sh: Fix misspellings of macro names. This causes the test to fail, so someone with libtool expertise needs to look into this. * t/tap-no-spurious-numbers.sh (highno): Fix misspelling of shell var.
Karl Berry 70127e61 2024-02-03T08:38:49 test: make bug-reporting message on test failure more explicit. From https://bugs.gnu.org/68855. * NEWS: mention this. * lib/am/check.am ($(TEST_SUITE_LOG)) <please_report>: new fn, new wording. * t/ax/test-defs.in (failure_footer_text): new wording. (failure_footer_text_colorized): add red. * t/ax/tap-summary-aux.sh (failure_footer): use failure_footer_text*. * t/tap-doc2.sh: change grep. * t/testsuite-summary-color.sh: likewise. * t/testsuite-summary-count-many.sh: likewise. * t/testsuite-summary-count.sh: likewise. * t/testsuite-summary-reference-log.sh: likewise.
Karl Berry d550b6ee 2024-01-27T18:07:48 test: output system information after summary report. From https://bugs.gnu.org/68746. * lib/am/check.am ($(TEST_SUITE_LOG)) <output_system_information>: new shell fn, called inline to write basic os information into test-suite.log. Also, question (do not change) the ".. contents:: :depth: 2" line; what is this for?
Paul Eggert b80b5c47 2024-01-01T11:29:06 maint: make update-copyright
Mike Frysinger f15852df 2023-01-03T21:46:38 rm: convert more cases to am__rm_f Fixes automake bug https://bugs.gnu.org/10828. Clean up a few more cases where we were doing `test ... || rm ...` to avoid calling `rm -f` without arguments by leveraging am__rm_f. These were harder to find in the source due to their constructed nature. The clean programs rules in particular were much more complicated than they needed to be. This logic boiled down to two things: delete the list of programs, and then delete the list without the exeext suffix, but only if the list of programs is non-empty. The check-TESTS rule was converted to am__rm_f, but a simplification was missed where the $list variable is inlined. * bin/automake.in: Delete test -z logic and always call am__rm_f. * contrib/check-html.am: Use $(am__rm_f) helper. * doc/automake.texi: Update examples to match current behavior. * lib/am/check.am: Inline $list variable. * lib/am/progs.am: Rewrite rule to use $(am__rm_f).
Mike Frysinger efc7dc40 2023-01-03T23:30:29 check: drop unused trs_list variable When the code that used this variable was removed, the variable itself was left behind. Clean that up now too. * lib/am/check.am: Delete trs_list.
Mike Frysinger 34bdde96 2023-01-04T02:00:14 maint: make update-copyright
Mike Frysinger 3099097d 2022-02-17T03:50:55 rm: handle -f w/no arguments gracefully Fixes automake bug https://bugs.gnu.org/10828. Delete the configure check that would abort if `rm -f` does not work, and delete the plans to make this a hard requirement in the future. Instead, test to see if `rm -f` fails w/out arguments. If it does, define am__rm_f such that it always passes at least the "" argument when deleting files. If it doesn't fail, then we can omit the "". Then go through lib/am/ and update places where we use the pattern `test -z ... || rm -f ...` and replace with $(am__rm_f). * NEWS: Mention support for `rm -f` w/out arguments. * PLANS/rm-f-without-args.txt: Remove. * lib/am/check.am: Use new $(am__rm_f) helper. * lib/am/clean.am: Likewise. * lib/am/header-vars.am: Likewise. * lib/am/inst-vars.am: Likewise. * lib/am/libs.am: Likewise. * lib/am/ltlib.am: Likewise. * lib/am/progs.am: Likewise. * lib/am/texinfos.am: Likewise. * m4/init.m4: Delete `rm -f` checks and call _AM_PROG_RM_F. * m4/rmf.m4: Define new _AM_PROG_RM_F macro. * t/rm-f-probe.sh: Update test.
Jim Meyering 6c8ff6a8 2022-01-12T14:15:12 maint: make update-copyright
Jim Meyering a470a47f 2021-07-11T19:19:42 maint: make update-copyright
Karl Berry 216d1834 2020-06-06T15:42:54 automake: support AM_TESTSUITE_SUMMARY_HEADER override. This change handles https://bugs.gnu.org/11745. * lib/am/check.am (AM_TESTSUITE_SUMMARY_HEADER): new variable. Default value is " for $(PACKAGE_STRING)", including quotes, to keep the default output the same. ($(TEST_SUITE_LOG)): use it, unquoted. * doc/automake.texi (Scripts-based Testsuites): document it. * NEWS: mention it. * t/testsuite-summary-header.sh: new test. * t/list-of-tests.mk (handwritten_tests): add it. * t/ax/testsuite-summary-checks.sh: fix typo.
Jim Meyering cf27a3df 2020-01-01T11:44:41 maint: make update-copyright
Paul Eggert 5ae02cc8 2019-10-14T13:46:55 maint: make update-copyright
Mathieu Lirzin bbaa4cdc 2018-01-04T16:19:30 maint: Update copyright years to 2018 This update has been made with 'make update-copyright'.
Mathieu Lirzin c2757b97 2017-09-19T13:43:07 maint: Reset master
Paul Eggert 3562e384 2017-09-16T13:03:36 Prefer https: URLs In Gnulib, Emacs, etc. we are changing ftp: and http: URLs to use https:, to discourage man-in-the-middle attacks when downloading software. The attached patch propagates these changes upstream to Automake. This patch does not affect files that Automake is downstream of, which I'll patch separately. Althouth the resources are not secret, plain HTTP is vulnerable to malicious routers that tamper with responses from GNU servers, and this sort of thing is all too common when people in some other countries browse US-based websites. See, for example: Aceto G, Botta A, Pescapé A, Awan MF, Ahmad T, Qaisar S. Analyzing internet censorship in Pakistan. RTSI 2016. https://dx.doi.org/10.1109/RTSI.2016.7740626 HTTPS is not a complete solution here, but it can be a significant help. The GNU project regularly serves up code to users, so we should take some care here.
Paul Eggert 199e7a44 2017-09-16T13:03:36 Prefer https: URLs In Gnulib, Emacs, etc. we are changing ftp: and http: URLs to use https:, to discourage man-in-the-middle attacks when downloading software. The attached patch propagates these changes upstream to Automake. This patch does not affect files that Automake is downstream of, which I'll patch separately. Althouth the resources are not secret, plain HTTP is vulnerable to malicious routers that tamper with responses from GNU servers, and this sort of thing is all too common when people in some other countries browse US-based websites. See, for example: Aceto G, Botta A, Pescapé A, Awan MF, Ahmad T, Qaisar S. Analyzing internet censorship in Pakistan. RTSI 2016. https://dx.doi.org/10.1109/RTSI.2016.7740626 HTTPS is not a complete solution here, but it can be a significant help. The GNU project regularly serves up code to users, so we should take some care here.
Mathieu Lirzin 8555e7b8 2017-07-16T00:11:07 check: Make 'check-TESTS' target depend on test dependencies The 'check' and 'recheck' targets are already depending on the progams and scripts required to run the test suite. The 'check-TESTS' target while not in the public API might be used by uninformed users, so it makes sense to add the same dependency to it. This fixes bug#27186. * lib/am/check.am [!%?SERIAL_TESTS%] (check-TESTS): Add dependency to '%CHECK_DEPS%'. * THANKS: Update.
Mathieu Lirzin d8add592 2017-03-02T18:55:53 maint: Update copyright years to 2017. This update has been made with 'make update-copyright'.
Jim Meyering 1370ce5f 2017-01-01T08:34:49 maint: update copyright dates for 2017 * all files: Run this command, using update-copyright from gnulib: UPDATE_COPYRIGHT_FORCE=1 \ UPDATE_COPYRIGHT_USE_INTERVALS=2 \ UPDATE_COPYRIGHT_MAX_LINE_LENGTH=79 \ update-copyright $(git ls-files)
Stefano Lattarini 5de75f07 2015-01-05T22:48:33 maint: update copyright years to 2015 (branch 'micro') Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
Stefano Lattarini a78f63c5 2014-04-21T15:10:54 maint: update copyright years We've been in 2014 already for few months now... Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
Stefano Lattarini ad9804e1 2013-12-30T23:21:03 parallel-tests: avoid possible implicit "make all" in test-suite.log rule This change fixes automake bug#16302. * lib/am/check.am ($(TEST_SUITE_LOG)): Avoid running "make $redo_logs" when $redo_logs expands to empty, since in that case we are actually ending up invoking a full "make all". That shouldn't be required, and can cause slowdowns for people implementing their extra "laziness wrappers" around check-TESTS (automake bug#16302). * NEWS: Update. Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
Stefano Lattarini 66ab18ae 2013-05-10T10:50:05 cosmetics: fix few typos, grammaros and missing whitespace * lib/am/*.am: In comments in some of these files. Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
Stefano Lattarini 7df8b28c 2012-12-31T18:18:37 maint: update copyright year for 2013 (in branch maint) Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
Stefano Lattarini c6cc3802 2012-08-13T14:05:40 Merge branch 'maint' * maint: tests: rework tests on AM_PATH_PYTHON cosmetics: fix typos and references in comments typofix: in a test diagnostic readme: fixlets to HACKING Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
Stefano Lattarini cbc0fc85 2012-08-13T12:10:46 cosmetics: fix typos and references in comments * lib/am/check.am: Here. * doc/automake.texi: And here. * t/*.sh: And in several of these tests. Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
Stefano Lattarini 2d5571e7 2012-07-22T16:38:47 check: support colorized testsuite output by default Fixes automake bug#11855. With this change, we make the 'color-tests' option enabled by default. That option remains still silently accepted as a no-op, for backward compatibility. The developer of a package is still free to disable testsuite coloring on a per-makefile basis (by adding "AM_COLOR_TESTS = no" to the Makefile.am) or on a whole-project basis (by AC_SUBST'ing AM_COLOR_TESTS to "no" in configure.ac). But now the user will be able to request the testsuite output to be colorized, if he really wants to: # With GNU make: make AM_COLOR_TESTS=yes check # With non-GNU make: make AM_COLOR_TESTS=yes AM_MAKEFLAGS=AM_COLOR_TESTS=yes check * NEWS: Update. * doc/automake.texi: Updated, and some related minor reformatting and rewording. * automake.in (handle_tests): No need to pass the transform '%COLOR%' when processing "check.am". * lib/am/parallel-tests.am: Remove use of the '%?COLOR%' transform; just act as if it were unconditionally TRUE. * t/color.sh: No need to specify 'color-tests' in 'AUTOMAKE_OPTIONS' nor in 'AM_INIT_AUTOMAKE'. * t/ax/testsuite-summary-checks.sh: Likewise. * t/ax/tap-summary-aux.sh: Likewise. * t/color2.sh: Likewise. Also, ensure that colorized testsuite output can be disabled by default by calling "AC_SUBST([AM_COLOR_TESTS], [no])". * t/tap-realtime.sh: Define 'AM_COLOR_TESTS' to "no" in Makefile.am, to avoid spurious colorization of the output due to the use of the 'expect' program. * t/color-tests-opt.sh: New test, check that the 'color-tests' option is still recognized as a no-op. * t/list-of-tests.mk: Add the new test. Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
Stefano Lattarini f2436f5b 2012-07-01T18:01:35 Merge branch 'maint' * maint: tests: don't trust the exit status of "make -k" for non-GNU makes parallel-tests: reimplement fix for bug#11791 tests setup: unset CDPATH tests setup: more namespace safeness tests setup: remove an unused variable tests setup: less hard-coding of the test subdirectory
Stefano Lattarini d5443e4c 2012-07-01T12:56:22 parallel-tests: reimplement fix for bug#11791 * lib/am/check.am: Here. The new implementation is shorter, slightly more efficient (requiring less forks), less brittle in the face of signals or unexpected interruptions in the make process, and should also be easier to merge in the 'ng/master' branch (as of now, due to the difficulties in merging our previous version of the fix in the 'ng/master' codebase, Automake-NG still lacks a fix for bug#11791). Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
Stefano Lattarini ca9b4d33 2012-07-01T10:14:23 Merge branch 'maint' * maint: parallel-tests: silence an overly verbose recipe tests: fix some uses of 'Exit', where 'exit' should now be used instead maintcheck: test scripts should be executable, check for that
Stefano Lattarini 167f68f4 2012-06-30T23:37:57 parallel-tests: silence an overly verbose recipe * lib/am/check.am (check-TESTS): Here, the part of the recipe removing the stale '.log' and '.trs' files. Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
Stefano Lattarini 5fb73b05 2012-06-30T22:17:22 Merge branch 'maint' * maint: parallel-tests: recipes for "check" and "recheck" are separated again
Stefano Lattarini e18ac4b8 2012-06-30T21:32:20 Merge branch 'maint' * maint: parallel-tests: "recheck" behaves better in case of compilation failures scripts: quote 'like this', not `like this' Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
Stefano Lattarini 6c9b282b 2012-06-30T20:53:39 parallel-tests: recipes for "check" and "recheck" are separated again * lib/am/check.am: Here. They have distinctly diverged recently, and the reduction in code duplication obtained keeping their recipes united is not anymore worth the extra complications. Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
Stefano Lattarini 2a3e45d6 2012-06-27T12:47:17 parallel-tests: "recheck" behaves better in case of compilation failures With this change, the "recheck" target behaves better in the face of build failures related to previously failed tests. For example, if a test is a compiled program that must be rerun by "make recheck", and its compilation fails, that test will still be rerun by further "make recheck" invocations. Previously, its '.log' and '.trs' would have both been lost, so that the test would have not been re-run. This change fixes automake bug#11791. * NEWS: Update. * lib/am/check.am (recheck, check-TESTS): Adjust to cater to scenario described above. * t/parallel-tests-recheck-pr11791.sh: New test. * t/list-of-tests.mk: Add it. Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
Stefano Lattarini 488de31e 2012-06-21T13:15:05 Merge branch 'maint' * maint: docs: avoid failures with development version of Texinfo (4.13.90) typofix: s/test derivers/test drivers/ in check.am comments typofix: s/env/even/ in comments in GNUmakefile
Stefano Lattarini 9c6ee0cd 2012-06-21T11:43:54 typofix: s/test derivers/test drivers/ in check.am comments Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
Stefano Lattarini 5e771b27 2012-05-18T18:00:51 check: the parallel-tests driver is now the default The old serial testsuite driver is still supported, and can be activated by the 'serial-tests' option (introduced by commit 'v1.11-1989-gdeb7773' of 2012-02-20, "automake: new option 'serial-tests'"). * lib/Automake/Options.pm: Now, the 'parallel-tests' option is a no-op (but still explicitly recognized, for backward compatibility), while the 'serial-tests' option is recognized and registered. * automake.in: Adjust to ensure that the parallel test harness is used by default; mostly, this amounts at using "! option('serial-tests')" where "option('parallel-tests')" was used before. * NEWS, doc/automake.texi: Update. * configure.ac (AM_INIT_AUTOMAKE): Drop 'parallel-tests' option, is redundant now. * defs: In the generated AM_INIT_AUTOMAKE call, use the 'serial-tests' when the serial testsuite harness is to be used, and no option otherwise. * lib/am/check.am: Use (static) conditional '%SERIAL_TESTS%' instead of conditional '%PARALLEL_TESTS%'. * lib/test-driver: Adjust heading comments. * t/ax/extract-testsuite-summary.pl: Likewise. * t/check-exported-srcdir.sh: Likewise. * t/check-subst.sh: Likewise. * t/java-compile-run-nested.sh: Likewise. * t/parallel-tests-dry-run-1.sh: Likewise. * t/parallel-tests-dry-run-2.sh: Likewise. * t/parallel-tests-interrupt.sh: Likewise. * t/parallel-tests-many.sh: Likewise. * t/test-metadata-global-log.sh: Likewise. * t/tests-environment-backcompat.sh: Likewise. * m4/init.m4 (AM_INIT_AUTOMAKE): Adjust a comment. * t/check8.sh: Likewise. * t/check-tests-in-builddir.sh: Likewise. * t/java-compile-run-flat.sh: Adjust a botched comment. * t/ax/tap-summary-aux.sh: Remove explicit use of the 'parallel-tests' option. * t/ax/testsuite-summary-checks.sh: Likewise. * t/aclocal-path-precedence.sh: Likewise. * t/dist-aux-many-subdirs.sh: Likewise. * t/parallel-tests-no-color-in-log.sh: Likewise. * t/parallel-tests-reset-term.sh: Likewise. * t/parallel-tests-subdir.sh: Likewise. * t/check-no-test-driver.sh: Fix setup to adjust to the changed semantics. Update heading comments. * t/parallel-tests-driver-install.sh: Adjust to the new semantic. * t/test-driver-is-distributed.sh: Simplified according to the new semantics. Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
Stefano Lattarini a16b838b 2012-05-05T16:39:10 Merge branch 'parallel-tests-performance' into maint * parallel-tests-performance: parallel-tests: separate different logs with an empty line
Stefano Lattarini 952d91c0 2012-05-05T16:33:36 parallel-tests: separate different logs with an empty line After commit v1.12-21-g5eeb366 "parallel-tests: optimize global log creation", the log files copied in the global log are not anymore separated by a blank line, causing unclear output like this: FAIL: test1 =========== output from test 1 FAIL: test2 =========== output from test 2 where we would want something like this: FAIL: test1 =========== output from test 1 FAIL: test2 =========== output from test 2 Fix this regression. * lib/am/check.am (am__create_global_log): Print an extra empty line after having displayed the content of a '.log' file. Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
Stefano Lattarini f9dbb0a6 2012-05-02T12:51:09 Merge branch 'parallel-tests-performance' into maint * parallel-tests-performance: parallel-tests: save few forks when possible parallel-tests: optimize global log creation parallel-tests: optimize 'recheck' target for speed coverage: add performance tests on some parallel-tests aspects
Stefano Lattarini 4c2dd6e0 2012-04-30T22:18:24 parallel-tests: save few forks when possible * lib/am/check.am (am__check_pre): Save some forks, by analyzing more carefully the value of '$@' to decide how to extract its dirname part, and whether that should be created as a directory. Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
Stefano Lattarini b105d40d 2012-04-30T20:44:50 color-tests: coloring can be forced on non-ANSI terminals as well Before this change, colorization of testsuite output was suppressed whenever the terminal was recognized to be a "dumb" one, incapable of handling ANSI coloring (i.e., when the environment variable TERM had a value of "dumb"). This happened even when the AM_COLOR_TESTS variable was set to a value of "always". Such a behaviour was suboptimal and slightly confusing; in fact, if a user wants to force coloring of testsuite output that is being redirected to a regular file, he should be able to do so even if his terminal is not capable of handling ANSI colors -- in fact, such terminal wouldn't be involved with the testsuite output in any way, so why should it be allowed to influence it? Thus, we now enable coloring of test output whenever AM_COLOR_TESTS is set to "always", irrespective of the value of the TERM environment variable. * NEWS: Update. * lib/am/check.am [%?COLOR%] (am__tty_colors): Activate colorization of testsuite output whenever AM_COLOR_TESTS has the value of "always". * t/ax/tap-summary-aux.sh: Export the TERM environment variable to "dumb" when forcing colorization of the testsuite output; this should *not* prevent such colorization from taking place, and we want to check that this expectation really holds. * t/ax/testsuite-summary-checks.sh: Likewise. * t/color.sh: Likewise, and adjust some grepping checks. * t/tap-color.sh: Likewise. Also, remove redundant "make check" invocation since we are at it. * t/color2.sh: Likewise, and check that exporting TERM=dumb actually prevents testsuite output colorization when AM_COLOR_TESTS is unset. * t/parallel-tests-reset-term.sh: Relax, to prevent it from failing spuriously due to the new semantic. Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
Stefano Lattarini 5eeb3660 2012-02-26T00:30:18 parallel-tests: optimize global log creation With this change, the time required to execute the test case 'testsuite-summary-speed.sh' has dropped as follows: + Slow Debian i686 system, 1 core x 1.5 GHz, 768 MB of RAM, GNU make 3.81: - 1 run: 4 minutes => 30/40 seconds + Fast Solaris 10 i686 system, 4 cores x 3 GHz, 20 GB of RAM, Solaris CCS make: - 2 runs: 6 minutes => 30 seconds + Very fast Fedora ppc64 system, 64 cores x 3.5 GHz, 64 GB of RAM, GNU make 3.82: - 5 runs: 7 minutes => 1 minute and 10 seconds * lib/am/check.am ($(TEST_SUITE_LOG)): Optimize for speed, by avoiding lots of forks with the help of ... (am__create_global_log): ... this new internal variable, basically defining a smart awk program, and ... (am__global_test_result_rx, am__copy_in_global_log_rx): ... these new internal variables, used by the one above. (am__rst_section): Remove as obsolete * t/rst-formatting.trs: Remove. * t/list-of-tests.mk: Update. * t/testsuite-summary-count-many.sh: Improve its stress testing by using 1 million tests rather than just 5 thousands: we can afford this with our new optimization, and still have the test completed in a reasonable time. Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
Stefano Lattarini f5733aab 2012-04-26T22:25:35 parallel-tests: optimize 'recheck' target for speed With this change, the time required to execute the test case 'testsuite-recheck-speed.sh' has dropped as follows: + Slow Debian i686 system, 1 core x 1.5 GHz, 768 MB of RAM, GNU make 3.81: - 1 run: 6 minutes => 40 seconds + Fast Solaris 10 i686 system, 4 cores x 3 GHz, 20 GB of RAM, Solaris CCS make: - 4 runs: 3 minutes => 1 minute + Very fast Fedora ppc64 system, 64 cores x 3.5 GHz, 64 GB of RAM, GNU make 3.82: - 5 runs: 4 minutes => 1 minute 30 seconds * lib/am/check.am (recheck): Optimize for speed, by avoiding lots of forks with the help of ... (am__list_recheck_tests): ... this new internal variable, basically defining a proper awk program, and ... (am__recheck_rx): ... this other new internal variable, used by the one above. * t/test-trs-recover2.sh: Relax by not checking for a very corner case ('.log' and '.trs' files both unreadable) that we don't handle anymore. Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
Stefano Lattarini f46359b4 2012-04-18T11:40:15 parallel-tests: 'recheck' must depend on 'all' Fixes automake bug#11252. When a developer experience one or more failures in the testsuite, a good workflow is for him to modify its program's sources to fix the bug thus revealed, run "make recheck" to verify that the change has indeed solved the testsuite failures previously experienced, and then run "make check" to verify that the change has not introduced any new failure or regression. Unfortunately, this apparently natural workflow couldn't have worked until now, since the Automake-provided 'recheck' target (which didn't depend on 'all') wouldn't have causes the program to be recompiled, and the failed tests would have thus been run with the older, buggy version of the program, failing the same way as before. * lib/am/check.am (recheck): Depend on 'all'. * t/parallel-tests-recheck-depends-on-all.sh: New test. * t/list-of-tests.mk: Add it. * NEWS: Update. Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
Stefano Lattarini 944ec778 2012-04-14T09:55:41 parallel-tests: print error messages on stderr, not stdout * lib/am/check.am ($(TEST_SUITE_LOG)): Be sure to print error messages on stderr, not on stdout. Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
Stefano Lattarini de58d25e 2012-02-23T15:15:26 am: do not quote `like this', as per GCS recommendation This patch converts the automake-provided '*.am' fragments, and related files, to the use of new quoting format 'like this' or "like this" rather than `like this'. * lib/am/check.am: Update quoting format throughout, in comments and diagnostic. Some related rewordings, reformatting, and removal of redundant commands since we are at it. * lib/am/configure.am: Likewise. * lib/am/dejagnu.am: Likewise. * lib/am/depend2.am: Likewise. * lib/am/distdir.am: Likewise. * lib/am/inst-vars.am: Likewise. * lib/am/install.am: Likewise. * lib/am/lang-compile.am: Likewise. * lib/am/lisp.am: Likewise. * lib/am/ltlib.am: Likewise. * lib/am/mans.am: Likewise. * lib/am/progs.am: Likewise. * lib/am/remake-hdr.am: Likewise. * lib/am/subdirs.am: Likewise. * lib/am/tags.am: Likewise. * lib/am/texi-vers.am: Likewise. * lib/am/texibuild.am: Likewise. * lib/am/texinfos.am: Likewise. * lib/am/yacc.am: Likewise.
Stefano Lattarini 641a5a4b 2012-02-16T10:46:23 maint: run "make update-copyright"
Stefano Lattarini b2c1adfe 2012-02-14T21:23:04 check: fix test-suite.log formatting with NetBSD sed * lib/am/check.am (am__rst_section): Tweak to ensure en extra final newline is printed also with NetBSD sed. Issue revealed by a failure of 'test-metadata-global-result.test'. * tests/rst-formatting.test: New test, checking the internal variables '$(am__rst_title)' and '$(am__rst_section)'. * tests/list-of-tests.mk: Add it.
Stefano Lattarini d029740b 2012-02-05T21:22:01 parallel-tests: more uses for '$(am__make_dryrun)' With this change we refactor some recipes of the parallel-tests harness to make them use, where appropriate, the new internal variable '$(am__make_dryrun)' (introduced in the 'maint' branch) when they need to determine whether make is running in "dry mode", i.e., with "make -n". This allows for some simplifications and for improved uniformity. * lib/am/check.am (check-TESTS recheck): Use $(am__make_dryrun) and simplify accordingly. (am--redo-logs): Delete this internal helper recipe, merged into ... ($(TEST_SUITE_LOG)): ... this, with the help of $(am__make_dryrun). recipe accordingly.
Stefano Lattarini da0964eb 2012-01-23T21:02:05 refactor: new variable telling whether make is running in "dry mode" We define a new variable '$(am__make_dryrun)', that can be used in recipes to determine whether make is running in "dry mode" (as with "make -n") or not. This is useful in rules that invoke make recursively, and are thus executed also with "make -n" -- either because they are declared as dependencies to '.MAKE' (NetBSD make) or because their recipes contain the "$(MAKE)" string (GNU and Solaris make). * lib/am/header-vars.am (am__make_dryrun): New variable. * lib/am/check.am (check-TESTS recheck): Use it, and simplify recipe accordingly. * lib/am/distdir.am (distdir): Likewise. * lib/am/lisp.am ($(am__ELCFILES)): Likewise.
Stefano Lattarini 0a9d201c 2012-01-17T15:06:05 check: move ".log -> .html" conversion in contrib That feature has been deprecated in the 1.11.x series, and marked for removal from the automake core in the 1.12 release. Here we implement such removal. Reference: <http://lists.gnu.org/archive/html/automake/2012-01/msg00005.html> See also commit 'v1.11-650-g20f2ac9'. * doc/automake.texi, NEWS: Update. * lib/am/check.am: Don't include 'check-html.am' anymore. * lib/am/check-html.am: Move ... * contrib/check-html.am: ... to this, and adjust comments. * lib/am/Makefile.am (dist_am_DATA): Update. * contrib/Makefile.am (EXTRA_DIST): Likewise. * tests/parallel-tests2.test: Adjust. * tests/parallel-tests-dryrun.test: Remove checks on the ".log -> .html" conversion. * tests/test-driver-custom-html.test: Remove as obsolete. * tests/test-driver-custom-no-html.test: Likewise. * tests/tap-html.test: Likewise. * tests/list-of-tests.mk: Don't list removed tests anymore.
Stefano Lattarini 9928ea5c 2012-01-16T20:06:08 Merge branch 'maint' * maint: recheck: fix interaction with "make -n" vala: avoid potential useless remakes (minor bugfix) vala: enhance tests recheck: behave better with non-GNU make check: separate .log -> .html conversion from core testsuite harness docs: deprecate .log -> .html conversion by parallel-tests tests: list some forgotten test cases in $(TESTS) maintcheck: consistency of list of test scripts build: explicitly declare some targets as .PHONY maint: remove obsolete/broken maintainer targets build: improve silencing of automake build system tests: move list of tests in its own Makefile fragment + Extra non-trivial edits: * tests/parallel-tests2.test: Obvious edits to merge the slightly inconsistent enhancements to coverage done in master with those done in maint. * lib/am/check.am (check-TESTS, recheck): Sweeping and rather complex changes to merge the considerable divergences between maint and master in a way that is meaningful and, where possible, retains the semantic changes from both maint and master.
Stefano Lattarini 8fec2399 2012-01-13T22:39:47 recheck: fix interaction with "make -n" * lib/am/check.am (recheck): Ensure the recipe does not erroneously remove '.log' files when running under "make -n". For the sake of NetBSD make, this also means that ... (.MAKE): ... this cannot depend on 'recheck' anymore. * tests/parallel-tests-dryrun.test: New test. * tests/list-of-tests.mk: Add it.
Stefano Lattarini d9a7182e 2012-01-13T18:53:43 recheck: behave better with non-GNU make * automake.in (handle_tests): Also substitute '%CHECK_DEPS%' with the value of the '@check' array, containing the list of early test dependencies like '$(check_SCRIPTS)', $(check_PROGRAMS)', etc. * lib/am/check.am (recheck): Explicitly depend on '%CHECK_DEPS%'. (check, recheck): Unify the implementation of their recipes, with different code paths taken depending on the name of the target. This makes the implementation of "recheck" less brittle in the process (especially when non-GNU make and AM_MAKEFLAGS overriding are involved). * lib/am/check-html.am (check-html, recheck-html): Likewise. * tests/check.test: Adjust to avoid spurious failure. * tests/parallel-tests2.test: Improve coverage.
Stefano Lattarini 20f2ac90 2012-01-13T17:56:13 check: separate .log -> .html conversion from core testsuite harness That feature is to be deprecated in the 1.11.x series, and removed from the automake core in the 1.12 release, where it will instead be offered in a semi-independent extra '*.am' fragment. So let's start better separating the .log -> .html conversion from the "core code" of the parallel-tests harness. Reference: <http://lists.gnu.org/archive/html/automake/2012-01/msg00005.html> * lib/am/check.am (.log.html, check-html, recheck-html): Move these targets ... * automake.in (handle_tests): ... and the initialization of the TEST_SUITE_HTML variable and the cleaning of the $(TEST_SUITE_HTML) file ... * lib/am/check-html.am: ... in this new file, with related (minor) refactorings, enhancements and simplifications. * lib/am/check.am (.MAKE. PHONY, AM_RECURSIVE_TARGETS): Adjust. * lib/am/Makefile.am (dist_am_DATA): Add 'check-html.am'. * tests/parallel-tests2.test: Improve coverage.
Stefano Lattarini d1fbc759 2012-01-05T20:09:24 Merge branch 'maint' * maint: parallel-tests: avoid issue with overly long lines in sed input Additional edits: * NEWS: Remove entry from merged commit, since that will also be present in the NEWS file of automake 1.11.3. * lib/am/check.am (check-TESTS, recheck, recheck-html): Rework merged changes to adapt them to the different form these recipes has assumed in the master branch (w.r.t. the maint branch).
Stefano Lattarini e3b0e124 2012-01-05T15:13:30 parallel-tests: avoid issue with overly long lines in sed input See automake bug#10437: <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=10437> and coreutils bug#10427: <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=10427#8> * lib/am/check.am (recheck, recheck-html): In order to strip trailing whitespace from the definition of the `$list' variable, we used to invoke sed in a way that could cause it to get passed overly long input lines, causing spurious failures. So rework the logic of the recipe to avoid any sed invocation, relying on simpler shell idioms instead. (check-TESTS): Reorganize the recipe to be more similar to the one of `recheck', for consistency and simplicity. * NEWS: Update. Report and analysis by Paul Eggert.
Stefano Lattarini 94aa5869 2011-08-18T21:53:34 Merge branch 'master' into test-protocols * master: * THANKS (Daniel Richard G.): Update e-mail address. java: avoid compilation errors when CLASSPATH is empty parallel-tests: no more spurious successes for FreeBSD make hacking: we now require autoconf 2.68
Stefano Lattarini 8c4d0f06 2011-08-18T21:47:11 Merge branch 'maint' * maint: * THANKS (Daniel Richard G.): Update e-mail address. java: avoid compilation errors when CLASSPATH is empty parallel-tests: no more spurious successes for FreeBSD make hacking: we now require autoconf 2.68
Stefano Lattarini b3afcecf 2011-08-12T18:29:28 parallel-tests: no more spurious successes for FreeBSD make Work around a bug of FreeBSD make bug that was causing the automake-generated "check" target to complete with success even if some tests failed; this happened only when FreeBSD make was run in concurrent mode (as in, e.g., "make -j2 check"). The bug is not present in NetBSD make. This change fixes automake bug#9245: <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9245> See also FreeBSD PR bin/159730: <http://www.freebsd.org/cgi/query-pr.cgi?pr=159730> * lib/am/check.am [%?PARALLEL_TESTS%] $(TEST_SUITE_LOG): Use a more "safe" (and apparently redundant) idiom to exit with error, so that the non-zero exit status is picked up also by FreeBSD make when it's running in concurrent mode. * NEWS: Update. * tests/check-concurrency-bug9245.test: New test. * tests/Makefile.am (TESTS): Update.
Stefano Lattarini 7fff2880 2011-08-10T15:37:27 test harness: avoid possible fork bomb This fixes a regression w.r.t. the master branch, exposed by test 'parallel-tests-fork-bomb.test'. * lib/am/check.am (am--redo-logs): Detect possible infinite recursion due to a test log in $(TEST_LOGS) being the same as $(TEST_SUITE_LOG). * tests/parallel-tests-fork-bomb.test: Enhance and extend a little now that we have more explicit semantics.
Stefano Lattarini 02dfe16d 2011-08-09T23:01:23 test harness: recursive make invocations must use $(AM_MAKEFLAGS) * lib/am/check.am ($(TEST_SUITE_LOG)): Pass $(AM_MAKEFLAGS) to $(MAKE) when issuing a recursive make call. Problem detected by the `sc_tests_make_without_am_makeflags' maintainer check.
Stefano Lattarini 1e0238ad 2011-08-08T13:18:50 test harness: more minor VPATH-rewrite issues * tests/am/check.am (am__set_TESTS_bases, $(TEST_SUITE_LOGS), am--redo-logs, recheck, recheck-html): Cosmetic fixlets to minimize the risk of unwanted VPATH rewrites. (check-TESTS): Likewise, and normalize trailing whitespace since we are at it. Bugs exposed by test cases `check6-p.test' and `check7-p.test'.
Stefano Lattarini bf15d302 2011-08-08T12:50:12 test harness: "make -n check" works correctly with Solaris make To avoid some recipes being executed with "make -n", we used the trick of issuing recursive make calls not directly with $(MAKE), but with a simple indirection instead: ... am__stealth_MAKE = $(MAKE) ... target: ... $(am__stealth_MAKE) ... ... This hack worked with GNU make, but, unfortunately, not with Solaris make. The present change removes the hack, and uses more proper idioms to avoid unwanted recipe executions with "make -n". Bug exposed by test case `parallel-tests-dry-run.test'. * lib/am/check.am (am__stealth_MAKE): Remove, no more used. (.log.trs): Break recipe in multiple shell invocations. ($(TEST_SUITE_LOG)): Likewise, and move some corner-case parts of the recipe to ... (am--redo-logs): ... this new internal helper .PHONY target.
Stefano Lattarini 232dcc6d 2011-08-08T09:31:03 test harness: fixlet in workaround for VPATH rewrite issue Fixes a blunder in previous commit `v1.11-983-gfda3de5'. * lib/am/check.am (am__set_TESTS_bases): Add forgotten end-of-line anchor to sed expression. In comments, add reference to ... * tests/test-trs-basic.test: ... this test, which has been adapted and tightened (and tweaked).
Stefano Lattarini fda3de57 2011-08-08T01:23:42 test harness: work around a VPATH rewrite issue * lib/am/check.am (am__TEST_BASES): Removed, it's role taken over by ... (am__set_TESTS_bases): ... these new variable. ($(TEST_SUITE_LOG): Use it, to avoid VPATH rewrite issues. * automake.in (handle_tests): Update the code for the cleanup of the `.trs' file to use `$(TEST_LOGS)' instead of `$(am__TEST_BASES)'. * tests/test-harness-vpath-rewrite.test: New test, better exposing the problem fixed by this change * tests/Makefile.am (TESTS): Add it. * tests/test-trs-basic.test: Update and extend.
Stefano Lattarini 152fcd20 2011-08-08T00:16:56 parallel-tests: work around Solaris XPG4 make segfault Issue exposed by test `posixsubst-tests-p.test', and similar to the problem solved by commit `v1.11-159-ge7aa360'. * lib/am/check.am [%?PARALLEL_TESTS%] (check-TESTS): Trim trailing whitespace from $list, to avoid triggering a nasty bug (potential segfault) on Solaris XPG4 make and Heirloom make.
Stefano Lattarini a0d4d97a 2011-08-05T14:16:19 check: update comments to reflect recent heavy changes * lib/am/check.am [%?PARALLEL_TESTS%]: Update the comments to reflect the recent changes. Remove quite many comments that were merely duplicating excerpts from the Automake manual.
Stefano Lattarini 4839ca92 2011-08-05T12:08:20 check: small cleanups and refactorings in test harness and drivers * lib/tap-driver (yn): New subroutine, converts a boolean value to a "yes" or "no" string. (write_test_results): Use it, and related reformatting. * lib/test-driver (trap): Also remove the `.trs' file on signals, for extra safety. (fatal): Remove this function, it's never used. ($logfile, $trsfile): Renamed ... ($log_file, $trs_file): ... to these, for clarity, and in order to be more consistent with the `tap-driver' script. Improve a couple of comments. * automake.in (handle_tests): Don't define anymore the now-obsolete make macro `$(TEST_LOGS_TMP)', nor add it to the list of files to be removed upon "make mostlyclean". * lib/am/check.am ($(TEST_SUITE_LOG)): New shell function `f_ok', tells whether a path refers to an existing, regular, readable file. Use it throughout. (recheck): Be safer w.r.t. make implementation that run recipes with `errexit' shell flag active.
Stefano Lattarini 956c58ab 2011-08-05T09:28:17 test harness: be aware of more metadata, simplify test drivers * lib/am/check.am ($(TEST_SUITE_LOG)): When producing the global test log, take into account the new metadata field `:global-test-result:, to write nicely formatted RST section titles, instead of leaving this chore to the individual test drivers. (am_rst_section): Re-introduce this variable, as removed in commit `v1.11-870-ga27c9c4'. * lib/test-driver, lib/tap-driver: Write the global test result as metadata in the `.trs' file, not as (part of) a formatted RST subsection title in the `.log' file. Related simplifications; in particular, get rid of temporary files usage. * doc/automake.texi (Log files generation and test results recording): Document the new metadata. * tests/check12.test: Update. * tests/parallel-tests-harderror.test: Likewise. * tests/parallel-tests-interrupt.test: Likewise. * tests/parallel-tests-log-override-2.test: Likewise. * tests/parallel-tests-log-override-recheck.test: Likewise. * tests/parallel-tests-unreadable.test: Likewise. * tests/tap-global-result.test: Likewise. * tests/test-metadata-results.test: Likewise. * tests/test-log.test: Likewise, and another minor unrelated fix. * tests/test-metadata-global-result.test: New test. * tests/Makefile.am (TESTS): Update.
Stefano Lattarini 0c81b43f 2011-07-28T12:25:22 test harness: use new `.trs' files to hold test metadata With this change, the test harness will keep test metadata in dedicated `.trs' files, instead of having them embedded into the `.log' files. This allows for easier forward-compatibility and extension of test metadata, and for more flexibility in the format of the `.log' files. Note that this change makes the `:end-metadata:' field obsolete. * doc/automake.texi (Parallel Test Harness, Log files generation and test results recording): Document the new APIs and behaviour; some related minor rewordings and fixlets. * NEWS: Update. * automake.in (handle_tests): When bringing in the content of `check2.am', substitute %BASE% with the basename of the `.log' file being created by a rule. Add the generated `.trs' files to the list of files to be cleaned by "make mostlyclean". * lib/am/check.am (am__test_driver_flags): Rename ... (am__common_driver_flags): ... to this, and remove the flags `--test-name' and `--log-file' from it: they are now define in the proper rules in `check2.am'. (am__TEST_BASES): New internal variable, holding the names of the tests, with any registered extension removed. (am__stealth_MAKE): New internal variable, can be used instead of $(MAKE) in recipes requiring a recursive call to make, but which are not intended to be executed by "make -n". (.log.trs): New suffix rule, to recover from deletion of `.trs' files. ($(TEST_SUITE_LOG)): Almost completely rewritten to follow the new API of "test logs in `.log' files, test metadata in `.trs' files". It goes to some length to work correctly in face of unreadable or missing `.log' and `.trs' files, and to error out with proper error messages when this is not possible. [%?PARALLEL_TESTS%] (check-TESTS): Also remove relevant "stale" `.trs' files (in addition to `.log files) before remaking the $(TEST_SUITE_LOG). (recheck, recheck-html): Look for the `:recheck:' field in the `.trs' files, not in the `.log' files. * lib/am/check2.am (?GENERIC?%EXT%.log, ?!GENERIC?%OBJ%): Adjust the call to the test driver, in particularly passing the new option `--trs-file'. [%am__EXEEXT%] (?GENERIC?%EXT%$(EXEEXT).log): Likewise. * lib/tap-driver ($USAGE): Adjust the help screen. (Getopt::Long::GetOptions): Handle the `--trs-file' option, through the use of ... ($trs_file): ... this new global variable. (finish): Write metadata for the test run to `$trs_file' rather then to `$log_file', through the use of ... (write_test_results): ... this new function. * lib/test-driver (print_usage): Update the help screen. (Option parsing): Handle the `--trs-file' option, through the use of ... ($resfile): ... this new global variable. (Main code): Write metadata for the test run to `$trsfile' rather than to `$logfile'. Minor related adjustments to comments. * tests/.gitignore: Ignore `*.trs' files. * tests/parallel-tests-unreadable-log.test: Moved ... * tests/parallel-tests-unreadable.test: ... to this, and extended to also check the semantics for unreadable `.trs' files. * tests/test-driver-end-metadata.test: Deleted as obsolete. * tests/test-driver-metadata-no-leading-space.test: Likewise. * tests/test-driver-global-log.test: Renamed ... * tests/test-metadata-global-log.test: ... to this, and modified as to verify the new APIs and semantics. * tests/test-driver-recheck: Renamed ... * tests/test-metadata-recheck.test: ... to this, and modified likewise. * tests/parallel-tests-once.test: New test. * tests/parallel-tests-make-n.test: Likewise. * test-metadata-results.test: Likewise. * test-missing.test: Likewise. * test-missing2.test: Likewise. * test-trs-basic.test: Likewise. * test-trs-recover.test: Likewise. * test-trs-recover2.test: Likewise. * tests/Makefile.am (TESTS): Update.
Stefano Lattarini 4014b502 2011-07-26T14:51:20 test harness: allow more metadata in log files This change reworks and improves the parallel test harness to use more specialized reStructuredText fields in the log files (instead of relying on specially-placed of "magic lines" and more indirect semantical formatting); the new fields are the following: - ":recheck:": tell whether the associated test will have to be re-run by "make recheck"; - ":copy-in-global-log:": tell whether the content of the log file should be copied in the "global log" `test-suite.log'; - ":end-metadata:", which inhibits the scanning of the rest of the log file (for what concerns test metadata). Also, the special `:test-result:' value "END" has been removed, superseded by the new `:end-metadata:' field. * doc/automake.texi (Log files generation and test results recording): Document the new API and semantics. Remove or fix some obsolete comments. * lib/am/check.am ($(TEST_SUITE_LOG), recheck, recheck-html): Adjust comments and code. * lib/tap-driver (finish): Adjust, with the help of ... (must_recheck, copy_in_global_log): ... these new functions. * lib/test-driver (Main code): Adjust, with the help of ... ($recheck, $gcopy): ... these new variables. * tests/trivial-test-driver: Update to obey the new APIs. * tests/test-driver-recheck.test: Likewise. * tests/test-driver-global-log.test: Likewise. * tests/tap-passthrough.test: Relax the test, avoiding to check what is written in `test-suite.log'; such check has been moved ... * tests/tap-global-log.test: ... in this new test, and extended. * tests/test-driver-metadata-no-leading-space.test: New test. * tests/test-driver-end-test-results.test: Removed, it checked the old APIs; superseded by ... * tests/test-driver-end-metadata.test: ... this new test. * tests/tap-log.test: Improve syncing with ... * tests/test-log.test: ... this new test. * tests/parallel-tests.test: Remove some duplication w.r.t. this last new test. Updated heading comments. * tests/Makefile.am (TESTS): Update.
Stefano Lattarini b9e9d546 2011-07-24T16:47:24 simple tests: support developer-defined fd redirections Motivated by coreutils bug#8846, and related discussions: <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8846> <http://lists.gnu.org/archive/html/bug-autoconf/2011-06/msg00002.html> In those threads it has been shown how problematic it can be to do portable file descriptor redirections in the testsuite when using the Automake testsuite harness. This change should remedy to that situation. * lib/am/check2.am (?GENERIC?%EXT%$(EXEEXT).log, ?GENERIC?%EXT%.log, ?!GENERIC?%OBJ%): Append redirection defined in $(AM_TESTS_FD_REDIRECT) to the command-line invocations of the test scripts. * lib/am/check.am [!%?PARALLEL_TESTS%] $(check-TESTS): Likewise. * NEWS: Update. * doc/automake.texi (Script-based Testsuites): Document the new feature. * tests/check-fd-redirect.test: New test. * tests/parallel-tests-fd-redirect.test: Likewise. * tests/parallel-tests-am_tests_environment.test: Remove checks about the use of redirections in AM_TESTS_ENVIRONMENT: they would check deprecated (if not undefined) behaviour now. Strengthen a couple of still valid checks, to keep the test more in sync with the documentation. Improve debugging information. * tests/Makefile.am (TESTS): Update.
Stefano Lattarini eaac33fe 2011-07-01T11:12:31 parallel-tests: simplify testsuite summary Prefer a more deterministic, "tabular" format for the testsuite summary, always listing the numbers of passed, failed, xfailed, xpassed, skipped and errored tests, even when these numbers are zero. This simplify the logic of testsuite summary creation, makes it more easily machine-parseable, and will probably allow for easier addition of new kinds of test results in the future. * lib/am/check.am (am__tty_colors_dummy): New make variable, to reduce code duplication. Extracted from previous versions of $(am__tty_colors), and extended by defining two new variables `$mgn' and `$brg'. [%?COLOR%, %!?COLOR%] (am__tty_colors): Use that new variable. (am__text_box): Delete, is not needed anymore. ($(TEST_SUITE_LOG)): Rewrite associated rules to implement the new testsuite summary format. * NEWS: Update. * tests/check10.test: Don't run with the parallel-tests harness too, that makes no sense anymore. * tests/color.test: Update and adjust. * tests/color2.test: Likewise. * tests/parallel-tests.test: Likewise. * tests/parallel-tests3.test: Likewise. * tests/parallel-tests6.test: Likewise. * tests/parallel-tests9.test: Likewise. * tests/parallel-tests-unreadable-log.test: Likewise. * tests/parallel-tests-empty-testlogs.test: Likewise. * tests/parallel-tests-log-override-recheck.test: Likewise. * tests/parallel-tests-no-spurious-summary.test: Likewise. * tests/test-driver-custom-multitest.test: Likewise. * tests/test-driver-end-test-results.test: Likewise. * tests/parallel-tests-no-color-in-log.test: New test. * tests/testsuite-summary-color.test: Likewise. * tests/testsuite-summary-count.test: Likewise. * tests/testsuite-summary-count-many.test: Likewise. * tests/testsuite-summary-reference-log.test: Likewise. * tests/testsuite-summary-checks.sh: New auxiliary script, used by the new tests above. * tests/extract-testsuite-summary: Likewise. * tests/trivial-test-driver: Optimize for speed when there are lots of of tests. * tests/Makefile.am (EXTRA_DIST): Distribute them. (testsuite-summary-color.log, testsuite-summary-count.log): Depend on them. (testsuite-summary-count-many.log): Depend on the auxiliary scripts 'trivial-test-driver' and 'extract-testsuite-summary'. (TESTS): Update.
Stefano Lattarini 15400bca 2011-06-30T14:55:43 parallel-tests: new recognized test result 'ERROR' * lib/am/check.am ($(TEST_SUITE_LOG)): Recognize a new test result `ERROR'. Use it when encountering unreadable test logs (previously a simple `FAIL' was used in this situations). * lib/test-driver: Set the global test result to `ERROR' when the test exit status is 99. When doing colorized output, color `ERROR' results in magenta. * doc/automake.texi (Log files generation and test results recording): Update by also listing `ERROR' among the list of valid `:test-results:' arguments. * NEWS: Update. * tests/trivial-test-driver: Update. * tests/parallel-tests.test: Likewise. * tests/parallel-tests-harderror.test: Likewise. * tests/parallel-tests-no-spurious-summary.test: Likewise. * tests/test-driver-global-log.test: Likewise. * tests/test-driver-recheck.test: Likewise. * tests/test-driver-custom-multitest-recheck.test: Likewise. * tests/test-driver-custom-multitest-recheck2.test: Likewise. * tests/test-driver-custom-multitest.test: Likewise. * tests/test-driver-custom-no-html.test: Likewise. * tests/test-driver-end-test-results.test: Likewise. * tests/color.test: Likewise. Also, make stricter, and also test from VPATH. * tests/color2.test: Likewise, and improve syncing with color.test. * tests/parallel-tests-exit-statuses.test: New test. * tests/parallel-tests-console-output.test: Likewise. * tests/Makefile.am (TESTS): Update.
Stefano Lattarini 61b2bb0d 2011-06-29T16:02:52 parallel-tests: make parsing of test results safer The new code for parsing the testsuite-generated `.log' files, as introduced in commit `v1.11-872-gc96b881', considers each `:test-result:' field anywhere in a `.log' file as a declaration of a test result, and accounts for it as such in the testsuite summary. Unfortunately this could easily cause spurious test failures being reported in the testsuite summary. This happened in practice with the Automake's own testsuite; for example: $ make check TESTS='check12-p.test'; echo exit: $? ... PASS: check12-p.test ===================================== 4 of 5 tests failed See tests/test-suite.log Please report to bug-automake@gnu.org ===================================== make[2]: *** [test-suite.log] Error 1 make: *** [check-am] Error 2 exit: 2 This change introduces a new special `:test-result:' "END", that, when seen, prevents the rest of the log file from being parsed. For more information, refer to the thread: <http://lists.gnu.org/archive/html/automake-patches/2011-06/msg00199.html> * lib/am/check.am ($(TEST_SUITE_LOG)): Stop the parsing of a log file as soon as the special ":test-result:END" directive is seen. Related changes and enhancements. * lib/test-driver: Protect the rest of the log after the result lined with a ":test-result:END" directive. * tests/parallel-tests-no-spurious-summary.test: New test. * tests/test-driver-end-test-results.test: Likewise. * tests/Makefile.am (TESTS): Update.
Stefano Lattarini 14027413 2011-06-22T21:59:34 docs: document custom test drivers and protocols * doc/automake.texi (Simple Tests): Note that the TESTS_ENVIRONMENT use suggested here is not portable to 'parallel-tests'. (Simple Tests using parallel-tests): Document new restrictions on the uses of TESTS_ENVIRONMENT and AM_TESTS_ENVIRONMENT. (Custom Test Drivers): New section and node. (Overview of Custom Test Drivers Support): New subsection. (Declaring Custom Test Drivers in @file{Makefile.am}): Likewise. (APIs for Custom Test Drivers): Likewise. (Options): Update description of color-tests. * lib/am/check ($(TEST_SUITE_LOG)): Remove comments that have been moved into the manual. (recheck, recheck-html): Minor adjustments to better conform to the documentation (this should cause no semantic changes w.r.t. the former behaviour); minor improvements and extensions to existing comments. * tests/test-driver-create-log-dir.test: New test. * tests/test-driver-strip-vpath.test: Likewise. * tests/test-driver-global-log.test: Likewise. * tests/test-driver-recheck.test: Likewise. * tests/Makefile.am (TESTS): Update.
Stefano Lattarini c96b881f 2011-06-03T22:50:56 parallel-tests: allow each test to have multiple results With this change, we improve the code creating the `test-suite.log' global log and the console testsuite summary to make it able to grasp multiple results per test script. This is required in order to introduce the planned support for test protocols, like TAP and SubUnit, which can indeed run multiple tests per test script, each with its individual result. The implementation makes use of a custom reStructuredText field `:test-result:'. Note that no new documentation is added by this change; that is be left for follow-up changes. * lib/check.am ($(TEST_SUITE_LOG)): When processing .log files, recognize a report of a test's result only if it is declared with the custom `:test-result:' reStructuredText field placed at the beginning of a line. Extend and add explanatory comments. (recheck, recheck-html): Add explanatory comments. * lib/test-driver: Write an appropriate reStructuredText field `:test-result:' in the generated log file. Use a reStructuredText transition to better separate the test outcome report from the test script's registered output. Improve comments. * tests/test-driver-custom-xfail-tests.test: Adjust. * tests/parallel-tests7.test: Adjust. * tests/parallel-tests-empty-testlogs.test: New test. * tests/parallel-tests-recheck-override.test: Likewise. * tests/parallel-tests2.test: Extend and keep more in-sync with ... * tests/test-driver-custom-html.test: ... this new related test. * tests/test-driver-custom-no-html.test: New test. * tests/test-driver-custom-multitest.test: Likewise. * tests/test-driver-custom-multitest-recheck.test: Likewise. * tests/test-driver-custom-multitest-recheck2.test: Likewise. * tests/trivial-test-driver: New file, used by the last four tests above. * tests/Makefile.am (TESTS): Update. (EXTRA_DIST): Distribute `trivial-test-driver'. (test-driver-custom-multitest.log): Depend on `trivial-test-driver'. (test-driver-custom-multitest-recheck.log): Likewise. (test-driver-custom-multitest-recheck2.log): Likewise. (test-driver-custom-html.log): Likewise.
Stefano Lattarini a27c9c49 2011-05-10T11:30:05 parallel-tests: add auxiliary script 'test-driver', refactor This refactoring should cause no API of functionality change, and is meant only to simplify the future implementation of TAP and SubUnit testsuite drivers. More precisely, our roadmap is to move most of the "testsuite driving" features out of the Automake-generated Makefiles, and into external scripts with well-defined interfaces. This will allow the user to define its own personalized testsuite drivers, and will also offer us a framework upon which to implement our new TAP and SubUnit drivers, all in a very unobtrusive way and retaining an high degree of code reuse and backward-compatibility. * lib/test-driver: New auxiliary script. * lib/Makefile.am (dist_SCRIPT_DATA): Add it. * automake.in (handle_tests): Require the new auxiliary script `test-driver', and define a new internal makefile variable `$(am__test_driver)', used to call it. Perform new substitution on `DRIVER' when processing the `check2.am' file. * lib/check.am (am__tty_colors): Define new shell variable `$am__color_tests'. (am__rst_section): Removed, its role taken over by the new `test-driver' script. (am__test_driver_flags): New variable, contains the command line options passed to `test-driver'. (am__check_pre): Do not deal with temporary files and exit traps anymore, as the `test-driver' script takes care of that now. Define shell variable `$am__enable_hard_errors', used by `$(am__test_driver_flags)'. Reorder so that we don't need to save and restore the value of the `TERM' environment variable anymore. Other related adjustments. (am__check_post): Remove, as its role has been completely taken over by the `test-driver' script. * am/check2.am (?GENERIC?%EXT%$(EXEEXT).log, ?GENERIC?%EXT%.log, ?!GENERIC?%OBJ%): Call the test script through the Automake substituted `%DRIVER%', and honor the command-line options in `$(am__test_driver_flags)'. Do not call the obsoleted `$(am__check_post)' anymore. * doc/automake.texi (Auxiliary Programs): Mention the new `test-driver' script. (Optional): Mention `test-driver' in AC_CONFIG_AUX_DIR. Since we are at it, break the list of auxiliary scripts by placing one per line, to simplify potential future additions of new scripts. * tests/check.test: Adjust. * tests/check2.test : Likewise. * tests/check3.test : Likewise. * tests/check4.test : Likewise. * tests/check10.test: Likewise. * tests/color.test: Likewise. * tests/color2.test: Likewise. * tests/comment9.test: Likewise. * tests/dejagnu.test: Likewise. * tests/exeext4.test: Likewise. * tests/maken3.test: Likewise. * tests/maken4.test: Likewise. * tests/parallel-tests-interrupt.test: Likewise. * tests/posixsubst-tests.test: Likewise. * tests/repeated-options.test: Likewise. * tests/check-no-test-driver.test: New test. * tests/parallel-test-driver-install.test: Likewise. * tests/Makefile.am (TESTS): Update. * NEWS: Update.
Stefano Lattarini 583daa1e 2011-06-20T10:46:09 Merge branch 'maint' * maint: check: don't use multi-line coloring for the report ansi2knr: deprecate, it will go away in the next major release docs: primary/prefix combination "pkglib_PROGRAMS" is now invalid docs: replace obsolete @vindex entry with a useful one docs: AM_DISTCHECK_CONFIGURE_FLAGS is for corner cases
Bert Wesarg b8c2b699 2011-06-17T21:59:52 check: don't use multi-line coloring for the report "less -R" can't handle multi-line coloring as it is done for the check reports of the serial and parallel testsuite, because of performance reasons. Thus, color each line of the check report by its own. * lib/am/check.am (am__text_box): Accept colors for lines, and color each line by its own. [%?PARALLEL_TESTS%] $(TEST_SUITE_LOG): Let am__text_box handle the line coloring. [!%?PARALLEL_TESTS%] $(check-TESTS): Color each report line by its own. * THANKS: Update.
Stefano Lattarini 12f48faf 2011-04-18T00:52:03 check: rename AM_TESTS_SETUP -> AM_TESTS_ENVIRONMENT The AM_TESTS_SETUP naming was not a good one after all. It may be technically more correct than AM_TESTS_ENVIRONMENT, but the latter is a better one simply because it is easier to remember, and even if you've never heard of it and only know the semantics of TESTS_ENVIRONMENT, you can have a straightforward way to figure out how AM_TESTS_ENVIRONMENT would work. * tests/check.am (am__check_pre): Update. * doc/automake.in (Simple Tests using parallel-tests): Update. * tests/parallel-tests-am_tests_setup.test: Renamed ... * tests/parallel-tests-am_tests_environment.test: ... to this, and updated. * tests/Makefile.am (TESTS): Update. Suggestion and motivation by Ralf Wildenhues.
Stefano Lattarini 95bbdf16 2011-01-24T11:34:32 check: new developer-reserved AM_TESTS_SETUP variable For reference, see the discussion at: <http://lists.gnu.org/archive/html/automake-patches/2011-01/msg00213.html> * lib/am/check.am [%?PARALLEL_TESTS%] (am__check_pre): Pass also $(AM_TESTS_SETUP). Comments updated, and some typos fixed. * doc/automake.texi (Simple Tests using parallel-tests): Document AM_TESTS_SETUP. Reorder some of the existing documentation a bit. * tests/parallel-tests-am_tests_setup.test: New test. From a suggestion by Ralf Wildenhues.
Ralf Wildenhues e9b44d82 2011-01-22T11:16:06 Add comment to check-TESTS rule working around make 3.80 bug. * lib/am/check.am (check-TESTS): Update comment. Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
Ralf Wildenhues 080efc94 2011-01-16T13:48:17 parallel-tests: work around Tru64/OSF 5.1 sh read bugs. * lib/am/check.am ($(TEST_SUITE_LOG), recheck, recheck-html): Test file readability before redirecting input from it, to avoid exiting Tru64/OSF 5.1 sh which treats read as special builtin. * tests/parallel-tests-unreadable-log.test: New test. * tests/Makefile.am (TESTS): Update. * NEWS: Update. Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
Ralf Wildenhues b8c1325a 2010-10-03T11:42:12 Document and fix expansion of variables before rules. * doc/automake.texi (General Operation): Document that variables are expanded before rules. * lib/am/check.am (am__check_post): Reword a bit so it does not get matched as a rule. Suggestion by Ben Pfaff. Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
Ralf Wildenhues 614b643e 2010-09-26T09:21:14 Revert "parallel-tests: avoid command-line length limit issue." This reverts commit 24e3b4ee2f8cb9f72dd94a05a893f3d4e88b7835, because it re-opened the bug fixed by v1.11-10-g218e678. 2010-09-07 Ralf Wildenhues <Ralf.Wildenhues@gmx.de> parallel-tests: avoid command-line length limit issue. * automake.in (handle_tests): New argument $makefile, new substitution %MAKEFILE%. (generate_makefile): Adjust. * lib/am/check.am [%?PARALLEL_TESTS%] (check-TESTS): Pass sanitized TEST_LOGS value as makefile snippet on standard input to $(MAKE), to avoid exceeding the command line limit on w32 (MSYS). * NEWS: Update. Report by Bob Friesenhahn. Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
Ralf Wildenhues 24e3b4ee 2010-09-07T04:38:08 parallel-tests: avoid command-line length limit issue. * automake.in (handle_tests): New argument $makefile, new substitution %MAKEFILE%. (generate_makefile): Adjust. * lib/am/check.am [%?PARALLEL_TESTS%] (check-TESTS): Pass sanitized TEST_LOGS value as makefile snippet on standard input to $(MAKE), to avoid exceeding the command line limit on w32 (MSYS). * NEWS: Update. Report by Bob Friesenhahn.
Ralf Wildenhues ccb31f6a 2010-09-07T04:35:49 Posix 2008 requires make to set errexit. * lib/am/check.am: Update comment.
Stefano Lattarini e7aa360a 2010-08-06T00:40:55 Work around a nasty bug (segfault) of Solaris make. * lib/am/check.am (recheck, recheck-html): Trim trailing spaces from $list, to avoid triggering a nasty bug (potential segfault) on Solaris make.
Dmitry V. Levin dbfabdfc 2010-01-30T22:21:19 Fix exit status of signal handlers in shell scripts. The value of `$?' on entrance to signal handlers in shell scripts cannot be relied upon, so set the exit code explicitly to 128 + SIG<SIGNAL>. * lib/am/check.am (am__check_pre): Use `exit 143' in signal handler. * lib/elisp-comp: Likewise. * lib/install-sh: Likewise. * lib/ylwrap: Likewise. Also, fix script to trap signal 13, not 3. * NEWS, THANKS: Update. Bug report, analysis, and initial patch by Dmitry V. Levin. Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
Ralf Wildenhues 218e678b 2009-05-24T09:03:00 parallel-tests: avoid GNU make 3.80 substitution bug. * lib/am/check.am [PARALLEL_TESTS] (check-TESTS): Remove any `.log' entries from `$(TEST_LOGS)' even if the list is nonempty, to work around GNU make 3.80 substitution reference issue with trailing white space in the variable. * tests/parallel-tests10.test: New test. * tests/parallel-tests6.test: Update comment. * tests/Makefile.am: Update. * NEWS: Update. Report by Bob Friesenhahn. Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
kmx.io     mail     discord kmxgit v0.4.0