kc3-lang/automake/t/parallel-tests-basics.sh

• Show log

  Commit

• Hash : ffd4ecc9
  Author :
  Date : 2013-05-20T11:23:01
  

  tests: better idiom to override make macro defs on the cmdline
  
  We now do so with a new wrapper shell function 'run_make()', which
  relies on the use of AM_MAKEFLAGS, avoiding the use of the '-e' make
  option.  The use of that option (that causes the environment variables
  to take precedence over the macro definitions in the Makefile) has
  proved to be quite brittle in the past, causing annoying and sometimes
  problematic spurious failures.  This has worsened in some still
  unpublished work going on in development branches.  It's time to solve
  the issue once and for all.
  
  Note that we do not convert all uses of $MAKE in the testsuite right
  away; we might do so in follow-up changes, with leisure, to avoid a
  "patch bomb" effect (this commit is already too much of a bomb itself).
  What we do in this commit is to get rid of all "$MAKE -e" invocations.
  
  We admit that the implementation and feature-set of 'run_make()' are
  far from perfect, but good enough for our current purposes.  We'll
  improve 'run_make()' if and when the need arises.
  
  * syntax-checks.mk (sc_tests_no_make_e): New syntax check, guard against
  the use of "$MAKE -e".
  (syntax_check_rules): Add it.
  (sc_tests_overriding_macros_on_cmdline): Adjust.
  (lint): New, alias for 'maintainer-check', for lazy typists.  Idea
  backported from the 'maint' branch (Automake 1.13a).
  * t/ax/am-test-lib.sh (run_make): New function.  Run $MAKE with the
  given command-line arguments, handling command-line override of variable
  definitions in a smart way (using AM_MAKEFLAGS if a non-GNU make
  implementation is detected to be in use).
  (useless_vpath_rebuild): Adjust to use 'run_make', to avoid a spurious
  maintainer check failure.
  (yl_distcheck): Use 'run_make' rather than bare '$MAKE'.
  (single_quote, append_single_quoted, is_valid_varname): New auxiliary
  function, used, directly or indirectly, by it.
  * Many tests: Adjust to avoid the use of "$MAKE -e", and prefer the
  use of 'run_make' in few other contexts as well, where it makes sense.
  Other minor fixlets while at it.
  
  Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
  

Download

• t/parallel-tests-basics.sh

• #! /bin/sh
  # Copyright (C) 2009-2013 Free Software Foundation, Inc.
  #
  # This program is free software; you can redistribute it and/or modify
  # it under the terms of the GNU General Public License as published by
  # the Free Software Foundation; either version 2, or (at your option)
  # any later version.
  #
  # This program is distributed in the hope that it will be useful,
  # but WITHOUT ANY WARRANTY; without even the implied warranty of
  # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
  # GNU General Public License for more details.
  #
  # You should have received a copy of the GNU General Public License
  # along with this program.  If not, see <http://www.gnu.org/licenses/>.
  
  # Basic checks on parallel-tests support:
  #  - console output
  #  - log files, and what goes in 'test-suite.log'
  #  - make clean
  #  - dependencies between tests
  #  - TESTS redefinition at runtime
  #  - TEST_LOGS redefinition at runtime
  #  - RECHECK_LOGS redefinition at runtime
  
  . test-init.sh
  
  cat >> configure.ac << 'END'
  AC_OUTPUT
  END
  
  cat > Makefile.am << 'END'
  TESTS = foo.test bar.test baz.test
  XFAIL_TESTS = bar.test
  foo.log: bar.log
  bar.log: baz.log
  END
  
  # foo.test and bar.test sleep to ensure their logs are always strictly newer
  # than the logs of their prerequisites, for HP-UX make.  The quoting pleases
  # maintainer-check.
  cat > foo.test <<'END'
  #! /bin/sh
  echo "this is $0"
  sleep '1'
  exit 0
  END
  cat > bar.test <<'END'
  #! /bin/sh
  echo "this is $0"
  sleep '1'
  exit 99
  END
  cat > baz.test <<'END'
  #! /bin/sh
  echo "this is $0"
  exit 1
  END
  chmod a+x foo.test bar.test baz.test
  
  $ACLOCAL
  $AUTOCONF
  $AUTOMAKE -a
  
  ./configure
  
  $MAKE check >stdout && { cat stdout; exit 1; }
  cat stdout
  count_test_results total=3 pass=1 fail=1 skip=0 xfail=0 xpass=0 error=1
  test -f test-suite.log
  cat test-suite.log
  test $(grep -c '^FAIL:' test-suite.log) -eq 1
  test $(grep -c '^ERROR:' test-suite.log) -eq 1
  $EGREP '^(X?PASS|XFAIL|SKIP)' test-suite.log && exit 1
  test -f baz.log
  test -f bar.log
  test -f foo.log
  
  $MAKE clean
  test ! -e baz.log
  test ! -e bar.log
  test ! -e foo.log
  test ! -e test-suite.log
  
  # Check dependencies: baz.test needs to run before bar.test,
  # but foo.test is not needed.
  # Note that this usage has a problem: the summary will only
  # take bar.log into account, because the $(TEST_SUITE_LOG) rule
  # does not "see" baz.log.  Hmm.
  run_make TESTS='bar.test' check >stdout && { cat stdout; exit 1; }
  cat stdout
  grep '^FAIL: baz\.test$' stdout
  grep '^ERROR: bar\.test$' stdout
  
  test -f baz.log
  test -f bar.log
  test ! -e foo.log
  test -f test-suite.log
  
  # Upon a lazy rerun, foo.test should be run, but the others shouldn't.
  # Note that the lazy rerun still exits with a failure, due to the previous
  # test failures.
  # Note that the previous test and this one taken together expose the timing
  # issue that requires the check-TESTS rule to always remove TEST_SUITE_LOG
  # before running the tests lazily.
  run_make RECHECK_LOGS= check > stdout && { cat stdout; exit 1; }
  cat stdout
  test -f foo.log
  grep '^PASS: foo\.test$' stdout
  grep bar.test stdout && exit 1
  grep baz.test stdout && exit 1
  grep '^# PASS: *1$' stdout
  grep '^# FAIL: *1$' stdout
  grep '^# ERROR: *1$' stdout
  
  # Now, explicitly retry with all test logs already updated, and ensure
  # that the summary is still displayed.
  run_make RECHECK_LOGS= check > stdout && { cat stdout; exit 1; }
  cat stdout
  grep foo.test stdout && exit 1
  grep bar.test stdout && exit 1
  grep baz.test stdout && exit 1
  grep '^# PASS: *1$' stdout
  grep '^# FAIL: *1$' stdout
  grep '^# ERROR: *1$' stdout
  
  # Lazily rerunning only foo should only rerun this one test.
  run_make RECHECK_LOGS=foo.log check > stdout && { cat stdout; exit 1; }
  cat stdout
  grep foo.test stdout
  grep bar.test stdout && exit 1
  grep baz.test stdout && exit 1
  grep '^# PASS: *1$' stdout
  grep '^# FAIL: *1$' stdout
  grep '^# ERROR: *1$' stdout
  
  $MAKE clean
  run_make TEST_LOGS=baz.log check > stdout && { cat stdout; exit 1; }
  cat stdout
  grep foo.test stdout && exit 1
  grep bar.test stdout && exit 1
  grep baz.test stdout
  
  $MAKE clean
  run_make TESTS=baz.test check > stdout && { cat stdout; exit 1; }
  cat stdout
  grep foo.test stdout && exit 1
  grep bar.test stdout && exit 1
  grep baz.test stdout
  
  :
  

Download