kc3-lang/automake/tests

Diff

  • Show log

    Commit

  • Hash : b7d67d55
    Author : Stefano Lattarini
    Date : 2011-12-14T10:35:04 

    tests: better handling of gettext and libtool requirements

This change fixes automake bug#9807.

Before this change, the automake testsuite only looked for the
`.m4' files containing libtool and gettext macros definitions in
the directory `${prefix}/share/aclocal' (and in the directories
specified by the `dirlist' file in there, if any), where ${prefix}
was the configure-time automake installation prefix (defaulting
to `/usr/local').

This approach had various shortcomings and disadvantages.  Let's
briefly describe the three major ones.

First, on most GNU/Linux systems, a libtool or gettext installed
from distro-provided packages (e.g., by dpkg on Debian/Ubuntu, or
by rmp on RedHat/Fedora) would have `/usr', not `/usr/local', as
its ${prefix}; so, trying to run the automake testsuite with a
simple "./configure && make && make check" would have failed to
execute the libtool and gettext tests on most GNU/Linux distros.
It's true that it was quite easy to work around this issue, by
creating a proper `/usr/local/share/aclocal/dirlist' file with
an entry pointing to `/usr/share/aclocal' (a workaround in fact
used by most automake developers); but the typical user wasn't
aware of the necessity of this trick, so the libtool and gettext
tests was usually skipped on testsuite runs "in the wild", thus
needlessly reducing coverage.

Second, the older testsuite behaviour made more difficult for
the developers to run the testsuite with non-default libtool or
gettext.  For example, assume the developer is working on a system
that has a default libtool version 1.5 installed in the /usr/local
hierarchy; to improve coverage, the developer installs also a more
modern libtool version, say 2.4, in its home directory, let's say
in ~/libtool-2.4; he then tries to run the automake testsuite with
this more modern libtool by doing an (apparently) simple:
  $ PATH=$HOME/libtool-2.4:$PATH make check
But the automake testsuite would still look for libtool macros in
/usr/local/share/aclocal, not in ~/libtool-2.4/share/aclocal, so
the wrong version of the macros would be picked up, and the tests
would either fail spuriously or (which would be worse) pass without
truly covering the libtool version the developers was thinking to
be testing with.
Worse again, the automake testsuite would *unconditionally* look
for libtool macros in /usr/local/share/aclocal, so even something
like:
  $ export ACLOCAL_PATH=$HOME/libtool-2.4/share/aclocal
  $ PATH=$HOME/libtool-2.4:$PATH make check
wouldn't work.

Third and last, during a "make distcheck", automake is configured
with a ${prefix} pointing to a proper subdirectory of the build
directory (usually `pwd`/_inst), which gets created on-the-fly;
in this case, with the old approach, the automake testsuite never
found the libtool and gettext macro files, ans so the libtool and
gettext tests was *always* skipped in a "make distcheck".

* tests/libtool-macros.test: New helper test, looking (with the
help of the `libtoolize' script) for libtool macro files required
by most libtool tests, and making them easily accessible.
* tests/gettext-macros.test: New helper test, looking (with the
help of the `libtoolize' script) for libtool macro files required
by most libtool tests, and making them easily accessible.
* tests/defs.in: Update to make it rely on the results and setups
of `libtool-macros.test' and `gettext-macros.test'.
* tests/Makefile.am: Declare dependency of all the logs of libtool
tests from `libtool-macros.log', and all the logs of gettext tests
from `gettext-macros.log'.
(TESTS): Add the new tests.

  • README

  • 			    The Automake test suite


User interface
==============


Running all tests
-----------------

  make check

  You can use `-jN' for faster completion (it even helps on a
  uniprocessor system, due to unavoidable sleep delays, as
  noted below).


Interpretation
--------------

  Successes:
    PASS  - success
    XFAIL - expected failure

  Failures:
    FAIL  - failure
    XPASS - unexpected success

  Other:
    SKIP  - skipped tests (third party tools not available)


Getting details from failures
-----------------------------

  Each test is a shell script, and by default is run by /bin/sh.
  In a non-VPATH build you can run them directly, they will be verbose.
  By default, verbose output of a test foo.test is retained in the log
  file foo.log.  A summary log is created in the file test-suite.log.

  You can limit the set of files using the TESTS variable, and enable
  detailed test output at the end of the test run with the VERBOSE
  variable:

    env VERBOSE=x TESTS='first.test second.test ...' make -e check


Supported shells
----------------

  The test scripts are written with portability in mind, so that they
  should run with any decent Bourne-compatible shell.

  However, some care must be used with Zsh, since, when not directly
  starting in Bourne-compatibility mode, it has some incompatibilities
  in the handling of `$0' which conflict with our usage, and which have
  no easy workaround.  Thus, if you want to run a test script, say
  foo.test, with Zsh, you *can't* simply do `zsh foo.test', but you
  *must* resort to:
    zsh -o no_function_argzero foo.test

  Note that this problem does not occur if zsh is executed through a
  symlink with a basename of `sh', since in that case it starts
  in Bourne compatibility mode.  So you should be perfectly safe when
  /bin/sh is zsh.


Reporting failures
------------------

  Send verbose output, i.e., the contents of test-suite.log, of failing
  tests to <bug-automake@gnu.org>, along with the usual version numbers
  (which Automake, which Autoconf, which operating system, which make
  version, which shell, etc.)



Writing test cases
==================


Do
--

  If you plan to fix a bug, write the test case first.  This way you'll
  make sure the test catches the bug, and that it succeeds once you have
  fixed the bug.

  Add a copyright/license paragraph.

  Explain what the test does.

  Cite the PR number (if any), and the original reporter (if any), so
  we can find or ask for information if needed.

  If a test checks examples or idioms given in the documentation, make
  sure the documentation reference them appropriately in comments, as in:
    @c Keep in sync with autodist-config-headers.test.
    @example
    ...
    @end example

  Use `required=...' for required tools.  Do not explicitly require
  tools which can be taken for granted because they're listed in the
  GNU Coding Standards (for example, `gzip').

  Include ./defs in every test script (see existing tests for examples
  of how to do this).

  Use the `skip_' function to skip tests, with a meaningful message if
  possible.  Where convenient, use the `warn_' function to print generic
  warnings, the `fail_' function for test failures, and the `fatal_'
  function for hard errors.  In case a hard error is due to a failed
  set-up of a test scenario, you can use the `framework_fail_' function
  instead.

  For tests that use the `parallel-tests' Automake option, set the shell
  variable `parallel_tests' to "yes" before including ./defs.  Also,
  do not use for them a name that ends in `-p.test', since that would
  risk to clash with automatically-generated tests.  For tests that are
  *not* meant to work with the `parallel-tests' Automake option (these
  should be very very few), set the shell variable `parallel_tests' to
  "no" before including ./defs.

  ./defs sets a skeleton configure.in.  If possible, append to this
  file.  In some cases you'll have to overwrite it, but this should
  be the exception.  Note that configure.in registers Makefile.in
  but do not output anything by default.  If you need ./configure
  to create Makefile, append AC_OUTPUT to configure.in.

  Use `set -e' to catch failures you might not have thought of.

  End the test script with a `:' or `Exit 0'.  Otherwise, when somebody
  changes the test by adding a failing command after the last command,
  the test will spuriously fail because $? is nonzero at the end.
  Note that this is relevant also for tests using `set -e', if they
  contain commands like "grep ... Makefile.in && Exit 1" (and there
  are indeed a lot of such tests).

  Use $ACLOCAL, $AUTOMAKE, $AUTOCONF, $AUTOUPDATE, $AUTOHEADER,
  $PERL, $MAKE, $EGREP, and $FGREP, instead of the corresponding
  commands.

  Use $sleep when you have to make sure that some file is newer
  than another.

  Use `cat' or `grep' to display (part of) files that may be
  interesting for debugging, so that when a user send a verbose
  output we don't have to ask him for more details.  Display stderr
  output on the stderr file descriptor.  If some redirected command
  is likely to fail, and `set -e' is in effect, display its output
  even in the failure case, before exiting.

  Use `Exit' rather than `exit' to abort a test.

  Use `$PATH_SEPARATOR', not hard-coded `:', as the separator of
  PATH's entries.

  It's more important to make sure that a feature works, than
  make sure that Automake's output looks correct.  It might look
  correct and still fail to work.  In other words, prefer
  running `make' over grepping `Makefile.in' (or do both).

  If you run $AUTOMAKE or $AUTOCONF several times in the same test
  and change `configure.in' by the meantime, do
    rm -rf autom4te.cache
  before the following runs.  On fast machines the new `configure.in'
  could otherwise have the same timestamp as the old `autom4te.cache'.
  Alternatively, use `--force' for subsequent runs of the tools.

  Use filenames with two consecutive spaces when testing that some
  code preserves filenames with spaces.  This will catch errors like
  `echo $filename | ...`.

  Before commit: make sure the test is executable, add the tests to
  TESTS in Makefile.am, add it to XFAIL_TESTS in addition if needed,
  write a ChangeLog entry, send the diff to <automake-patches@gnu.org>.


Do not
------

  Do not test an Automake error with `$AUTOMAKE && Exit 1', or in three
  years we'll discover that this test failed for some other bogus reason.
  This happened many times.  Better use something like
     AUTOMAKE_fails
     grep 'expected diagnostic' stderr
  (Note this doesn't prevent the test from failing for another
  reason, but at least it makes sure the original error is still
  here.)

  Do not override Makefile variables using make arguments, as in e.g.:
    $MAKE prefix=/opt install
  This is not portable for recursive targets (targets that call a
  sub-make may not pass `prefix=/opt' along).  Use the following
  instead:
    prefix=/opt $MAKE -e install

kmx.io     mail     discord kmxgit v0.4.0