Hash :
eaac33fe
Author :
Date :
2011-07-01T11:12:31
parallel-tests: simplify testsuite summary Prefer a more deterministic, "tabular" format for the testsuite summary, always listing the numbers of passed, failed, xfailed, xpassed, skipped and errored tests, even when these numbers are zero. This simplify the logic of testsuite summary creation, makes it more easily machine-parseable, and will probably allow for easier addition of new kinds of test results in the future. * lib/am/check.am (am__tty_colors_dummy): New make variable, to reduce code duplication. Extracted from previous versions of $(am__tty_colors), and extended by defining two new variables `$mgn' and `$brg'. [%?COLOR%, %!?COLOR%] (am__tty_colors): Use that new variable. (am__text_box): Delete, is not needed anymore. ($(TEST_SUITE_LOG)): Rewrite associated rules to implement the new testsuite summary format. * NEWS: Update. * tests/check10.test: Don't run with the parallel-tests harness too, that makes no sense anymore. * tests/color.test: Update and adjust. * tests/color2.test: Likewise. * tests/parallel-tests.test: Likewise. * tests/parallel-tests3.test: Likewise. * tests/parallel-tests6.test: Likewise. * tests/parallel-tests9.test: Likewise. * tests/parallel-tests-unreadable-log.test: Likewise. * tests/parallel-tests-empty-testlogs.test: Likewise. * tests/parallel-tests-log-override-recheck.test: Likewise. * tests/parallel-tests-no-spurious-summary.test: Likewise. * tests/test-driver-custom-multitest.test: Likewise. * tests/test-driver-end-test-results.test: Likewise. * tests/parallel-tests-no-color-in-log.test: New test. * tests/testsuite-summary-color.test: Likewise. * tests/testsuite-summary-count.test: Likewise. * tests/testsuite-summary-count-many.test: Likewise. * tests/testsuite-summary-reference-log.test: Likewise. * tests/testsuite-summary-checks.sh: New auxiliary script, used by the new tests above. * tests/extract-testsuite-summary: Likewise. * tests/trivial-test-driver: Optimize for speed when there are lots of of tests. * tests/Makefile.am (EXTRA_DIST): Distribute them. (testsuite-summary-color.log, testsuite-summary-count.log): Depend on them. (testsuite-summary-count-many.log): Depend on the auxiliary scripts 'trivial-test-driver' and 'extract-testsuite-summary'. (TESTS): Update.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105
#! /bin/sh
# Copyright (C) 2011 Free Software Foundation, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2, or (at your option)
# any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# Check test counts in the testsuite summary, with test drivers allowing
# multiple test results per test script, and for a huge number of tests.
# Incidentally, this test also checks that the testsuite summary doesn't
# give any bug-report address if it's not defined.
parallel_tests=yes
. ./defs || Exit 1
for s in trivial-test-driver extract-testsuite-summary; do
cp "$testsrcdir/$s" . || fatal_ "failed to fetch auxiliary script $s"
done
br='============================================================================'
header="\
${br}
Testsuite summary for $me 1.0
${br}"
footer="\
${br}
See ./test-suite.log
${br}"
echo AC_OUTPUT >> configure.in
cat > Makefile.am << 'END'
TEST_LOG_DRIVER = $(SHELL) $(srcdir)/trivial-test-driver
TESTS = all.test
# Without this, the test driver will be horrendously slow.
END
cat > all.test <<'END'
#!/bin/sh
cat results.txt || { echo ERROR: weird; exit 99; }
END
chmod a+x all.test
$PERL -w -e '
use warnings FATAL => "all";
use strict;
# FIXME: we would like this to be 1000 or even 10000, but the current
# implementation is too slow to handle that :-(
my $base = 5;
my %count = (
TOTAL => $base * 1000,
PASS => $base * 700,
SKIP => $base * 200,
XFAIL => $base * 80,
FAIL => $base * 10,
XPASS => $base * 7,
ERROR => $base * 3,
);
my @results = qw/PASS SKIP XFAIL FAIL XPASS ERROR/;
open (RES, ">results.txt") or die "opening results.txt: $!\n";
open (CNT, ">count.txt") or die "opening count.txt: $!\n";
printf CNT "# %-6s %d\n", "TOTAL:", $count{TOTAL};
for my $res (@results)
{
my $uc_res = uc $res;
print STDERR "Generating list of $res ...\n";
for (1..$count{$res})
{
print RES "$uc_res: $_\n";
}
printf CNT "# %-6s %d\n", $res . ":", $count{$res};
}
'
(echo "$header" && cat count.txt && echo "$footer") > summary.exp
$ACLOCAL
$AUTOMAKE -a
$AUTOCONF
./configure
($MAKE check || : > make.fail) | tee stdout
test -f make.fail
$PERL -w extract-testsuite-summary stdout > summary.got
cat summary.exp
cat summary.got
diff summary.exp summary.got || Exit 1
: