Next: , Previous: Introduction to TAP, Up: Using the TAP test protocol

15.4.2 Use TAP with the Automake test harness

Currently, the TAP driver that comes with Automake requires some by-hand steps on the developer's part (this situation should hopefully be improved in future Automake versions). You'll have to grab the script from the Automake distribution by hand, copy it in your source tree, add a call to AC_PROG_AWK in to search for a proper awk program, and use the Automake support for third-party test drivers to instruct the harness to use the script and that awk program to run your TAP-producing tests. See the example below for clarification.

Apart from the options common to all the Automake test drivers (see Command-line arguments for test drivers), the supports the following options, whose names are chosen for enhanced compatibility with the prove utility.

Causes the test driver to ignore the exit status of the test scripts; by default, the driver will report an error if the script exits with a non-zero status. This option has effect also on non-zero exit statuses due to termination by a signal.
Instruct the test driver to display TAP diagnostic (i.e., lines beginning with the ‘#’ character) in the testsuite progress output too; by default, TAP diagnostic is only copied to the .log file.
Revert the effects of --comments.
Instruct the test driver to merge the test scripts' standard error into their standard output. This is necessary if you want to ensure that diagnostics from the test scripts are displayed in the correct order relative to test results; this can be of great help in debugging (especially if your test scripts are shell scripts run with shell tracing active). As a downside, this option might cause the test harness to get confused if anything that appears on standard error looks like a test result.
Revert the effects of --merge.
Change the string that introduces TAP diagnostic from the default value of “#” to STRING. This can be useful if your TAP-based test scripts produce verbose output on which they have limited control (because, say, the output comes from other tools invoked in the scripts), and it might contain text that gets spuriously interpreted as TAP diagnostic: such an issue can be solved by redefining the string that activates TAP diagnostic to a value you know won't appear by chance in the tests' output. Note however that this feature is non-standard, as the “official” TAP protocol does not allow for such a customization; so don't use it if you can avoid it.

Here is an example of how the TAP driver can be set up and used.

     % cat
     AC_INIT([GNU Try Tap], [1.0], [])
     AM_INIT_AUTOMAKE([foreign parallel-tests -Wall -Werror])
     % cat
     TESTS = foo.test bar.test baz.test
     % cat foo.test
     echo 1..4 # Number of tests to be executed.
     echo 'ok 1 - Swallows fly'
     echo 'not ok 2 - Caterpillars fly # TODO metamorphosis in progress'
     echo 'ok 3 - Pigs fly # SKIP not enough acid'
     echo '# I just love word plays ...'
     echo 'ok 4 - Flies fly too :-)'
     % cat bar.test
     echo 1..3
     echo 'not ok 1 - Bummer, this test has failed.'
     echo 'ok 2 - This passed though.'
     echo 'Bail out! Ennui kicking in, sorry...'
     echo 'ok 3 - This will not be seen.'
     % cat baz.test
     echo 1..1
     echo ok 1
     # Exit with error, even if all the tests have been successful.
     exit 7
     % cp PREFIX/share/automake-APIVERSION/ .
     % autoreconf -vi && ./configure && make check
     PASS: foo.test 1 - Swallows fly
     XFAIL: foo.test 2 - Caterpillars fly # TODO metamorphosis in progress
     SKIP: foo.test 3 - Pigs fly # SKIP not enough acid
     PASS: foo.test 4 - Flies fly too :-)
     FAIL: bar.test 1 - Bummer, this test has failed.
     PASS: bar.test 2 - This passed though.
     ERROR: bar.test - Bail out! Ennui kicking in, sorry...
     PASS: baz.test 1
     ERROR: baz.test - exited with status 7
     Please report to
     % echo exit status: $?
     exit status: 1
     % env TEST_LOG_DRIVER_FLAGS='--comments --ignore-exit' \
           TESTS='foo.test baz.test' make -e check
     PASS: foo.test 1 - Swallows fly
     XFAIL: foo.test 2 - Caterpillars fly # TODO metamorphosis in progress
     SKIP: foo.test 3 - Pigs fly # SKIP not enough acid
     # foo.test: I just love word plays...
     PASS: foo.test 4 - Flies fly too :-)
     PASS: baz.test 1
     % echo exit status: $?
     exit status: 0