15.3.1 Overview of Custom Test Drivers Support

Starting from Automake version 1.12, the parallel test harness allows the package authors to use third-party custom test drivers, in case the default ones are inadequate for their purposes, or do not support their testing protocol of choice.

A custom test driver is expected to properly run the test programs passed to it (including the command-line arguments passed to those programs, if any), to analyze their execution and outcome, to create the .log and .trs files associated to these test runs, and to display the test results on the console. It is responsibility of the author of the test driver to ensure that it implements all the above steps meaningfully and correctly; Automake isn't and can't be of any help here. On the other hand, the Automake-provided code for testsuite summary generation offers support for test drivers allowing several test results per test script, if they take care to register such results properly (see Log files generation and test results recording).

The exact details of how test scripts' results are to be determined and analyzed is left to the individual drivers. Some drivers might only consider the test script exit status (this is done for example by the default test driver used by the parallel test harness, described in the previous section). Other drivers might implement more complex and advanced test protocols, which might require them to parse and interpreter the output emitted by the test script they're running (examples of such protocols are TAP and SubUnit).

It's very important to note that, even when using custom test drivers, most of the infrastructure described in the previous section about the parallel harness remains in place; this includes:

• list of test scripts defined in TESTS, and overridable at runtime through the redefinition of TESTS or TEST_LOGS;
• concurrency through the use of make's option -j;
• per-test .log and .trs files, and generation of a summary .log file from them;
• recheck target, RECHECK_LOGS variable, and lazy reruns of tests;
• inter-test dependencies;
• support for check_* variables (check_PROGRAMS, check_LIBRARIES, ...);
• use of VERBOSE environment variable to get verbose output on testsuite failures;
• definition and honoring of TESTS_ENVIRONMENT, AM_TESTS_ENVIRONMENT and AM_TESTS_FD_REDIRECT variables;
• definition of generic and extension-specific LOG_COMPILER and LOG_FLAGS variables.

On the other hand, the exact semantics of how (and if) color-tests, XFAIL_TESTS, and hard errors are supported and handled is left to the individual test drivers.