Next: Testsuite progress output, Previous: Command-line arguments for test drivers, Up: API for Custom Test Drivers
The test driver must correctly generate the files specified by the --log-file and --trs-file option (even when the tested program fails or crashes).
The .log file should ideally contain all the output produced by the tested program, plus optionally other information that might facilitate debugging or analysis of bug reports. Apart from that, its format is basically free.
The .trs file is used to register some metadata through the use of custom reStructuredText fields. This metadata is expected to be employed in various ways by the parallel test harness; for example, to count the test results when printing the testsuite summary, or to decide which tests to re-run upon make reheck. Unrecognized metadata in a .trs file is currently ignored by the harness, but this might change in the future. The list of currently recognized metadata follows.
:test-result:fields can be present in the same .trs file; this is done in order to support test protocols that allow a single test script to run more test cases.
The only recognized test results are currently
ERROR. These results,
when declared with
:test-result:, can be optionally followed by
text holding the name and/or a brief description of the corresponding
test; the parallel-tests harness will ignore such extra text when
generating test-suite.log and preparing the testsuite summary.
no, then the corresponding test script will not be run upon a make recheck. What happens when two or more
:recheck:fields are present in the same .trs file is undefined behaviour.
no, then the content of the .log file will not be copied into the global test-suite.log. We allow to forsake such copying because, while it can be useful in debugging and analysis of bug report, it can also be just a waste of space in normal situations, e.g., when a test script is successful. What happens when two or more
:copy-in-global-log:fields are present in the same .trs file is undefined behaviour.
$(TEST_SUITE_LOG), so it's quite free-form. For example, a test script which run 10 test cases, 6 of which pass and 4 of which are skipped, could reasonably have a
PASS/SKIPvalue for this field, while a test script which run 19 successful tests and one failed test could have an
ALMOST PASSEDvalue. What happens when two or more
:test-global-result:fields are present in the same .trs file is undefined behaviour.
Let's see a small example. Assume a .trs file contains the following lines:
:test-result: PASS server starts :global-log-copy: no :test-result: PASS HTTP/1.1 request :test-result: FAIL HTTP/1.0 request :recheck: yes :test-result: SKIP HTTPS request (TLS library wasn't available) :test-result: PASS server stops
Then the corresponding test script will be re-run by make check, will contribute with five test results to the testsuite summary (three of these tests being successful, one failed, and one skipped), and the content of the corresponding .log file will not be copied in the global log file test-suite.log.