[ < ] [ > ]   [ << ] [ Up ] [ >> ]         [Top] [Contents] [Index] [ ? ]

9.29 gauche.test - Unit Testing

Module: gauche.test

Defines a set of functions to write test scripts. A test script will look like this:

 
(use gauche.test)
(test-start "my feature")
(load "my-feature")  ; load your program
(import my-feature)  ; if your program defines a module.

(test-module 'my-feature) ; tests consistency in your module.

(test-section "feature group 1")
(test "feature 1-1" EXPECT (lambda () TEST-BODY))
(test "feature 1-2" EXPECT (lambda () TEST-BODY))
 …

(test-section "feature group 2")
(define test-data ...)
(test "feature 2-1" EXPECT (lambda () TEST-BODY))
(test "feature 2-2" (test-error) (lambda () TEST-THAT-SIGNALS-ERROR))
 …

(test-end :exit-on-failure #t)

With this convention, you can run test both interactively or in batch. To run a test interactively, just load the file and it reports a result of each test, as well as the summary of failed test at the end. To run a test in batch, it is convenient to redirect the stdout to some file If stdout is redirected to other than tty, all the verbose logs will go there, and only a small amount of messages go to stderr.

It is recommended to have a "check" target always in Makefile of your module/program, so that the user of your program can run a test easily. The rule may look like this:

 
check :
        gosh my-feature-test.scm > test.log

Structuring a test file

Function: test-start module-name

Initializes internal state and prints a log header. This should be called before any tests. Module-name is used only for logging purpose.

Function: test-section section-name

Marks beginning of the group of tests. This is just for logging.

Function: test-log fmtstr args …

This is also just for logging. Creates a formatted string with fmrstr and args just like format, then write it to the current output port, with prefix ;; and newline at the end.

With the typical Makefile settings, where you redirect stdout of test scripts to a log file, the message only goes to the log file.

Using this, you can dump information that can’t be automatically tested but may be useful for troubleshooting. For example, you get a mysterious test failure reports you can’t reproduce on your machine, and suspect some aspects of the running systems may unpredictably affect the test result. You can put test-log in the test code to dump such parameters, and ask the reporter to run the test again and analyze the log.

Function: test-end :key exit-on-failure

Prints out list of failed tests. If exit-on-failure is #f or omitted, this procedure returns the number of failed tests.

Otherwise, this function terminates the gosh process by exit. If a fixnum is given to exit-on-failure it becomes the process’s exit status; if other true value is given, the exit status will be 1.

Function: test-record-file file

Suppose you have several test scripts. Normally you run them as a group and what you want to know is a concise summary of the whole results, instead of each result of individual test files.

A test record file is an auxiliary file used to gather summary of the result. It holds a one-line summary of tests like this:

 
Total:  9939 tests,  9939 passed,     0 failed,     0 aborted.

When a test record file exists, test-start reads and parses it, and remembers the numbers. Then test-end adds the count of the results and writes them back to the same test record file.

If you writes the check target in your makefile as follows, you will get the final one-line summary every time you run make check, assuming that ‘test1.scm’, ‘test2.scm’, and ‘test3.scm’ all has (test-record-file "test.record") before a call to test-start.

 
check:
        @rm -f test.record test.log
        gosh test1.scm >> test.log
        gosh test2.scm >> test.log
        gosh test3.scm >> test.log
        @cat test.record

Note that to make test-record-file work, it must be placed before the call to test-start.

Alternatively, you can use the envionment variable GAUCHE_TEST_RECORD_FILE to specify the test record file.

Environment Variable: GAUCHE_TEST_RECORD_FILE

If this environment varaible is set when the test script is run, its value is used as the name of the test record file.

If the test script calls test-record-file, it takes precedence and this environment variable is ignored.

Function: test-summary-check

If the test record file is set (either by test-record-file or the enviornment variable GAUCHE_TEST_RECORD_FILE), read it, and then exit with status 1 if the record has nonzero failure count and/or nonzero abort count. If the test record file isn’t set, this procedure does nothing.

This is useful when you have multiple test scripts and you want to let make fail if any of tests fails, but not before all test script is run. If you make every test script use :exit-on-failure of test-end, then make stops immediately after the script that fails. Instead, you avoid using :exit-on-failure, but use the test record file and for the last thing you can call this function:

 
check:
   rm -f $GAUCHE_TEST_RECORD_FILE test.log
   gosh test1.scm >> test.log
   gosh test2.scm >> test.log
   cat $GAUCHE_TEST_RECORD_FILE /dev/null
   gosh -ugauche.test -Etest-summary-check -Eexit

By this, make will run all the test script no matter how many of them fails (since gosh exits with status 0), but detect an error since the last line of gosh call exits with status 1 if there has been any failure.

Individual tests

Macro: test* name expected expr :optional check

A convenience macro that wraps expr by lambda.

 
(test* name expected expr)
  ≡ (test name expected (lambda () expr))
Function: test name expected thunk :optional check

Calls thunk, and checks its result fits expected using a procedure check, which is called as follows:

 
(check expected result-of-thunk)

It should return #t if the given result agrees with the expected value, or #f otherwise. The default check procedure is test-check, explained below. It compares expected and result-of-thunk with equal?, except when expected is some of special case test objects. (See “testing ambiguous results” and “testing abnormal cases” paragraphs below for this special treatment.)

One typical usage of the custom check procedure is to compare inexact numbers tolerating small error.

 
(test "test 1" (/ 3.141592653589 4)
      (lambda () (atan 1))
      (lambda (expected result)
        (< (abs (- expected result)) 1.0e-10)))

Name is a name of the test, for the logging purpose.

When thunk signals an uncaptured error, it is catched and yields a special error object <test-error>. You can check it with another error object created by test-error function to see if it is an expected type of error. See the entry of test-error below for the details.

Function: test-check expected result :optional fallback

The default procedure test and test* use to check the result of the test expression conforms the expected value. By default, test-check just compares expected and result with a procedure fallback, which is defaulted to equal?. test-check behaves differently if expected is one of special test objects described below.

Testing ambiguous results

Function: test-one-of choice …

Sometimes the result of test expression depends on various external environment, and you cannot put an exact expected value. This procedure supports to write such tests conveniently.

Returns a special object represents either one of choices. The default check procedure, test-check, recognizes the object when it is passed in the expected argument, and returns true if any one of choice … passes the check against the result.

For example, the following test passes if proc returns either 1 or 2.

 
(test* "proc returns either 1 or 2" (test-one-of 1 2) (proc))

Testing abnormal cases

Function: test-error :optional (condition-type <error>)

Returns a new <test-error> object that mathes with other <test-error> object with the given condition-type.

The test-check procedure treats <test-error> objects specially. When err-expected and err-actual are <test-error> objects, (test-check err-expected err-actual) returns #t if err-expected’s condition type is the same as or supertype of err-actual’s.

For example, if you want to test a call to foo raises an <io-error> (or its condition subtype), you can write as the following example:

 
(test "see if foo raises <io-error>" (test-error <io-error>) (foo))
Variable: *test-error*

(Deprecated) Bounded to an instance of <test-error> with condition type <error>. This is only provided for the backward compatibility; new code should use test-error procedure above.

Variable: *test-report-error*

If this variable is true, the test routine prints stack trace to the current error port when it captures an error. It is useful when you got an unexpected test-error object and want to check out where the error is occurring.

This variable is initialized by the environment variable GAUCHE_TEST_REPORT_ERROR when the gauche.test module is loaded. For example, you can use the environment variable to check out an unexpected error from your test script as follows (the value of the environment variable doesn’t matter).

 
env GAUCHE_TEST_REPORT_ERROR=1 gosh mytest.scm

Testing modules

Function: test-module module :key allow-undefined bypass-arity-check

Module must be a symbol module name or a module. This procedure performs some heuristic consistency check of the given module: (1) whether symbols set as autoload in module can actually be loaded, (2) whether the symbols declared in export list is actually defined in the module, (3) whether the global variable referenced within functions are all defined, and (4) if a global variable is used as a function, whether the number of arguments given to it is consistent to the actual function. Although not perfect, this test can catch some careless misses.

Sometimes you have a global vaiable that may not be defined depending on compiler options or platforms, and you check its existence at runtime before using it. The undefined variable reference check by test-module doesn’t follow such logic, and reports an error whenever it finds your code referring to undefined variable. In such case, you can give a list of symbols to the allow-undefined keyword argument; the test will excludes them from the check.

The arity check may also raise false positives, if the module count on a behavior of global procedures that will be modified after the module is loaded (e.g. a method with different number of arguments can be added to a generic function after the module is loaded, which would make the code valid.) If you know you’re doing right thing and need to suppress the false positives, pass a list of names of the functions to bypass-arity-check keyword arguments.


[ < ] [ > ]   [ << ] [ Up ] [ >> ]

This document was generated on July 19, 2014 using texi2html 1.82.