3 SchemeUnit API

(require (planet schematics/schemeunit:3))

3.1 Overview of SchemeUnit

There are three basic data types in SchemeUnit:

• A check is the basic unit of a test. As the name suggests, it checks some condition is true.

• A test case is a group of checks that form one conceptual unit. If any check within the case fails, the entire case fails.

• A test suite is a group of test cases and test suites that has a name.

3.2 Checks

Checks are the basic building block of SchemeUnit. A check checks some condition. If the condition holds the check evaluates to #t. If the condition doesn’t hold the check raises an instance of exn:test:check with information detailing the failure.

Although checks are implemented as macros, which is necessary to grab source location, they are conceptually functions. This means, for instance, checks always evaluate their arguments. You can use check as first class functions, though you will lose precision in the reported source locations if you do so.

The following are the basic checks SchemeUnit provides. You can create your own checks using define-check.

For example, the following check succeeds:

(check < 2 3)

For example, the following checks all fail:

(check-eq? (list 1) (list 1) "allocated data not eq?")

(check-not-eq? 1 1 "integers are eq?")

(check-eqv? 1 1.0 "not eqv?")

(check-equal? 1 1.0 "not equal?")

(check-not-equal? (list 1) (list 1) "equal?")

Here’s an example that passes and an example that fails:

(check-pred string? "I work")

(check-pred number? "I fail")

Here’s an example that passes and an example that fails:

(check-= 1.0 1.01 0.01 "I work")

(check-= 1.0 1.01 0.005 "I fail")

For example, the following checks all fail:

(check-true 1)

(check-false 1)

(check-not-false #f)

Here are two example, one showing a test that succeeds, and one showing a common error:

(check-exn exn?

(lambda ()

(raise (make-exn "Hi there"

(current-continuation-marks)))))

; Forgot to wrap the expression in a thunk. Don't do this!

(check-exn exn?

(raise (make-exn "Hi there"

(current-continuation-marks))))

The following check will succeed:

(check-regexp-match "a+bba" "aaaaaabba")

This check will fail:

(check-regexp-match "a+bba" "aaaabbba")

3.2.1 Augmenting Information on Check Failure

When an check fails it stores information including the name of the check, the location and message (if available), the expression the check is called with, and the parameters to the check. Additional information can be stored by using the with-check-info* function, and the with-check-info macro.

The are several predefined functions that create check information structures with predefined names. This avoids misspelling errors:

procedure (make-check-name name) → check-info?

name : string?

(make-check-params params) → check-info?

params : (listof any)

(make-check-location loc) → check-info?

loc : (list any (or/c number? #f) (or/c number? #f) (or/c number? #f) (or/c number? #f))

(make-check-expression msg) → check-info?

msg : any

(make-check-message msg) → check-info?

msg : string?

(make-check-actual param) → check-info?

param : any

(make-check-expected param) → check-info?

param : any

Example:

(with-check-info*

(list (make-check-info 'time (current-seconds)))

(lambda () (check = 1 2)))

When this check fails the message

time: <current-seconds-at-time-of-running-check>

will be printed along with the usual information on an check failure.

Example:

(for-each

(lambda (elt)

(with-check-info

(('current-element elt))

(check-pred odd? elt)))

(list 1 3 5 7 8))

When this test fails the message

current-element: 8

will be displayed along with the usual information on an check failure.

3.2.2 Custom Checks

Custom checks can be defined using define-check and its variants. To effectively use these macros it is useful to understand a few details about a check’s evaluation model.

Firstly, a check should be considered a function, even though most uses are actually macros. In particular, checks always evaluate their arguments exactly once before executing any expressions in the body of the checks. Hence if you wish to write checks that evalute user defined code that code must be wrapped in a thunk (a function of no arguments) by the user. The predefined check-exn is an example of this type of check.

It is also useful to understand how the check information stack operates. The stack is stored in a parameter and the with-check-info forms evaluate to calls to parameterize. Hence check information has lexical scope. For this reason simple checks (see below) cannot usefully contain calls to with-check-info to report additional information. All checks created using define-simple-check or define-check grab some information by default: the name of the checks and the values of the parameters. Additionally the macro forms of checks grab location information and the expressions passed as parameters.

Example:

To define a check check-odd?

(define-simple-check (check-odd? number)

(odd? number))

We can use these checks in the usual way:

(check-odd? 3)  ; Success

(check-odd? 2)  ; Failure

Examples:

Here’s the first form, where we use a predefined predicate to construct a binary check:

(define-binary-check (check-char=? char=? actual expected))

In use:

(check-char=? (read-char a-port) #\a)

If the expression is more complicated the second form should be used. For example, below we define a binary check that tests a number if within 0.01 of the expected value:

(define-binary-check (check-in-tolerance actual expected)

(< (abs (- actual expected)) 0.01))

3.2.3 The Check Evaluation Context

The semantics of checks are determined by the parameters current-check-around and current-check-handler. Other testing form such as test-begin and test-suite change the value of these parameters.

3.3 Compound Testing Forms

3.3.1 Test Cases

As programs increase in complexity the unit of testing grows beyond a single check. For example, it may be the case that if one check fails it doesn’t make sense to run another. To solve this problem compound testing forms can be used to group expressions. If any expression in a group fails (by raising an exception) the remaining expressions will not be evaluated.

For example, in the following code the world is not destroyed as the preceding check fails:

(test-begin

(check-eq? 'a 'b)

; This line won't be run

(destroy-the-world))

Here’s the above example rewritten to use test-case so the test can be named.

(test-case

"Example test"

(check-eq? 'a 'b)

; This line won't be run

(destroy-the-world))

3.3.2 Test Suites

Test cases can themselves be grouped into test suites. A test suite can contain both test cases and test suites. Unlike a check or test case, a test suite is not immediately run. Instead use one of the functions described in User Interfaces or Programmatically Running Tests and Inspecting Results.

The before-thunk and after-thunk are optional thunks (functions are no argument). They are run before and after the tests are run, respectively.

Unlike a check or test case, a test suite is not immediately run. Instead use one of the functions described in User Interfaces or Programmatically Running Tests and Inspecting Results.

For example, here is a test suite that displays Before before any tests are run, and After when the tests have finished.

(test-suite

"An example suite"

#:before (lambda () (display "Before"))

#:after  (lambda () (display "After"))

(test-case

"An example test"

(check-eq? 1 1)))

3.3.2.1 Utilities for Defining Test Suites

There are some macros that simplify the common cases of defining test suites:

For example, this code creates a binding for the name example-suite as well as creating a test suite with the name "example-suite":

(define-test-suite example-suite

(check = 1 1))

Finally, there is the test-suite* macro, which defines a test suite and test cases using a shorthand syntax:

As far I know no-one uses this macro, so it might disappear in future versions of SchemeUnit.

3.3.3 Compound Testing Evaluation Context

Just like with checks, there are several parameters that control the semantics of compound testing forms.

3.4 Test Control Flow

The before, after, and around macros allow you to specify code that is always run before, after, or around expressions in a test case.

Example:

The test below checks that the file test.dat contains the string "foo". The before action writes to this file. The after action deletes it.

(around

(with-output-to-file "test.dat"

(lambda ()

(write "foo")))

(with-input-from-file "test.dat"

(lambda ()

(check-equal? "foo" (read))))

(delete-file "test.dat"))

3.5 Miscellaneous Utilities

The require/expose macro allows you to access bindings that a module does not provide. It is useful for testing the private functions of modules.

Note that require/expose can be a bit fragile, especially when mixed with compiled code. Use at your own risk!

This example gets make-failure-test, which is defined in a SchemeUnit test:

(require/expose (planet schematics/schemeunit:3/check-test) (make-failure-test))

3.6 User Interfaces

SchemeUnit provides a textual and a graphical user interface

3.6.1 Textual User Interface

(require (planet schematics/schemeunit:3/text-ui))

The textual UI is in the text-ui module. It is run via the run-tests function

The optional verbosity is one of 'quiet, 'normal, or 'verbose. Quiet output displays only the number of successes, failures, and errors. Normal reporting suppresses some extraneous check information (such as the expression). Verbose reports all information.

run-tests returns the number of unsuccessful tests.

3.6.2 Graphical User Interface

The GUI has not yet been updated to this version of SchemeUnit.

3.7 Programmatically Running Tests and Inspecting Results

SchemeUnit provides an API for running tests, from which custom UIs can be created.

3.7.1 Result Types

3.7.2 Functions to Run Tests

Example:

(run-test

(test-suite

"Dummy"

(test-case "Dummy" (check-equal? 1 2))))

This function is useful for writing custom folds (and hence UIs) over test results without you having to take care of all the expected setup and teardown. For example, fold-test-results will run test suite before and after actions for you. However it is still flexible enough, via its keyword arguments, to do almost anything that foldts can. Hence it should be used in preference to foldts.

result-fn is a function from the results of run (defaults to a test-result) and the seed to a new seed

Seed is any value

Test is a test-case or test-suite

Run is a function from a test case name (string) and action (thunk) to any values.

FDown is a function from a test suite name (string) and the seed, to a new seed

FUp is a function from a test suite name (string) and the seed, to a new seed.

Examples:

The following code counts the number of successes

(define (count-successes test)

(fold-test-results

(lambda (result seed)

(if (test-success? result)

(add1 seed)

seed))

0

test))

The following code returns the symbol 'burp instead of running test cases. Note how the result-fn receives the value of run.

(define (burp test)

(fold-test-results

(lambda (result seed) (cons result seed))

null

test

#:run (lambda (name action) 'burp)))

Fdown is a function of test suite, test suite name, before action, after action, and the seed. It is run when a test suite is encountered on the way down the tree (pre-order).

Fup is a function of test suite, test suite name, before action, after action, the seed at the current level, and the seed returned by the children. It is run on the way up the tree (post-order).

Fhere is a function of the test case, test case name, the test case action, and the seed. (Note that this might change in the near future to just the test case. This change would be to allow fhere to discriminate subtypes of test-case, which in turn would allow test cases that are, for example, ignored).

Example:

Here’s the implementation of fold-test-results in terms of foldts:

(define (fold-test-results suite-fn case-fn seed test)

(foldts

(lambda (suite name before after seed)

(before)

(suite-fn name seed))

(lambda (suite name before after seed kid-seed)

(after)

kid-seed)

(lambda (case name action seed)

(case-fn

(run-test-case name action)

seed))

seed

test))

If you’re used to folds you’ll probably be a bit surprised that the functions you pass to foldts receive both the structure they operate on, and the contents of that structure. This is indeed unusual. It is done to allow subtypes of test-case and test-suite to be run in customised ways. For example, you might define subtypes of test case that are ignored (not run), or have their execution time recorded, and so on. To do so the functions that run the test cases need to know what type the test case has, and hence is is necessary to provide this information.

If you’ve made it this far you truly are a master SchemeUnit hacker. As a bonus prize we’ll just mention that the code in hash-monad.ss and monad.ss might be of interest for constructing user interfaces. The API is still in flux, so isn’t documented here. However, do look at the implementation of run-tests for examples of use.