Version: 2:0
Overeasy: Racket Language Test Engine
| (require (planet neil/overeasy:2:0)) |
1 Introduction
Overeasy is a software test engine for the Racket programming language. It
designed for all of:
rapid interactive testing of expressions in the REPL;
unit testing of individual modules; and
running hierarchical sets of individual module unit tests at once.
An individual test case, or test, is specified by the programmer with the test syntax, and evaluation of that syntax causes the test to be
run. Properties that are checked by tests are:
values of expressions (single value, or multiple value);
exceptions raised; and
output to current-output-port and current-error-port.
Some checking is also done to help protect test suites from
crashing due to errors in the setup of the test itself, such as errors in
evaluating an expression that provides an expected value for a test.
A future version of Overeasy might permit the properties that are
tested to be extensible, such as for testing network state or other
resources.
For the properties checked by tests, in most cases, the programmer
can specify both an expected value and a predicate, or checker, for comparing expected and actual values. Note that, if the predicate
is not an equality predicate of some kind, then the “expected” would be a
misnomer, and “argument to the predicate” would be more accurate. The
actual test syntax does not include the word “expected.” Specification of
expected exceptions is diferent from values and output ports, in that only the
predicate is specified, with no separate expected or argument value. All these
have have reasonable defaults whenever possible.
1.1 Examples
Here’s a simple test, with the first argument the expression under test, and
the other argument the expected value.
> (test (+ 1 2 3) 6)
How the results of tests are reported varies depending on how the tests
are run. For purposes of these examples, we will pretend we are running tests
in the simplest way. In this way, tests that fail produce one-line
error-messages to current-error-port, which in DrRacket show up as red italic text by default. Tests that
pass in this way do not produce any message at all. So, our first example
above, does not produce any message.
Now, for a test that fails:
> (test (+ 1 2 3) 7) Test FAIL : Value 6 did not match expected value 7 by equal?.
That’s a quick way to do a test in a REPL or when you’re otherwise in a
hurry, but if you’re reviewing a report of failed tests for one or more
modules, you’d probably like a more descriptive way of seeing which tests
failed. That’s what the test ID is for, and to specify it, we use the #:id keyword arguments in our test:
> (test #:id 'simple-addition (+ 1 2 3) 7) Test FAIL simple-addition : Value 6 did not match expected value 7 by equal?.
Quick note on syntax. The above is actually shorthand syntax. In the
non-shorthand syntax, every argument to test has a keyword, so the above is actually shorthand for:
(test #:id 'simple-addition #:code (+ 1 2 3) #:val 7)
#:code and #:val are used so often that the keywords can be left off, so long as there are no other keyword arguments before them, other than #:id.
You can even leave off the #:id keyword, so long as you have both code and val expressions, also without keywords. So, the above example has equivalent shorthand:
(test 'simple-addition (+ 1 2 3) 7)
In the rest of these examples, we’ll use the shorthand syntax, because
it’s quicker to type, and getting rid of the #:code and #:val keywords also makes less-common keyword arguments stand out.
So far, we’ve been checking the values of code, and we haven’t yet dealt
in exceptions. Exceptions, such as due to programming errors in the code being
tested, can also be reported:
> (test (+ 1 (error "help!") 3) 3) Test FAIL : Got exception #(struct:exn:fail "help!"), but expected value 3.
And if an exception is the correct behavior, instead of specifying an
expected value, we can use #:exn to specify predicate just like for with-handlers:
> (test (+ 1 (error "help!") 3) #:exn exn:fail?)
That test passed. But if our code under test doesn’t throw an exception
matched by our #:exn predicate, that’s a test failure:
> (test (+ 1 2 3) #:exn exn:fail?) Test FAIL : Got value 6, but expected exception matched by predicate exn:fail?.
Of course, when you want finer discrimination of exceptions than, say,exn:fail? or exn:fail:filesystem?, you can write a custom predicate that uses exn-message or other information, and supply it to test’s #:exn.
Multiple values are supported:
> (test (begin 1 2 3) (values 1 2 3)) Test FAIL : Value 3 did not match expected values (1 2 3) by equal?.
You might have noticed that a lot of the test failure messages say “by
equal?”. That’s referring to the default predicate, so, the following
test passes:
> (test (string-append "a" "b" "c") "abc")
But we let’s say we wanted the expected and actual values to not only be equal? but to be eq? as well:
> (test (string-append "a" "b" "c") "abc" #:val-check eq?) Test FAIL : Value "abc" did not match expected value "abc" by eq?.
As mentioned earlier, the checker does not have to be an equality
predicate, and it can use whatever reasoning you like in rendering its verdict
on whether the actual value should be considered OK.
In addition to values an exceptions, test also intercepts and permits checking of current-output-port and current-error-port. By default, it assumes no output to either of those ports, which is
especially good for catching programming errors like neglecting to specify an
output port to a procedure for which the port is optional:
> (test (let ((o (open-output-string))) (display 'a o) (display 'b) (display 'c o) (get-output-string o)) "abc")
Test FAIL : Value "ac" did not match expected value "abc" by equal?. Out bytes #"b" did not match expected #"" by equal?.
Likewise, messages to current-error-port, such as warnings and errors from legacy code, are also caught by
default:
> (test (begin (fprintf (current-error-port) "%W%SYS$FROBINATOR_OVERHEAT\n") 0) 42)
Test FAIL : Value 0 did not match expected value 42 by equal?. Err bytes #"%W%SYS$FROBINATOR_OVERHEAT\n" did not match expected #"" by equal?.
Now we know why we’ve started getting 0, which information might have
gone unnoticed had our test engine not captured error port output: the
frobinator is failing, after all these years of valiant service.
With the #:out-check and #:err-check keyword arguments to test, you can specify predicates other than equal?. Also, by setting one of these predicates to #f, you can cause the output to be consumed but not stored and checked.
This is useful if, for example, the code produces large amounts of debugging
message output.
(test (begin (display "blah") (display "blah") (display "blah") (* 44 2)) 88 #:out-check #f)
There are some more tricks you can do with test. Most notably,
you’ll sometimes want to set up state in the system – Racket parameters, test
input files, whatever. Because the test syntax can appear anywhere normal Racket code can, you can set up this
state using normal Racket code. No special forms for setup and tear-down are
required, nor are they provided.
1.2 Report Backends
The architecture of Overeasy is designed to permit different backends for
reporting test results to be plugged in. Currently implemented backends are
for:
quick one-line error messages for tests that fail; and
more verbose textual report of all test cases run.
In the future, Web front-end and GUI backends might also be implemented.
The backend is dynamic context, so no changes to the files containing test
code is required to run the tests with a different backend.
1.3 Test Contexts and Test Sections
The architectural notion that permits the backends to be plugged in is
called the test context. Test context are nested dynamically, with each introduced
context having the previous context as a parent. The same test context notion
that permits backends for reporting to be introduced also permits test sections for grouping tests to be nested dynamically.
The dynamic nesting of test sections facilitates reporting of test results when
running unit tests for multiple modules together. Plugging in a backend for
reporting simply means establishing it as the first or topmost test context.
By default, if a test is run without a test context, then the one-line
error messages are used. If a test section context is introduced without a
parent context, such as would usually be the case for an individual
module’s unit tests, then the text report backend is plugged in by default.
One place you’ll want to use a section is for the unit tests for a
particular module. This groups the tests together if the module’s unit tests
are run in the context of a larger test suite, and it also provides a default
report context when the unit tests are run by themselves. You might want to
package the module’s unit tests in a procedure, for ease of use as part of a
test suite. (Unless you have rigged up something different, like by having require or dynamic-require simply run the tests, without needing to then invoke a provided
procedure. For illustration in this document, we’ll use procedures.) For
example, if you have a fruits module, in file fruits.rkt, then you might want to put its unit tests in a procedure in
file test-fruits.rkt, like so:
(define (test-fruits) (test-section #:id 'fruits (test #:id 'apple #:code (+ 1 2 3) #:val 6) (test #:id 'banana #:code (+ 4 5 6) #:val 6) (test #:id 'cherry #:code (+ 7 8 9) #:val 24)))
Notice that we put all the tests for module in fruits in test-section here, and gave it an ID. The ID didn’t have to be fruits like the module name; we could have called it fruity-unit-tests, fructose, or any other symbol.
Then let’s say we have a cars module, so in file some-cars-tests.rkt, we put this procedure:
(define (test-drive-cars) (test-section #:id 'cars (test 'delorean (+ 77 11) 88) (test 'ferrari (or (and #f 'i-cant-drive) 55) 55) (test 'magnum (+ 300 8) 308)))
Those unit test suites are used independently. Later, those
modules are integrated into a larger system, COLOSSUS. For running all the
unit tests for the modules of COLOSSUS, we add another module, which requires the other test modules, and invokes the each unit test procedure
within its own test section:
(test-section #:id 'colossus-components (test-fruits) (test-drive-cars))
Unless this is done within another test context, the result will be
to execute the tests in the default text report context. This produces a
report like:
;; START-TESTS |
;; |
;; START-TEST-SECTION colossus-components |
;; |
;; START-TEST-SECTION fruits |
;; |
;; TEST apple |
;; (+ 1 2 3) |
;; OK |
;; |
;; TEST banana |
;; (+ 4 5 6) |
;; *FAIL* Value 15 did not match expected value 6 by equal?. |
;; |
;; TEST cherry |
;; (+ 7 8 9) |
;; OK |
;; |
;; END-TEST-SECTION fruits |
;; |
;; START-TEST-SECTION cars |
;; |
;; TEST delorean |
;; (+ 77 11) |
;; OK |
;; |
;; TEST ferrari |
;; (or (and #f (quote i-cant-drive)) 55) |
;; OK |
;; |
;; TEST magnum |
;; (+ 300 8) |
;; OK |
;; |
;; END-TEST-SECTION cars |
;; |
;; END-TEST-SECTION colossus-components |
;; |
;; END-TESTS |
;; OK: 5 FAIL: 1 BROKEN: 0 |
;; SOME TESTS *FAILED*! |
|
The test sections here are nested only two deep, but test sections may
be nested to arbitrary depth. You can use test sections at each nested
subsystem, to organize the unit tests for a module into groups, to group
variations of generated test cases (e.g., if evaluating the same test form multiple times, with different values or state each time),
or other purposes.
By the way, the #:id keyword argument itself can be left out of the test-section syntax, when you prefer. So, the following two forms are
equivalent:
(test-section #:id 'math (test (+ 1 2) 3) ...)
(test-section 'math (test (+ 1 2) 3) ...)
1.4 Project Status and History
Work is ongoing, but Overeasy should be useful already. It is being
developed both as a useful tool, and as input to discussion in the Racket
developer community about unifying the various test engines.
This package does not yet provide an interface so that additional reporting
backends can be added. This is intentional, so that we can be comfortable
that the interface won’t be changing soon before others start developing to
it.
As a historical note, Overeasy is much superior to the author’s
2005 lightweight unit testing library, Testeez.
2 Interface
See above.
Deprecated. Alias for test-section.
See above.
3 History
- PLaneT 2:0 —
2012-06-11 Converted to McFly.
The default test context now raises an exception with syntax location info for failed test cases, rather than only writing a message to current-error-port.
The test syntax now preserves syntax location info better.
Added shorthand syntax (test ID CODE VAL RESTn ...).
The new name test-section is now preferred to the old name with-test-section.
In test-section syntax, the #:id keyword itself is now optional.
test-section may have no body forms.
- Version 0.1 —
PLaneT 1:0 — 2011-08-26 Initial release.
4 Legal
Copyright 2011–2012 Neil Van Dyke. This program is Free Software; Software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but without any warranty; without even the implied warranty of merchantability or fitness for a particular purpose. See http://www.gnu.org/licenses/ for details. For other licenses and consulting, please contact the author.