On this page:
2.1 Overview
2.2 Recording running times
benchmark-time
benchmark-time*
2.3 Analysing running times
faster?
as-fast?
faster*?
as-fast*?
2.4 Scheme Unit Integration
check-faster
check-as-fast
Version: 4.1.5.4

2 Benchmark API

 (require (planet schematics/benchmark:2))

2.1 Overview

2.2 Recording running times

The benchmark-time form records the running (CPU) time of some expressions.

(benchmark-time expr ...)

Runs the expr ten times, recording the CPU time for each run. Collects garbage between each call. Evaluates to a vector of exact integers.

Example:

  (benchmark-time (expt 20 20))

The same functionality is also available as a function

(benchmark-time* thunk)  
(vectorof
exact-integer?)
  thunk : (-> any)

Runs the thunk ten times, recording the CPU time for each run. Collects garbage between each call. Evaluates to a vector of exact integers.

2.3 Analysing running times

There are a number of macros and functions that are useful for analysing run times.

(faster? [name1 expr1 ...] [name2 expr2 ...])

The faster? macros compares the execution time of the expressions qexpr1 ... and expr2 ... using benchmark-time, and expands to code that returns #t if the first block takes less mean time than the second, and #f otherwise.

The strings name1 and name2 are used in reporting the results.

(as-fast? [name1 expr1 ...] [name2 expr2 ...])

Like faster?, but #t is returned if the mean time is less than or equal.

Here’s an example of faster? comparing computing a big factorial to a constant:

  (faster?
   ["factorial" (factorial 2048)]
   ["constant"  5])

The output should be similar to this:

factorial was NOT faster than constant
------------------------------------------------------------
Statistics for factorial
Mean:     3.6999999999999997
Std dev.: 0.48304589153964794
Statistics for constant
Mean:     0.0
Std dev.: 0.0
factorial took +inf.0 the time of constant
------------------------------------------------------------
#f

If you want more control the faster*? and as-fast*? functions may be useful.

(faster*? name1 times1 name2 times2 [verbose?])  (or/c #t #f)
  name1 : string?
  times1 : (vectorof number?)
  name2 : string?
  times2 : (vectorof number?)
  verbose? : (or/c #t #f) = #t

Compares the vectors of runtimes times1 and times2, evaluating to #t if the mean time of times1 is less than times2.

The verbose? flag determines if output is displayed to current-output-port detailing the mean and standard deviation of the runtimes, and the ratio of the mean runtimes.

The name arguments name1 and name2 are used in the reported generated by a verbose call.

(as-fast*? name1 times1 name2 times2 [verbose?])  (or/c #t #f)
  name1 : string?
  times1 : (vectorof number?)
  name2 : string?
  times2 : (vectorof number?)
  verbose? : (or/c #t #f) = #t

Like faster*? except evalutes to #t if the mean time of times1 is less or equal to the mean time of times2.

Here’s the above example using faster*?:

  (faster*?
   "factorial" (benchmark-time (factorial 2048))
   "constant" (benchmark-time 5))

The output looks the same:

factorial was NOT faster than constant
------------------------------------------------------------
Statistics for factorial
Mean:     3.9
Std dev.: 0.31622776601683794
Statistics for constant
Mean:     0.10000000000000002
Std dev.: 0.316227766016838
factorial took 38.99999999999999 the time of constant
------------------------------------------------------------
#f

2.4 SchemeUnit Integration

If you’re optimising code it is useful to record your assumptions about optimisations with your other tests. This way if something changes to invalidate an optimisation you will be made aware of it. For this reason the Benchmark library includes features that integrate with SchemeUnit.

(check-faster name1 thunk1 name2 thunk2)  #t
  name1 : string?
  thunk1 : (-> any)
  name2 : string?
  thunk2 : (-> any)

A check that succeeds if the mean CPU time of thunk1 is less than that of thunk2. The names are used in reporting.

(check-as-fast name1 thunk1 name2 thunk2)  #t
  name1 : string?
  thunk1 : (-> any)
  name2 : string?
  thunk2 : (-> any)

A check that succeeds if the mean CPU time of thunk1 is less than or equal that of thunk2. The names are used in reporting.