#lang scribble/doc @(require scribble/manual) @(require scribble/eval) @(require "base.ss") @(require (for-label scheme/base "base.ss" "../main.ss")) @title[#:tag "api"]{Benchmark API} @defmodule[(planet schematics/benchmark:2)] @section{Overview} @section{Recording running times} The @scheme[benchmark-time] form records the running (CPU) time of some expressions. @defform[(benchmark-time expr ...)]{Runs the @scheme[expr] ten times, recording the CPU time for each run. Collects garbage between each call. Evaluates to a vector of exact integers.} Example: @schemeblock[(benchmark-time (expt 20 20))] The same functionality is also available as a function @defproc[(benchmark-time* (thunk (-> any))) (vectorof exact-integer?)]{ Runs the @scheme[thunk] ten times, recording the CPU time for each run. Collects garbage between each call. Evaluates to a vector of exact integers.} @section{Analysing running times} There are a number of macros and functions that are useful for analysing run times. @defform[(faster? [name1 expr1 ...] [name2 expr2 ...])]{The @scheme[faster?] macros compares the execution time of the expressions q@scheme[expr1 ...] and @scheme[expr2 ...] using @scheme[benchmark-time], and expands to code that returns @scheme[#t] if the first block takes less mean time than the second, and @scheme[#f] otherwise. The strings @scheme[name1] and @scheme[name2] are used in reporting the results.} @defform[(as-fast? [name1 expr1 ...] [name2 expr2 ...])]{Like @scheme[faster?], but @scheme[#t] is returned if the mean time is less than or equal. } Here's an example of @scheme[faster?] comparing computing a big factorial to a constant: @schemeblock[ (faster? ["factorial" (factorial 2048)] ["constant" 5]) ] The output should be similar to this: @verbatim{ factorial was NOT faster than constant ------------------------------------------------------------ Statistics for factorial Mean: 3.6999999999999997 Std dev.: 0.48304589153964794 Statistics for constant Mean: 0.0 Std dev.: 0.0 factorial took +inf.0 the time of constant ------------------------------------------------------------ #f } If you want more control the @scheme[faster*?] and @scheme[as-fast*?] functions may be useful. @defproc[(faster*? (name1 string?) (times1 (vectorof number?)) (name2 string?) (times2 (vectorof number?)) [verbose? (or/c #t #f) #t]) (or/c #t #f)]{ Compares the vectors of runtimes @scheme[times1] and @scheme[times2], evaluating to @scheme[#t] if the mean time of @scheme[times1] is less than @scheme[times2]. The @scheme[verbose?] flag determines if output is displayed to @scheme[current-output-port] detailing the mean and standard deviation of the runtimes, and the ratio of the mean runtimes. The name arguments @scheme[name1] and @scheme[name2] are used in the reported generated by a verbose call. } @defproc[(as-fast*? (name1 string?) (times1 (vectorof number?)) (name2 string?) (times2 (vectorof number?)) [verbose? (or/c #t #f) #t]) (or/c #t #f)]{ Like @scheme[faster*?] except evalutes to @scheme[#t] if the mean time of @scheme[times1] is less or equal to the mean time of @scheme[times2]. } Here's the above example using @scheme[faster*?]: @schemeblock[ (faster*? "factorial" (benchmark-time (factorial 2048)) "constant" (benchmark-time 5)) ] The output looks the same: @verbatim{ factorial was NOT faster than constant ------------------------------------------------------------ Statistics for factorial Mean: 3.9 Std dev.: 0.31622776601683794 Statistics for constant Mean: 0.10000000000000002 Std dev.: 0.316227766016838 factorial took 38.99999999999999 the time of constant ------------------------------------------------------------ #f } @section{SchemeUnit Integration} If you're optimising code it is useful to record your assumptions about optimisations with your other tests. This way if something changes to invalidate an optimisation you will be made aware of it. For this reason the Benchmark library includes features that integrate with SchemeUnit. @defproc[(check-faster (name1 string?) (thunk1 (-> any)) (name2 string?) (thunk2 (-> any))) #t]{ A check that succeeds if the mean CPU time of @scheme[thunk1] is less than that of @scheme[thunk2]. The names are used in reporting. } @defproc[(check-as-fast (name1 string?) (thunk1 (-> any)) (name2 string?) (thunk2 (-> any))) #t]{ A check that succeeds if the mean CPU time of @scheme[thunk1] is less than or equal that of @scheme[thunk2]. The names are used in reporting. }