Next: Introduction, Previous: (dir), Up: (dir) [Contents][Index]
This is the trivial-benchmark Reference Manual, version 2.0.0, generated automatically by Declt version 4.0 beta 2 "William Riker" on Mon Aug 15 06:02:26 2022 GMT+0.
Next: Systems, Previous: The trivial-benchmark Reference Manual, Up: The trivial-benchmark Reference Manual [Contents][Index]
Frequently I want to do a quick benchmark comparison of my functions. TIME
is nice to get some data, but it's limited to a single run so there isn't really much of a statistical value in it. Trivial-Benchmark runs a block of code many times and outputs some statistical data for it. On SBCL this includes the data from TIME
, for all other implementations just the REAL- and RUN-TIME data. However, you can extend the system by adding your own metric
s to it, or even by adding additional statistical compute
ations.
For basic throwaway benchmarking, the with-timing
macro should suffice:
(benchmark:with-timing (1000)
(+ 1 1))
However, you can also do more complex timing using make-timer
and with-sampling
. The former creates a new timer object (with an optional list of metrics to sample) and the latter collects one sample for each metric of the timer for the duration of the body forms.
(defvar *timer* (benchmark:make-timer))
(benchmark:with-sampling (*timer*)
(+ 1 1))
(benchmark:with-sampling (*timer*)
(expt 10 100))
(benchmark:report *timer*)
(benchmark:reset *timer*)
(benchmark:report *timer*)
Sample Output:
If you're interested in adding additional metrics, you'll want to take a look at the metric
class, as well as the related methods, start
stop
discard
commit
take-sample
samples
sample-size
condense
reduce-samples
compute
report
reset
. For a basic metric
, you only need to implement start
, stop
, commit
, samples
, and take-sample
. The other functions have standard methods that will do their computations based on those five.
If you have a function that returns a current state of your sample and simply want to have the metric show the delta between start
and stop
, you can use define-delta-metric
.
(define-delta-metric (run-time internal-time-units-per-second)
(get-internal-run-time))
You can also implement a new computation type that is used to display the different metrics in the table. For that, simply write a method on compute
with the name of your computation as an eql-specializer and then push the name onto *default-computations*
so that it's always displayed when using report
. As an example, the maximum is simply calculated as follows:
(defmethod compute ((x (eql :maximum)) (metric metric))
(reduce-samples metric #'max))
Analogous to unit testing, performance testing or benchmark testing is a useful way to measure performance regressions in a code base. Trivial-Benchmark has preliminary support for defining and running benchmarks.
First, a benchmark package needs to be defined, which is essentially a normal Common Lisp package with some additional bookkeeping.
(define-benchmark-package #:my-app-benchmarks
(:use #:my-app)
(:export #:run-benchmarks))
The macro define-benchmark-package
automatically uses the COMMON-LISP
and the TRIVIAL-BENCHMARK
packages.
After the package has been defined, you can define benchmarks like normal Lisp function definitions using the define-benchmark
macro. Within the define-benchmark
macro, you use with-benchmark-sampling
around forms that you wish to measure.
(define-benchmark measure-trig ()
(declare (optimize speed))
(loop :repeat 100000
:for r := (random 1.0d0)
:do (with-benchmark-sampling
(+ (sin r)
(cos r)
(tan r)))))
The macro with-benchmark-sampling
can be called many times throughout the body of the define-benchmark
, and even called within functions called within the body. In other words, you're able to write something like this:
(defun trig-trial (r)
(with-benchmark-sampling
(+ (sin r) (cos r) (tan r))))
(defun inv-trig-trial (r)
(with-benchmark-sampling
(+ (asin r) (acos r) (atan r))))
(define-benchmark measure-all-trig ()
(declare (optimize speed))
(loop :repeat 100000
:for r := (random 1.0d0)
:do (trig-trial r)
(inv-trig-trial r)))
You can run the benchmarks of a package by doing:
(run-package-benchmarks :package ':my-app-benchmarks
:verbose t)
You can elide the :package
argument if you're in the package already. Running this will give:
> (run-package-benchmarks :package ':my-app-benchmarks
:verbose t)
Benchmarking MEASURE-ALL-TRIG...done.
Benchmarking BENCH-TRIG...done.
#<HASH-TABLE :TEST EQ :COUNT 2 {1002FD8243}>
The returned hash table contains a summary of all of the statistics, which you can further process as needed (e.g., record, plot, etc.). The table maps the benchmark names to an alist of metrics and their computed statistics. We can inspect this easily enough:
> (alexandria:hash-table-alist *)
((BENCH-TRIG
(REAL-TIME :SAMPLES 100000 :TOTAL 52/125 :MINIMUM 0 :MAXIMUM 3/1000 :MEDIAN 0
:AVERAGE 13/3125000 :DEVIATION 6.482819e-5)
(RUN-TIME :SAMPLES 100000 :TOTAL 383/1000 :MINIMUM 0 :MAXIMUM 1/1000 :MEDIAN
0 :AVERAGE 383/100000000 :DEVIATION 6.176837e-5)
(USER-RUN-TIME :SAMPLES 100000 :TOTAL 52663/250000 :MINIMUM 1/1000000
:MAXIMUM 53/1000000 :MEDIAN 1/500000 :AVERAGE 52663/25000000000 :DEVIATION
9.770944e-7)
(SYSTEM-RUN-TIME :SAMPLES 100000 :TOTAL 94299/500000 :MINIMUM 1/1000000
:MAXIMUM 87/1000000 :MEDIAN 1/500000 :AVERAGE 94299/50000000000 :DEVIATION
1.1286627e-6)
(PAGE-FAULTS :SAMPLES 100000 :TOTAL 0 :MINIMUM 0 :MAXIMUM 0 :MEDIAN 0
:AVERAGE 0 :DEVIATION 0.0)
(GC-RUN-TIME :SAMPLES 100000 :TOTAL 0 :MINIMUM 0 :MAXIMUM 0 :MEDIAN 0
:AVERAGE 0 :DEVIATION 0.0)
(BYTES-CONSED :SAMPLES 100000 :TOTAL 0 :MINIMUM 0 :MAXIMUM 0 :MEDIAN 0
:AVERAGE 0 :DEVIATION 0.0)
(EVAL-CALLS :SAMPLES 100000 :TOTAL 0 :MINIMUM 0 :MAXIMUM 0 :MEDIAN 0 :AVERAGE
0 :DEVIATION 0.0))
(MEASURE-ALL-TRIG
(REAL-TIME :SAMPLES 200000 :TOTAL 102/125 :MINIMUM 0 :MAXIMUM 1/200 :MEDIAN 0
:AVERAGE 51/12500000 :DEVIATION 6.4601496e-5)
(RUN-TIME :SAMPLES 200000 :TOTAL 77/100 :MINIMUM 0 :MAXIMUM 1/1000 :MEDIAN 0
:AVERAGE 77/20000000 :DEVIATION 6.192881e-5)
(USER-RUN-TIME :SAMPLES 200000 :TOTAL 41371/100000 :MINIMUM 1/1000000
:MAXIMUM 1/12500 :MEDIAN 1/500000 :AVERAGE 41371/20000000000 :DEVIATION
1.2130914e-6)
(SYSTEM-RUN-TIME :SAMPLES 200000 :TOTAL 363543/1000000 :MINIMUM 1/1000000
:MAXIMUM 11/100000 :MEDIAN 1/500000 :AVERAGE 363543/200000000000 :DEVIATION
1.4713012e-6)
(PAGE-FAULTS :SAMPLES 200000 :TOTAL 0 :MINIMUM 0 :MAXIMUM 0 :MEDIAN 0
:AVERAGE 0 :DEVIATION 0.0)
(GC-RUN-TIME :SAMPLES 200000 :TOTAL 0 :MINIMUM 0 :MAXIMUM 0 :MEDIAN 0
:AVERAGE 0 :DEVIATION 0.0)
(BYTES-CONSED :SAMPLES 200000 :TOTAL 15106048 :MINIMUM 0 :MAXIMUM 131072
:MEDIAN 0 :AVERAGE 236032/3125 :DEVIATION 1595.1276)
(EVAL-CALLS :SAMPLES 200000 :TOTAL 0 :MINIMUM 0 :MAXIMUM 0 :MEDIAN 0 :AVERAGE
0 :DEVIATION 0.0)))
Next: Files, Previous: Introduction, Up: The trivial-benchmark Reference Manual [Contents][Index]
The main system appears first, followed by any subsystem dependency.
An easy to use benchmarking system.
Nicolas Hafner <shinmera@tymoon.eu>
Nicolas Hafner <shinmera@tymoon.eu>
(GIT https://github.com/Shinmera/trivial-benchmark.git)
zlib
2.0.0
alexandria (system).
Next: Packages, Previous: Systems, Up: The trivial-benchmark Reference Manual [Contents][Index]
Files are sorted by type and then listed depth-first from the systems components trees.
Next: trivial-benchmark/package.lisp, Previous: Lisp, Up: Lisp [Contents][Index]
trivial-benchmark (system).
Next: trivial-benchmark/toolkit.lisp, Previous: trivial-benchmark/trivial-benchmark.asd, Up: Lisp [Contents][Index]
trivial-benchmark (system).
Next: trivial-benchmark/timer.lisp, Previous: trivial-benchmark/package.lisp, Up: Lisp [Contents][Index]
package.lisp (file).
trivial-benchmark (system).
Next: trivial-benchmark/samples.lisp, Previous: trivial-benchmark/toolkit.lisp, Up: Lisp [Contents][Index]
toolkit.lisp (file).
trivial-benchmark (system).
Next: trivial-benchmark/suite.lisp, Previous: trivial-benchmark/timer.lisp, Up: Lisp [Contents][Index]
timer.lisp (file).
trivial-benchmark (system).
Previous: trivial-benchmark/samples.lisp, Up: Lisp [Contents][Index]
samples.lisp (file).
trivial-benchmark (system).
Next: Definitions, Previous: Files, Up: The trivial-benchmark Reference Manual [Contents][Index]
Packages are listed by definition order.
common-lisp.
Next: Indexes, Previous: Packages, Up: The trivial-benchmark Reference Manual [Contents][Index]
Definitions are sorted by export status, category, package, and then by lexicographic order.
Next: Internals, Previous: Definitions, Up: Definitions [Contents][Index]
Next: Macros, Previous: Public Interface, Up: Public Interface [Contents][Index]
The list of computation-names used to print the REPORT table.
The list of class-names used to populate a TIMER by default.
Next: Ordinary functions, Previous: Special variables, Up: Public Interface [Contents][Index]
Define a benchmark named NAME. The structure of this macro is that of DEFUN, except that WITH-BENCHMARK-SAMPLING can be called to collect metrics on the enclosed forms.
ARGS must be a lambda list that does not require arguments present at the call site.
Define a package as if by DEFPACKAGE, except that both the COMMON-LISP and TRIVIAL-BENCHMARK packages are used, and that this package is designated as one containing benchmarks.
Shortcut to define a DELTA-METRIC.
The SAMPLE-POINT-FORMS should return a number to use to calculate a delta.
NAME can be either just a NAME or a list of the form (NAME UNITS) where
UNITS is the number the sample points are divided by when saving them.
For example, if you sample a function of milliseconds, but want to report
in seconds, set UNITS to 1000. Doing this instead of dividing directly in
the form avoids potential consing when running into bignums.
See DELTA-METRIC
Binds METRIC-VAR to each metric of TIMER and then evaluates FORMS. Returns the value of RESULT-FORM after the loop.
Benchmark the execution of the forms FORMS within the context of a DEFINE-BENCH macro. Acts like PROGN.
Takes a sample for the evaluation time of FORMS and stores it in the timer given by TIMER-FORM.
Acts like a PROGN.
Specifically, START is called, then FORMS are evaluated. If an error occurs within the body,
DISCARD is called on the timer, otherwise COMMIT is called once the body exits.
See START
See DISCARD
See COMMIT
Evaluates FORMS N times, using WITH-SAMPLING on the value of TIMER-FORM each iteration.
At the end, runs REPORT on the timer with STREAM and COMPUTATIONS.
See WITH-SAMPLING
Next: Generic functions, Previous: Macros, Up: Public Interface [Contents][Index]
Find the benchmark package designated by PACKAGE-DESIGNATOR and return it, otherwise return NIL if it doesn’t exist.
Creates a TIMER object using the given METRIC-TYPES
Maps the metrics in TIMER, calling FUNCTION with each.
Prints a table (each list in TABLE being a row) with proper spacing.
Rounds NUM to N digits after the dot.
Run all of the benchmarks associated with the benchmark package PACKAGE, the current one by default.
Return a hash table mapping benchmark names to their metrics represented as lists. In particular, it will be an alist whose CARs are metrics and whose CDRs are plists of computations.
VERBOSE is a generalized boolean that controls output verbosity while testing.
Returns T if A and B denote the same type (are subtypes of one another)
If one of the arguments does not denote a type, the result of TYPE-OF is used in their place.
Next: Standalone methods, Previous: Ordinary functions, Up: Public Interface [Contents][Index]
Commit the current sample of METRIC. If the metric is running, call STOP first.
Compute a value of the statistical computation THING for METRIC based on its current samples.
Turn the SAMPLE value into a usable number.
Discard the current sample of METRIC. If the metric is running, call STOP first.
Returns the metric of TYPE in TIMER if any. The metric must match the type by TYPE=
Sets the METRIC in TIMER.
The metric is replaced if it is found in the timer by TYPE= comparison.
Returns the types of metrics in TIMER.
Returns a list of metrics stored in TIMER.
Apply FUNCTION to the samples stored in METRIC in a REDUCE fashion.
Print a report of all COMPUTATIONS for THING to STREAM
STREAM can be one of the following:
T — Print to *standard-output*
NIL — Print to a string and return it.
STREAM — Print to the stream
Reset the METRIC and remove all its samples.
Returns T if the metric is currently sampling.
See START
See STOP
Return the number of samples stored in METRIC.
Return a sequence of committed samples stored in METRIC.
automatically generated reader method
automatically generated writer method
Begin a sample for METRIC. Sets RUNNING to T.
Returns the value stored when START was called on the DELTA-METRIC.
automatically generated reader method
automatically generated writer method
Return a current sampling value for METRIC.
Note that not all metrics must implement a method for this function. It is perfectly plausible for a metric to skip this method if it cannot provide a sample value at any point in time.
Samples SB-IMPL::*EVAL-CALLS*.
Samples SB-IMPL::GET-BYTES-CONSED.
Samples SB-IMPL::*GC-RUN-TIME* in seconds.
Samples the third value (page-faults) of SB-SYS:GET-SYSTEM-INFO in seconds.
Samples the second value (system-run-time) of SB-SYS:GET-SYSTEM-INFO in seconds.
Samples the first value (user-run-time) of SB-SYS:GET-SYSTEM-INFO in seconds.
Samples the results of GET-INTERNAL-RUN-TIME in seconds.
Next: Classes, Previous: Generic functions, Up: Public Interface [Contents][Index]
Previous: Standalone methods, Up: Public Interface [Contents][Index]
Samples SB-IMPL::GET-BYTES-CONSED.
1
Samples SB-IMPL::ELAPSED-CYCLES.
A LISTED-METRIC that calculates a sample point according to the delta between the START and COMMIT.
Sub-classes of this must implement the TAKE-SAMPLE method.
See LISTED-METRIC
Samples SB-IMPL::*EVAL-CALLS*.
1
Samples SB-IMPL::*GC-RUN-TIME* in seconds.
1000
A class container for sampling information.
:running
Samples the third value (page-faults) of SB-SYS:GET-SYSTEM-INFO in seconds.
1
Samples the results of GET-INTERNAL-REAL-TIME in seconds.
internal-time-units-per-second
Samples the results of GET-INTERNAL-RUN-TIME in seconds.
internal-time-units-per-second
Samples the second value (system-run-time) of SB-SYS:GET-SYSTEM-INFO in seconds.
1000000
Class container for a set of METRICS.
:metrics
Samples the first value (user-run-time) of SB-SYS:GET-SYSTEM-INFO in seconds.
1000000
A METRIC that implements its sample storage as a vector.
SAMPLES is SETF-able for this class.
See METRIC
(make-array 1000 :adjustable t :fill-pointer 0)
:samples
Previous: Public Interface, Up: Definitions [Contents][Index]
Next: Ordinary functions, Previous: Internals, Up: Internals [Contents][Index]
A table mapping package objects (which are benchmark packages) to a list of fbound symbols.
Next: Generic functions, Previous: Special variables, Up: Internals [Contents][Index]
Previous: Ordinary functions, Up: Internals [Contents][Index]
automatically generated reader method
h0.
automatically generated writer method
h0.
automatically generated reader method
h1.
automatically generated writer method
h1.
automatically generated reader method
l0.
automatically generated writer method
l0.
automatically generated reader method
l1.
automatically generated writer method
l1.
Stop the sample for METRIC. Sets RUNNING to NIL.
automatically generated reader method
automatically generated writer method
automatically generated reader method
automatically generated reader method
automatically generated reader method
automatically generated reader method
automatically generated reader method
automatically generated reader method
automatically generated reader method
automatically generated reader method
automatically generated reader method
automatically generated writer method
automatically generated writer method
automatically generated writer method
automatically generated writer method
automatically generated writer method
automatically generated writer method
automatically generated writer method
automatically generated writer method
automatically generated writer method
Previous: Definitions, Up: The trivial-benchmark Reference Manual [Contents][Index]
Jump to: | (
C D F G I M P R S T U W |
---|
Jump to: | (
C D F G I M P R S T U W |
---|
Next: Data types, Previous: Functions, Up: Indexes [Contents][Index]
Jump to: | *
H L M R S U |
---|
Jump to: | *
H L M R S U |
---|
Jump to: | B C D E F G M P R S T U V |
---|
Jump to: | B C D E F G M P R S T U V |
---|