The trivial-benchmark Reference Manual

Table of Contents

The trivial-benchmark Reference Manual

This is the trivial-benchmark Reference Manual, version 2.0.0, generated automatically by Declt version 2.4 "Will Decker" on Wed Jun 20 12:41:01 2018 GMT+0.

1 Introduction

About Trivial-Benchmark

Frequently I want to do a quick benchmark comparison of my functions. TIME is nice to get some data, but it's limited to a single run so there isn't really much of a statistical value in it. Trivial-Benchmark runs a block of code many times and outputs some statistical data for it. On SBCL this includes the data from TIME, for all other implementations just the REAL- and RUN-TIME data. However, you can extend the system by adding your own metrics to it, or even by adding additional statistical computeations.

Measurement How-To

For basic throwaway benchmarking, the with-timing macro should suffice:

(benchmark:with-timing (1000)
  (+ 1 1))

However, you can also do more complex timing using make-timer and with-sampling. The former creates a new timer object (with an optional list of metrics to sample) and the latter collects one sample for each metric of the timer for the duration of the body forms.

(defvar *timer* (benchmark:make-timer))

(benchmark:with-sampling (*timer*)
  (+ 1 1))

(benchmark:with-sampling (*timer*)
  (expt 10 100))

(benchmark:report *timer*)

(benchmark:reset *timer*)

(benchmark:report *timer*)

Sample Output:

Extending Trivial-Benchmark

If you're interested in adding additional metrics, you'll want to take a look at the metric class, as well as the related methods, start stop discard commit take-sample samples sample-size condense reduce-samples compute report reset. For a basic metric, you only need to implement start, stop, commit, samples, and take-sample. The other functions have standard methods that will do their computations based on those five.

If you have a function that returns a current state of your sample and simply want to have the metric show the delta between start and stop, you can use define-delta-metric.

(define-delta-metric (run-time internal-time-units-per-second)
  (get-internal-run-time))

You can also implement a new computation type that is used to display the different metrics in the table. For that, simply write a method on compute with the name of your computation as an eql-specializer and then push the name onto *default-computations* so that it's always displayed when using report. As an example, the maximum is simply calculated as follows:

(defmethod compute ((x (eql :maximum)) (metric metric))
  (reduce-samples metric #'max))

Benchmark Suites

Analogous to unit testing, performance testing or benchmark testing is a useful way to measure performance regressions in a code base. Trivial-Benchmark has preliminary support for defining and running benchmarks.

First, a benchmark package needs to be defined, which is essentially a normal Common Lisp package with some additional bookkeeping.

(define-benchmark-package #:my-app-benchmarks
  (:use #:my-app)
  (:export #:run-benchmarks))

The macro define-benchmark-package automatically uses the COMMON-LISP and the TRIVIAL-BENCHMARK packages.

After the package has been defined, you can define benchmarks like normal Lisp function definitions using the define-benchmark macro. Within the define-benchmark macro, you use with-benchmark-sampling around forms that you wish to measure.

(define-benchmark measure-trig ()
  (declare (optimize speed))
  (loop :repeat 100000
        :for r := (random 1.0d0)
        :do (with-benchmark-sampling
              (+ (sin r)
                 (cos r)
                 (tan r)))))

The macro with-benchmark-sampling can be called many times throughout the body of the define-benchmark, and even called within functions called within the body. In other words, you're able to write something like this:

(defun trig-trial (r)
  (with-benchmark-sampling
    (+ (sin r) (cos r) (tan r))))

(defun inv-trig-trial (r)
  (with-benchmark-sampling
    (+ (asin r) (acos r) (atan r))))

(define-benchmark measure-all-trig ()
  (declare (optimize speed))
  (loop :repeat 100000
        :for r := (random 1.0d0)
        :do (trig-trial r)
            (inv-trig-trial r)))

You can run the benchmarks of a package by doing:

(run-package-benchmarks :package ':my-app-benchmarks
                        :verbose t)

You can elide the :package argument if you're in the package already. Running this will give:

> (run-package-benchmarks :package ':my-app-benchmarks 
                          :verbose t)
Benchmarking MEASURE-ALL-TRIG...done.
Benchmarking BENCH-TRIG...done.
#<HASH-TABLE :TEST EQ :COUNT 2 {1002FD8243}>

The returned hash table contains a summary of all of the statistics, which you can further process as needed (e.g., record, plot, etc.). The table maps the benchmark names to an alist of metrics and their computed statistics. We can inspect this easily enough:

> (alexandria:hash-table-alist *)
((BENCH-TRIG
  (REAL-TIME :SAMPLES 100000 :TOTAL 52/125 :MINIMUM 0 :MAXIMUM 3/1000 :MEDIAN 0
   :AVERAGE 13/3125000 :DEVIATION 6.482819e-5)
  (RUN-TIME :SAMPLES 100000 :TOTAL 383/1000 :MINIMUM 0 :MAXIMUM 1/1000 :MEDIAN
   0 :AVERAGE 383/100000000 :DEVIATION 6.176837e-5)
  (USER-RUN-TIME :SAMPLES 100000 :TOTAL 52663/250000 :MINIMUM 1/1000000
   :MAXIMUM 53/1000000 :MEDIAN 1/500000 :AVERAGE 52663/25000000000 :DEVIATION
   9.770944e-7)
  (SYSTEM-RUN-TIME :SAMPLES 100000 :TOTAL 94299/500000 :MINIMUM 1/1000000
   :MAXIMUM 87/1000000 :MEDIAN 1/500000 :AVERAGE 94299/50000000000 :DEVIATION
   1.1286627e-6)
  (PAGE-FAULTS :SAMPLES 100000 :TOTAL 0 :MINIMUM 0 :MAXIMUM 0 :MEDIAN 0
   :AVERAGE 0 :DEVIATION 0.0)
  (GC-RUN-TIME :SAMPLES 100000 :TOTAL 0 :MINIMUM 0 :MAXIMUM 0 :MEDIAN 0
   :AVERAGE 0 :DEVIATION 0.0)
  (BYTES-CONSED :SAMPLES 100000 :TOTAL 0 :MINIMUM 0 :MAXIMUM 0 :MEDIAN 0
   :AVERAGE 0 :DEVIATION 0.0)
  (EVAL-CALLS :SAMPLES 100000 :TOTAL 0 :MINIMUM 0 :MAXIMUM 0 :MEDIAN 0 :AVERAGE
   0 :DEVIATION 0.0))
 (MEASURE-ALL-TRIG
  (REAL-TIME :SAMPLES 200000 :TOTAL 102/125 :MINIMUM 0 :MAXIMUM 1/200 :MEDIAN 0
   :AVERAGE 51/12500000 :DEVIATION 6.4601496e-5)
  (RUN-TIME :SAMPLES 200000 :TOTAL 77/100 :MINIMUM 0 :MAXIMUM 1/1000 :MEDIAN 0
   :AVERAGE 77/20000000 :DEVIATION 6.192881e-5)
  (USER-RUN-TIME :SAMPLES 200000 :TOTAL 41371/100000 :MINIMUM 1/1000000
   :MAXIMUM 1/12500 :MEDIAN 1/500000 :AVERAGE 41371/20000000000 :DEVIATION
   1.2130914e-6)
  (SYSTEM-RUN-TIME :SAMPLES 200000 :TOTAL 363543/1000000 :MINIMUM 1/1000000
   :MAXIMUM 11/100000 :MEDIAN 1/500000 :AVERAGE 363543/200000000000 :DEVIATION
   1.4713012e-6)
  (PAGE-FAULTS :SAMPLES 200000 :TOTAL 0 :MINIMUM 0 :MAXIMUM 0 :MEDIAN 0
   :AVERAGE 0 :DEVIATION 0.0)
  (GC-RUN-TIME :SAMPLES 200000 :TOTAL 0 :MINIMUM 0 :MAXIMUM 0 :MEDIAN 0
   :AVERAGE 0 :DEVIATION 0.0)
  (BYTES-CONSED :SAMPLES 200000 :TOTAL 15106048 :MINIMUM 0 :MAXIMUM 131072
   :MEDIAN 0 :AVERAGE 236032/3125 :DEVIATION 1595.1276)
  (EVAL-CALLS :SAMPLES 200000 :TOTAL 0 :MINIMUM 0 :MAXIMUM 0 :MEDIAN 0 :AVERAGE
   0 :DEVIATION 0.0)))

2 Systems

The main system appears first, followed by any subsystem dependency.

2.1 trivial-benchmark

Maintainer

Nicolas Hafner <shinmera@tymoon.eu>

Author

Nicolas Hafner <shinmera@tymoon.eu>

License

Artistic

Description

An easy to use benchmarking system.

Version

2.0.0

Dependency

alexandria

Source

trivial-benchmark.asd (file)

Components

• package.lisp (file)
• toolkit.lisp (file)
• timer.lisp (file)
• samples.lisp (file)
• suite.lisp (file)

3 Files

Files are sorted by type and then listed depth-first from the systems components trees.

3.1 Lisp

3.1.1 trivial-benchmark.asd

Location

trivial-benchmark.asd

Systems

trivial-benchmark (system)

3.1.2 trivial-benchmark/package.lisp

Parent

trivial-benchmark (system)

Location

package.lisp

Packages

trivial-benchmark

3.1.3 trivial-benchmark/toolkit.lisp

Dependency

package.lisp (file)

Parent

trivial-benchmark (system)

Location

toolkit.lisp

Exported Definitions

• print-table (function)
• round-to (function)
• type= (function)

3.1.4 trivial-benchmark/timer.lisp

Dependency

toolkit.lisp (file)

Parent

trivial-benchmark (system)

Location

timer.lisp

Exported Definitions

• *default-computations* (special variable)
• *default-metrics* (special variable)
• commit (generic function)
• commit (method)
• commit (method)
• compute (generic function)
• compute (method)
• compute (method)
• compute (method)
• compute (method)
• compute (method)
• compute (method)
• compute (method)
• compute (method)
• compute (method)
• condense (generic function)
• condense (method)
• discard (generic function)
• discard (method)
• discard (method)
• do-metrics (macro)
• make-timer (function)
• map-metrics (function)
• metric (generic function)
• metric (method)
• (setf metric) (generic function)
• (setf metric) (method)
• metric (class)
• metric-types (generic function)
• metric-types (method)
• metrics (generic function)
• metrics (method)
• (setf metrics) (method)
• reduce-samples (generic function)
• reduce-samples (method)
• report (generic function)
• report (method)
• reset (generic function)
• reset (method)
• running (generic function)
• running (method)
• (setf running) (method)
• sample-size (generic function)
• sample-size (method)
• samples (generic function)
• start (generic function)
• start (method)
• start (method)
• take-sample (generic function)
• take-sample (method)
• timer (class)
• with-sampling (macro)
• with-timing (macro)

Internal Definitions

• format-timer-stats (function)
• report-to (generic function)
• report-to (method)
• report-to (method)
• report-to (method)
• report-to (method)
• stop (generic function)
• stop (method)
• stop (method)
• stop (method)

3.1.5 trivial-benchmark/samples.lisp

Dependency

timer.lisp (file)

Parent

trivial-benchmark (system)

Location

samples.lisp

Exported Definitions

• bytes-consed (class)
• commit (method)
• commit (method)
• cpu-cycles (class)
• define-delta-metric (macro)
• delta-metric (class)
• discard (method)
• discard (method)
• eval-calls (class)
• gc-run-time (class)
• page-faults (class)
• real-time (class)
• reset (method)
• reset (method)
• run-time (class)
• samples (method)
• (setf samples) (method)
• start (method)
• start (method)
• starting-value (generic function)
• starting-value (method)
• (setf starting-value) (method)
• system-run-time (class)
• take-sample (method)
• take-sample (method)
• take-sample (method)
• take-sample (method)
• take-sample (method)
• take-sample (method)
• take-sample (method)
• take-sample (method)
• user-run-time (class)
• vector-metric (class)

Internal Definitions

• cpu-cycles-h0 (method)
• (setf cpu-cycles-h0) (method)
• cpu-cycles-h1 (method)
• (setf cpu-cycles-h1) (method)
• cpu-cycles-l0 (method)
• (setf cpu-cycles-l0) (method)
• cpu-cycles-l1 (method)
• (setf cpu-cycles-l1) (method)
• stop (method)
• stop (method)
• stopping-value (method)
• (setf stopping-value) (method)
• units (method)
• (setf units) (method)
• units (method)
• (setf units) (method)
• units (method)
• (setf units) (method)
• units (method)
• (setf units) (method)
• units (method)
• (setf units) (method)
• units (method)
• (setf units) (method)
• units (method)
• (setf units) (method)
• units (method)
• (setf units) (method)
• units (method)
• (setf units) (method)

3.1.6 trivial-benchmark/suite.lisp

Dependency

samples.lisp (file)

Parent

trivial-benchmark (system)

Location

suite.lisp

Exported Definitions

• define-benchmark (macro)
• define-benchmark-package (macro)
• find-benchmark-package (function)
• run-package-benchmarks (function)
• with-benchmark-sampling (macro)

Internal Definitions

• *benchmark-packages* (special variable)
• *current-benchmark* (special variable)
• *current-suite-report* (special variable)
• *current-timer* (special variable)
• metrics-alist (function)

4 Packages

Packages are listed by definition order.

4.1 trivial-benchmark

Source

package.lisp (file)

Nicknames

• benchmark
• org.shirakumo.trivial-benchmark

Use List

common-lisp

Exported Definitions

• *default-computations* (special variable)
• *default-metrics* (special variable)
• bytes-consed (class)
• commit (generic function)
• commit (method)
• commit (method)
• commit (method)
• commit (method)
• compute (generic function)
• compute (method)
• compute (method)
• compute (method)
• compute (method)
• compute (method)
• compute (method)
• compute (method)
• compute (method)
• compute (method)
• condense (generic function)
• condense (method)
• cpu-cycles (class)
• define-benchmark (macro)
• define-benchmark-package (macro)
• define-delta-metric (macro)
• delta-metric (class)
• discard (generic function)
• discard (method)
• discard (method)
• discard (method)
• discard (method)
• do-metrics (macro)
• eval-calls (class)
• find-benchmark-package (function)
• gc-run-time (class)
• make-timer (function)
• map-metrics (function)
• metric (generic function)
• metric (method)
• (setf metric) (generic function)
• (setf metric) (method)
• metric (class)
• metric-types (generic function)
• metric-types (method)
• metrics (generic function)
• metrics (method)
• (setf metrics) (generic function)
• (setf metrics) (method)
• page-faults (class)
• print-table (function)
• real-time (class)
• reduce-samples (generic function)
• reduce-samples (method)
• report (generic function)
• report (method)
• reset (generic function)
• reset (method)
• reset (method)
• reset (method)
• round-to (function)
• run-package-benchmarks (function)
• run-time (class)
• running (generic function)
• running (method)
• (setf running) (generic function)
• (setf running) (method)
• sample-size (generic function)
• sample-size (method)
• samples (generic function)
• samples (method)
• (setf samples) (generic function)
• (setf samples) (method)
• start (generic function)
• start (method)
• start (method)
• start (method)
• start (method)
• starting-value (generic function)
• starting-value (method)
• (setf starting-value) (generic function)
• (setf starting-value) (method)
• system-run-time (class)
• take-sample (generic function)
• take-sample (method)
• take-sample (method)
• take-sample (method)
• take-sample (method)
• take-sample (method)
• take-sample (method)
• take-sample (method)
• take-sample (method)
• take-sample (method)
• timer (class)
• type= (function)
• user-run-time (class)
• vector-metric (class)
• with-benchmark-sampling (macro)
• with-sampling (macro)
• with-timing (macro)

Internal Definitions

• *benchmark-packages* (special variable)
• *current-benchmark* (special variable)
• *current-suite-report* (special variable)
• *current-timer* (special variable)
• cpu-cycles-h0 (generic function)
• cpu-cycles-h0 (method)
• (setf cpu-cycles-h0) (method)
• (setf cpu-cycles-h0) (generic function)
• cpu-cycles-h1 (generic function)
• cpu-cycles-h1 (method)
• (setf cpu-cycles-h1) (method)
• (setf cpu-cycles-h1) (generic function)
• cpu-cycles-l0 (generic function)
• cpu-cycles-l0 (method)
• (setf cpu-cycles-l0) (method)
• (setf cpu-cycles-l0) (generic function)
• cpu-cycles-l1 (generic function)
• cpu-cycles-l1 (method)
• (setf cpu-cycles-l1) (method)
• (setf cpu-cycles-l1) (generic function)
• format-timer-stats (function)
• metrics-alist (function)
• report-to (generic function)
• report-to (method)
• report-to (method)
• report-to (method)
• report-to (method)
• stop (generic function)
• stop (method)
• stop (method)
• stop (method)
• stop (method)
• stop (method)
• stopping-value (generic function)
• stopping-value (method)
• (setf stopping-value) (method)
• (setf stopping-value) (generic function)
• units (generic function)
• units (method)
• (setf units) (method)
• units (method)
• (setf units) (method)
• units (method)
• (setf units) (method)
• units (method)
• (setf units) (method)
• units (method)
• (setf units) (method)
• units (method)
• (setf units) (method)
• units (method)
• (setf units) (method)
• units (method)
• (setf units) (method)
• units (method)
• (setf units) (method)
• (setf units) (generic function)

5 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.

5.1 Exported definitions

5.1.1 Special variables

Special Variable: *default-computations*

The list of computation-names used to print the REPORT table.

Package

trivial-benchmark

Source

timer.lisp (file)

Special Variable: *default-metrics*

The list of class-names used to populate a TIMER by default.

Package

trivial-benchmark

Source

timer.lisp (file)

5.1.2 Macros

Macro: define-benchmark NAME ARGS &body BODY

Define a benchmark named NAME. The structure of this macro is that of DEFUN, except that WITH-BENCHMARK-SAMPLING can be called to collect metrics on the enclosed forms.

ARGS must be a lambda list that does not require arguments present at the call site.

Package

trivial-benchmark

Source

suite.lisp (file)

Macro: define-benchmark-package NAME &body PACKAGE-OPTIONS

Define a package as if by DEFPACKAGE, except that both the COMMON-LISP and TRIVIAL-BENCHMARK packages are used, and that this package is designated as one containing benchmarks.

Package

trivial-benchmark

Source

suite.lisp (file)

Macro: define-delta-metric NAME &body SAMPLE-POINT-FORMS

Shortcut to define a DELTA-METRIC.
The SAMPLE-POINT-FORMS should return a number to use to calculate a delta. NAME can be either just a NAME or a list of the form (NAME UNITS) where UNITS is the number the sample points are divided by when saving them. For example, if you sample a function of milliseconds, but want to report in seconds, set UNITS to 1000. Doing this instead of dividing directly in the form avoids potential consing when running into bignums.

See DELTA-METRIC

Package

trivial-benchmark

Source

samples.lisp (file)

Macro: do-metrics (METRIC-VAR TIMER &optional RESULT-FORM) &body FORMS

Binds METRIC-VAR to each metric of TIMER and then evaluates FORMS. Returns the value of RESULT-FORM after the loop.

Package

trivial-benchmark

Source

timer.lisp (file)

Macro: with-benchmark-sampling &body FORMS

Benchmark the execution of the forms FORMS within the context of a DEFINE-BENCH macro. Acts like PROGN.

Package

trivial-benchmark

Source

suite.lisp (file)

Macro: with-sampling (TIMER-FORM) &body FORMS

Takes a sample for the evaluation time of FORMS and stores it in the timer given by TIMER-FORM. Acts like a PROGN.

Specifically, START is called, then FORMS are evaluated. If an error occurs within the body, DISCARD is called on the timer, otherwise COMMIT is called once the body exits.

See START
See DISCARD
See COMMIT

Package

trivial-benchmark

Source

timer.lisp (file)

Macro: with-timing (N &optional TIMER-FORM STREAM COMPUTATIONS) &body FORMS

Evaluates FORMS N times, using WITH-SAMPLING on the value of TIMER-FORM each iteration. At the end, runs REPORT on the timer with STREAM and COMPUTATIONS.

See WITH-SAMPLING

Package

trivial-benchmark

Source

timer.lisp (file)

5.1.3 Functions

Function: find-benchmark-package PACKAGE-DESIGNATOR

Find the benchmark package designated by PACKAGE-DESIGNATOR and return it, otherwise return NIL if it doesn’t exist.

Package

trivial-benchmark

Source

suite.lisp (file)

Function: make-timer &optional METRIC-TYPES

Creates a TIMER object using the given METRIC-TYPES

Package

trivial-benchmark

Source

timer.lisp (file)

Function: map-metrics TIMER FUNCTION

Maps the metrics in TIMER, calling FUNCTION with each.

Package

trivial-benchmark

Source

timer.lisp (file)

Function: print-table TABLE &key STREAM PADDING

Prints a table (each list in TABLE being a row) with proper spacing.

Package

trivial-benchmark

Source

toolkit.lisp (file)

Function: round-to NUM N

Rounds NUM to N digits after the dot.

Package

trivial-benchmark

Source

toolkit.lisp (file)

Function: run-package-benchmarks &key PACKAGE VERBOSE

Run all of the benchmarks associated with the benchmark package PACKAGE, the current one by default.

Return a hash table mapping benchmark names to their metrics represented as lists. In particular, it will be an alist whose CARs are metrics and whose CDRs are plists of computations.

VERBOSE is a generalized boolean that controls output verbosity while testing.

Package

trivial-benchmark

Source

suite.lisp (file)

Function: type= A B

Returns T if A and B denote the same type (are subtypes of one another)
If one of the arguments does not denote a type, the result of TYPE-OF is used in their place.

Package

trivial-benchmark

Source

toolkit.lisp (file)

5.1.4 Generic functions

Generic Function: commit METRIC

Commit the current sample of METRIC. If the metric is running, call STOP first.

Package

trivial-benchmark

Source

timer.lisp (file)

Methods

Method: commit (METRIC cpu-cycles)

Source

samples.lisp (file)

Method: commit (METRIC delta-metric)

Source

samples.lisp (file)

Method: commit (TIMER timer)

Method: commit (METRIC metric) before

Generic Function: compute THING METRIC

Compute a value of the statistical computation THING for METRIC based on its current samples.

Package

trivial-benchmark

Source

timer.lisp (file)

Methods

Method: compute (X (eql count)) (METRIC metric)

Method: compute (X (eql samples)) (METRIC metric)

Method: compute (X (eql total)) (METRIC metric)

Method: compute (X (eql minimum)) (METRIC metric)

Method: compute (X (eql maximum)) (METRIC metric)

Method: compute (X (eql median)) (METRIC metric)

Method: compute (X (eql average)) (METRIC metric)

Method: compute (X (eql deviation)) (METRIC metric)

Method: compute (COMPUTATIONS list) (METRIC metric)

Generic Function: condense SAMPLE

Turn the SAMPLE value into a usable number.

Package

trivial-benchmark

Source

timer.lisp (file)

Methods

Method: condense THING

Generic Function: discard METRIC

Discard the current sample of METRIC. If the metric is running, call STOP first.

Package

trivial-benchmark

Source

timer.lisp (file)

Methods

Method: discard (METRIC cpu-cycles)

Source

samples.lisp (file)

Method: discard (METRIC delta-metric)

Source

samples.lisp (file)

Method: discard (TIMER timer)

Method: discard (METRIC metric) before

Generic Function: metric TYPE TIMER

Returns the metric of TYPE in TIMER if any. The metric must match the type by TYPE=

Package

trivial-benchmark

Source

timer.lisp (file)

Writer

(setf metric) (generic function)

Methods

Method: metric (TYPE symbol) (TIMER timer)

Generic Function: (setf metric) METRIC TIMER

Sets the METRIC in TIMER.
The metric is replaced if it is found in the timer by TYPE= comparison.

Package

trivial-benchmark

Source

timer.lisp (file)

Reader

metric (generic function)

Methods

Method: (setf metric) (METRIC metric) (TIMER timer)

Generic Function: metric-types TIMER

Returns the types of metrics in TIMER.

Package

trivial-benchmark

Source

timer.lisp (file)

Methods

Method: metric-types (TIMER timer)

Generic Function: metrics TIMER

Returns a list of metrics stored in TIMER.

Package

trivial-benchmark

Source

timer.lisp (file)

Writer

(setf metrics) (generic function)

Methods

Method: metrics (TIMER timer)

automatically generated reader method

Generic Function: (setf metrics) NEW-VALUE OBJECT

Package

trivial-benchmark

Reader

metrics (generic function)

Methods

Method: (setf metrics) NEW-VALUE (TIMER timer)

automatically generated writer method

Source

timer.lisp (file)

Generic Function: reduce-samples METRIC FUNCTION

Apply FUNCTION to the samples stored in METRIC in a REDUCE fashion.

Package

trivial-benchmark

Source

timer.lisp (file)

Methods

Method: reduce-samples METRIC FUNCTION

Generic Function: report THING &key STREAM COMPUTATIONS

Print a report of all COMPUTATIONS for THING to STREAM

STREAM can be one of the following:
T — Print to *standard-output*
NIL — Print to a string and return it. STREAM — Print to the stream

Package

trivial-benchmark

Source

timer.lisp (file)

Methods

Method: report THING &key STREAM COMPUTATIONS

Generic Function: reset METRIC

Reset the METRIC and remove all its samples.

Package

trivial-benchmark

Source

timer.lisp (file)

Methods

Method: reset (METRIC vector-metric) after

Source

samples.lisp (file)

Method: reset (METRIC vector-metric)

Source

samples.lisp (file)

Method: reset (TIMER timer)

Generic Function: running METRIC

Returns T if the metric is currently sampling.

See START
See STOP

Package

trivial-benchmark

Source

timer.lisp (file)

Writer

(setf running) (generic function)

Methods

Method: running (METRIC metric)

automatically generated reader method

Generic Function: (setf running) NEW-VALUE OBJECT

Package

trivial-benchmark

Reader

running (generic function)

Methods

Method: (setf running) NEW-VALUE (METRIC metric)

automatically generated writer method

Source

timer.lisp (file)

Generic Function: sample-size METRIC

Return the number of samples stored in METRIC.

Package

trivial-benchmark

Source

timer.lisp (file)

Methods

Method: sample-size METRIC

Generic Function: samples METRIC

Return a sequence of committed samples stored in METRIC.

Package

trivial-benchmark

Source

timer.lisp (file)

Writer

(setf samples) (generic function)

Methods

Method: samples (VECTOR-METRIC vector-metric)

automatically generated reader method

Source

samples.lisp (file)

Generic Function: (setf samples) NEW-VALUE OBJECT

Package

trivial-benchmark

Reader

samples (generic function)

Methods

Method: (setf samples) NEW-VALUE (VECTOR-METRIC vector-metric)

automatically generated writer method

Source

samples.lisp (file)

Generic Function: start METRIC

Begin a sample for METRIC. Sets RUNNING to T.

Package

trivial-benchmark

Source

timer.lisp (file)

Methods

Method: start (METRIC cpu-cycles)

Source

samples.lisp (file)

Method: start (METRIC delta-metric)

Source

samples.lisp (file)

Method: start (TIMER timer)

Method: start (METRIC metric) around

Generic Function: starting-value DELTA-METRIC

Returns the value stored when START was called on the DELTA-METRIC.

Package

trivial-benchmark

Source

samples.lisp (file)

Writer

(setf starting-value) (generic function)

Methods

Method: starting-value (DELTA-METRIC delta-metric)

automatically generated reader method

Generic Function: (setf starting-value) NEW-VALUE OBJECT

Package

trivial-benchmark

Reader

starting-value (generic function)

Methods

Method: (setf starting-value) NEW-VALUE (DELTA-METRIC delta-metric)

automatically generated writer method

Source

samples.lisp (file)

Generic Function: take-sample METRIC

Return a current sampling value for METRIC.

Note that not all metrics must implement a method for this function. It is perfectly plausible for a metric to skip this method if it cannot provide a sample value at any point in time.

Package

trivial-benchmark

Source

timer.lisp (file)

Methods

Method: take-sample (METRIC eval-calls)

Samples SB-IMPL::*EVAL-CALLS*.

Source

samples.lisp (file)

Method: take-sample (METRIC bytes-consed)

Samples SB-IMPL::GET-BYTES-CONSED.

Source

samples.lisp (file)

Method: take-sample (METRIC gc-run-time)

Samples SB-IMPL::*GC-RUN-TIME* in seconds.

Source

samples.lisp (file)

Method: take-sample (METRIC page-faults)

Samples the third value (page-faults) of SB-SYS:GET-SYSTEM-INFO in seconds.

Source

samples.lisp (file)

Method: take-sample (METRIC system-run-time)

Samples the second value (system-run-time) of SB-SYS:GET-SYSTEM-INFO in seconds.

Source

samples.lisp (file)

Method: take-sample (METRIC user-run-time)

Samples the first value (user-run-time) of SB-SYS:GET-SYSTEM-INFO in seconds.

Source

samples.lisp (file)

Method: take-sample (METRIC run-time)

Samples the results of GET-INTERNAL-RUN-TIME in seconds.

Source

samples.lisp (file)

Method: take-sample (METRIC real-time)

Samples the results of GET-INTERNAL-REAL-TIME in seconds.

Source

samples.lisp (file)

Method: take-sample (TIMER timer)

5.1.5 Classes

Class: bytes-consed ()

Samples SB-IMPL::GET-BYTES-CONSED.

Package

trivial-benchmark

Source

samples.lisp (file)

Direct superclasses

delta-metric (class)

Direct methods

• take-sample (method)
• units (method)
• units (method)

Direct slots

Slot: units

Initform

1

Readers

units (generic function)

Writers

(setf units) (generic function)

Class: cpu-cycles ()

Samples SB-IMPL::ELAPSED-CYCLES.

Package

trivial-benchmark

Source

samples.lisp (file)

Direct superclasses

vector-metric (class)

Direct methods

• commit (method)
• discard (method)
• stop (method)
• start (method)
• cpu-cycles-l1 (method)
• cpu-cycles-l1 (method)
• cpu-cycles-h1 (method)
• cpu-cycles-h1 (method)
• cpu-cycles-l0 (method)
• cpu-cycles-l0 (method)
• cpu-cycles-h0 (method)
• cpu-cycles-h0 (method)

Direct slots

Slot: h0

Readers

cpu-cycles-h0 (generic function)

Writers

(setf cpu-cycles-h0) (generic function)

Slot: l0

Readers

cpu-cycles-l0 (generic function)

Writers

(setf cpu-cycles-l0) (generic function)

Slot: h1

Readers

cpu-cycles-h1 (generic function)

Writers

(setf cpu-cycles-h1) (generic function)

Slot: l1

Readers

cpu-cycles-l1 (generic function)

Writers

(setf cpu-cycles-l1) (generic function)

Class: delta-metric ()

A LISTED-METRIC that calculates a sample point according to the delta between the START and COMMIT. Sub-classes of this must implement the TAKE-SAMPLE method.

See LISTED-METRIC

Package

trivial-benchmark

Source

samples.lisp (file)

Direct superclasses

vector-metric (class)

Direct subclasses

• real-time (class)
• run-time (class)
• user-run-time (class)
• system-run-time (class)
• page-faults (class)
• gc-run-time (class)
• bytes-consed (class)
• eval-calls (class)

Direct methods

• commit (method)
• discard (method)
• stop (method)
• start (method)
• units (method)
• units (method)
• stopping-value (method)
• stopping-value (method)
• (setf starting-value) (method)
• starting-value (method)

Direct slots

Slot: starting-value

Readers

starting-value (generic function)

Writers

(setf starting-value) (generic function)

Slot: stopping-value

Readers

stopping-value (generic function)

Writers

(setf stopping-value) (generic function)

Slot: units

Initform

1

Readers

units (generic function)

Writers

(setf units) (generic function)

Class: eval-calls ()

Samples SB-IMPL::*EVAL-CALLS*.

Package

trivial-benchmark

Source

samples.lisp (file)

Direct superclasses

delta-metric (class)

Direct methods

• take-sample (method)
• units (method)
• units (method)

Direct slots

Slot: units

Initform

1

Readers

units (generic function)

Writers

(setf units) (generic function)

Class: gc-run-time ()

Samples SB-IMPL::*GC-RUN-TIME* in seconds.

Package

trivial-benchmark

Source

samples.lisp (file)

Direct superclasses

delta-metric (class)

Direct methods

• take-sample (method)
• units (method)
• units (method)

Direct slots

Slot: units

Initform

1000

Readers

units (generic function)

Writers

(setf units) (generic function)

Class: metric ()

A class container for sampling information.

Package

trivial-benchmark

Source

timer.lisp (file)

Direct superclasses

standard-object (class)

Direct subclasses

vector-metric (class)

Direct methods

• (setf metric) (method)
• report-to (method)
• compute (method)
• compute (method)
• compute (method)
• compute (method)
• compute (method)
• compute (method)
• compute (method)
• compute (method)
• compute (method)
• commit (method)
• discard (method)
• stop (method)
• stop (method)
• start (method)
• print-object (method)
• (setf running) (method)
• running (method)

Direct slots

Slot: running

Initargs

:running

Readers

running (generic function)

Writers

(setf running) (generic function)

Class: page-faults ()

Samples the third value (page-faults) of SB-SYS:GET-SYSTEM-INFO in seconds.

Package

trivial-benchmark

Source

samples.lisp (file)

Direct superclasses

delta-metric (class)

Direct methods

• take-sample (method)
• units (method)
• units (method)

Direct slots

Slot: units

Initform

1

Readers

units (generic function)

Writers

(setf units) (generic function)

Class: real-time ()

Samples the results of GET-INTERNAL-REAL-TIME in seconds.

Package

trivial-benchmark

Source

samples.lisp (file)

Direct superclasses

delta-metric (class)

Direct methods

• take-sample (method)
• units (method)
• units (method)

Direct slots

Slot: units

Initform

internal-time-units-per-second

Readers

units (generic function)

Writers

(setf units) (generic function)

Class: run-time ()

Samples the results of GET-INTERNAL-RUN-TIME in seconds.

Package

trivial-benchmark

Source

samples.lisp (file)

Direct superclasses

delta-metric (class)

Direct methods

• take-sample (method)
• units (method)
• units (method)

Direct slots

Slot: units

Initform

internal-time-units-per-second

Readers

units (generic function)

Writers

(setf units) (generic function)

Class: system-run-time ()

Samples the second value (system-run-time) of SB-SYS:GET-SYSTEM-INFO in seconds.

Package

trivial-benchmark

Source

samples.lisp (file)

Direct superclasses

delta-metric (class)

Direct methods

• take-sample (method)
• units (method)
• units (method)

Direct slots

Slot: units

Initform

1000000

Readers

units (generic function)

Writers

(setf units) (generic function)

Class: timer ()

Class container for a set of METRICS.

Package

trivial-benchmark

Source

timer.lisp (file)

Direct superclasses

standard-object (class)

Direct methods

• reset (method)
• report-to (method)
• commit (method)
• discard (method)
• stop (method)
• start (method)
• take-sample (method)
• describe-object (method)
• print-object (method)
• initialize-instance (method)
• metric-types (method)
• (setf metric) (method)
• metric (method)
• (setf metrics) (method)
• metrics (method)

Direct slots

Slot: metrics

Initargs

:metrics

Readers

metrics (generic function)

Writers

(setf metrics) (generic function)

Class: user-run-time ()

Samples the first value (user-run-time) of SB-SYS:GET-SYSTEM-INFO in seconds.

Package

trivial-benchmark

Source

samples.lisp (file)

Direct superclasses

delta-metric (class)

Direct methods

• take-sample (method)
• units (method)
• units (method)

Direct slots

Slot: units

Initform

1000000

Readers

units (generic function)

Writers

(setf units) (generic function)

Class: vector-metric ()

A METRIC that implements its sample storage as a vector. SAMPLES is SETF-able for this class.

See METRIC

Package

trivial-benchmark

Source

samples.lisp (file)

Direct superclasses

metric (class)

Direct subclasses

• delta-metric (class)
• cpu-cycles (class)

Direct methods

• reset (method)
• reset (method)
• (setf samples) (method)
• samples (method)

Direct slots

Slot: samples

Initargs

:samples

Initform

(make-array 1000 :adjustable t :fill-pointer 0)

Readers

samples (generic function)

Writers

(setf samples) (generic function)

5.2 Internal definitions

5.2.1 Special variables

Special Variable: *benchmark-packages*

A table mapping package objects (which are benchmark packages) to a list of fbound symbols.

Package

trivial-benchmark

Source

suite.lisp (file)

Special Variable: *current-benchmark*

Package

trivial-benchmark

Source

suite.lisp (file)

Special Variable: *current-suite-report*

Package

trivial-benchmark

Source

suite.lisp (file)

Special Variable: *current-timer*

Package

trivial-benchmark

Source

suite.lisp (file)

5.2.2 Functions

Function: format-timer-stats STREAM TIMER &optional COMPUTATIONS

Package

trivial-benchmark

Source

timer.lisp (file)

Function: metrics-alist TIMER &optional COMPUTATIONS

Package

trivial-benchmark

Source

suite.lisp (file)

5.2.3 Generic functions

Generic Function: cpu-cycles-h0 OBJECT

Generic Function: (setf cpu-cycles-h0) NEW-VALUE OBJECT

Package

trivial-benchmark

Methods

Method: cpu-cycles-h0 (CPU-CYCLES cpu-cycles)

automatically generated reader method

Source

samples.lisp (file)

Method: (setf cpu-cycles-h0) NEW-VALUE (CPU-CYCLES cpu-cycles)

automatically generated writer method

Source

samples.lisp (file)

Generic Function: cpu-cycles-h1 OBJECT

Generic Function: (setf cpu-cycles-h1) NEW-VALUE OBJECT

Package

trivial-benchmark

Methods

Method: cpu-cycles-h1 (CPU-CYCLES cpu-cycles)

automatically generated reader method

Source

samples.lisp (file)

Method: (setf cpu-cycles-h1) NEW-VALUE (CPU-CYCLES cpu-cycles)

automatically generated writer method

Source

samples.lisp (file)

Generic Function: cpu-cycles-l0 OBJECT

Generic Function: (setf cpu-cycles-l0) NEW-VALUE OBJECT

Package

trivial-benchmark

Methods

Method: cpu-cycles-l0 (CPU-CYCLES cpu-cycles)

automatically generated reader method

Source

samples.lisp (file)

Method: (setf cpu-cycles-l0) NEW-VALUE (CPU-CYCLES cpu-cycles)

automatically generated writer method

Source

samples.lisp (file)

Generic Function: cpu-cycles-l1 OBJECT

Generic Function: (setf cpu-cycles-l1) NEW-VALUE OBJECT

Package

trivial-benchmark

Methods

Method: cpu-cycles-l1 (CPU-CYCLES cpu-cycles)

automatically generated reader method

Source

samples.lisp (file)

Method: (setf cpu-cycles-l1) NEW-VALUE (CPU-CYCLES cpu-cycles)

automatically generated writer method

Source

samples.lisp (file)

Generic Function: report-to STREAM THING &key COMPUTATIONS

Package

trivial-benchmark

Source

timer.lisp (file)

Methods

Method: report-to (STREAM stream) (TIMER timer) &key COMPUTATIONS

Method: report-to (STREAM (eql t)) THING &key COMPUTATIONS

Method: report-to (STRING (eql nil)) THING &key COMPUTATIONS

Method: report-to (STREAM stream) (METRIC metric) &key COMPUTATIONS

Generic Function: stop METRIC

Stop the sample for METRIC. Sets RUNNING to NIL.

Package

trivial-benchmark

Source

timer.lisp (file)

Methods

Method: stop (METRIC cpu-cycles)

Source

samples.lisp (file)

Method: stop (METRIC delta-metric)

Source

samples.lisp (file)

Method: stop (TIMER timer)

Method: stop (METRIC metric) around

Method: stop (METRIC metric)

Generic Function: stopping-value OBJECT

Generic Function: (setf stopping-value) NEW-VALUE OBJECT

Package

trivial-benchmark

Methods

Method: stopping-value (DELTA-METRIC delta-metric)

automatically generated reader method

Source

samples.lisp (file)

Method: (setf stopping-value) NEW-VALUE (DELTA-METRIC delta-metric)

automatically generated writer method

Source

samples.lisp (file)

Generic Function: units OBJECT

Generic Function: (setf units) NEW-VALUE OBJECT

Package

trivial-benchmark

Methods

Method: units (EVAL-CALLS eval-calls)

automatically generated reader method

Source

samples.lisp (file)

Method: (setf units) NEW-VALUE (EVAL-CALLS eval-calls)

automatically generated writer method

Source

samples.lisp (file)

Method: units (BYTES-CONSED bytes-consed)

automatically generated reader method

Source

samples.lisp (file)

Method: (setf units) NEW-VALUE (BYTES-CONSED bytes-consed)

automatically generated writer method

Source

samples.lisp (file)

Method: units (GC-RUN-TIME gc-run-time)

automatically generated reader method

Source

samples.lisp (file)

Method: (setf units) NEW-VALUE (GC-RUN-TIME gc-run-time)

automatically generated writer method

Source

samples.lisp (file)

Method: units (PAGE-FAULTS page-faults)

automatically generated reader method

Source

samples.lisp (file)

Method: (setf units) NEW-VALUE (PAGE-FAULTS page-faults)

automatically generated writer method

Source

samples.lisp (file)

Method: units (SYSTEM-RUN-TIME system-run-time)

automatically generated reader method

Source

samples.lisp (file)

Method: (setf units) NEW-VALUE (SYSTEM-RUN-TIME system-run-time)

automatically generated writer method

Source

samples.lisp (file)

Method: units (USER-RUN-TIME user-run-time)

automatically generated reader method

Source

samples.lisp (file)

Method: (setf units) NEW-VALUE (USER-RUN-TIME user-run-time)

automatically generated writer method

Source

samples.lisp (file)

Method: units (RUN-TIME run-time)

automatically generated reader method

Source

samples.lisp (file)

Method: (setf units) NEW-VALUE (RUN-TIME run-time)

automatically generated writer method

Source

samples.lisp (file)

Method: units (REAL-TIME real-time)

automatically generated reader method

Source

samples.lisp (file)

Method: (setf units) NEW-VALUE (REAL-TIME real-time)

automatically generated writer method

Source

samples.lisp (file)

Method: units (DELTA-METRIC delta-metric)

automatically generated reader method

Source

samples.lisp (file)

Method: (setf units) NEW-VALUE (DELTA-METRIC delta-metric)

automatically generated writer method

Source

samples.lisp (file)

Appendix A Indexes

A.1 Concepts

A.2 Functions