This is the fiveam Reference Manual, version 1.4.2, generated automatically by Declt version 4.0 beta 2 "William Riker" on Mon Feb 26 16:25:29 2024 GMT+0.
The main system appears first, followed by any subsystem dependency.
fiveamA simple regression testing framework
Edward Marco Baringer <mb@bese.it>
BSD
1.4.2
alexandria (system).
net.didierverna.asdf-flv (system).
trivial-backtrace (system).
package.lisp (file).
utils.lisp (file).
check.lisp (file).
fixture.lisp (file).
classes.lisp (file).
random.lisp (file).
test.lisp (file).
explain.lisp (file).
suite.lisp (file).
run.lisp (file).
Files are sorted by type and then listed depth-first from the systems components trees.
fiveam/fiveam.asdfiveam/package.lispfiveam/utils.lispfiveam/check.lispfiveam/fixture.lispfiveam/classes.lispfiveam/random.lispfiveam/test.lispfiveam/explain.lispfiveam/suite.lispfiveam/run.lispfiveam/utils.lisppackage.lisp (file).
fiveam (system).
acond2 (macro).
aif (macro).
binding (function).
check-required (function).
def-special-environment (macro).
dolist* (macro).
if-bind (macro).
list-match (function).
list-match-case (macro).
make-collector (function).
partitionx (function).
vars (function).
varsymp (function).
fiveam/check.lisppackage.lisp (file).
utils.lisp (file).
fiveam (system).
actual-condition (reader method).
(setf actual-condition) (writer method).
add-result (function).
bind-run-state (macro).
bind-run-state* (macro).
check-failure (condition).
process-failure (function).
reason (reader method).
reason (reader method).
(setf reason) (writer method).
(setf reason) (writer method).
test-case (reader method).
test-case (reader method).
(setf test-case) (writer method).
(setf test-case) (writer method).
test-expr (reader method).
test-expr (reader method).
(setf test-expr) (writer method).
(setf test-expr) (writer method).
test-failure (class).
test-failure-p (generic function).
test-passed (class).
test-passed-p (generic function).
test-result (class).
test-skipped (class).
test-skipped-p (generic function).
unexpected-test-failure (class).
with-*test-dribble* (macro).
with-run-state (macro).
fiveam/fixture.lisppackage.lisp (file).
fiveam (system).
def-fixture (macro).
get-fixture (function).
(setf get-fixture) (function).
rem-fixture (function).
with-fixture (macro).
*fixture* (special variable).
fiveam/classes.lisppackage.lisp (file).
fiveam (system).
print-object (method).
print-object (method).
%test-names (reader method).
(setf %test-names) (writer method).
%tests (reader method).
(setf %tests) (writer method).
collect-profiling-info (reader method).
(setf collect-profiling-info) (writer method).
depends-on (reader method).
(setf depends-on) (writer method).
description (reader method).
(setf description) (writer method).
detailed-text-explainer (class).
explainer (class).
name (reader method).
(setf name) (writer method).
profiling-info (reader method).
(setf profiling-info) (writer method).
runtime-package (reader method).
(setf runtime-package) (writer method).
simple-text-explainer (class).
status (reader method).
(setf status) (writer method).
test-bundle (class).
test-case (class).
test-lambda (reader method).
(setf test-lambda) (writer method).
test-suite (reader method).
(setf test-suite) (writer method).
test-suite (class).
testable-object (class).
tests (reader method).
(setf tests) (writer method).
text-explainer (class).
fiveam/random.lisppackage.lisp (file).
check.lisp (file).
fiveam (system).
*max-trials* (special variable).
*num-trials* (special variable).
for-all (macro).
for-all* (macro).
gen-buffer (function).
gen-character (function).
gen-float (function).
gen-integer (function).
gen-list (function).
gen-one-element (function).
gen-string (function).
gen-tree (function).
dbind-rec (macro).
for-all-test-failed (class).
for-all-test-failed-p (generic function).
for-all-test-never-run (class).
for-all-test-no-tests (class).
for-all-test-passed (class).
for-all-test-result (class).
perform-random-testing (function).
perform-random-testing/run-once (function).
reason (method).
fiveam/test.lisppackage.lisp (file).
fixture.lisp (file).
classes.lisp (file).
fiveam (system).
*default-test-compilation-time* (special variable).
*run-test-when-defined* (special variable).
def-test (macro).
get-test (function).
(setf get-test) (function).
rem-test (function).
test (macro).
test-names (function).
*test* (special variable).
register-test (function).
fiveam/explain.lisppackage.lisp (file).
utils.lisp (file).
check.lisp (file).
classes.lisp (file).
random.lisp (file).
fiveam (system).
*verbose-failures* (special variable).
explain (generic function).
partition-results (function).
fiveam/suite.lisppackage.lisp (file).
test.lisp (file).
classes.lisp (file).
fiveam (system).
def-suite (macro).
def-suite* (macro).
in-suite (macro).
in-suite* (macro).
make-suite (function).
%in-suite (macro).
*suite* (special variable).
*toplevel-suites* (special variable).
list-all-suites (function).
suite-emptyp (generic function).
fiveam/run.lisppackage.lisp (file).
check.lisp (file).
classes.lisp (file).
test.lisp (file).
explain.lisp (file).
suite.lisp (file).
fiveam (system).
! (function).
!! (function).
!!! (function).
*debug-on-error* (special variable).
*debug-on-failure* (special variable).
*on-error* (special variable).
*on-failure* (special variable).
*print-names* (special variable).
debug! (function).
explain! (function).
results-status (function).
run (function).
run! (function).
run-all-tests (function).
%run (generic function).
*!!!* (special variable).
*!!* (special variable).
*!* (special variable).
*initial-!* (special variable).
*run-queue* (special variable).
*test-dribble-indent* (special variable).
circular-dependency (condition).
import-testing-symbols (function).
on-problem-action (type).
resolve-dependencies (generic function).
return-result-list (function).
run-resolving-dependencies (generic function).
run-test-lambda (generic function).
Packages are listed by definition order.
it.bese.fiveam5am
fiveam
alexandria.
common-lisp.
! (function).
!! (function).
!!! (function).
*debug-on-error* (special variable).
*debug-on-failure* (special variable).
*default-test-compilation-time* (special variable).
*max-trials* (special variable).
*num-trials* (special variable).
*on-error* (special variable).
*on-failure* (special variable).
*print-names* (special variable).
*run-test-when-defined* (special variable).
*test-dribble* (special variable).
*verbose-failures* (special variable).
debug! (function).
def-fixture (macro).
def-suite (macro).
def-suite* (macro).
def-test (macro).
explain (generic function).
explain! (function).
fail (macro).
finishes (macro).
for-all (macro).
for-all* (macro).
gen-buffer (function).
gen-character (function).
gen-float (function).
gen-integer (function).
gen-list (function).
gen-one-element (function).
gen-string (function).
gen-tree (function).
get-fixture (function).
(setf get-fixture) (function).
get-test (function).
(setf get-test) (function).
in-suite (macro).
in-suite* (macro).
is (macro).
is-every (macro).
is-false (macro).
is-true (macro).
make-suite (function).
pass (macro).
rem-fixture (function).
rem-test (function).
results-status (function).
run (function).
run! (function).
run-all-tests (function).
signals (macro).
skip (macro).
test (macro).
test-names (function).
with-fixture (macro).
%in-suite (macro).
%run (generic function).
%test-names (generic reader).
(setf %test-names) (generic writer).
%tests (generic reader).
(setf %tests) (generic writer).
*!!!* (special variable).
*!!* (special variable).
*!* (special variable).
*fixture* (special variable).
*initial-!* (special variable).
*run-queue* (special variable).
*suite* (special variable).
*test* (special variable).
*test-dribble-indent* (special variable).
*toplevel-suites* (special variable).
acond2 (macro).
actual-condition (generic reader).
(setf actual-condition) (generic writer).
add-result (function).
aif (macro).
bind-run-state (macro).
bind-run-state* (macro).
binding (function).
check-failure (condition).
check-required (function).
circular-dependency (condition).
collect-profiling-info (generic reader).
(setf collect-profiling-info) (generic writer).
dbind-rec (macro).
def-special-environment (macro).
depends-on (generic reader).
(setf depends-on) (generic writer).
description (generic reader).
(setf description) (generic writer).
detailed-text-explainer (class).
dolist* (macro).
explainer (class).
for-all-test-failed (class).
for-all-test-failed-p (generic function).
for-all-test-never-run (class).
for-all-test-no-tests (class).
for-all-test-passed (class).
for-all-test-result (class).
if-bind (macro).
import-testing-symbols (function).
list-all-suites (function).
list-match (function).
list-match-case (macro).
make-collector (function).
name (generic reader).
(setf name) (generic writer).
on-problem-action (type).
partition-results (function).
partitionx (function).
perform-random-testing (function).
perform-random-testing/run-once (function).
process-failure (function).
profiling-info (generic reader).
(setf profiling-info) (generic writer).
reason (generic function).
(setf reason) (generic writer).
register-test (function).
resolve-dependencies (generic function).
return-result-list (function).
run-resolving-dependencies (generic function).
run-test-lambda (generic function).
runtime-package (generic reader).
(setf runtime-package) (generic writer).
simple-text-explainer (class).
status (generic reader).
(setf status) (generic writer).
suite-emptyp (generic function).
test-bundle (class).
test-case (generic reader).
(setf test-case) (generic writer).
test-case (class).
test-expr (generic reader).
(setf test-expr) (generic writer).
test-failure (class).
test-failure-p (generic function).
test-lambda (generic reader).
(setf test-lambda) (generic writer).
test-passed (class).
test-passed-p (generic function).
test-result (class).
test-skipped (class).
test-skipped-p (generic function).
test-suite (generic reader).
(setf test-suite) (generic writer).
test-suite (class).
testable-object (class).
tests (generic reader).
(setf tests) (generic writer).
text-explainer (class).
unexpected-test-failure (class).
vars (function).
varsymp (function).
with-*test-dribble* (macro).
with-run-state (macro).
Definitions are sorted by export status, category, package, and then by lexicographic order.
T if we should drop into the debugger on error, NIL otherwise. OBSOLETE: superseded by *ON-ERROR*
T if we should drop into the debugger on a failing check, NIL otherwise. OBSOLETE: superseded by *ON-FAILURE*
Number of total times we attempt to run the body of the
FOR-ALL test including when the body is skipped due to failed
guard conditions.
Since we have guard conditions we may get into infinite loops where the test code is never run due to the guards never returning true. This second run limit prevents that.
Number of times we attempt to run the body of the FOR-ALL test.
The action to perform on error:
- :DEBUG if we should drop into the debugger
- :BACKTRACE to print a backtrace
- NIL to simply continue
The action to perform on check failure:
- :DEBUG if we should drop into the debugger
- :BACKTRACE to print a backtrace
- NIL to simply continue
T if we should print test running progress, NIL otherwise.
When non-NIL tests are run as soon as they are defined.
T if we should print the expression failing, NIL otherwise.
Defines a fixture named NAME. A fixture is very much like a
macro but is used only for simple templating. A fixture created
with DEF-FIXTURE is a macro which can use the special macrolet
&BODY to specify where the body should go.
See Also: WITH-FIXTURE
Define a new test-suite named NAME.
IN (a symbol), if provided, causes this suite te be nested in the suite named by IN. NB: This macro is built on top of make-suite, as such it, like make-suite, will overrwrite any existing suite named NAME.
Create a test named NAME.
NAME is the symbol which names the test.
DEPENDS-ON is a list of the form:
(AND . test-names) - This test is run only if all of the tests
in TEST-NAMES have passed, otherwise a single test-skipped
result is generated.
(OR . test-names) - If any of TEST-NAMES has passed this test is
run, otherwise a test-skipped result is generated.
(NOT test-name) - This is test is run only if TEST-NAME failed.
AND, OR and NOT can be combined to produce complex dependencies.
If DEPENDS-ON is a symbol it is interpreted as ‘(AND
,depends-on), this is accomadate the common case of one test
depending on another.
FIXTURE specifies a fixture to wrap the body in.
If PROFILE is T profiling information will be collected as well.
Simply generate a FAIL.
Generates a pass if BODY executes to normal completion. In other words if body does signal, return-from or throw this test fails.
Bind BINDINGS to random variables and test BODY *num-trials* times.
BINDINGS is a list of binding forms, each element is a list
of (BINDING VALUE &optional GUARD). Value, which is evaluated
once when the for-all is evaluated, must return a generator which
be called each time BODY is evaluated. BINDING is either a symbol
or a list which will be passed to destructuring-bind. GUARD is a
form which, if present, stops BODY from executing when IT returns
NIL. The GUARDS are evaluated after all the random data has been
generated and they can refer to the current value of any
binding. NB: Generator forms, unlike guard forms, can not contain
references to the bound variables.
Examples:
(for-all ((a (gen-integer)))
(is (integerp a)))
(for-all ((a (gen-integer) (plusp a)))
(is (integerp a))
(is (plusp a)))
(for-all ((less (gen-integer))
(more (gen-integer) (< less more)))
(is (<= less more)))
(for-all (((a b) (gen-two-integers)))
(is (integerp a))
(is (integerp b)))
Bind BINDINGS to random variables and test BODY *num-trials* times.
Works like FOR-ALL but variables are defined sequentially - subsequent
generator bindings may use value of a previous generator.
Example:
(for-all* ((a (gen-integer))
(b (gen-integer :min a :max (+ a 20)))
(is (<= a b)))
Set the *suite* special variable so that all tests defined
after the execution of this form are, unless specified otherwise,
in the test-suite named SUITE-NAME.
See also: DEF-SUITE *SUITE*
Just like in-suite, but silently creates missing suites.
The DWIM checking operator.
If TEST returns a true value a test-passed result is generated, otherwise a test-failure result is generated. The reason, unless REASON-ARGS is provided, is generated based on the form of TEST:
(predicate expected actual) - Means that we want to check
whether, according to PREDICATE, the ACTUAL value is
in fact what we EXPECTED.
(predicate value) - Means that we want to ensure that VALUE
satisfies PREDICATE.
Wrapping the TEST form in a NOT simply produces a negated reason string.
The input is either a list of lists, or a list of pairs. Generates (is (,predicate ,expr ,value)) for each pair of elements or (is (,predicate ,expr ,value) ,@reason) for each list.
Generates a pass if CONDITION returns false, generates a failure otherwise. Like IS-TRUE, and unlike IS, IS-FALSE does not inspect CONDITION to determine what reason to give it case of test failure
Like IS this check generates a pass if CONDITION returns true and a failure if CONDITION returns false. Unlike IS this check does not inspect CONDITION to determine how to report the failure.
Simply generate a PASS.
Generates a pass if BODY signals a condition of type CONDITION. BODY is evaluated in a block named NIL, CONDITION is not evaluated.
Generates a TEST-SKIPPED result.
Create a test named NAME. If NAME is a list it must be of the
form:
(name &key depends-on suite fixture compile-at profile)
NAME is the symbol which names the test.
DEPENDS-ON is a list of the form:
(AND . test-names) - This test is run only if all of the tests
in TEST-NAMES have passed, otherwise a single test-skipped
result is generated.
(OR . test-names) - If any of TEST-NAMES has passed this test is
run, otherwise a test-skipped result is generated.
(NOT test-name) - This is test is run only if TEST-NAME failed.
AND, OR and NOT can be combined to produce complex dependencies.
If DEPENDS-ON is a symbol it is interpreted as ‘(AND
,depends-on), this is accomadate the common case of one test
depending on another.
FIXTURE specifies a fixture to wrap the body in.
If PROFILE is T profiling information will be collected as well.
Insert BODY into the fixture named FIXTURE-NAME.
See Also: DEF-FIXTURE
Rerun the most recently run test and explain the results.
Rerun the second most recently run test and explain the results.
Rerun the third most recently run test and explain the results.
Calls (run! test-spec) but enters the debugger if any kind of error happens.
Explain the results of RESULT-LIST using a detailed-text-explainer with output going to *test-dribble*. Return a boolean indicating whether no tests failed.
Returns a generator of characters.
CODE must be a generator of random integers. ALPHANUMERICP, if non-NIL, limits the returned chars to those which pass alphanumericp.
Returns a generator which produces floats of type TYPE. BOUND, if specified, constrains the results to be in the range (-BOUND, BOUND).
Returns a generator which produces random integers greater than or equal to MIN and less than or equal to MAX.
Returns a generator which produces random lists. LENGTH must be an integer generator and ELEMENTS must be a generator which produces objects.
Returns a generator which produces random strings. LENGTH must be a generator which produces integers, ELEMENTS must be a generator which produces characters of type ELEMENT-TYPE.
Returns a generator which produces random trees. SIZE controls the approximate size of the tree, but don’t try anything above 30, you have been warned. ELEMENTS must be a generator which will produce the elements.
Create a new test suite object.
Overrides any existing suite named NAME.
Given a list of test results (generated while running a test) return true if no results are of type TEST-FAILURE. Returns second and third values, which are the set of failed tests and skipped tests respectively.
Run the test specified by TEST-SPEC.
TEST-SPEC can be either a symbol naming a test or test suite, or a testable-object object. This function changes the operations performed by the !, !! and !!! functions.
Equivalent to (explain! (run TEST-SPEC)).
Runs all defined test suites, T if all tests passed and NIL otherwise. SUMMARY can be :END to print a summary at the end, :SUITE to print it after each suite or NIL to skip explanations.
Given a list of test results report write to stream detailed human readable statistics regarding the results.
simple-text-explainer) results &optional stream recursive-depth) ¶detailed-text-explainer) results &optional stream recursive-depth) ¶testable-object) stream) ¶test-bundle) stream) ¶Lookup table mapping fixture names to fixture objects.
List of test waiting to be run.
The current test suite object
Lookup table mapping test (and test suite) names to objects.
Used to indent tests and test suites in their parent suite
Just like IF-BIND but the var is always IT.
Define two macros for dealing with groups or related special variables.
ACCESSOR is defined as a macro: (defmacro ACCESSOR (VARS &rest
BODY)). Each element of VARS will be bound to the
current (dynamic) value of the special variable.
BINDER is defined as a macro for introducing (and binding new)
special variables. It is basically a readable LET form with the
prorpe declarations appended to the body. The first argument to
BINDER must be a form suitable as the first argument to LET.
ACCESSOR defaults to a new symbol in the same package as NAME
which is the concatenation of "WITH-" NAME. BINDER is built as
"BIND-" and BINDER* is BINDER "*".
Like DOLIST but destructuring-binds the elements of LIST.
If ITERATOR is a symbol then dolist* is just like dolist EXCEPT that it creates a fresh binding.
Anaphoric IF control structure.
VAR (a symbol) will be bound to the primary value of TEST. If TEST returns a true value then THEN will be executed, otherwise ELSE will be executed.
Create a TEST-RESULT object of type RESULT-TYPE passing it the initialize args MAKE-INSTANCE-ARGS and add the resulting object to the list of test results.
Returns an unordered LIST of all suites.
Create a collector function.
A Collector function will collect, into a list, all the values passed to it in the order in which they were passed. If the callector function is called without arguments it returns the current list of values.
Run the test function TEST-LAMBDA and return a list of all test results generated, does not modify the special environment variable RESULT-LIST.
Internal method for running a test. Does not
update the status of the tests nor the special variables !,
!!, !!!
symbol)) ¶test-suite)) ¶list)) ¶test-bundle)) ¶automatically generated reader method
test-bundle)) ¶automatically generated writer method
test-bundle)) ¶automatically generated reader method
test-bundle)) ¶automatically generated writer method
unexpected-test-failure)) ¶automatically generated reader method
unexpected-test-failure)) ¶automatically generated writer method
testable-object)) ¶testable-object)) ¶When T profiling
information will be collected when the
test is run.
testable-object)) ¶testable-object)) ¶The list of AND, OR, NOT forms specifying when to run this test.
testable-object)) ¶testable-object)) ¶The textual description of this test object.
for-all-test-failed)) ¶testable-object)) ¶testable-object)) ¶A symbol naming this test object.
name.
testable-object)) ¶testable-object)) ¶An object representing how
much time and memory where used by the
test.
for-all-test-failed)) ¶check-failure)) ¶test-result)) ¶automatically generated reader method
check-failure)) ¶test-result)) ¶automatically generated writer method
Given a dependency spec determine if the spec
is satisfied or not, this will generally involve running other
tests. If the dependency spec can be satisfied the test is also
run.
testable-object)) ¶testable-object)) ¶A symbol specifying the current status
of this test. Either: T - this test (and all its
dependencies, have passed. NIL - this test
failed (either it failed or its dependecies weren’t
met. :circular this test has a circular dependency
and was skipped. Or :depends-not-satisfied or :resolving
symbol)) ¶test-suite)) ¶check-failure)) ¶test-result)) ¶automatically generated reader method
check-failure)) ¶test-result)) ¶automatically generated writer method
check-failure)) ¶test-result)) ¶automatically generated reader method
check-failure)) ¶test-result)) ¶automatically generated writer method
test-failure)) ¶test-passed)) ¶test-skipped)) ¶test-suite)) ¶test-suite)) ¶The hash table mapping names to test
objects in this suite. The values in this hash table
can be either test-cases or other test-suites.
Signaled when a check fails.
Condition signaled when a circular dependency between test-cases has been detected.
error.
:test-case
common-lisp.
:random-state
A test case is a single, named, collection of
checks.
A test case is the smallest organizational element which can be
run individually. Every test case has a name, which is a symbol,
a description and a test lambda. The test lambda is a regular
funcall’able function which should use the various checking
macros to collect results.
Every test case is part of a suite, when a suite is not
explicitly specified (either via the :SUITE parameter to the TEST
macro or the global variable *SUITE*) the test is inserted into
the global suite named NIL.
Sometimes we want to run a certain test only if another test has
passed. FiveAM allows us to specify the ways in which one test is
dependent on another.
- AND Run this test only if all the named tests passed.
- OR Run this test if at least one of the named tests passed.
- NOT Run this test only if another test has failed.
FiveAM considers a test to have passed if all the checks executed
were successful, otherwise we consider the test a failure.
When a test is not run due to it’s dependencies having failed a test-skipped result is added to the results.
The function to run.
:test-lambda
By default it stores *package* from the time this test was defined (macroexpanded).
:runtime-package
The test-suite associated with this test
:test-suite
Class for unsuccessful checks.
Class for successful checks.
All checking macros will generate an object of type TEST-RESULT.
A test which was not run. Usually this is due to
unsatisfied dependencies, but users can decide to skip the test when
appropriate.
A test suite is a collection of tests or test suites.
Test suites serve to organize tests into groups so that the
developer can chose to run some tests and not just one or
all. Like tests test suites have a name and a description.
Test suites, like tests, can be part of other test suites, this
allows the developer to create a hierarchy of tests where sub
trees can be singularly run.
Running a test suite has the effect of running every test (or suite) in the suite.
The hash table mapping names to test
objects in this suite. The values in this hash table
can be either test-cases or other test-suites.
(make-instance (quote it.bese.fiveam::test-bundle))
The textual description of this test object.
:description
The list of AND, OR, NOT forms specifying when to run this test.
:depends-on
A symbol specifying the current status
of this test. Either: T - this test (and all its
dependencies, have passed. NIL - this test
failed (either it failed or its dependecies weren’t
met. :circular this test has a circular dependency
and was skipped. Or :depends-not-satisfied or :resolving
:unknown
:status
An object representing how
much time and memory where used by the
test.
When T profiling
information will be collected when the
test is run.
:collect-profiling-info
Represents the result of a test which neither
passed nor failed, but signaled an error we couldn’t deal
with.
Note: This is very different than a SIGNALS check which instead creates a TEST-PASSED or TEST-FAILURE object.
:condition
| Jump to: | !
%
(
A B C D E F G I L M N P R S T V W |
|---|
| Jump to: | !
%
(
A B C D E F G I L M N P R S T V W |
|---|
| Jump to: | *
A C D F N P R S T |
|---|
| Jump to: | *
A C D F N P R S T |
|---|
| Jump to: | C D E F I O P R S T U |
|---|
| Jump to: | C D E F I O P R S T U |
|---|