Next: Introduction, Previous: (dir), Up: (dir) [Contents][Index]
This is the kaputt Reference Manual, generated automatically by Declt version 3.0 "Montgomery Scott" on Mon Apr 19 14:57:01 2021 GMT+0.
| • Introduction | What kaputt is all about | |
| • Systems | The systems documentation | |
| • Files | The files documentation | |
| • Packages | The packages documentation | |
| • Definitions | The symbols documentation | |
| • Indexes | Concepts, functions, variables and data types |
Kaputt is a test framework for Common Lisp that focuses on the following features:
Kaputt is simple, it only defines three abstractions testcase, assertion and protocol and does not add any artefact on the backtrace when errors occur.
Kaputt is extensible, it is possible to add problem-specific assertions to make tests more informative.
Kaputt fits well interactive development.
See Comparison to some-other available test frameworks below for a more detailed motivation of the need for a new test framework.
Kaputt is not yet available in the Quicklisp distribution, so
Kaputt needs to be installed manually. Clone the repository in a
directory listed in the ql:*local-project-directories* parameter and
(ql:register-local-projects) so that you can (require "kaputt").
Clone the repository in a place where ASDF can find it, for instance
~/common-lisp if you are using a modern ASDF module with its default
configuration. Then (require "kaputt").
This is illustrated by the file example.lisp from the
distribution. We describe an interactive session based on the content
of this file. After evaluating
(defpackage #:kaputt/example
(:use #:common-lisp #:kaputt)
(:export
#:run-all-tests))
(in-package #:kaputt/example)
(define-testcase cl-strings/string-downcase-turns-nil-into-a-string ()
(assert-string= "nil" (string-downcase nil)))
We can move to the REPL to evaluate:
CL-USER> (in-package #:kaputt/example)
#<PACKAGE "KAPUTT/EXAMPLE">
KAPUTT/EXAMPLE> (cl-strings/string-downcase-turns-nil-into-a-string)
.
Test suite ran 1 assertions split across 1 test cases.
Success: 1/1 (100%)
Failure: 0/1 (0%)
T
Let's define a few more tests by evaluating a further form from
example.lisp but we modify it to display how Kaputt handles failing
assertions:
(define-testcase cl-strings/string-upcase-turns-nil-into-a-string ()
(assert-string= "nil" (string-upcase nil)))
The evaluation of the function
KAPUTT/EXAMPLE> (cl-strings/string-upcase-turns-nil-into-a-string)
summons the debugger:
Test assertion failed:
(ASSERT-STRING= "nil" (STRING-UPCASE NIL))
In this call, the composed forms in argument position evaluate as:
(STRING-UPCASE NIL) => "NIL"
The assertion ASSERT-STRING= is defined by the body forms
(STRING= STRING1 STRING2 :START1 START1 :END1 END1 :START2 START2 :END2 END2)
It has no high-level description.
[Condition of type KAPUTT::ASSERTION-FAILED]
Restarts:
0: [CONTINUE] Record a failure for ASSERT-STRING= and continue testing.
1: [IGNORE] Record a success for ASSERT-STRING= and continue testing.
2: [RETRY] Retry ASSERT-STRING=.
3: [SKIP] Skip the rest of test case CL-STRINGS/STRING-UPCASE-TURNS-NIL-INTO-A-STRING and continue testing.
4: [RETRY] Retry SLIME REPL evaluation request.
5: [*ABORT] Return to SLIME's top level.
--more--
We can fix the testcase by formulating a reasonable expectation:
(define-testcase cl-strings/string-upcase-turns-nil-into-a-string ()
(assert-string= "NIL" (string-upcase nil)))
and 4. [RETRY] the SLIME REPL evaluation request to see the tests
complete. The report describing the error condition met by the
failing assertion can be customised freely when defining a new assertion.
We can finally organise our tests in a testsuite by defining a test case calling other test cases:
(define-testcase run-all-tests ()
(cl-strings/string-downcase-turns-nil-into-a-string)
(cl-strings/string-upcase-turns-nil-into-a-string))
It is possible to define parametrised testcases. See in kaputt.lisp
various example of advanced usages of the define-assertion function.
This snippets show an example of ASF system definition for a systeme depending on Kaputt:
(asdf:defsystem #:kaputt/example
:description "An Example for the Kaputt Test Framework"
:author "Michaël Le Barbier"
:license "MIT"
:depends-on (#:kaputt)
:components
((:file "example")))
Before writing Kaputt I worked with Stefil and FiveAM which in my experience were:
These other test frameworks make provision to organise tests hierarchically in test suites. In Kaputt a test case is also a normal Lisp function and test suites can be modelled by test cases calling several test cases or test suites in sequence. There is no need for specific support in the test framework to define this hierarchical organisation.
These other test frameworks make provision to define test fixtures
(test setup and tearoff). In Kaputt since a test case is also a
function, the usual WITH-* idiom can be used to define test fixtures
and therefore the test framework does not need to define specific
support for these.
These other test frameworks are built around an IS macro which is
presented as a generic “do what I mean” comparison operator. I found
this not convenient to use in specific situations, e.g. when testing
numerical algorithms and was very unsatisified with the kludges and
workarounds I needed to build to still use the IS macro and found
the resulting code rather convoluted and uninformative.
In Kaputt tests are built around assertions and new assertions can
be defined with the DEFINE-ASSERTION macro.
These other frameworks using the “do what I mean” comparison approach were not showing me the information I needed to make a first diagnostic of error conditions met and I spent a lot of time evaluating various expressions in the more-or-less right backtrace frame.
In Kaputt we have total control of the report produced by the error condition triggered by error failures, so that reports are much more informative and lead to quicker error r
Next: Files, Previous: Introduction, Up: Top [Contents][Index]
The main system appears first, followed by any subsystem dependency.
| • The kaputt system |
Michaël Le Barbier
MIT
A Simple Interactive Test Framework for Common Lisp
kaputt.asd (file)
kaputt.lisp (file)
Files are sorted by type and then listed depth-first from the systems components trees.
| • Lisp files |
| • The kaputt.asd file | ||
| • The kaputt/kaputt.lisp file |
Next: The kaputt/kaputt․lisp file, Previous: Lisp files, Up: Lisp files [Contents][Index]
kaputt.asd
kaputt (system)
Previous: The kaputt․asd file, Up: Lisp files [Contents][Index]
kaputt (system)
kaputt.lisp
Next: Definitions, Previous: Files, Up: Top [Contents][Index]
Packages are listed by definition order.
| • The kaputt package |
The Kaputt Test Framework.
kaputt.lisp (file)
common-lisp
Definitions are sorted by export status, category, package, and then by lexicographic order.
| • Exported definitions | ||
| • Internal definitions |
Next: Internal definitions, Previous: Definitions, Up: Definitions [Contents][Index]
| • Exported special variables | ||
| • Exported macros | ||
| • Exported functions | ||
| • Exported generic functions | ||
| • Exported classes |
Next: Exported macros, Previous: Exported definitions, Up: Exported definitions [Contents][Index]
The protocol class to use when running test suites.
kaputt.lisp (file)
Next: Exported functions, Previous: Exported special variables, Up: Exported definitions [Contents][Index]
Define an assertion function ASSERTION-NAME, accepting ASSERTION-ARGS with BODY.
The BODY is interpreted as computing a generalised boolean value,
which triggers an ASSERTION-FAILED when this boolean is NIL.
The ASSERTION-NAME must start with ASSERT-, ASSERT<, ASSERT> or ASSERT=.
When the first BODY form is a string, this string is used as an informational message
when reporting the corresponding error condition. It is also used as a documentaion
string for the created function.
When the next BODY form starts with the keyword :REPORT, then the rest of that form must be function of a STREAM. This function is then used to generate an informational message when reporting the corresponding error condition. This allows to add dynamic context to this informational messsage, like the value of some variables.
kaputt.lisp (file)
Define a test case function TESTCASE-NAME, accepting TESTCASE-ARGS with BODY.
The BODY is examined and assertions spotted in it are wrapped with extra code
installing restarts and triggering protocol events.
Test cases are allowed to be nested. A toplevel test case is a test suite and triggers testsuite protocol events when beginning and ending. The return value of a testcase is a boolean which is true iff the current testsuite has experienced a failed assertion. Thus, even if a test case does not experience any failure, a NIL value is returned if a previous test case in the current test suite has experienced a failure.
kaputt.lisp (file)
Next: Exported generic functions, Previous: Exported macros, Up: Exported definitions [Contents][Index]
The assertion (ASSERT-EQ A B) is true, iff A and B satisfy the EQ predicate.
kaputt.lisp (file)
The assertion (ASSERT-EQL A B) is true, iff A and B satisfy the EQL predicate.
kaputt.lisp (file)
The assertion (ASSERT-EQUAL A B) is true, iff A and B satisfy the EQUAL predicate.
kaputt.lisp (file)
The assertion (ASSERT-FLOAT-IS-APPROXIMATELY-EQUAL FLOAT1 FLOAT2) is true iff FLOAT1 and FLOAT2 are in a neighbourhood whose size is based on the magnitude orders of FLOAT1 and FLOAT2 and the floating point precision.
kaputt.lisp (file)
The assertion (ASSERT-FLOAT-IS-DEFINITELY-GREATER-THAN FLOAT1 FLOAT2) is true iff FLOAT1 is greater than FLOAT2 and not in a neighbourhood of FLOAT2 whose diameter is controlled by the INACCURACY, the magnitude orders of FLOAT1 and FLOAT2 and the floating point precision.
kaputt.lisp (file)
The assertion (ASSERT-FLOAT-IS-DEFINITELY-LESS-THAN FLOAT1 FLOAT2) is true iff FLOAT2
is greater than FLOAT1 and not in a neighbourhood of FLOAT1 whose diameter is controlled
by the INACCURACY, the magnitude orders of FLOAT1 and FLOAT2 and the floating point precision.
kaputt.lisp (file)
The assertion (ASSERT-FLOAT-IS-ESSENTIALLY-EQUAL FLOAT1 FLOAT2) is true iff FLOAT1 and FLOAT2 are in a neighbourhood whose size is based on the magnitude orders of FLOAT1 and FLOAT2 and the floating point precision.
kaputt.lisp (file)
The assertion (ASSERT-NIL EXPR) is true, iff EXPR is NIL.
kaputt.lisp (file)
The assertion (ASSERT-SET-EQUAL LIST1 LIST2) is true iff LIST1 denotes the same set as LIST2.
kaputt.lisp (file)
The assertion (ASSERT-STRING-EQUAL STRING1 STRING2) is true, iff STRING1 and STRING2
satisfy the STRING-EQUAL predicate.
This assertion supports the same keyword parameters as STRING-EQUAL.
kaputt.lisp (file)
The assertion (ASSERT-STRING< STRING1 STRING2) is true, iff STRING1 and STRING2
satisfy the STRING< predicate.
This assertion supports the same keyword parameters as STRING<.
kaputt.lisp (file)
The assertion (ASSERT-STRING<= STRING1 STRING2) is true, iff STRING1 and STRING2
satisfy the STRING<= predicate.
This assertion supports the same keyword parameters as STRING<=.
kaputt.lisp (file)
The assertion (ASSERT-STRING= STRING1 STRING2) is true, iff STRING1 and STRING2
satisfy the STRING= predicate.
This assertion supports the same keyword parameters as STRING=.
kaputt.lisp (file)
The assertion (ASSERT-STRING> STRING1 STRING2) is true, iff STRING1 and STRING2
satisfy the STRING> predicate.
This assertion supports the same keyword parameters as STRING>.
kaputt.lisp (file)
The assertion (ASSERT-STRING>= STRING1 STRING2) is true, iff STRING1 and STRING2
satisfy the STRING>= predicate.
This assertion supports the same keyword parameters as STRING>=.
kaputt.lisp (file)
The assertion (ASSERT-SUBSETP LIST1 LIST2) is true iff LIST1 is a subset of LIST2.
kaputt.lisp (file)
The assertion (ASSERT-T EXPR) is true, iff EXPR is a true generalised boolean.
kaputt.lisp (file)
The assertion (ASSERT-TYPE EXPR TYPE) is true, iff EXPR evaluates to a value of type TYPE.
kaputt.lisp (file)
The assertion (ASSERT-VECTOR-EQUAL VECTOR1 VECTOR2) is true iff VECTOR1 and VECTOR2 are equal.
Vectors are equal if they have the same length and have equal elements at each
index. The equality predicate used to compare elements is either EQL or the predicate
provided by the :TEST keyword argument.
kaputt.lisp (file)
The assertion (ASSERT< A B) is true, iff A and B satisfy the < predicate.
kaputt.lisp (file)
The assertion (ASSERT<= A B) is true, iff A and B satisfy the <= predicate.
kaputt.lisp (file)
The assertion (ASSERT= A B) is true, iff A and B satisfy the = predicate.
kaputt.lisp (file)
The assertion (ASSERT> A B) is true, iff A and B satisfy the > predicate.
kaputt.lisp (file)
The assertion (ASSERT>= A B) is true, iff A and B satisfy the >= predicate.
kaputt.lisp (file)
Next: Exported classes, Previous: Exported functions, Up: Exported definitions [Contents][Index]
This event is sent to a protocol when an assertion begins.
kaputt.lisp (file)
progn (short method combination)
Options: :most-specific-first
This event is sent to a protocol when an assertion ends.
kaputt.lisp (file)
progn (short method combination)
Options: :most-specific-first
This method tells if a test suite has never recorded a failure.
kaputt.lisp (file)
This event is sent to a protocol when a test case begins.
kaputt.lisp (file)
progn (short method combination)
Options: :most-specific-first
This event is sent to a protocol when a test case ends.
kaputt.lisp (file)
progn (short method combination)
Options: :most-specific-first
This event is sent to a protocol, when a test suite starts. This is the appropriate place to prepare output files, register starting time and so on.
kaputt.lisp (file)
progn (short method combination)
Options: :most-specific-first
This event is sent to a protocol when a test suite ends. This is the appopriate place to close output files, display aggregated reports.
kaputt.lisp (file)
progn (short method combination)
Options: :most-specific-first
Previous: Exported generic functions, Up: Exported definitions [Contents][Index]
This class models test suite execution protocols.
When running a test suite, the progress and results of the execution
are recorded in a PROTOCOL instance.
Descendants of this class provide more functionalities.
kaputt.lisp (file)
standard-object (class)
t
kaputt::*batch-mode*
A count protocol counts TESTCASE, ASSERTION, SUCCESS and FAILURE.
kaputt.lisp (file)
protocol (class)
protocol-dotta (class)
0
0
0
0
A dotta protocol reports assertion progress with dots and capital letter E, for success and errors respectively. At the end of a testsuite, it prints basic counts describing the current testsuite and a detailed failure report.
kaputt.lisp (file)
70
A protocol record keeps track of all failures encountered in a test suite and prints a detailed list of the failures when the test suite finishes.
kaputt.lisp (file)
protocol-verbose (class)
protocol-dotta (class)
A trace protocol reports each event sent to it.
kaputt.lisp (file)
protocol-verbose (class)
A verbose protocol owns a STREAM-OUTPUT.
kaputt.lisp (file)
protocol (class)
*standard-output*
Previous: Exported definitions, Up: Definitions [Contents][Index]
| • Internal special variables | ||
| • Internal macros | ||
| • Internal functions | ||
| • Internal generic functions | ||
| • Internal conditions |
Next: Internal macros, Previous: Internal definitions, Up: Internal definitions [Contents][Index]
When *BATCH-MODE* is NIL, the debugger will pop-up on failed assertions, unless the operator has required to scroll through errors when restarting on a previous error of the current testcase.
kaputt.lisp (file)
The current protocol.
kaputt.lisp (file)
kaputt.lisp (file)
The list of argument values used to call the current assertion in a test case.
kaputt.lisp (file)
The form used to call the current assertion in a test case.
kaputt.lisp (file)
Next: Internal functions, Previous: Internal special variables, Up: Internal definitions [Contents][Index]
kaputt.lisp (file)
Next: Internal generic functions, Previous: Internal macros, Up: Internal definitions [Contents][Index]
Walks through BODY-FORMS and wrap assertion forms in a RESTART-CASE.
kaputt.lisp (file)
kaputt.lisp (file)
kaputt.lisp (file)
kaputt.lisp (file)
Next: Internal conditions, Previous: Internal functions, Up: Internal definitions [Contents][Index]
kaputt.lisp (file)
kaputt.lisp (file)
kaputt.lisp (file)
This method tells if a test suite is in batch mode.
kaputt.lisp (file)
This method enables batch mode for a test suite.
kaputt.lisp (file)
Previous: Internal generic functions, Up: Internal definitions [Contents][Index]
kaputt.lisp (file)
serious-condition (condition)
:assertion-description
assertion-description (generic function)
(setf assertion-description) (generic function)
:assertion-form
assertion-form (generic function)
(setf assertion-form) (generic function)
:assertion-args
assertion-args (generic function)
(setf assertion-args) (generic function)
Previous: Definitions, Up: Top [Contents][Index]
| • Concept index | ||
| • Function index | ||
| • Variable index | ||
| • Data type index |
Next: Function index, Previous: Indexes, Up: Indexes [Contents][Index]
| Jump to: | F K L |
|---|
| Jump to: | F K L |
|---|
Next: Variable index, Previous: Concept index, Up: Indexes [Contents][Index]
| Jump to: | (
A D F G M P R |
|---|
| Jump to: | (
A D F G M P R |
|---|
Next: Data type index, Previous: Function index, Up: Indexes [Contents][Index]
| Jump to: | *
A B C F S T W |
|---|
| Jump to: | *
A B C F S T W |
|---|
Previous: Variable index, Up: Indexes [Contents][Index]
| Jump to: | A C K P S |
|---|
| Jump to: | A C K P S |
|---|