The should-test Reference Manual

Table of Contents

Next: , Previous: , Up: (dir)   [Contents][Index]

The should-test Reference Manual

This is the should-test Reference Manual, version 1.0.0, generated automatically by Declt version 2.3 "Robert April" on Wed Mar 14 04:34:28 2018 GMT+0.


Next: , Previous: , Up: Top   [Contents][Index]

1 Introduction

SHOULD-TEST - a Common Lisp Testing Library

(you should test even Common Lisp code sometimes)

SHOULD-TEST is a minimal yet feature-rich Common Lisp test framework.

SHOULD-TEST is methodology-agnostic and non-opinionated, i.e. it doesn't care what kind of test approach you'd like to take (like unit testing, random testing a la quickcheck or something else).

It's just built from first principles to facilitate the following activities:

The library is at the rather early stages of its development, but it's actively used in the support of RUTILS, CL-REDIS, and some of my in-house projects.

Usage

Defining tests

Test are defined with deftest:

(deftest some-fn ()
  (should be = 1 (some-fn 2))
  (should be = 2 (some-fn 1)))

Being run, deftest returns either T or NIL as primary value. Secondary and third values in case of NIL are lists of:

should is a macro that takes care of checking assertions. If the assertion doesn't hold, should signals a condition of types should-failed or should-erred which are aggregated by deftest.

Also, should returns either T or NIL and a list of a failed expression with expected and actual outputs as values.

Under the hood it calls the generic function should-check and passes it a keyword produced from the first symbol (in this case, :be), a test predicate (here, '=), and a tested expression as thunk (here it will be e.g. (lambda () (some-fn 1))), and expected results if any. If multiple expected results are given, like in (should be eql nil #{:failed 1} (some-other-fn :dummy)), it means that multiple values are expected. As you see, the keyword and test predicate are passed unevaluated, so you can't use expressions here.

The pre-defined types of assertions are be, signal and print-to. They check correspondingly:

Obviously, it's possible to define more assertion types as methods of should-check generic function.

deftest and should write summary of test results to *test-output* (by default bound to *standard-output*). The var *verbose* (default T) controls if the summary contains full failure reports or just test names.

Tests are defined as lambda-functions attached to a symbol's test property, so (deftest some-fn ... will do the following:

(setf (get some-fn 'test)
      (lambda () ...))

Running tests

To run the tests, use test. Without arguments, it runs all the tests in the current package. Given a :package argument it will do the same for that package, and given a :test argument it will run that individual test.

In case of individual test's failure it will return NIL and a list of failed assertions and a list of assertions, that triggered uncaught errors.

In case of failed test of a package it will return NIL and 2 hash-tables holding the same lists as above keyed by failed test's names.

As you see, the system uses a somewhat recursive protocol for test results:

So the structure of the summary, returned from test, will be the following:

#{
  failed-test-1 ((failed-assertion-1 expected actual)
                 (failed-assertion-2 ...
  failed-test-2 ...
 }

(#{} are RUTILS literal hash-table delimiters)

There's also :failed key to test that will re-test only tests which failed at their last run.

Usage patterns

As SHOULD-TEST is agnostic, it doesn't impose any restrictions on how each individual project organizes its tests. Yet, having established patterns and best-practices never hearts. This section collects some of them.

There's no restriction on naming tests. Though it seems like a good approach to name them the same as functions they test. As for generic functions, it, probably, makes sense to have different tests for different methods. In this case I add some suffix to the test's name to indicate which method is tested (like transform-string for one of the methods of gf transform that is specialized for the string class of arguments).

As for code organization, I use the following directory structure of the typical project:

/project-root
 |----src
 |    `----module
 |         `-----file.lisp
 `----test
      |----some-general-tests.lisp
      `----module
           `-----file-test.lisp

I also usually place the tests in the same package as the code they test but protect them with #+dev guard, so that in production environment they are not compiled and loaded altogether.

ASDF provides a way to define the standard for testing a system that can be invoked with asdf:test-system. The easiest way to hook into this facility is to define the following method for asdf:test-op somewhere either in package.lisp or in some common file in the test module (in the example above: some-general-tests.lisp):

(defmethod asdf:perform ((o asdf:test-op)
                         (s (eql (asdf:find-system <your-system>))))
  (asdf:load-system <your-system>)
  (st:test :package <your-package>))
  t)

Quickstart

As the project just got started it's not in quicklisp. So to add it as an ASDF-dependency you have manually download/clone the project. The other option is to just take the file src/should-test.lisp and drop it into your project. It's designed to be self-contained: it contains the package definition and implements the core features of the framework.

Requirements

Self-testing

There's a minimal test suite defined in src/self-test.lisp. The test suite is also hooked to asdf:test-op for the should-test system.

License

Copyright (c) 2013-2015 Vsevolod Dyomkin vseloved@gmail.com

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

Except as contained in this notice, the name(s) of the above copyright holders shall not be used in advertising or otherwise to promote the sale, use or other dealings in this Software without prior written authorization.

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.


Next: , Previous: , Up: Top   [Contents][Index]

2 Systems

The main system appears first, followed by any subsystem dependency.


Previous: , Up: Systems   [Contents][Index]

2.1 should-test

Maintainer

Vsevolod Dyomkin <vseloved@gmail.com>

Author

Vsevolod Dyomkin <vseloved@gmail.com>

License

MIT

Description

Minimal yet feature-rich Common Lisp test framework.

Version

1.0.0

Dependencies
Source

should-test.asd (file)

Components

Next: , Previous: , Up: Top   [Contents][Index]

3 Files

Files are sorted by type and then listed depth-first from the systems components trees.


Previous: , Up: Files   [Contents][Index]

3.1 Lisp


Next: , Previous: , Up: Lisp files   [Contents][Index]

3.1.1 should-test.asd

Location

should-test.asd

Systems

should-test (system)


Next: , Previous: , Up: Lisp files   [Contents][Index]

3.1.2 should-test/should-test.lisp

Parent

should-test (system)

Location

should-test.lisp

Packages

should-test

Exported Definitions
Internal Definitions

should-checked-rez (method)


Next: , Previous: , Up: Lisp files   [Contents][Index]

3.1.3 should-test/self-test.lisp

Dependency

should-test.lisp (file)

Parent

should-test (system)

Location

self-test.lisp


Previous: , Up: Lisp files   [Contents][Index]

3.1.4 should-test/xunit.lisp

Dependency

self-test.lisp (file)

Parent

should-test (system)

Location

xunit.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Top   [Contents][Index]

4 Packages

Packages are listed by definition order.


Previous: , Up: Packages   [Contents][Index]

4.1 should-test

Source

should-test.lisp (file)

Nickname

st

Use List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Top   [Contents][Index]

5 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.


Next: , Previous: , Up: Definitions   [Contents][Index]

5.1 Exported definitions


Next: , Previous: , Up: Exported definitions   [Contents][Index]

5.1.1 Special variables

Special Variable: *test-output*

Stream to print test results.

Package

should-test

Source

should-test.lisp (file)

Special Variable: *verbose*
Package

should-test

Source

should-test.lisp (file)

Special Variable: *xml-output*
Package

should-test

Source

xunit.lisp (file)


Next: , Previous: , Up: Exported definitions   [Contents][Index]

5.1.2 Macros

Macro: deftest NAME () &body BODY

Define a NAMEd test which is a function
that treats each form in its BODY as an assertion to be checked and prints some information to the output.
The result of this function is a boolean indicating
if any of the assertions has failed.
In case of failure second value is a list of failure descriptions, returned from assertions,
and the third value is a list of uncaught errors if any.

Package

should-test

Source

should-test.lisp (file)

Macro: should KEY TEST &rest EXPECTED-AND-TESTEE

Define an individual test from:

- a comparison TEST
- EXPECTED values
- an operation that needs to be tested (TESTEE)

KEY is used to determine, which kind of results processing is needed (implemented by generic function SHOULD-CHECK methods).
The simplest key is BE that just checks for equality.
Another pre-defined key is SIGNAL, which intercepts conditions.

Package

should-test

Source

should-test.lisp (file)


Next: , Previous: , Up: Exported definitions   [Contents][Index]

5.1.3 Functions

Function: test &key PACKAGE TEST FAILED

Run a scpecific TEST or all tests defined in PACKAGE (defaults to current).

Returns T if all tests pass or 3 values:

- NIL
- a hash-table of failed tests with their failed assertions’ lists
- a hash-table of tests that have signalled uncaught errors with these errors

If FAILED is set reruns only tests failed at last run.

Package

should-test

Source

should-test.lisp (file)

Function: test-for-xunit OUT &rest ARGS &key ID PACKAGE TEST

Like TEST but writes xunit-style XML test report to OUT.

Package

should-test

Source

xunit.lisp (file)

Function: undeftest NAME

Remove test from symbol NAME.

Package

should-test

Source

should-test.lisp (file)


Next: , Previous: , Up: Exported definitions   [Contents][Index]

5.1.4 Generic functions

Generic Function: should-check KEY TEST FN &rest EXPECTED

Specific processing for SHOULD based on KEY.
FN’s output values are matched to EXPECTED values (if they are given). Up to 2 values are returned:

- if the test passed (T or NIL)
- in case of failure - actual result

Package

should-test

Source

should-test.lisp (file)

Methods
Method: should-check (KEY (eql print-to)) STREAM-SYM FN &rest EXPECTED
Method: should-check (KEY (eql signal)) TEST FN &rest EXPECTED
Method: should-check (KEY (eql be)) TEST FN &rest EXPECTED
Generic Function: should-format OBJ

Format appropriately for test output.

Package

should-test

Source

should-test.lisp (file)

Methods
Method: should-format OBJ around
Method: should-format OBJ
Method: should-format (OBJ hash-table)
Method: should-format (OBJ list)

Previous: , Up: Exported definitions   [Contents][Index]

5.1.5 Conditions

Condition: should-checked ()
Package

should-test

Source

should-test.lisp (file)

Direct superclasses

condition (condition)

Direct subclasses
Direct methods

should-checked-rez (method)

Direct slots
Slot: rez
Initargs

:rez

Readers

should-checked-rez (generic function)

Condition: should-erred ()
Package

should-test

Source

should-test.lisp (file)

Direct superclasses

should-checked (condition)

Condition: should-failed ()
Package

should-test

Source

should-test.lisp (file)

Direct superclasses

should-checked (condition)

Condition: should-test-error ()
Package

should-test

Source

should-test.lisp (file)

Direct superclasses

simple-error (condition)

Condition: should-test-redefinition-warning ()
Package

should-test

Source

should-test.lisp (file)

Direct superclasses

style-warning (condition)

Direct slots
Slot: name
Initargs

:name


Previous: , Up: Definitions   [Contents][Index]

5.2 Internal definitions


Next: , Previous: , Up: Internal definitions   [Contents][Index]

5.2.1 Macros

Macro: xml TAG &rest ATTRS-&-BODY
Package

should-test

Source

xunit.lisp (file)


Next: , Previous: , Up: Internal definitions   [Contents][Index]

5.2.2 Functions

Function: xml-escape VALUE
Package

should-test

Source

xunit.lisp (file)


Previous: , Up: Internal definitions   [Contents][Index]

5.2.3 Generic functions

Generic Function: should-checked-rez CONDITION
Package

should-test

Methods
Method: should-checked-rez (CONDITION should-checked)
Source

should-test.lisp (file)


Previous: , Up: Top   [Contents][Index]

Appendix A Indexes


Next: , Previous: , Up: Indexes   [Contents][Index]

A.1 Concepts

Jump to:   F   L   S  
Index Entry  Section

F
File, Lisp, should-test.asd: The should-test<dot>asd file
File, Lisp, should-test/self-test.lisp: The should-test/self-test<dot>lisp file
File, Lisp, should-test/should-test.lisp: The should-test/should-test<dot>lisp file
File, Lisp, should-test/xunit.lisp: The should-test/xunit<dot>lisp file

L
Lisp File, should-test.asd: The should-test<dot>asd file
Lisp File, should-test/self-test.lisp: The should-test/self-test<dot>lisp file
Lisp File, should-test/should-test.lisp: The should-test/should-test<dot>lisp file
Lisp File, should-test/xunit.lisp: The should-test/xunit<dot>lisp file

S
should-test.asd: The should-test<dot>asd file
should-test/self-test.lisp: The should-test/self-test<dot>lisp file
should-test/should-test.lisp: The should-test/should-test<dot>lisp file
should-test/xunit.lisp: The should-test/xunit<dot>lisp file

Jump to:   F   L   S  

Next: , Previous: , Up: Indexes   [Contents][Index]

A.2 Functions

Jump to:   D   F   G   M   S   T   U   X  
Index Entry  Section

D
deftest: Exported macros

F
Function, test: Exported functions
Function, test-for-xunit: Exported functions
Function, undeftest: Exported functions
Function, xml-escape: Internal functions

G
Generic Function, should-check: Exported generic functions
Generic Function, should-checked-rez: Internal generic functions
Generic Function, should-format: Exported generic functions

M
Macro, deftest: Exported macros
Macro, should: Exported macros
Macro, xml: Internal macros
Method, should-check: Exported generic functions
Method, should-check: Exported generic functions
Method, should-check: Exported generic functions
Method, should-checked-rez: Internal generic functions
Method, should-format: Exported generic functions
Method, should-format: Exported generic functions
Method, should-format: Exported generic functions
Method, should-format: Exported generic functions

S
should: Exported macros
should-check: Exported generic functions
should-check: Exported generic functions
should-check: Exported generic functions
should-check: Exported generic functions
should-checked-rez: Internal generic functions
should-checked-rez: Internal generic functions
should-format: Exported generic functions
should-format: Exported generic functions
should-format: Exported generic functions
should-format: Exported generic functions
should-format: Exported generic functions

T
test: Exported functions
test-for-xunit: Exported functions

U
undeftest: Exported functions

X
xml: Internal macros
xml-escape: Internal functions

Jump to:   D   F   G   M   S   T   U   X  

Next: , Previous: , Up: Indexes   [Contents][Index]

A.3 Variables

Jump to:   *  
N   R   S  
Index Entry  Section

*
*test-output*: Exported special variables
*verbose*: Exported special variables
*xml-output*: Exported special variables

N
name: Exported conditions

R
rez: Exported conditions

S
Slot, name: Exported conditions
Slot, rez: Exported conditions
Special Variable, *test-output*: Exported special variables
Special Variable, *verbose*: Exported special variables
Special Variable, *xml-output*: Exported special variables

Jump to:   *  
N   R   S  

Previous: , Up: Indexes   [Contents][Index]

A.4 Data types

Jump to:   C   P   S  
Index Entry  Section

C
Condition, should-checked: Exported conditions
Condition, should-erred: Exported conditions
Condition, should-failed: Exported conditions
Condition, should-test-error: Exported conditions
Condition, should-test-redefinition-warning: Exported conditions

P
Package, should-test: The should-test package

S
should-checked: Exported conditions
should-erred: Exported conditions
should-failed: Exported conditions
should-test: The should-test system
should-test: The should-test package
should-test-error: Exported conditions
should-test-redefinition-warning: Exported conditions
System, should-test: The should-test system

Jump to:   C   P   S