The testbild Reference Manual

Table of Contents

Next: , Previous: , Up: (dir)   [Contents][Index]

The testbild Reference Manual

This is the testbild Reference Manual, version 0.0.1, generated automatically by Declt version 2.3 "Robert April" on Wed Mar 14 04:38:59 2018 GMT+0.


Next: , Previous: , Up: Top   [Contents][Index]

1 Introduction

testbild - Universal test output production and consumption facility
====================================================================
testbild is a Common Lisp library designed to provide a common interface for
[Unit Testing] [test] output. Currently it supports [TAP] [tap] (versions 12,
13) and [xUnit] [xunit] style output.

Support for TAP consumption is planned.

testbild is completely written in Common Lisp and licensed under the
[GPLv3+] [gpl]. Please see the file COPYING in the top-level directory of the
distribution tarball or see the link above if you didn't receive a copy along
with this file.


Installation
------------
Please see file INSTALL.


Usage
-----
If you're using a testing framework, maybe it already offers support for
testbild. If not, go and push its authors to check it out or maybe even provide
them with patches that integrate the framework with testbild yourself!

In case you're one of said authors or doing the testing all by yourself, read
on:

Generally, testbild is comprised by a set of CLOS classes, `producer` being the
root class. It defines the minimal interface which is mostly influenced by TAP,
the output style known for offering most capabilities, but implements void
default methods for all generics save from `emit-result` which is the bare
minimum to implement for a test output class. `producer` also (re)sets and
internal test counter upon (re)initialization available via the accessor
`tests-run`, thus no additional effort has to be put into test producers relying
on test counters.

Output producer implementations should only differ in instance initialization
arguments and keywords allowed as values for `directive`, one of `emit-result`'s
`&key` arguments, hence being the only things differing between subclasses to be
taken care of by its user.

Integrating low-level test code with testbild should be done as follows:

* Generate a fresh test producer instance that is tied to the execution of a
  test suite, file etc., e.g. by creating a closure or using a special variable
* Couple the start of test code execution with `init-test`
* Same for termination and `finalize-test`
* To comply with TAP, have the user define a "test plan" no matter which output
  they choose and execute `emit-plan` right after `init-test` or right before
  `finalize-test`
* Use and/or provide `emit-comment` and `emit-bailout` as desired

A typical use case might be as follows:

    CL-USER> (let ((producer (make-instance 'tap-producer)))
               (init-test producer)
               (emit-plan producer :plan-argument 3) ; simple plan, three tests planned
               (emit-result producer :success (= (+ 1 2) 3) :description "test whether 1+2 equals 3")
               (emit-comment producer "Math is fun!")
               (emit-result producer :success (eql nil 42) :description "42 is the answer!" :directive :todo :reason "But we're still in hunt of the question..")
               (emit-result producer :success t :directive :skip :reason "Skipping stuff is also supported")
               (finalize-test producer))
    TAP version 13
    1..3
    ok 1 - test whether 1+2 equals 3
    # Math is fun!
    not ok 2 - 42 is the answer! # TODO But we're still in hunt of the question..
    ok 3 # SKIP Skipping stuff is also supported

If _x-unit-producer_ was used instead, this would have been the output:

    .IS
on STDOUT and

    Math is fun!
on STDERR as xUnit has no native support for comments.


Missing Stuff
-------------
* TAP
  * Support for [YAMLish] [yamlish] diagnostics
  * Consumer part
* Other formats
  * Story / BDD
  * TestDox

History
-------
While there are many nice testing framework for Common Lisp and writing your own
test code is especially easy with the most powerful language on earth, most if
not all of said frameworks use an inferior report style usually found in and
associated with the xUnit family of unit testing frameworks, even though there
is a formally specified, clearly superior alternative available: TAP, Test
Anything Protocol.

In lack of TAP support _Testbild_, which is the German word for (TV) test
pattern but also interpretable as (overall) "test appearance", was written to
help both test and test framework authors exploit the features of TAP without
losing support for different reporting styles.


Links and References
--------------------
Homepage:   
Hacking:  

[gpl]: http://www.gnu.org/licenses/gpl-3.0-standalone.html
[test]: http://en.wikipedia.org/wiki/Unit_testing
[tap]: http://testanything.org/
[xunit]: http://en.wikipedia.org/wiki/XUnit
[yamlish]: http://testanything.org/wiki/index.php?title=YAMLish&oldid=1930



Copyright
---------
Copyright (C) 2010  Alexander Kahl   
This file is part of testbild.  
testbild is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3 of the License, or
(at your option) any later version.

testbild is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see .


Next: , Previous: , Up: Top   [Contents][Index]

2 Systems

The main system appears first, followed by any subsystem dependency.


Previous: , Up: Systems   [Contents][Index]

2.1 testbild

Author

Alexander Kahl <e-user@fsfe.org>

License

GPLv3+

Description

Universal test output production and consumption facility

Version

0.0.1

Dependencies
Source

testbild.asd (file)

Component

src (module)


Next: , Previous: , Up: Top   [Contents][Index]

3 Modules

Modules are listed depth-first from the system components tree.


Previous: , Up: Modules   [Contents][Index]

3.1 testbild/src

Parent

testbild (system)

Location

src/

Components

Next: , Previous: , Up: Top   [Contents][Index]

4 Files

Files are sorted by type and then listed depth-first from the systems components trees.


Previous: , Up: Files   [Contents][Index]

4.1 Lisp


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.1 testbild.asd

Location

testbild.asd

Systems

testbild (system)

Packages

testbild-system


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.2 testbild/src/package.lisp

Parent

src (module)

Location

src/package.lisp

Packages

testbild


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.3 testbild/src/producer.lisp

Dependency

package.lisp (file)

Parent

src (module)

Location

src/producer.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.4 testbild/src/tap.lisp

Dependencies
Parent

src (module)

Location

src/tap.lisp

Exported Definitions

Previous: , Up: Lisp files   [Contents][Index]

4.1.5 testbild/src/x-unit.lisp

Dependencies
Parent

src (module)

Location

src/x-unit.lisp

Exported Definitions

Next: , Previous: , Up: Top   [Contents][Index]

5 Packages

Packages are listed by definition order.


Next: , Previous: , Up: Packages   [Contents][Index]

5.1 testbild-system

Source

testbild.asd

Use List

Previous: , Up: Packages   [Contents][Index]

5.2 testbild

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Top   [Contents][Index]

6 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.


Next: , Previous: , Up: Definitions   [Contents][Index]

6.1 Exported definitions


Next: , Previous: , Up: Exported definitions   [Contents][Index]

6.1.1 Generic functions

Generic Function: emit-bailout PRODUCER &optional REASON

Emit a bail-out, if available for this kind of producer.

Package

testbild

Source

producer.lisp (file)

Methods
Method: emit-bailout PRODUCER &optional REASON
Generic Function: emit-comment PRODUCER COMMENT

Emit a comment, if available for this kind of producer.

Package

testbild

Source

producer.lisp (file)

Methods
Method: emit-comment (PRODUCER x-unit-producer) COMMENT

emit-comment producer comment => nil

xUnit has no support for comments so we use STDERR.

Source

x-unit.lisp (file)

Method: emit-comment (PRODUCER tap-producer) COMMENT

emit-comment producer comment => comment-list

TAP comments are delimited by hash marks and EOLs so COMMENT is split by all known EOL delimiter combinations and output with a hash mark in front each.

Source

tap.lisp (file)

Method: emit-comment PRODUCER COMMENT
Generic Function: emit-plan PRODUCER &key PLAN PLAN-ARGUMENT &allow-other-keys

A so-called test plan is used e.g. by TAP as a means of cross-checking proper test suite execution.

Package

testbild

Source

producer.lisp (file)

Methods
Method: emit-plan (PRODUCER tap-producer) &key PLAN PLAN-ARGUMENT &allow-other-keys

emit-plan producer &key (plan :simple) plan-argument => nil

TAP plans can be of type :SIMPLE or :SKIP, the latter means to skip all tests within the suite.

Source

tap.lisp (file)

Method: emit-plan (PRODUCER tap-producer) &key PLAN PLAN-ARGUMENT before

emit-plan :before producer &key plan plan-argument => nil

TAP test plans must always begin at the beginning of a line and start with "1..".

Source

tap.lisp (file)

Method: emit-plan PRODUCER &key PLAN PLAN-ARGUMENT &allow-other-keys
Generic Function: emit-result PRODUCER &key SUCCESS DESCRIPTION DIRECTIVE REASON &allow-other-keys

Emit the result from running an test / an assertion.

Package

testbild

Source

producer.lisp (file)

Methods
Method: emit-result (PRODUCER x-unit-producer) &key SUCCESS DESCRIPTION DIRECTIVE REASON &allow-other-keys

emit-result producer &key (success t) description directive reason &allow-other-keys => nil

xUnit output consists of single characters per assertion.

Source

x-unit.lisp (file)

Method: emit-result (PRODUCER x-unit-producer) &key SUCCESS DESCRIPTION DIRECTIVE REASON &allow-other-keys before

emit-result :before producer &key success description directive reason &allow-other-keys => nil

Ensure output proceeds on a fresh line after FILL-COLUMN test assertions, if set.

Source

x-unit.lisp (file)

Method: emit-result (PRODUCER tap-producer) &key SUCCESS DESCRIPTION DIRECTIVE REASON &allow-other-keys

emit-result producer &key (success t) description directive reason &allow-other-keys => nil

TAP style test result emitter. DIRECTIVE types :TODO and :SKIP are supported,
:error is ignored.

Source

tap.lisp (file)

Method: emit-result (PRODUCER tap-producer) &key SUCCESS DESCRIPTION DIRECTIVE REASON &allow-other-keys before

emit-result :before producer &key success description directive reason &allow-other-keys => nil

TAP test results must always begin at the beginning of a line.

Source

tap.lisp (file)

Method: emit-result (PRODUCER producer) &key SUCCESS DESCRIPTION DIRECTIVE REASON &allow-other-keys before

emit-result :before producer &key success description directive reason &allow-other-keys => number

Increase the recorded number of tests by one.

Generic Function: finalize-test PRODUCER

Emit final test output.

Package

testbild

Source

producer.lisp (file)

Methods
Method: finalize-test (PRODUCER x-unit-producer)

finalize-test producer => nil

xUnit output always ends with a line feed.

Source

x-unit.lisp (file)

Method: finalize-test PRODUCER
Generic Function: init-test PRODUCER

Test initialization / header output.

Package

testbild

Source

producer.lisp (file)

Methods
Method: init-test (PRODUCER x-unit-producer)

init-test producer => nil

xUnit output always starts on a fresh line.

Source

x-unit.lisp (file)

Method: init-test (PRODUCER tap-producer)

init-test producer => nil

TAP version 13 and higher supports an optional version line in the header.

Source

tap.lisp (file)

Method: init-test PRODUCER
Generic Function: producer-fill-column OBJECT
Package

testbild

Methods
Method: producer-fill-column (X-UNIT-PRODUCER x-unit-producer)

Line feeds will be inserted after FILL-COLUMN characters if set.

Source

x-unit.lisp (file)

Generic Function: supported-tap-versions OBJECT
Package

testbild

Methods
Method: supported-tap-versions (TAP-PRODUCER tap-producer)

List of supported TAP versions.

Source

tap.lisp (file)

Generic Function: tap-version OBJECT
Package

testbild

Methods
Method: tap-version (TAP-PRODUCER tap-producer)

TAP output version.

Source

tap.lisp (file)

Generic Function: tests-run OBJECT
Generic Function: (setf tests-run) NEW-VALUE OBJECT
Package

testbild

Methods
Method: tests-run (PRODUCER producer)
Method: (setf tests-run) NEW-VALUE (PRODUCER producer)

Recorded number of test results emitted using this producer.

Source

producer.lisp (file)


Previous: , Up: Exported definitions   [Contents][Index]

6.1.2 Classes

Class: producer ()

Base class for test output producers.

Package

testbild

Source

producer.lisp (file)

Direct superclasses

standard-object (class)

Direct subclasses
Direct methods
Direct slots
Slot: stream

Stream to use for test output

Initargs

:stream

Initform

*standard-output*

Readers

test-producer-stream (generic function)

Writers

(setf test-producer-stream) (generic function)

Slot: tests-run

Recorded number of test results emitted using this producer.

Initform

0

Readers

tests-run (generic function)

Writers

(setf tests-run) (generic function)

Class: tap-producer ()

Producer for TAP (Test Anything Protocol) output.

Package

testbild

Source

tap.lisp (file)

Direct superclasses

producer (class)

Direct methods
Direct slots
Slot: version

TAP output version.

Initargs

:version

Initform

13

Readers

tap-version (generic function)

Slot: supported-versions

List of supported TAP versions.

Allocation

:class

Initform

(quote (12 13))

Readers

supported-tap-versions (generic function)

Class: x-unit-producer ()

Producer for xUnit style test output.

Package

testbild

Source

x-unit.lisp (file)

Direct superclasses

producer (class)

Direct methods
Direct slots
Slot: fill-column

Line feeds will be inserted after FILL-COLUMN characters if set.

Initargs

:fill-column

Readers

producer-fill-column (generic function)


Previous: , Up: Definitions   [Contents][Index]

6.2 Internal definitions


Previous: , Up: Internal definitions   [Contents][Index]

6.2.1 Generic functions

Generic Function: test-producer-stream OBJECT
Generic Function: (setf test-producer-stream) NEW-VALUE OBJECT
Package

testbild

Methods
Method: test-producer-stream (PRODUCER producer)
Method: (setf test-producer-stream) NEW-VALUE (PRODUCER producer)

Stream to use for test output

Source

producer.lisp (file)


Previous: , Up: Top   [Contents][Index]

Appendix A Indexes


Next: , Previous: , Up: Indexes   [Contents][Index]

A.1 Concepts

Jump to:   F   L   M   T  
Index Entry  Section

F
File, Lisp, testbild.asd: The testbild<dot>asd file
File, Lisp, testbild/src/package.lisp: The testbild/src/package<dot>lisp file
File, Lisp, testbild/src/producer.lisp: The testbild/src/producer<dot>lisp file
File, Lisp, testbild/src/tap.lisp: The testbild/src/tap<dot>lisp file
File, Lisp, testbild/src/x-unit.lisp: The testbild/src/x-unit<dot>lisp file

L
Lisp File, testbild.asd: The testbild<dot>asd file
Lisp File, testbild/src/package.lisp: The testbild/src/package<dot>lisp file
Lisp File, testbild/src/producer.lisp: The testbild/src/producer<dot>lisp file
Lisp File, testbild/src/tap.lisp: The testbild/src/tap<dot>lisp file
Lisp File, testbild/src/x-unit.lisp: The testbild/src/x-unit<dot>lisp file

M
Module, testbild/src: The testbild/src module

T
testbild.asd: The testbild<dot>asd file
testbild/src: The testbild/src module
testbild/src/package.lisp: The testbild/src/package<dot>lisp file
testbild/src/producer.lisp: The testbild/src/producer<dot>lisp file
testbild/src/tap.lisp: The testbild/src/tap<dot>lisp file
testbild/src/x-unit.lisp: The testbild/src/x-unit<dot>lisp file

Jump to:   F   L   M   T  

Next: , Previous: , Up: Indexes   [Contents][Index]

A.2 Functions

Jump to:   (  
E   F   G   I   M   P   S   T  
Index Entry  Section

(
(setf test-producer-stream): Internal generic functions
(setf test-producer-stream): Internal generic functions
(setf tests-run): Exported generic functions
(setf tests-run): Exported generic functions

E
emit-bailout: Exported generic functions
emit-bailout: Exported generic functions
emit-comment: Exported generic functions
emit-comment: Exported generic functions
emit-comment: Exported generic functions
emit-comment: Exported generic functions
emit-plan: Exported generic functions
emit-plan: Exported generic functions
emit-plan: Exported generic functions
emit-plan: Exported generic functions
emit-result: Exported generic functions
emit-result: Exported generic functions
emit-result: Exported generic functions
emit-result: Exported generic functions
emit-result: Exported generic functions
emit-result: Exported generic functions

F
finalize-test: Exported generic functions
finalize-test: Exported generic functions
finalize-test: Exported generic functions

G
Generic Function, (setf test-producer-stream): Internal generic functions
Generic Function, (setf tests-run): Exported generic functions
Generic Function, emit-bailout: Exported generic functions
Generic Function, emit-comment: Exported generic functions
Generic Function, emit-plan: Exported generic functions
Generic Function, emit-result: Exported generic functions
Generic Function, finalize-test: Exported generic functions
Generic Function, init-test: Exported generic functions
Generic Function, producer-fill-column: Exported generic functions
Generic Function, supported-tap-versions: Exported generic functions
Generic Function, tap-version: Exported generic functions
Generic Function, test-producer-stream: Internal generic functions
Generic Function, tests-run: Exported generic functions

I
init-test: Exported generic functions
init-test: Exported generic functions
init-test: Exported generic functions
init-test: Exported generic functions

M
Method, (setf test-producer-stream): Internal generic functions
Method, (setf tests-run): Exported generic functions
Method, emit-bailout: Exported generic functions
Method, emit-comment: Exported generic functions
Method, emit-comment: Exported generic functions
Method, emit-comment: Exported generic functions
Method, emit-plan: Exported generic functions
Method, emit-plan: Exported generic functions
Method, emit-plan: Exported generic functions
Method, emit-result: Exported generic functions
Method, emit-result: Exported generic functions
Method, emit-result: Exported generic functions
Method, emit-result: Exported generic functions
Method, emit-result: Exported generic functions
Method, finalize-test: Exported generic functions
Method, finalize-test: Exported generic functions
Method, init-test: Exported generic functions
Method, init-test: Exported generic functions
Method, init-test: Exported generic functions
Method, producer-fill-column: Exported generic functions
Method, supported-tap-versions: Exported generic functions
Method, tap-version: Exported generic functions
Method, test-producer-stream: Internal generic functions
Method, tests-run: Exported generic functions

P
producer-fill-column: Exported generic functions
producer-fill-column: Exported generic functions

S
supported-tap-versions: Exported generic functions
supported-tap-versions: Exported generic functions

T
tap-version: Exported generic functions
tap-version: Exported generic functions
test-producer-stream: Internal generic functions
test-producer-stream: Internal generic functions
tests-run: Exported generic functions
tests-run: Exported generic functions

Jump to:   (  
E   F   G   I   M   P   S   T  

Next: , Previous: , Up: Indexes   [Contents][Index]

A.3 Variables

Jump to:   F   S   T   V  
Index Entry  Section

F
fill-column: Exported classes

S
Slot, fill-column: Exported classes
Slot, stream: Exported classes
Slot, supported-versions: Exported classes
Slot, tests-run: Exported classes
Slot, version: Exported classes
stream: Exported classes
supported-versions: Exported classes

T
tests-run: Exported classes

V
version: Exported classes

Jump to:   F   S   T   V  

Previous: , Up: Indexes   [Contents][Index]

A.4 Data types

Jump to:   C   P   S   T   X  
Index Entry  Section

C
Class, producer: Exported classes
Class, tap-producer: Exported classes
Class, x-unit-producer: Exported classes

P
Package, testbild: The testbild package
Package, testbild-system: The testbild-system package
producer: Exported classes

S
System, testbild: The testbild system

T
tap-producer: Exported classes
testbild: The testbild system
testbild: The testbild package
testbild-system: The testbild-system package

X
x-unit-producer: Exported classes

Jump to:   C   P   S   T   X