CLUnit
CLUnit is a Common Lisp unit testing framework. It is designed to be easy to use so that you can quickly start testing.
CLUnit provides a rich set of features aimed at improving your unit testing experience:
- Multiple inheritance for test suites allows you to group tests into hierarchies.
- Composes the test results of each test run into a single report.
- Allows redefinition of inline functions and macros without having to redefine your tests.
- Supports composable test suite fixtures.
- Allows for an interactive testing process which gives you access to the test environment.
- Provides visual feeback of the unit test progress.
- Extensible test reporting. Builtin support for default reporting and TAP output.
Table of Contents
Documentation
The API documentation can be found here. If you are interested in the story behind CLUnit you can find it here.
Tutorial
For this tutorial we will write a unit test for our hypothetical number crunching library. Our unit test contains four test suites and five test cases. The image below gives a visual representation of our test hierarchy.
Creating the test hierarchy
To create the test suite hierarchy shown in the image we use the following code.
;; Test suite for all number operation tests.
(defsuite NumberSuite ())
;; Test suite for floating point operations
(defsuite FloatSuite (NumberSuite))
;; Test suite for integer operations.
(defsuite IntegerSuite (NumberSuite))
; Test suite for boolean operations on numbers.
(defsuite BooleanSuite (FloatSuite IntegerSuite))
Defining a test
Next we define two test cases for our FloatSuite and IntegerSuite test suites. Each test contains two assertion forms.
ASSERT-TRUE considers the assertion a pass if the expression supplied to it returns true and (ASSERT-EQUALITY value expression) considers the assertion to have passed if (FUNCALL clunit:*clunit-equality-test* value expression) returns true.
;; Define a test called TEST-INT1
(deftest test-int1 (IntegerSuite)
(assert-true (= 1 -1))
(assert-equality 4 (+ 2 2)))
;; Define a test called TEST-FLOAT1
(deftest test-float1 (FloatSuite)
(assert-true (= 1.0 -1.0))
(assert-equality 4.0 (+ 2.0 2.0)))
Running our first test
To run an individual test you use the form (RUN-TEST 'test-name), however we are interested in running the entire test suite so we shall use the form (RUN-SUITE 'test-name).
(run-suite 'NumberSuite)
PROGRESS:
=========
NUMBERSUITE: (Test Suite)
INTEGERSUITE: (Test Suite)
TEST-INT1: F.
BOOLEANSUITE: (Test Suite)
FLOATSUITE: (Test Suite)
TEST-FLOAT1: F.
BOOLEANSUITE: (Test Suite)
FAILURE DETAILS:
================
NUMBERSUITE -> INTEGERSUITE: (Test Suite)
TEST-INT1: Expression: (= 1 -1)
Expected: T
Returned: NIL
NUMBERSUITE -> FLOATSUITE: (Test Suite)
TEST-FLOAT1: Expression: (= 1.0 -1.0)
Expected: T
Returned: NIL
SUMMARY:
========
Test functions:
Executed: 2
Skipped: 0
Tested 4 assertions.
Passed: 2/4 ( 50.0%)
Failed: 2/4 ( 50.0%)
Errors: 0/4 ( 0.0%)
The functions RUN-SUITE and RUN-TEST by default print out the test progress as they execute. The function RUN-SUITE first executes the tests referred to by the test suite argument. After executing the test cases, it then executes the child test suites. This process is repeated recursively until all the test cases and test suites in the hierarchy have been executed.
During the test progress, the names of test suites and test cases are printed using a visual format that shows how the hierarchy is nested. Each assertion in a test case prints either . for a pass or F for a failure. An E is printed when an error occurs in the test, when an E is seen it means there is a serious problem in your code and the rest of the test case is skipped.
The progress report can be switched off by setting the :report-progress keyword argument to NIL for example:
(run-suite 'NumberSuite :report-progress nil)
FAILURE DETAILS:
================
NUMBERSUITE -> INTEGERSUITE: (Test Suite)
TEST-INT1: Expression: (= 1 -1)
Expected: T
Returned: NIL
NUMBERSUITE -> FLOATSUITE: (Test Suite)
TEST-FLOAT1: Expression: (= 1.0 -1.0)
Expected: T
Returned: NIL
SUMMARY:
========
Test functions:
Executed: 2
Skipped: 0
Tested 4 assertions.
Passed: 2/4 ( 50.0%)
Failed: 2/4 ( 50.0%)
Errors: 0/4 ( 0.0%)
Rerunning failed tests
CLUnit provides a convenient function (RERUN-FAILED-TESTS) for rerunning only the tests that failed the last time. When the failed tests are fixed, you do the normal run to make sure all tests pass.
Changing the report format
The functions RUN-SUITE and RUN-TEST return a CLUNIT-REPORT object, the default PRINT-OBJECT method prints the aggregated details of the unit test failures and a test summary.
The variable clunit:*clunit-report-format* controls the output format of the unit test results. Possible values are :default, :tap or NIL. The value :tap sets the reporting to TAP output.
(setf clunit:*clunit-report-format* :tap)
(run-suite 'NumberSuite :report-progress nil)
TAP version 13
1..4
not ok 1
# Suite: NUMBERSUITE -> INTEGERSUITE
# Test: TEST-INT1
# Expression: (= 1 -1)
# Expected: T
# Returned: NIL
ok 2
not ok 3
# Suite: NUMBERSUITE -> FLOATSUITE
# Test: TEST-FLOAT1
# Expression: (= 1.0 -1.0)
# Expected: T
# Returned: NIL
ok 4
Controlling diagnostic output
Apart from the mandatory arguments, the assertion forms accept additional arguments that will be printed if an error occurs. An assertion macro call then takes the the following form:
(ASSERT-TRUE expression [form1] [form2] ... [formN])
If the assertion fails the form and its value is printed. However, if the form is a string only the value is printed so this can be used to insert comments into the diagnostic output.
(deftest test-suiteless ()
(let ((a 1) (b 2) (c 3))
(assert-true (= a b c) "This assertion is meant to fail." a b c )))
(run-test 'test-suiteless :report-progress nil)
FAILURE DETAILS:
================
TEST-SUITELESS: Expression: (= A B C)
Expected: T
Returned: NIL
This assertion is meant to fail.
A => 1
B => 2
C => 3
SUMMARY:
========
Test functions:
Executed: 1
Skipped: 0
Tested 1 assertion.
Passed: 0/1 ( 0.0%)
Failed: 1/1 (100.0%)
Errors: 0/1 ( 0.0%)
Interactive tests
The functions RUN-SUITE and RUN-TEST accept another keyword argument called :use-debugger. If this argument is set to true, the unit test will fall into the debugger if an assesrtion fails.
(run-suite 'NumberSuite :report-progress nil :use-debugger t)
Debugger invoked on a CLUNIT::ASSERTION-CONDITION:
TEST-INT1: Expression: (= 1 -1)
Expected: T
Returned: NIL
Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL.
restarts (invokable by number or by possibly-abbreviated name):
0: [CONTINUE ] Continue with TEST-INT1 test.
1: [CONTINUE-WITHOUT-DEBUGGING] Continue unit test without interactive debugging.
2: [SKIP-ASSERTION ] Skip assertion in test TEST-INT1.
3: Continue unit test without interactive debugging.
4: [SKIP-TEST ] Skip test TEST-INT1.
5: [SKIP-SUITE ] Skip test suite INTEGERSUITE.
6: Skip test suite NUMBERSUITE.
7: [CANCEL-UNIT-TEST ] Cancel unit test execution.
8: [ABORT ] Exit debugger, returning to top level.
Defining and using fixtures
Fixtures are used in unit tests to setup a consistent context/enviroment in which tests can be executed so that results are reproducible. CLUnit allows you to define fixtures for each test suite.
The form (deffixture suite (plug) . body) defines a code template that is wrapped around the code of each test case and test suite that are executed by that test suite at runtime. The test case body is plugged into the template at the position identified by PLUG. Fixtures are expanded at runtime, so the fixture that will wrap around a test depends on the test suite call stack.
;; Fixture Definitions
(deffixture IntegerSuite (@body)
(let ((x 0) (y 1) (z 2))
@body))
(deffixture FloatSuite (@body)
(let ((x 0.0) (y 1.0) (z 2.0))
@body)))
;; Test Case Definition
(deftest test-bool1 (BooleanSuite)
(assert-true (< x y z))
(assert-true (= x y z) x y z))
(run-suite 'NumberSuite)
PROGRESS:
=========
NUMBERSUITE: (Test Suite)
INTEGERSUITE: (Test Suite)
TEST-INT1: F.
BOOLEANSUITE: (Test Suite)
TEST-BOOL1: .F
FLOATSUITE: (Test Suite)
TEST-FLOAT1: F.
BOOLEANSUITE: (Test Suite)
TEST-BOOL1: .F
FAILURE DETAILS:
================
NUMBERSUITE -> FLOATSUITE -> BOOLEANSUITE: (Test Suite)
TEST-BOOL1: Expression: (= X Y Z)
Expected: T
Returned: NIL
X => 0.0
Y => 1.0
Z => 2.0
This test is meant to fail.
NUMBERSUITE -> FLOATSUITE: (Test Suite)
TEST-FLOAT1: Expression: (= 1.0 -1.0)
Expected: T
Returned: NIL
NUMBERSUITE -> INTEGERSUITE -> BOOLEANSUITE: (Test Suite)
TEST-BOOL1: Expression: (= X Y Z)
Expected: T
Returned: NIL
X => 0
Y => 1
Z => 2
This test is meant to fail.
NUMBERSUITE -> INTEGERSUITE: (Test Suite)
TEST-INT1: Expression: (= 1 -1)
Expected: T
Returned: NIL
SUMMARY:
========
Test functions:
Executed: 4
Skipped: 0
Tested 8 assertions.
Passed: 4/8 ( 50.0%)
Failed: 4/8 ( 50.0%)
Errors: 0/8 ( 0.0%)
Short circuiting a unit test
CLUnit provides an easy way of short circuiting a unit test. The functions RUN-SUITE and RUN-TEST accept another keyword argument called :stop-on-fail. If this argument is set to true, the rest of the unit test is cancelled when any assertion fails or an error occurs.
(run-suite 'NumberSuite :stop-on-fail t)
PROGRESS:
=========
NUMBERSUITE: (Test Suite)
INTEGERSUITE: (Test Suite)
TEST-INT1: F
FAILURE DETAILS:
================
NUMBERSUITE -> INTEGERSUITE: (Test Suite)
TEST-INT1: Expression: (= 1 -1)
Expected: T
Returned: NIL
SUMMARY:
========
Test functions:
Executed: 1
Skipped: 0
Tested 1 assertion.
Passed: 0/1 ( 0.0%)
Failed: 1/1 (100.0%)
Errors: 0/1 ( 0.0%)
Test dependency
Since the execution of tests is unordered, what we need is a way of specifying that one test depends on another test passing. CLUnit provides a way of doing this via the DEFTEST macro.
;; Define a test case not associated with any test suite
;; and with no dependencies.
(deftest name () . body)
;; Define a test case associated with test suites: suite1 ... suiteN.
(deftest name (suite1 suite2 ... suiteN) . body)
;; Define a test case associated with test suites: suite1 ... suiteN
;; that depends on tests: test1 ... testN.
(deftest name ((suite1 suite2 ... suiteN) (test1 test2 ... testN)) . body)
A test case with dependencies will be queued until all the test cases it depends on have been run. If all the test cases pass the queued test is executed otherwise its skipped.
(defsuite arithmetic ())
(defsuite arithmetic-* (arithmetic))
(defsuite arithmetic-+ (arithmetic))
(deftest test-mult0 (arithmetic-*)
(assert-condition arithmetic-error (/ 1 0)))
;; TEST-MULT1 depends on TEST-MULT0.
(deftest test-mult1 ((arithmetic-*) (test-mult0))
(assert-fail "This is a forced fail by ~S." :derp))
;; TEST-ADD1 depends on TEST-MULT1.
(deftest test-add1 ((arithmetic-+) (test-mult1))
(assert-true t))
(run-suite 'arithmetic)
PROGRESS:
=========
ARITHMETIC: (Test Suite)
ARITHMETIC-+: (Test Suite)
TEST-ADD1: [QUEUED]
ARITHMETIC-*: (Test Suite)
TEST-MULT1: [QUEUED]
TEST-MULT0: ..
QUEUED TESTS:
=============
TEST-MULT1: F
TEST-ADD1: [SKIPPED]
FAILURE DETAILS:
================
ARITHMETIC -> ARITHMETIC-*: (Test Suite)
TEST-MULT1: This is a forced fail by :DERP.
SUMMARY:
========
Test functions:
Executed: 2
Skipped: 1
Tested 3 assertions.
Passed: 2/3 ( 66.7%)
Failed: 1/3 ( 33.3%)
Errors: 0/3 ( 0.0%)