Testing Conventions

If you haven't done any testing in Clojure yet it's useful to know that the default location for tests is <project>/test/<namespace>/namespace_file>_test.clj. If you're using Lein it will automatically create a testing file and namespace for you in this location.

Lets says that we had a project called TurtleGame, we'd have the following:

• TurtleGame/src/TurtleGame/core.clj
• TurtleGame/test/TurtleGame/core-test.clj

Installing Expectations

Expectations is a testing library that makes it easier to build tests. It provides a simpler format for tests and provides a set of assertions that make it easier to define tests. There's a stand alone version, but I'm using expectations/clojure-test . This is compatible with normal clojure.test so you can use other tools that think you're using the standard testing system (e.g. a test runner).

Add the following to your projects project.clj:

:dependencies [ ;; your other dependencies
                [expectations/clojure-test "1.2.1"]
                [pjstadig/humane-test-output  "0.10.0"]
              ]
:injections   [ ;; any other injections
                (require 'pjstadig.humane-test-output)
                (pjstadig.humane-test-output/activate!)
              ]

In the project's testing namespace (e.g. <project directory>/test/<project namespace>/core_test.clj) add the following:

(ns someproject.core-test
  (:require
            [clojure.test :refer :all]
            [expectations.clojure.test :refer [defexpect expect expecting more more-> more-of]]
            [someproject.core :refer :all]))

;; tests go here

Humane test output

The output from Clojure.test can be pretty indistinct so there's a few libraries out there to help improve the output. The first of these is humane-test-output.

We added it to the project.clj higher up. Technically if you just want to use expectations/clojure-test from the command line (e.g. lein test or bat-test) then you just need to install humane-test-output you don't need to inject it as we did in the project.clj. That's because expectations/clojure-test will find it on the classpath for you and will use it. However, I also run my tests from within the REPL and want the output to be the same as when I run lein bat-test on the commandline.

The impact is that it changes a simple test like this:

(deftest addition-test
    (is (= 2 (+ 2 2)) "Should be equal to 4"))

;; standard clojure.test output
Should be equal to 4
expected: 2
  actual: [4]

;; using humane test output you get a diff
Should be equal to 4
expected: 2
  actual: 4
      diff: - 2
            + 4

Bat-test and Pretty test runner

By default Lein is able to run your tests on the commandline but the stacktraces on failures are pretty ugly, it don't support colour and there's no option to continuously run the tests.

bat-test is a test runner that creates a 'good test runner set-up' by using a set of other libraries. It does what I want it to do, and you don't have to configure anything - it's weakness is that there isn't really any documentation. Today I would probably have a look at Kaocha which has received a lot of interest.

Bat-test was the most fully-features and easiest to get going as it includes pretty, eftest and cloverage - this is worth knowing so you don't also set these up.

Pretty is a library to improve how stacktraces show information and it provides colour. Eftest is a test runner, bat-test takes advantage of it's capabilities. And, cloverage tells you about test coverage.

As I use bat-test for every project I have it installed in my ~/lein/profiles.clj but it can also go into the project specific project.clj

:plugins [ ;; your other plugins
           [metosin/bat-test "0.4.4"]
           [io.aviso/pretty  "0.1.37"]
         ]
:middleware [ ;; your lein middleware
              io.aviso.lein-pretty/inject
            ]
:dependencies [ ;; your other dependencies
                [io.aviso/pretty "0.1.37"]
              ]
:injections [ ;; any injections you have
              (require 'io.aviso.repl)
            ]

Most of these settings are for Pretty which is a library to make tracebacks more usable on Clojure. Have a look at my post on Lein development experience for more on this set-up.

Running tests

There are two ways to run tests, from the command line and within the REPL. From the command line you can do:

lein bat-test once

lein bat-test :only <project-namespace>.core-test/name-of-test

lein bat-test auto

In the first two you're running tests in the same way that lein test works, but the last option auto lets you continuously run the tests. Commonly, I have a tmux split and have my tests running automatically so I can see them all the time.

In the REPL you can just import the namespace and then run a specific test

;; in vim I do :%Eval then switch to the repl and run
(someproject.core-test/name-of-test)

If you want to run all your test continuously in the REPL then you'll need to use Eftest fully, see the Github page for more.

Writing tests with Expectations

With all the prerequisites set-up we can start writing some tests. Expectations simplifies the syntax for writing equality tests. The simplest way is:

(defexpect basic-equality
    (expect 1 (+ 1 1) "Numbers weren't equal"))

The defexpect is just like a defn we provide a name for the set of tests. Individual assertions are written with the expect line, the assertion we're making and some text that will be printed as part of the failure. When this triggers I get:

FAIL in someproject.core-test/basic-equality (core_test.clj:10)
Numbers weren't equal
expected: 1
actual: [2]

15/15   100% [==================================================]  ETA: 00:00

Ran 15 tests in 0.064 seconds
50 assertions, 1 failure, 0 errors.

In the example above the plus + function is being called, but normally you'd call your own function. For example:

(defexpect fn-adder-basic-test
    (expect int? (someproject.core/fn-adder) "Expecting an integer to be returned")
    (expect 2 (someproject.core/fn-adder 1 1) "Expecting adding 1 and 1 to be 2"))

We'll come back to using a predicate test like int? later, for now lets look at how we can test the other types.

Aside from numbers you can also test strings for equality:

(defexpect string-user-name-test
    (expect "John Smith" (str "John" "Williams")))

The output I get is:

FAIL in passtui.core-test/string-user-name-test (core_test.clj:11)

expected: "John Smith"
  actual: ["John Williams"]

Equality in collections

Expectations (and by extension expectations/clojure.test) has fantastic support for testing the equality of lists, maps and vectors. Here a simple example:

(defexpect simple-vector-test
    (expect [1 2] (vector 1 2) "Expecting return of a vector containing 1 and 2")
    (expect ["Jack" "Leo"] (vector "Jack" "Jill")))

;; output from lein bat-test or REPL:
;; expected: ["Jack" "Leo"]
;;   actual: [["Jack" "Jill"]]

We can do the same thing in sets and maps:

(expect      {:number 2 :name "Jack" :species :cat}
    (assoc {} :number 1 :name "Jack" :species :cat))

 ;; expected: {:number 2, :name "Jack", :species :cat}
 ;;  actual: {:number 1, :name "Jack", :species :cat}
 ;;    diff: - {:number 2}
 ;;          + {:number 1}

Finally, the most complex situation is a nested collection:

; a complex vector with a nested map of data
(expect [{:name "Jack" :species "cat" :colour "ginger" :age 8}
         {:name "Leo" :species "lion" :colour "brown" :age 4}]
        (vector {:name "Jack" :species "cat" :colour "ginger" :age 8}
                {:name "Leo" :species "lion" :colour "brown" :age 4}))

;; the output from either lein test or in the REPL is:
;; expected: [{:name "Jack", :species "cat", :colour "ginger", :age 8}
;;            {:name "Leo", :species "lion", :colour "brown", :age 4}]
;;   actual: [{:name "Jack", :species "cat", :colour "ginger", :age 8}
;;            {:name "Leo", :species "lama", :colour "brown", :age 4}]
;;     diff: - [nil {:species "lion"}]
;;           + [nil {:species "lama"}]

It's also possible to look for a specific value in a collection:

;; expect a key and value {:foo 1} in a map
(expect {:foo 1} (in {:foo 1 :cat 4}))  ;;=> success

Assertions using predicates

Aside from equality you can also assert that the returned value will pass any predicate test. For example:

(expect even? (+ 1 2) "Returned number should be even")

Here were testing that the number that's returned from the + function will be even.

Testing other than equality

I found it a bit confusing when you want to test something where the natural outcome is not equality. For a simple example, lets say we want to test that one number is smaller than another:

; initially tried this - but it won't work
(defexpect digit-test
    (expect >= 5 (+ 1 9)))

(defexpect digit-test
    (expect true (>= 5 (+ 1 9)) "Expected 5 to be larger number"))


;; FAIL in (digit-test) (form-init16529601607872859898.clj:2)
;; Expected 5 to be larger number
;; expected: true
;; actual: false
;; diff: - true

As expect is testing for true the way to do this is to wrap it so that you return true: in this case we've done the equality test in another step. However, as you can see it's not really giving you useful output.

The other option is to use the standard clojure test version:

(defexpect digi-test
    (is (>= 5 (+ 1 9)) "Expecting 5 to be larger number"))

;; FAIL in (digi-test1) (form-init16529601607872859898.clj:2)
;; Expecting 5 to be larger number
;; expected: (>= 5 (+ 1 9))
;; actual: (not (>= 5 10))

The clojure.test version gives slightly better output for me so I've been using that. It's one of the nice parts of expectations/clojure-test is that you can use both types.

I've missed out a few different ways of testing including function equality, testing types, regular expressions, exceptions and spec. These are all covered in the docs and in the blog posts.

More, More-> and More-of Testing

Lets say that you want to call your function and then assert some things about the value that it returns, you can group these together using the various more options. The simplest one is more :

(expect
    (more
        vector?
        not-empty) [1 2])

In this example we test whether the vector ([1 2]) is a vector and whether it is empty: as it is both of these the test passes as true.

The more-> is a threading option:

(expect (more-> 1 first
                3 last) [1 2 3])

The most useful one is more-of which lets you test the values of an item:

(expect (more-of returnval
            string? returnval
            empty? returnval
        ) (myproject/checker-fn 4)
        "Requested zero length, with options set, expecting empty string back"
)

In this example we're calling a function myproject/checker-fn with 4 as an argument, when it returns we use the returnval and we check if it is a string and empty. I find this structure easy to understand and it makes it simple to test a variety of attributes.

It also supports destructuring so you can do:

(expect (more-of [x :as all]
            vector?
            1 x) [1 2 3])

In this case we're sending the vector [1 2 3] and then we're testing the full vector and the first element.

From-each

We often want to test each value in a collection of values. The from-each macro lets you do this:

(expect string?
    (from-each [letter ["a" "b" 1]]
        letter
    )
)

;; FAIL
;; expected: (=? string? letter)
;; actual: (not (string? 1))

This is the simplest form where we provide a collect (["a" "b" 1]) and we destructure with letter. Then we check to see if each one is a string. It has the same options as doseq with :let and :when - so far I haven't needed these.

Resources

• Expectations

  A minimalist's unit testing framework.

• Clojure-expectations/clojure-test

  A version of the Expectations test library that is compatible with clojure.test. This means you can use all the tools that are compatible with clojure.test

• Jake McCrary - Testing Category

  Jake wrote lein-test-refresh which I don't use, but there's lots of great information in these articles.

• Example-based Unit Testing in Clojure

  Covers what to test, rather than how to test.

Summary

For me expectations/clojure-test is a great mix of working with the standard Clojure test runners so it's easy to fit into a workflow, while having the advantages of expectations simpler way of defining tests. So far it's working well for me. If you have thoughts on alternative libraries or runners please leave a comment!