Best way to run tests on file save

Learn
1

TL;DR: what is the best way to run tests on every file save?

On the project we have at work, I’d like to run the test suite at every file save. At the moment I’m doing it using stack test --file-watch. This works but it adds some seconds of overhead at every file save. I mean that when I save a file, stack is notified immediately, but the test execution starts 3/4 seconds after. This can be observed also if I run stack test without anything to compile. The whole command takes something around 6 seconds, while the test suite execution time is around 2 seconds.

I tried with cabal and I noticed the same behavior: running cabal test without anything to compile also takes around 6 seconds, while the test suite takes 2 seconds.

For context, the codebase includes 3 packages (a main one depending on two small ones), plus a test suite and some executables. It depends on several packages on Stackage/Hackage plus some other public and private projects. We are currently using GHC 9.10.3, stack 3.9.1, cabal 3.16.1.0.

Paying that ~4 seconds price at every file save is a bit of a burden and I would like to find a better way. Is there some cabal/stack option to avoid all the operations which are performed before the test execution? Is there some other tool that I could use instead of stack or cabal to run my test suite

2

People usually use ghci for this.

ghcid (note the ‘d’) has some flags for rerunning tests. Assuming your tests don’t need to run with optimisations.

I use ghci in emacs with a more generalised key shortcut that is “switch to whatever terminal buffer is on the screen, run Ctrl-C interrupt, resend the previous prompt” — similar flow to ghcid.

3

ghcid is indeed a good way of doing this; the --test argument can be used to run arbitrary GHCi commands after (re-)loading.

The way I like to structure things:

  • All the “real” code lives in a library (myproject:lib:myproject)
  • The “application” (myproject:exe:myproject) is just a thin wrapper that says something like main = MyProject.Main.main
  • The testsuite has its own main
  • An “outer” ghcid job loads and watches the libary part, reloading as usual, and restarting (--restart) whenever any of the files change that ghci itself doesn’t watch (myproject.cabal, cabal.project, etc.).
  • The “outer” ghcid then runs (via --test) an “inner” ghcid that loads and watches the testsuite: --test ':! ghcid --test Main.main -c cabal repl myproject:test:myproject-tests'

Put together, it looks something like this:

#!/bin/sh
ghcid \
    --restart myproject.cabal \
    --restart cabal.project \
    --test ':! ghcid --test "Main.main" -c cabal repl myproject:test:myproject-tests' \
    -c 'cabal repl myproject:lib:myproject'

Depending on which files you change, this will (usually) do the right thing:

  • If any of your test source files change, it will reload the testsuite (but not the library) and re-run tests. This is very fast.
  • If any of the library sources change, it will reload the library, restart the inner ghci, load the testsuite, and re-run tests. This is still pretty fast: the testsuite will be restarted, but the library is reloaded, and typically none of the testsuite itself has changed, so the restart can use cached build artifacts.
  • If cabal.project or myproject.cabal changes, it restarts everything, loads the library, then the testsuite, then runs the tests. This will take a bit longer (similar to plain cabal test), but since you don’t change those files frequently, that’s OK (and you can’t really avoid it anyway).

To restrict which tests are run, I use environment variables - I usually use tasty, which looks at environment variables such as TASTY_PATTERN to determine which tests to run. You can also use environment variables to change other testing options, such as the size and number of quickcheck tests, shrinking steps, success/failure reporting, etc.

With the above setup, this requires restarting the ghcid job to pick up those changed env vars; you can expand the pattern to use an env file, restarting the outer ghcid when that file changes, and re-sourcing the env file before restarting the inner ghcid, but I usually don’t bother, because I don’t really change those test patterns a lot.

4

I think it would be incredibly useful if something like your approach with the double ghcid could be added to ghcid itself! In particular since the approach that just defines main = Library.main is so widespread (and what I would argue best practice). It would then be nice if the tooling had good/out-of-the-box support for such best practices (rather than having to jump trough some hoops).

5

I think that cabal’s multirepl support (–enable-multi-repl) already lets you load the test and library components at the same time. We are living in the future.

Well, ideally it would be the past. But at least it’s… present.

6

Looks like you’re right. Without nesting ghcid jobs, we can rewrite the above script like this:

#!/bin/sh
ghcid \
    --restart myproject.cabal \
    -c 'cabal --enable-multi-repl repl myproject:test:myproject-test myproject:lib:myproject'

I’ve just tested this, and it seems to reload on changes to source files from either component (library or testsuite), as you’d expect.

7

Thanks all, this looks very nice!

For the moment I’m trying with

ghcid \
    --restart myproject.cabal \
    -c 'cabal --enable-multi-repl repl blue-moon:test:test blue-moon:lib:blue-moon --ghc-options="-threaded -rtsopts -with-rtsopts=-N -j8"' \
    --test "Main.main"

I have two questions:

  • when I launch the process, every file of the project is compiled, even if there is nothing to be recompiled
[  1 of 980] Compiling API.Common       ( /home/marcosh/projects/livtours/bm-backend/src/API/Common.hs, interpreted )[blue-moon-0.0.0-inplace]
  • when the tests run, only one thread is used
WARNING: Only one CPU core detected, make sure to compile your test suite with these ghc options:                     
         -threaded -rtsopts -with-rtsopts=-N               
         (This is important for correctness as well as speed, as a parallel test suite can find thread safety problems.)

Is there a way to tell ghcid to run tests on multiple threads?

8

Coming soon:TM: : Introduce a serialisable bytecode format and corresponding “bytecode” way

I hope somebody else can figure out an answer about multithreading.