Doctests as test-suite without `Custom` builds

Learn
#1

Having examples in documentation is useful, and checking whether examples actually work, is useful as well. The doctest package provides a way to achieve this, but it works by running a command on a source-tree, and hence can’t be easily integrated with a Cabal project as a test-suite, including a way to specify the dependencies doctests may have.

The cabal-doctest project provides a way to achieve this, by generating a module at build-time which contains meta-information required to run doctest as a test-suite, passing in all necessary GHC flags. However, it works by hooking into Cabal as a Custom build-type in Setup.hs.

This works fine, but has one major drawback: some Cabal features don’t work with Custom builds, e.g., the use of internal libraries is only supported with per-component builds (I remember Backpack to suffer the same fate).

Is there a way to have a doctest suite to run as a Cabal test-suite, without relying on Custom builds and hence keeping it possible to use internal libraries?

1 Like
#2

It’s absolutely possible to run doctest as a part of a test-suite. I’ve been doing this for years.

You can define a separate test stanza for doctest tests and then call doctest manually with the list of modules and extensions needed. A simple example from my Haskell course:

This approach has two downsides:

  1. You need to pass all modules explicitly. This can be easily solved by using the Glob package in big Haskell projects.
  2. You need to pass all extensions explicitly. Again, usually not a problem. Worst case, it’s 10 lines of some common extensions you use, that you write once and forget about doctest settings almost forever.
3 Likes
#3

Right, that’s also what I am using in one of my libraries:

To make it work well on CI I found it helps to run cabal build first to write a .ghc.environment file and then cabal test works:

#4

There is also cabal-docspec (also has precompiled binaries) which has been working great for me; you need neither build-type: Custom nor GHC environment files.

One usually runs it as a standalone binary; but I guess one also could run it in a test suite via process (although I haven’t tried).

#5

There’s no release of cabal-docspec on Hackage, so it’s a bit difficult to make things “just work”. Also, shelling out a test-runner feels very non-Haskell’y.

1 Like
#6

I did exactly this in the past, but ran into several issues which make this approach basically not work:

  • Whenever a module under test imports some module for a package which is not “globally” installed, but is (of course) listed as a build-depends of the code under test, the import fails. What cabal-doctest does (among other things) is pass provide all flags needed to pass to doctest (the function) to be able to find such packages.
  • When using cabal-doctest, functions that aren’t exported but do have some doctest attached to them are tested. With a plain doctest (the function) invocation, I end up with Variable not in scope errors.

I seem to remember some issues with QuickCheck/prop> tests as well in the past, with QuickCheck not being found (since, indeed, not globally installed).

1 Like
#7

Thanks! Indeed, when instructing Cabal to create a .ghc.environment.* file, a call to plain Test.DocTest.doctest in a test-suite seems to work. However, since this depends on a setting in cabal.project (or similar), running cabal test on a package downloaded from Hackage (which doesn’t include cabal.project) would fail, right? Similarly, the Hackage builder would fail as well (unless that one is configured to always write environment files).

#8

For what it’s worth, you can also run doctests with cabal repl --with-ghc=doctest. That seems to take care of setting the packages environment and enabling the language extensions.

1 Like
#9

Indeed. However, my end goal is for cabal test to “just work” and run all doctests, so using cabal repl won’t work.

#10

The Cabal 3.8 release added the code-generators field: cabal/Cabal-3.8.1.0.md at master · haskell/cabal · GitHub

This field should allow doctest runners and test-discovery tools to run using standard cabal test commands making use of this stanza, rather than needing custom builds. However, the existing tools will need to release updates to make use of this stanza! I would encourage PRs to cabal-doctest and other similar packages to make use of this functionality.

2 Likes
#11

I have the following alias for cabal build to always build the package locally with the environment files:

alias cbuild="cabal build -O0 --enable-tests --enable-benchmarks --write-ghc-environment-files=always"

And the following alias for cabal test to build with nice coloured output (because cabal defaults are not great):

alias ctest="cabal test -O0 --enable-tests --test-show-details=direct"

I don’t have your use case with the need to download packages from Hackage and running tests locally for them but I imagine the above two commands would work as well without the need to create cabal.project files.

Similarly, I have the following configure step on CI to make sure that the environment files are generated so that doctest can run smoothly

Thus being said, I personally think that the entire story with these environment files is massively overcomplicated and this is one of the examples why Haskell tooling is so bad.

Doctest is such a wonderful tool, it should be supported by Haskell tooling natively, it should work out-of-the box and you shouldn’t need to do all these dances to make it work correctly.

4 Likes
#12

And if you see the post right above yours, cabal provides a feature now for exactly that! The tooling just needs to be updated to take advantage of it.

1 Like
#13

I’m also a big fan of doctests and have been following the (somewhat bleak?) doctest situation in Haskell for a while now.

To give a little historical context (as I remember it), doctest was created quite a long time ago by @sol. It worked well for a lot of projects, but as @ChShersh says, it can be quite finicky when you have big, complicated projects. The various workarounds described above often fail in various ways depending on whether you’re trying to run doctests in stack, cabal-v1, cabal-v2, the Nixpkgs Haskell builder, haskell.nix, etc.

To fix the various problems, cabal-doctest was created by @phadej. It hooks into Cabal’s machinery in order to “just work” in a lot of cases. The big problem with it is that it uses Setup.hs to bootstrap itself. This doesn’t work well when cross-compiling Haskell. There are quite a lot of people against using Setup.hs for anything.

There have been a bunch of other solutions that have come out after cabal-doctest. I have an issue on one of my repos where I have been tracking various doctest-related solutions, and the pros and cons of each. Maybe it will give some other ideas:

4 Likes
#14

Yeah, I found out about this during my research on this subject. I might give it a stab, though it’s not high priority at the moment.