# Doctests as test-suite without \`Custom\` builds

**URL:** <https://discourse.haskell.org/t/doctests-as-test-suite-without-custom-builds/5771>\
**Category:** Learn\
**Created:** [February 9, 2023, 12:55pm UTC](https://discourse.haskell.org/t/doctests-as-test-suite-without-custom-builds/5771 "2023-02-09T12:55:13Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![NicolasT](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/nicolast/32/2872_2.png) [@NicolasT](https://discourse.haskell.org/u/NicolasT)\
**Post date:** [February 9, 2023, 12:55pm UTC](https://discourse.haskell.org/t/doctests-as-test-suite-without-custom-builds/5771/1 "2023-02-09T12:55:13Z")

</div>

Having examples in documentation is useful, and checking whether examples actually work, is useful as well. The `doctest` package provides a way to achieve this, but it works by running a command on a source-tree, and hence can’t be easily integrated with a Cabal project as a test-suite, including a way to specify the dependencies doctests may have.

The `cabal-doctest` project provides a way to achieve this, by generating a module at build-time which contains meta-information required to run `doctest` as a test-suite, passing in all necessary GHC flags. However, it works by hooking into Cabal as a `Custom` build-type in `Setup.hs`.

This works fine, but has one major drawback: some Cabal features don’t work with `Custom` builds, e.g., the use of internal libraries is only supported with per-component builds (I remember Backpack to suffer the same fate).

Is there a way to have a `doctest` suite to run as a Cabal test-suite, without relying on `Custom` builds and hence keeping it possible to use internal libraries?

---

<div class="post-metadata">

**Author:** ![ChShersh](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/chshersh/32/2551_2.png) [@ChShersh](https://discourse.haskell.org/u/ChShersh)\
**Post date:** [February 9, 2023, 1:27pm UTC](https://discourse.haskell.org/t/doctests-as-test-suite-without-custom-builds/5771/2 "2023-02-09T13:27:46Z")

</div>

It’s absolutely possible to run `doctest` as a part of a test-suite. I’ve been doing this for years.

You can define a separate `test` stanza for `doctest` tests and then call `doctest` manually with the list of modules and extensions needed. A simple example from my Haskell course:

- [exercises/exercises.cabal at 1f44e5f5f2f77a17e8fe4c62500702368de5830e · haskell-beginners-2022/exercises · GitHub](https://github.com/haskell-beginners-2022/exercises/blob/1f44e5f5f2f77a17e8fe4c62500702368de5830e/exercises.cabal#L103-L106)
- [exercises/DoctestLecture1.hs at 1f44e5f5f2f77a17e8fe4c62500702368de5830e · haskell-beginners-2022/exercises · GitHub](https://github.com/haskell-beginners-2022/exercises/blob/1f44e5f5f2f77a17e8fe4c62500702368de5830e/test/DoctestLecture1.hs)

This approach has two downsides:

1. You need to pass all modules explicitly. This can be easily solved by using the [Glob](https://hackage.haskell.org/package/Glob) package in big Haskell projects.
2. You need to pass all extensions explicitly. Again, usually not a problem. Worst case, it’s 10 lines of some common extensions you use, that you write once and forget about `doctest` settings almost forever.

---

<div class="post-metadata">

**Author:** ![nomeata](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/nomeata/32/1188_2.png) [@nomeata](https://discourse.haskell.org/u/nomeata)\
**Post date:** [February 9, 2023, 2:59pm UTC](https://discourse.haskell.org/t/doctests-as-test-suite-without-custom-builds/5771/3 "2023-02-09T14:59:40Z")

</div>

Right, that’s also what I am using in one of my libraries:

> <https://github.com/nomeata/haskell-candid/blob/6d60e4464bd4a206d91a6cdade8810a37b09c6d0/candid.cabal#L117-L128>

> <https://github.com/nomeata/haskell-candid/blob/master/doctests.hs>

To make it work well on CI I found it helps to run `cabal build` first to write a `.ghc.environment` file and then `cabal test` works:

> <https://github.com/nomeata/haskell-candid/blob/6d60e4464bd4a206d91a6cdade8810a37b09c6d0/.github/workflows/test.yml#L59-L69>

---

<div class="post-metadata">

**Author:** ![amesgen](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/amesgen/32/2083_2.png) [@amesgen](https://discourse.haskell.org/u/amesgen)\
**Post date:** [February 9, 2023, 4:19pm UTC](https://discourse.haskell.org/t/doctests-as-test-suite-without-custom-builds/5771/4 "2023-02-09T16:19:14Z")

</div>

There is also [cabal-docspec](https://github.com/phadej/cabal-extras/blob/master/cabal-docspec/MANUAL.md) (also has [precompiled binaries](https://github.com/phadej/cabal-extras/releases)) which has been working great for me; you need neither `build-type: Custom` nor GHC environment files.

One usually runs it as a standalone binary; but I guess one also could run it in a test suite via `process` (although I haven’t tried).

---

<div class="post-metadata">

**Author:** ![NicolasT](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/nicolast/32/2872_2.png) [@NicolasT](https://discourse.haskell.org/u/NicolasT)\
**Post date:** [February 9, 2023, 5:40pm UTC](https://discourse.haskell.org/t/doctests-as-test-suite-without-custom-builds/5771/5 "2023-02-09T17:40:57Z")

</div>

There’s no release of `cabal-docspec` on Hackage, so it’s a bit difficult to make things “just work”. Also, shelling out a test-runner feels very non-Haskell’y.

---

<div class="post-metadata">

**Author:** ![NicolasT](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/nicolast/32/2872_2.png) [@NicolasT](https://discourse.haskell.org/u/NicolasT)\
**Post date:** [February 9, 2023, 5:49pm UTC](https://discourse.haskell.org/t/doctests-as-test-suite-without-custom-builds/5771/6 "2023-02-09T17:49:59Z")

</div>

I did exactly this in the past, but ran into several issues which make this approach basically not work:

- Whenever a module under test imports some module for a package which is not “globally” installed, but is (of course) listed as a `build-depends` of the code under test, the import fails. What `cabal-doctest` does (among other things) is pass provide all flags needed to pass to `doctest` (the function) to be able to find such packages.
- When using `cabal-doctest`, functions that aren’t exported but do have some doctest attached to them are tested. With a plain `doctest` (the function) invocation, I end up with `Variable not in scope` errors.

I seem to remember some issues with `QuickCheck`/`prop>` tests as well in the past, with `QuickCheck` not being found (since, indeed, not globally installed).

---

<div class="post-metadata">

**Author:** ![NicolasT](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/nicolast/32/2872_2.png) [@NicolasT](https://discourse.haskell.org/u/NicolasT)\
**Post date:** [February 9, 2023, 6:25pm UTC](https://discourse.haskell.org/t/doctests-as-test-suite-without-custom-builds/5771/7 "2023-02-09T18:25:13Z")

</div>

Thanks! Indeed, when instructing Cabal to create a `.ghc.environment.*` file, a call to plain `Test.DocTest.doctest` in a test-suite seems to work. However, since this depends on a setting in `cabal.project` (or similar), running `cabal test` on a package downloaded from Hackage (which doesn’t include `cabal.project`) would fail, right? Similarly, the Hackage builder would fail as well (unless that one is configured to always write environment files).

---

<div class="post-metadata">

**Author:** ![tristanC](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/tristanc/32/827_2.png) [@tristanC](https://discourse.haskell.org/u/tristanC)\
**Post date:** [February 9, 2023, 6:41pm UTC](https://discourse.haskell.org/t/doctests-as-test-suite-without-custom-builds/5771/8 "2023-02-09T18:41:37Z")

</div>

For what it’s worth, you can also run doctests with `cabal repl --with-ghc=doctest`. That seems to take care of setting the packages environment and enabling the language extensions.

---

<div class="post-metadata">

**Author:** ![NicolasT](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/nicolast/32/2872_2.png) [@NicolasT](https://discourse.haskell.org/u/NicolasT)\
**Post date:** [February 9, 2023, 7:05pm UTC](https://discourse.haskell.org/t/doctests-as-test-suite-without-custom-builds/5771/9 "2023-02-09T19:05:30Z")

</div>

Indeed. However, my end goal is for `cabal test` to “just work” and run all doctests, so using `cabal repl` won’t work.

---

<div class="post-metadata">

**Author:** ![sclv](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/sclv/32/71_2.png) [@sclv](https://discourse.haskell.org/u/sclv)\
**Post date:** [February 9, 2023, 7:54pm UTC](https://discourse.haskell.org/t/doctests-as-test-suite-without-custom-builds/5771/10 "2023-02-09T19:54:38Z")

</div>

The Cabal 3.8 release added the `code-generators` field: [cabal/Cabal-3.8.1.0.md at master · haskell/cabal · GitHub](https://github.com/haskell/cabal/blob/master/release-notes/Cabal-3.8.1.0.md)

This field should allow doctest runners and test-discovery tools to run using standard cabal test commands making use of this stanza, rather than needing custom builds. However, the existing tools will need to release updates to make use of this stanza! I would encourage PRs to cabal-doctest and other similar packages to make use of this functionality.

---

<div class="post-metadata">

**Author:** ![ChShersh](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/chshersh/32/2551_2.png) [@ChShersh](https://discourse.haskell.org/u/ChShersh)\
**Post date:** [February 10, 2023, 9:55am UTC](https://discourse.haskell.org/t/doctests-as-test-suite-without-custom-builds/5771/11 "2023-02-10T09:55:25Z")

</div>

I have the following alias for `cabal build` to always build the package locally with the environment files:

```bash
alias cbuild="cabal build -O0 --enable-tests --enable-benchmarks --write-ghc-environment-files=always"

```

And the following alias for `cabal test` to build with nice coloured output (because cabal defaults are not great):

```bash
alias ctest="cabal test -O0 --enable-tests --test-show-details=direct"

```

I don’t have your use case with the need to download packages from Hackage and running tests locally for them but I imagine the above two commands would work as well without the need to create `cabal.project` files.

Similarly, I have the following _configure_ step on CI to make sure that the environment files are generated so that `doctest` can run smoothly

> <https://github.com/chshersh/iris/blob/588d81cf317d84558f18b0d63150c2c7ee8b7a80/.github/workflows/ci.yml#L51-L53>

* * *

Thus being said, I personally think that the entire story with these environment files is massively overcomplicated and this is one of the examples why Haskell tooling is so bad.

Doctest is such a wonderful tool, it should be supported by Haskell tooling natively, it should work out-of-the box and you shouldn’t need to do all these dances to make it work correctly.

---

<div class="post-metadata">

**Author:** ![sclv](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/sclv/32/71_2.png) [@sclv](https://discourse.haskell.org/u/sclv)\
**Post date:** [February 10, 2023, 5:50pm UTC](https://discourse.haskell.org/t/doctests-as-test-suite-without-custom-builds/5771/12 "2023-02-10T17:50:14Z")

</div>

> [@ChShersh](#):
>
> Doctest is such a wonderful tool, it should be supported by Haskell tooling natively, it should work out-of-the box and you shouldn’t need to do all these dances to make it work correctly.

And if you see the post right above yours, cabal provides a feature now for exactly that! The tooling just needs to be updated to take advantage of it.

---

<div class="post-metadata">

**Author:** ![cdepillabout](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/cdepillabout/32/1168_2.png) [@cdepillabout](https://discourse.haskell.org/u/cdepillabout)\
**Post date:** [February 11, 2023, 12:35am UTC](https://discourse.haskell.org/t/doctests-as-test-suite-without-custom-builds/5771/13 "2023-02-11T00:35:34Z")

</div>

I’m also a big fan of doctests and have been following the (somewhat bleak?) doctest situation in Haskell for a while now.

To give a little historical context (as I remember it), `doctest` was created quite a long time ago by @sol. It worked well for a lot of projects, but as @ChShersh says, it can be quite finicky when you have big, complicated projects. The various workarounds described above often fail in various ways depending on whether you’re trying to run doctests in `stack`, `cabal-v1`, `cabal-v2`, the Nixpkgs Haskell builder, `haskell.nix`, etc.

To fix the various problems, `cabal-doctest` was created by @phadej. It hooks into Cabal’s machinery in order to “just work” in a lot of cases. The big problem with it is that it uses `Setup.hs` to bootstrap itself. This doesn’t work well when cross-compiling Haskell. There are quite a lot of people against using `Setup.hs` for anything.

There have been a bunch of other solutions that have come out after `cabal-doctest`. I have an issue on one of my repos where I have been tracking various doctest-related solutions, and the pros and cons of each. Maybe it will give some other ideas:

> <https://github.com/cdepillabout/pretty-simple/issues/82>
>
> Currently, the custom Setup.hs used for doctests causes cross-compilation to fai…l due to a \[loingstanding cabal bug\](https://github.com/haskell/cabal/issues/1493).
> 
> I set \`write-ghc-environment-files\` to \`always\` in my \[\`cabal.project\`\](https://hub.darcs.net/vmchale/recursion/browse/cabal.project#3) and then just run \`doctest\` from the command-line as a workaround.

---

<div class="post-metadata">

**Author:** ![NicolasT](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/nicolast/32/2872_2.png) [@NicolasT](https://discourse.haskell.org/u/NicolasT)\
**Post date:** [February 11, 2023, 10:50pm UTC](https://discourse.haskell.org/t/doctests-as-test-suite-without-custom-builds/5771/14 "2023-02-11T22:50:26Z")

</div>

Yeah, I found out about this during my research on this subject. I might give it a stab, though it’s not high priority at the moment.

---

<div class="post-metadata">

**Author:** ![wolverian](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/wolverian/32/1387_2.png) [@wolverian](https://discourse.haskell.org/u/wolverian)\
**Post date:** [December 28, 2025, 7:40pm UTC](https://discourse.haskell.org/t/doctests-as-test-suite-without-custom-builds/5771/15 "2025-12-28T19:40:44Z")

</div>

Sorry for resurrecting such an old thread, but I was wondering if any progress has happened here for integrating doctests with Cabal using the `code-generators` field?