I have (finally) reached a point where I need to write tests for a small Haskell program I have written, which is being used in anger.
Are there any recommendations on what libraries / tools to use, if you are starting from nothing, today?
I will go searching and looking, but sometimes a(n opinionated) recommendation is a useful starting point.
The Stack project uses hspec and QuickCheck, but that may reflect its history. tasty says it is āmodernā and it also has āprovidersā for Hspec and QuickCheck tests.
doctest is handy. The tasty ecosystem is good. My shelltestrunner might be useful for cli tests. If you get into property testing later, try the variants like hedgehog or smallcheck, not just quickcheck.
I use and love hspec. Well documented, integration with QuickCheck/SmallCheck/HUnit if needed, batteries included.
My go-to toolkit is tasty as the harness, tasty-hunit for unit tests, tasty-golden for golden tests, and tasty-quickcheck for property-testing.
I maintain a list but it still needs fleshing out. I like the tasty testing universe and hedgehog for property testing (because of automatic shrinking and gens-as-values).
What program?
QuickCheck is wildly overrated, unless you write simple primitivesā¦ or have a ton of time writing smart generators for complex data (I did that for something seemingly simple like windows filepaths and it took a lot of time).
If itās cliā¦ ghcup recently got a test suite for the optparse-applicative part: ghcup-hs/test/optparse-test/InstallTest.hs at master Ā· haskell/ghcup-hs Ā· GitHub
That expresses our expectations of cli arguments and parsed types. Is that generally useful? No, unless breaking cli API is a major concern.
Itās a command line tool that looks up users and groups via LDAP (only useful within the company). It has reached the level of usage now that I might break it subtly and nobody will notice until something goes awry 
Nice, thanks for the pointer! That sounds useful for my case as well. If my program for instance starts not exiting with a non-zero return code in circumstances where it used to, itās a concern.
I forgot, if you are writing a CLI program and you need golden tests (so tests where you are comparing program output with a reference output stored in a file), the way to go is shelltestrunner.
Well documented, sensible syntax, output regexps, deals with both standard and error output.
It is a CLI program itself, installable from most Linux distributions rrepos, so it is useful even if you are not developing in Haskell.
Ah, nice - that sounds like it could cover a good deal of the (āintegrationā) tests I want to have, thanks!
I have been trying top get Hspec going, but Iām hitting a circular import where running the tests imports MainSpec which imports Main which imports MainSpec again, so I still need to wrap my head around that part to get other (āunitā) tests going.
You should take advantage of hspec-discover if you havenāt already. Tests should be run in their own testsuite stanza, so I am not sure why you are importing MainSpec from Main. (simple repo with tests).
Donāt hesistate to share what you are doing, sometimes it is just minor configuration problems.
I am using a Spec.hs which just has the hspec-discover line in it.
Maybe the two Mainās got confused?!
Anyway, I changed my Main.hs (which contained all the real code for my CLI tool) to another name, so I could import that in AnotherNameSpec.hs, and then I created a stub Main.hs for the CLI program that also imports AnotherName. That seems to work now.
If I understood the setup correctly, I can add this.
A good way to set up unit tests in a cabal package which provides an executable is like this:
lib-internal) in src/;lib-internal in exe/ (or app/);lib-internal in test/ātests donāt know about the executable.But as long as it works, other solutions will do fine too!
I got curious to plot historical popularity of test frameworks over entire Hackage:
Surprisingly the absolute number of packages using test-framework remains constant: I would expect it to decrease by migration to tasty, because test-framework is stalled for a decade.
hspec keeps leading over tasty, but its relative edge decreased from 40% in 2019 to 20% now.
Thankfully test-framework still receives minimal maintenance (version bumps) by (I believe) Hackage trustees, so the number will likely not drop. Only once this service to the community is discontinued, active maintainers will switch, and the number will drop to a lower constant which represents the unmaintained packages.
