Getting started with testing Haskell in 2024

I have (finally) reached a point where I need to write tests for a small Haskell program I have written, which is being used in anger.

Are there any recommendations on what libraries / tools to use, if you are starting from nothing, today?

I will go searching and looking, but sometimes a(n opinionated) recommendation is a useful starting point.

7 Likes

The Stack project uses hspec and QuickCheck, but that may reflect its history. tasty says it is ā€˜modernā€™ and it also has ā€˜providersā€™ for Hspec and QuickCheck tests.

1 Like

doctest is handy. The tasty ecosystem is good. My shelltestrunner might be useful for cli tests. If you get into property testing later, try the variants like hedgehog or smallcheck, not just quickcheck.

2 Likes

I use and love hspec. Well documented, integration with QuickCheck/SmallCheck/HUnit if needed, batteries included.

4 Likes

My go-to toolkit is tasty as the harness, tasty-hunit for unit tests, tasty-golden for golden tests, and tasty-quickcheck for property-testing.

6 Likes

I maintain a list but it still needs fleshing out. I like the tasty testing universe and hedgehog for property testing (because of automatic shrinking and gens-as-values).

3 Likes

What program?

QuickCheck is wildly overrated, unless you write simple primitivesā€¦ or have a ton of time writing smart generators for complex data (I did that for something seemingly simple like windows filepaths and it took a lot of time).

If itā€™s cliā€¦ ghcup recently got a test suite for the optparse-applicative part: ghcup-hs/test/optparse-test/InstallTest.hs at master Ā· haskell/ghcup-hs Ā· GitHub

That expresses our expectations of cli arguments and parsed types. Is that generally useful? No, unless breaking cli API is a major concern.

Itā€™s a command line tool that looks up users and groups via LDAP (only useful within the company). It has reached the level of usage now that I might break it subtly and nobody will notice until something goes awry :slight_smile:

Nice, thanks for the pointer! That sounds useful for my case as well. If my program for instance starts not exiting with a non-zero return code in circumstances where it used to, itā€™s a concern.

2 Likes

I forgot, if you are writing a CLI program and you need golden tests (so tests where you are comparing program output with a reference output stored in a file), the way to go is shelltestrunner.
Well documented, sensible syntax, output regexps, deals with both standard and error output.

It is a CLI program itself, installable from most Linux distributions rrepos, so it is useful even if you are not developing in Haskell.

4 Likes

Ah, nice - that sounds like it could cover a good deal of the (ā€œintegrationā€) tests I want to have, thanks!

I have been trying top get Hspec going, but Iā€™m hitting a circular import where running the tests imports MainSpec which imports Main which imports MainSpec again, so I still need to wrap my head around that part to get other (ā€œunitā€) tests going.

1 Like

You should take advantage of hspec-discover if you havenā€™t already. Tests should be run in their own testsuite stanza, so I am not sure why you are importing MainSpec from Main. (simple repo with tests).

Donā€™t hesistate to share what you are doing, sometimes it is just minor configuration problems.

I am using a Spec.hs which just has the hspec-discover line in it.

Maybe the two Mainā€™s got confused?!

Anyway, I changed my Main.hs (which contained all the real code for my CLI tool) to another name, so I could import that in AnotherNameSpec.hs, and then I created a stub Main.hs for the CLI program that also imports AnotherName. That seems to work now.

If I understood the setup correctly, I can add this.

A good way to set up unit tests in a cabal package which provides an executable is like this:

  • one internal library (let us callib lib-internal) in src/;
  • one executable which imports lib-internal in exe/ (or app/);
  • one testsuite which imports lib-internal in test/ā†tests donā€™t know about the executable.

But as long as it works, other solutions will do fine too!

1 Like

I got curious to plot historical popularity of test frameworks over entire Hackage:

Surprisingly the absolute number of packages using test-framework remains constant: I would expect it to decrease by migration to tasty, because test-framework is stalled for a decade.

hspec keeps leading over tasty, but its relative edge decreased from 40% in 2019 to 20% now.

9 Likes