[ANN] Skeletest - A new batteries-included, opinionated test framework

Announcements
#1

I’m excited to announce the first release of a test framework I’ve been musing about for over a year! Skeletest takes inspiration from pytest and jest, two test frameworks that IMO are some of the best test frameworks out there. Skeletest is batteries-included and opinionated, but it’s also extendable and hookable.

Skeletest features that no other Haskell test frameworks have:

  • Built-in explainable predicates
  • Show source code of failure (example)
  • Assert on values without a Show instance (example, snapshot)
  • P.con magic for checking predicates on fields in a data type, e.g. P.con User{name = P.eq "alice"} (example)
  • Markers to tag tests for selection from CLI or modify in hooks
    • For example, some Skeletest tests are marked as integration tests, which are skipped by default (e.g. not run in Hackage CI), and can be selected on CLI with @integration
  • Fixtures to share setup/provisioning logic (example test using this fixture)
  • Nicely formatted snapshot files for easy auditability (example)
  • Allow multiple snapshots in one test
  • Select tests by filepath
  • Test discovery comes out of the box
  • Built-in xfail/skip modifiers

How does it compare to existing test frameworks?

hspec
  • Can’t define custom flags
  • Can’t hook into test execution
  • No built-in support for golden tests. hspec-golden exists, but because hspec isn’t extendable, you can’t just add a --update flag, you have to run a separate hgold executable
tasty
  • Barebones framework, requires adding multiple libraries for unit testing, property testing, etc.
sydtest

Actually pretty solid, has a lot of nice features

  • Can’t define custom flags
  • Can’t hook into test execution
  • Built-in golden test
  • Fairly detailed failure messages, can’t do more complex predicates like explainable-predicates, though
  • Not a fan of random test order by default

Writing the tests for Skeletest itself has been such a pleasant experience, I’m really excited to see if that holds true in the wild as well. The project is only a month old, so it’s still rough and fresh, but I’m optimistic about what’s possible!

Note: To preempt some comments I know some of you will want to bring up :wink: — yes, this framework is intended to be the default test framework for a new build tool I’m experimenting with. I know people are very passionate about build tools and package managers and Cabal and Stack, and I do very much want to have that discussion, but let’s do that in a separate thread, after I flesh out my ideas a bit more.

29 Likes
#2

I was about to ask about comparing with hedgehog or quickcheck but I see you do in fact import hedgehog. Good call!

2 Likes
#3

Amazing! I’d love to see a column added here: GitHub - NorfairKing/sydtest: A modern testing framework for Haskell with good defaults and advanced testing features.

2 Likes
#4

You can’t not post a link to this!

1 Like
#5

@NorfairKing Sure, when the framework matures a bit :slight_smile: Don’t want to put the cart before the horse

@BebeSparkelSparkel The link is in the README :slight_smile:

1 Like
#6

What’s the magic behind P.con? How comes that it’s well-typed?

1 Like
#7

Good question!

Basically, skeletest uses both a preprocessor and GHC plugin. One of the things the plugin does is search for and replace P.con with an internal conMatches function before the typechecking phase.

2 Likes
#8

When multiple targets are specified, they are joined with or .

Interesting choice; when this question arisen for tasty, we decided on “and” semantics because of Allow --pattern|-p option to repeat by rhendric · Pull Request #380 · UnkindPartition/tasty · GitHub.

#9

Sure, perhaps AND is more “pure”, but IMO OR is more practical. I find it much less surprising for the following invocations to use OR:

skeletest test/MyLib/FooSpec.hs test/MyLib/BarSpec.hs

skeletest [fooFunc] [barFunc]
#10

This is fascinating! I’ve been a big fan of pytest, because it has so many features that make working with tests a dream. The idea of fixtures (and them being trees) imo is brilliant. Especially with included cleanup actions, they become very powerful.

It looks like you solved a big problem in pytest’s execution of fixtures: the fact that they’re implicit. Say you have a fixture called foo and want to use it in a test. The way you do it in pytest is by adding a parameter to the function named foo. Pytest magically figures out that you mean to use the fixture.

You solved it by demanding fixtures are explicitly retrieved by using the type class functions. Nice work!

When I have time, I’d love to take a closer look and see how this compares to pytest. Pytest has a billion features that I cannot expect a new project to instantly have, but from reading the README it looks very promising!

1 Like
#11

I got curious to see how close to P.con we can get without preprocessors / GHC plugins. It seems we can get fairly close so that

data MyRecord = MyRecord { fld1 :: Int, fld2 :: String, fld3 :: Double }
  deriving (Generic, EqOrPred)

main :: IO ()
main = print $
  MyRecord {fld1 = 3, fld2 = "foo", fld3 = pi} === MyRecord {fld1 = p odd, fld2 = "foo"}

prints True, where class EqOrPred and (===) are implemented in Pred.hs · GitHub.

2 Likes
#12

:joy: Certainly a bit cursed!

The preprocessor is already a requirement in order to do test discovery, so we have that regardless, and the plugin makes it easier to generate the main function than the preprocessor, so it’s already gonna be registered without this.

Plus, I want to do other things like show the function name in the failure message of P.>>>, which requires either a plugin or TH.

#13

Thanks for your work on this! I just ported a library from sydtest to skeletest to give it a spin. I really like how easy the fixtures are! The test output is nice too. The custom predicates were straightfoward to figure out (I had to hunt down (<<<)). It runs a little bit slower than sydtest I think. I’ll stick with it because I like the fixtures so much!

#14

Thanks for the trial! Yes, I haven’t done any performance testing, so would have to keep an eye on that

#15

Ooh, putStrLn output is inline with the test too. Huge improvement for me. I’m converted, thanks!

1 Like