# The sad state of property-based testing libraries

**URL:** <https://discourse.haskell.org/t/the-sad-state-of-property-based-testing-libraries/9880>\
**Category:** Links\
**Created:** [July 3, 2024, 9:58am UTC](https://discourse.haskell.org/t/the-sad-state-of-property-based-testing-libraries/9880 "2024-07-03T09:58:55Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![stevan](https://avatars.discourse-cdn.com/v4/letter/s/45deac/32.png) [@stevan](https://discourse.haskell.org/u/stevan)\
**Post date:** [July 3, 2024, 9:58am UTC](https://discourse.haskell.org/t/the-sad-state-of-property-based-testing-libraries/9880/1 "2024-07-03T09:58:55Z")

</div>

[https://stevana.github.io/the\_sad\_state\_of\_property-based\_testing\_libraries.html](https://stevana.github.io/the_sad_state_of_property-based_testing_libraries.html)

---

<div class="post-metadata">

**Author:** ![BurningWitness](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/burningwitness/32/3665_2.png) [@BurningWitness](https://discourse.haskell.org/u/BurningWitness)\
**Post date:** [July 3, 2024, 10:34am UTC](https://discourse.haskell.org/t/the-sad-state-of-property-based-testing-libraries/9880/2 "2024-07-03T10:34:43Z")

</div>

> [@stevan](#):
>
> It’s difficult to think of corner cases that you might have overlooked (because if you can then you probably wouldn’t have overlooked them in the first place)! This is where generating random inputs, a key feature of property-based testing, comes in. The idea being that random inputs will eventually hit corner cases.

I still don’t know if I’m missing something great here or if this approach is too contrived to be useful (I don’t use QuickCheck or similar in my personal projects).

The promise of “maybe some day one of these tests will fail because it hit a jackpot corner case” sure sounds cool, but it gives me way less confidence than an exhaustive list of tests that is handwritten to hit things I actually care about.

---

<div class="post-metadata">

**Author:** ![f-a](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/f-a/32/2740_2.png) [@f-a](https://discourse.haskell.org/u/f-a)\
**Post date:** [July 3, 2024, 10:56am UTC](https://discourse.haskell.org/t/the-sad-state-of-property-based-testing-libraries/9880/3 "2024-07-03T10:56:40Z")

</div>

> [@BurningWitness](#):
>
> The promise of “maybe some day one of these tests will fail because it hit a jackpot corner case” sure sounds cool, but it gives me way less confidence than an exhaustive list of tests that is handwritten to hit things I actually care about.

Try it and see!

In my experience the amount of bugs picked up by `QuickCheck` is substantial, and this helps with shedding light in those blind spots that happen when we write test “by hand”.  
You don’t even have to go for very complex properties to reap most benefits from QuickCheck.

Writing `QuickCheck` tests is not difficult _provided your datatypes are `Arbitrary` friendly_. So yes, you need a bit of pondering at the design stage.  
After that, I am confident the boon is worth ten time the hassle.

---

<div class="post-metadata">

**Author:** ![ReleaseCandidate](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/releasecandidate/32/2849_2.png) [@ReleaseCandidate](https://discourse.haskell.org/u/ReleaseCandidate)\
**Post date:** [July 3, 2024, 10:59am UTC](https://discourse.haskell.org/t/the-sad-state-of-property-based-testing-libraries/9880/4 "2024-07-03T10:59:04Z")

</div>

This list of features is strange, the IMHO single most important feature is good (or at least usable) shrinking. So for Haskell that means using Hedgehog.

> **[QuickCheck, Hedgehog, Validity](https://tech.fpcomplete.com/blog/quickcheck-hedgehog-validity/)**
>
> Property testing has so much potential to become a powerful tool for software development yet it remains mysterious to most developers. This blog explores ways to utilize the power of property testing as it pertains to Haskell. Learn about...

> [@BurningWitness](#):
>
> I still don’t know if I’m missing something great

Oh yes, you do. Sadly there aren’t many useful guides on how to use it for “real” code.  
I know of [The 'Property Based Testing' series | F# for fun and profit](https://fsharpforfunandprofit.com/series/property-based-testing/) especially the posts starting at 3  
and, although less specific:  
[Edit] Sorry, wrong video, I wanted to post this: [https://www.youtube.com/watch?v=wHJZ0icwSkc](https://www.youtube.com/watch?v=wHJZ0icwSkc)  
and his blog articles at [Property-Based Testing in a Screencast Editor: Introduction](https://wickstrom.tech/2019-03-02-property-based-testing-in-a-screencast-editor-introduction.html) and [Time Travelling and Fixing Bugs with Property-Based Testing](https://wickstrom.tech/2019-11-17-time-travelling-and-fixing-bugs-with-property-based-testing.html) instead of  
[https://www.youtube.com/watch?v=4bpc8NpNHRc](https://www.youtube.com/watch?v=4bpc8NpNHRc)

---

<div class="post-metadata">

**Author:** ![ReleaseCandidate](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/releasecandidate/32/2849_2.png) [@ReleaseCandidate](https://discourse.haskell.org/u/ReleaseCandidate)\
**Post date:** [July 3, 2024, 11:02am UTC](https://discourse.haskell.org/t/the-sad-state-of-property-based-testing-libraries/9880/5 "2024-07-03T11:02:24Z")

</div>

And I would really suggest _not_ using QuickCheck, but Hedgehog. QuickCheck was the first, and that _really_ shows.

Oh, there is also a new property testing library, which I haven’t tested [oh, what a pun!] yet: Falsify  
[https://well-typed.com/blog/2023/04/falsify/](https://well-typed.com/blog/2023/04/falsify/)

[![](https://img.youtube.com/vi/N0d7_MJmsKQ/maxresdefault.jpg "The Haskell Unfolder Episode 4: falsify") ](https://www.youtube.com/watch?v=N0d7_MJmsKQ)

---

<div class="post-metadata">

**Author:** ![stevan](https://avatars.discourse-cdn.com/v4/letter/s/45deac/32.png) [@stevan](https://discourse.haskell.org/u/stevan)\
**Post date:** [July 3, 2024, 11:24am UTC](https://discourse.haskell.org/t/the-sad-state-of-property-based-testing-libraries/9880/6 "2024-07-03T11:24:16Z")

</div>

> [@BurningWitness](#):
>
> “maybe some day one of these tests will fail because it hit a jackpot corner case”

With “eventually hit corner cases” I meant more “while generating 10000 tests in a second or two” rather than “after months and millions of generated tests”.

> [@BurningWitness](#):
>
> an exhaustive list of tests that is handwritten to hit things I actually care about.

How do you know that you’ve not forgotten to care about something :-)?

> [@ReleaseCandidate](#):
>
> This list of features is strange, the IMHO single most important feature is good (or at least usable) shrinking. So for Haskell that means using Hedgehog.

Depends what you are testing I suppose, I’m interested in testing stateful and concurrent systems rather than mere pure functions. Integrated shrinking a la Hedgehog/falsify won’t help much in that case. Since we are generating lists of commands to execute against the real system and the model, we can get shrinking of these lists for free by the library (making integrated shrinking less important).

---

<div class="post-metadata">

**Author:** ![ReleaseCandidate](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/releasecandidate/32/2849_2.png) [@ReleaseCandidate](https://discourse.haskell.org/u/ReleaseCandidate)\
**Post date:** [July 3, 2024, 11:51am UTC](https://discourse.haskell.org/t/the-sad-state-of-property-based-testing-libraries/9880/7 "2024-07-03T11:51:05Z")

</div>

Oh, that’s your blog. Could you please make the text bigger and the TOC a bit more contrasty? The difficulty to read it

> [@stevan](#):
>
> Depends what you are testing I suppose,

Of course. That’s why I found your title to be (too) click-baity.

---

<div class="post-metadata">

**Author:** ![jeukshi](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/jeukshi/32/4040_2.png) [@jeukshi](https://discourse.haskell.org/u/jeukshi)\
**Post date:** [July 3, 2024, 12:47pm UTC](https://discourse.haskell.org/t/the-sad-state-of-property-based-testing-libraries/9880/8 "2024-07-03T12:47:11Z")

</div>

> [@BurningWitness](#):
>
> The promise of “maybe some day one of these tests will fail because it hit a jackpot corner case” sure sounds cool, but it gives me way less confidence than an exhaustive list of tests that is handwritten to hit things I actually care about.

Well, if there is a chance that those tests will hit a jackpot, then there is a chance that your program will hit a jackpot during execution and fail. Also, there are cases where an exhaustive list of tests is just not realistic, like the de/serialization of arbitrary text.

But actually, I like this way of testing for a different reason: it switches your focus, from example execution, to things that will hold true for any possible execution. It is just a more general way of reasoning about code or a solution, and in my experience, it often leads to better design or understanding of a problem at hand.

---

<div class="post-metadata">

**Author:** ![brandonchinn178](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/brandonchinn178/32/5103_2.png) [@brandonchinn178](https://discourse.haskell.org/u/brandonchinn178)\
**Post date:** [July 3, 2024, 2:52pm UTC](https://discourse.haskell.org/t/the-sad-state-of-property-based-testing-libraries/9880/9 "2024-07-03T14:52:50Z")

</div>

How does falsify rank in the survey? I have my eye on that one. It doesn’t support monadic properties yet, but there’s a PR up for it

I also like hedgehog over quickcheck, but can someone explain how you still get good coverage if hedgehog forces you to specify a range? I would imagine it’s easier to miss edge cases like overflow/underflow

---

<div class="post-metadata">

**Author:** ![stevan](https://avatars.discourse-cdn.com/v4/letter/s/45deac/32.png) [@stevan](https://discourse.haskell.org/u/stevan)\
**Post date:** [July 3, 2024, 3:04pm UTC](https://discourse.haskell.org/t/the-sad-state-of-property-based-testing-libraries/9880/10 "2024-07-03T15:04:45Z")

</div>

> [@brandonchinn178](#):
>
> It doesn’t support monadic properties yet, but there’s a PR up for it

Support for monadic properties is a pre-requisite for stateful and parallel testing.

---

<div class="post-metadata">

**Author:** ![BurningWitness](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/burningwitness/32/3665_2.png) [@BurningWitness](https://discourse.haskell.org/u/BurningWitness)\
**Post date:** [July 3, 2024, 3:23pm UTC](https://discourse.haskell.org/t/the-sad-state-of-property-based-testing-libraries/9880/11 "2024-07-03T15:23:27Z")

</div>

> [@stevan](#):
>
> How do you know that you’ve not forgotten to care about something :-)?

I don’t, but property tests don’t solve that issue, I still have to define properties that hit the wanted cases often enough. The pessimistic view here is that I focus on executing working properties a thousand times each while losing track of the edges (since the very definitions now run in terms of properties).

> [@jeukshi](#):
>
> there are cases where an exhaustive list of tests is just not realistic, like the de/serialization of arbitrary text.

True, that’s why I tested radix trees in a similar way, handcrafting tree generators.

I think for parsers it’s more complex than that, because tests break into two types: testing control flow in general (where property-based testing catches internal state alterations) and testing primitive parsers (where side cases are easy to describe). That’s how `binary` did it ([`Action.hs`](https://github.com/kolmodin/binary/blob/2a712db14912dddccf3e2207815e30b9f3049514/tests/Action.hs) for properties, [`QC.hs`](https://github.com/kolmodin/binary/blob/2a712db14912dddccf3e2207815e30b9f3049514/tests/QC.hs) for equalities).

I guess my concern is that I consider property-based testing a specific tool for a job, so coming up with new libraries for shrinking and custom generator monad stacks feels overkill.

---

<div class="post-metadata">

**Author:** ![jeukshi](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/jeukshi/32/4040_2.png) [@jeukshi](https://discourse.haskell.org/u/jeukshi)\
**Post date:** [July 3, 2024, 8:54pm UTC](https://discourse.haskell.org/t/the-sad-state-of-property-based-testing-libraries/9880/12 "2024-07-03T20:54:44Z")

</div>

> [@BurningWitness](#):
>
> I guess my concern is that I consider property-based testing a specific tool for a job, so coming up with new libraries for shrinking and custom generator monad stacks feels overkill.

Is this really a valid concern, though? Software testing is ultimately about economic judgment; you want the best bang for the buck, and as you point out, it depends on a case-by-case basis. If it is ever a valid option, then why not have a better library providing it?

Plus, as I was trying to say, but failed to put it right, property-based testing makes you come up with an actual _specification_ of the thing under test (even if the tests themselves can’t really prove that spec). That makes it special, compared to most other example-based techniques.

---

<div class="post-metadata">

**Author:** ![hasufell](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/hasufell/32/1250_2.png) [@hasufell](https://discourse.haskell.org/u/hasufell)\
**Post date:** [July 4, 2024, 4:14am UTC](https://discourse.haskell.org/t/the-sad-state-of-property-based-testing-libraries/9880/13 "2024-07-04T04:14:59Z")

</div>

> [@BurningWitness](#):
>
> The promise of “maybe some day one of these tests will fail because it hit a jackpot corner case” sure sounds cool, but it gives me way less confidence than an exhaustive list of tests that is handwritten to hit things I actually care about.

This happened to me twice.

First at work, where we tested a financial algorithm that would produce odd results when a loan started in 1860 or something. We never figured out why.

Second in the `filepath` library, where I used property tests to assert that the new version is semantically equivalent to the old version (after the large `OsPath` rewrite): [filepath/tests/filepath-equivalent-tests/TestEquiv.hs at master · haskell/filepath · GitHub](https://github.com/haskell/filepath/blob/master/tests/filepath-equivalent-tests/TestEquiv.hs)

Every 3 months the CI would fail and something new popped up, until I was tired of it. The problem with property tests is that you need **really good generators**. Windows filepaths are very complex (unlike unix ones) and it’s almost impossible to hit interesting corner cases if you generate naive random ascii strings as filepaths.

So what I did then was to describe windows filepaths as an ABNF, write an ADT for it, then a generator for that ADT, tweaking the distribution a little to ensure it doesn’t exhaust on lists, but always “visits” all branches of the nested sum types… and finally have a “Show” instance that turns it back into a real filepath.

The result is here: [filepath/tests/filepath-equivalent-tests/Gen.hs at master · haskell/filepath · GitHub](https://github.com/haskell/filepath/blob/master/tests/filepath-equivalent-tests/Gen.hs#L44)

My conclusion is:

- property tests are rarely a substitute for unit tests (not all behaviors you want to test for are really “properties”)
- they are not very useful unless you write really good generators

---

<div class="post-metadata">

**Author:** ![tomjaguarpaw](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/tomjaguarpaw/32/1230_2.png) [@tomjaguarpaw](https://discourse.haskell.org/u/tomjaguarpaw)\
**Post date:** [July 4, 2024, 7:53am UTC](https://discourse.haskell.org/t/the-sad-state-of-property-based-testing-libraries/9880/14 "2024-07-04T07:53:46Z")

</div>

> [@hasufell](#):
>
> they are not very useful unless you write really good generators

It’s hard to summarize how I feel about generators, but maybe this does the job: “generators are often complicated enough to themselves need tests”.

---

<div class="post-metadata">

**Author:** ![ReleaseCandidate](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/releasecandidate/32/2849_2.png) [@ReleaseCandidate](https://discourse.haskell.org/u/ReleaseCandidate)\
**Post date:** [July 4, 2024, 8:43am UTC](https://discourse.haskell.org/t/the-sad-state-of-property-based-testing-libraries/9880/15 "2024-07-04T08:43:10Z")

</div>

Thinking about “necessary features”, generators where you can define frequencies or probabilities for sub-generators are definitely a must have. Which implies support for shrinking of these.

> [@brandonchinn178](#):
>
> can someone explain how you still get good coverage if hedgehog forces you to specify a range? I would imagine it’s easier to miss edge cases like overflow/underflow

If you include the limits, that’s not a problem?  
But I personally always use the explicit way by using `frequency`, something like this:  
Important to know is that Hedeghog shrinks `frequency` towards the first element of the list.

```haskell
frequency [ (1, 0.0)
          , (1, 1.0/0.0)
          , (1, -1.0/0.0)
          , (1, 0.0/0.0)
          , (30, float (linear SOME_FUNCTION_USING_FLOAT_RANGE))
]

```

The most obvious usage of these kind of generators are Unicode code points, as most of the people most of the time do not want all Unicode code points (or Unicode groups) with the same frequency.

```haskell
frequency [(10, alphaNum), (5, latin1), (1, HGen.unicode)]

```

With numbers or floats in special I also sometimes use different intervals with different frequencies, for floats this could be denormalized floats (floats, which are not of the form `1.?e-N` but `0.0...0?e-N`)

Something like

```haskell
frequency [ (1, 0)
          , (1, MAX_POS)
          , (1, MIN_NEG)
          , (10, int (linear MIN_NEG, MAX_POS))
          , (30, int (linear -50 -7)
          , (50, int (linear 123 12345678))

```

> [@hasufell](#):
>
> The problem with property tests is that you need **really good generators**.

Yes, this is also something where _all_(?) tutorials, … fall short.

> [@tomjaguarpaw](#):
>
> “generators are often complicated enough to themselves need tests”.

Yes, I actually often do that - at least “visually”, by looking at “enough” (whatever that means) generated values.

---

<div class="post-metadata">

**Author:** ![stevan](https://avatars.discourse-cdn.com/v4/letter/s/45deac/32.png) [@stevan](https://discourse.haskell.org/u/stevan)\
**Post date:** [July 4, 2024, 8:44am UTC](https://discourse.haskell.org/t/the-sad-state-of-property-based-testing-libraries/9880/16 "2024-07-04T08:44:48Z")

</div>

> [@hasufell](#):
>
> My conclusion is:
> 
> - property tests are rarely a substitute for unit tests (not all behaviors you want to test for are really “properties”)
> - they are not very useful unless you write really good generators

When testing pure functions, I agree with this conclusion. When testing stateful systems (which the post is about), less so – as there the property is the real system is faithful to fake/model. This covers all unit tests about the real system while at the same time contract testing the fake, thereby making it possible to use with confidence as part of integration tests further down the road.

Generators for commands tend to be simple compared to functionality they excerise (they are the inputs/API for your black box), and the library takes care of shrinking lists of commands already so you only have to focus on generating individual commands.

---

<div class="post-metadata">

**Author:** ![michaelpj](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/michaelpj/32/417_2.png) [@michaelpj](https://discourse.haskell.org/u/michaelpj)\
**Post date:** [July 4, 2024, 8:58am UTC](https://discourse.haskell.org/t/the-sad-state-of-property-based-testing-libraries/9880/17 "2024-07-04T08:58:26Z")

</div>

One time that property-based testing can really give you a lot of bang for very little buck is when you have an existing good model of the behaviour you want. For example, writing an optimized version of a data structure or algorithm. Testing against the original model removes the need to come up with interesting properties, since the property is always “behaves the same”, you just need to come up with generators that exercise interesting behaviour (which is non-trivial, as people have pointed out).

I once wrote a very-mutable circular buffer using lots of array indices and suchlike, and a single property test comparing it to an existing queue implementation caught _dozens_ of bugs, and since the generator would also produce “simple” operations, I didn’t even need to write any basic smoke/unit tests either. It was sweet.

(EDIT: I see steven said something similar, the point that generating lists of commands/operations is easier is also definitely a good one)

---

<div class="post-metadata">

**Author:** ![stevan](https://avatars.discourse-cdn.com/v4/letter/s/45deac/32.png) [@stevan](https://discourse.haskell.org/u/stevan)\
**Post date:** [July 4, 2024, 9:03am UTC](https://discourse.haskell.org/t/the-sad-state-of-property-based-testing-libraries/9880/18 "2024-07-04T09:03:25Z")

</div>

> [@michaelpj](#):
>
> I once wrote a very-mutable circular buffer using lots of array indices and suchlike, and a single property test comparing it to an existing queue implementation caught _dozens_ of bugs, and since the generator would also produce “simple” operations, I didn’t even need to write any basic smoke/unit tests either. It was sweet.

I have a similar [example](https://stevana.github.io/the_sad_state_of_property-based_testing_libraries.html#example-circular-buffer) in the post btw.

---

<div class="post-metadata">

**Author:** ![amigalemming](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/amigalemming/32/4319_2.png) [@amigalemming](https://discourse.haskell.org/u/amigalemming)\
**Post date:** [July 7, 2024, 5:24pm UTC](https://discourse.haskell.org/t/the-sad-state-of-property-based-testing-libraries/9880/19 "2024-07-07T17:24:06Z")

</div>

In your article I don’t see the sad state of property-based testing libraries, but a rich set of implementations to choose from. Thank you, for pointing me to them. I did not know of `quickcheck-state-machine` before.  
Using plain QuickCheck took me a lot of training and experience, too. I spend a long time with Arbitrary instances until I discovered the value of `forAll`. My next important discovery was `doctest`. Writing properties as documentation lowered the barrier for me a lot, to write property tests, at all. However, I wrote my own utility `doctest-extract`, that decouples doctesting from `Cabal` and `GHCi`.

---

<div class="post-metadata">

**Author:** ![ibizaman](https://avatars.discourse-cdn.com/v4/letter/i/9d8465/32.png) [@ibizaman](https://discourse.haskell.org/u/ibizaman)\
**Post date:** [July 8, 2024, 11:38am UTC](https://discourse.haskell.org/t/the-sad-state-of-property-based-testing-libraries/9880/20 "2024-07-08T11:38:39Z")

</div>

I had a great experience introducing the rapid library for stateful property testing at work. We’re a Golang shop. It allowed us to confidently switch MongoDB drivers. We added a comprehensive (as much as possible anyway) regression test suite to the existing MongoDB driver and then we could simply make the new MongoDB driver pass the same tests.

Even as I write it it sounds obvious, but in an enterprise context it makes sense to reiterate. I like to think about stateful property tests as allowing you to focus only on one method (in Go lingo) at a time. You think of the edge cases of that method in isolation. This let’s you tame combinatorial explosion.

I do agree that we did need to spend a lot of time crafting generators. Also, the rapid library has no classification and no frequency-based generator so we needed to be creative here. Still, it was definitely worth the effort.

I also saw the advantages called out in the [integration testing with contract tested fakes](https://stevana.github.io/the_sad_state_of_property-based_testing_libraries.html#integration-testing-with-contract-tested-fakes) section of the original post. Namely:

> There are other advantages to using fakes, for example we can use this fake in integration tests with components that depend on the software that we tested with the fake.

and

> One of the problems with integration testing against fakes is that the fake can be wrong. The standard solution to solve that problem is to [contract test](https://martinfowler.com/bliki/ContractTest.html) the fake to make sure that it is faithful to the software it’s supposed to be a fake of. We don’t have this problem, because our stateful and parallel property-based tests assure that the fake is faithful

I went into much more details in this 2 part blog post I wrote. [Post 1](https://blog.tiserbox.com/posts/2024-02-27-stateful-property-testing-in-go.html) and [Post 2](https://blog.tiserbox.com/posts/2024-07-08-make-good-stubs-with-stateful-property-testing.html). Thanks for prompting me to write part 2, I procrastinated for a long time 😃