Sam and Wouter interview Harry Goldstein, a researcher in property-based testing who works in PL, SE, and HCI. In this episode, we reflect on random generators, the find-a-friend model, interdisciplinary research, and how to have impact beyond your own research community.
7 Likes
I was surprised to hear how it was lamented that Haskell’s module system doesn’t enable generation of property tests for comparison of supposedly identical implementation. I’d imagine that either .hi or .hsig or other Backpack files could be used for this purpose, is there a reason they can’t?
Intuitively, it makes sense to me when Harry claimed (paraphrasing now) that people aren’t good at interpreting probabilistic textual figures and that they require it graphically. But what does it that entail?
I suppose that means Harry thinks the newly released Falsify doesn’t go far enough, even though it allows for labelling generator results and outputs the textual percentages of their occurrences. This was shown in Chris Warbo’s SK logic in egglog: part 3, falsifying myself.
It’s totally possible that a version of Mica could be implemented with Haskell’s module system, that’s a great point! I think first-class modules may have some advantages for this kind of testing (that’s why we originally did this work in OCaml), but I don’t think they’re necessary.
2 Likes
And yes, my claim is that the textual percentages provided by the new Falsify (and by the original QuickCheck) don’t go far enough. When you have three categories, it may be reasonable to try to parse
Label "result":
15.0000% Not equal
81.0000% Timeout
4.0000% True
but if you want to visualize a distribution over more categories, integer values, etc. you really want a visual chart of some kind. We talk about this a bit more in the Tyche paper.
The good news is that it should be really easy to support Tyche in Falsify and get the best of all worlds!
2 Likes