# HF Coordination for test framework IDE integration

**URL:** <https://discourse.haskell.org/t/hf-coordination-for-test-framework-ide-integration/5221>\
**Category:** Haskell Foundation\
**Created:** [October 25, 2022, 2:18pm UTC](https://discourse.haskell.org/t/hf-coordination-for-test-framework-ide-integration/5221 "2022-10-25T14:18:07Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![santiweight](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/santiweight/32/1187_2.png) [@santiweight](https://discourse.haskell.org/u/santiweight)\
**Post date:** [October 25, 2022, 2:18pm UTC](https://discourse.haskell.org/t/hf-coordination-for-test-framework-ide-integration/5221/1 "2022-10-25T14:18:07Z")

</div>

I’m bringing this one up as a pure conversation starter. I have very little idea what is involved and the value-effort balance of such a task.

All major languages have test integration in their IDE, where you can run single tests via the IDE by pressing a play button.

Here’s an [example for C#](https://stackoverflow.com/questions/41958510/how-to-run-all-tests-in-visual-studio-code).

It would be pretty invaluable to coordinate the efforts of major test frameworks (and minor also). It seems to me that a great place to start is the [`tasty`](https://hackage.haskell.org/package/tasty) framework, which already unifies all other major testing frameworks, and so provides a great entry point for such a task. [`tasty-discover`](https://hackage.haskell.org/package/tasty-discover) is relevant here

---

<div class="post-metadata">

**Author:** ![david-christiansen](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/david-christiansen/32/2404_2.png) [@david-christiansen](https://discourse.haskell.org/u/david-christiansen)\
**Post date:** [October 25, 2022, 4:54pm UTC](https://discourse.haskell.org/t/hf-coordination-for-test-framework-ide-integration/5221/2 "2022-10-25T16:54:30Z")

</div>

Having the little play button next to tests is really nice when working with Kotlin in IntelliJ. Here’s a potential minimal design for a solution in the spirit of Haskell that doesn’t rely on special language support for outside libraries.

Libraries like JUnit rely on Java annotations to unambiguously mark tests, but we don’t have anything like Java annotations in Haskell. We tend to use newtypes for this kind of situation.

What if there were something like this in a well-known location:

```haskell
data TestResult = Passed | Failed Text | Error Text

class IsTest a where
  runTest :: a -> IO TestResult

```

Any module-level definition whose type has a `IsTest` instance would be presented with a play button, and the play button would invoke `runTest` on it. Then, we could get something like the JUnit behavior with a micro-library:

```haskell
newtype Test = Test (IO ())
newtype TestFail = TestFail Text deriving Show via Text
instance Exception TestFail where
instance IsTest Test where
  runTest (Test impl) = (impl >> pure Passed) `catch` handleFailure `catch` handleOther
    where
      handleFailure (TestFail msg) = pure (Failed msg)
      handleOther e = pure (Error (show e))

 ▶️ myTest = Test $ do ...

```

Existing test frameworks could integrate very easily with this by providing `IsTest` instances.

---

<div class="post-metadata">

**Author:** ![adamgundry](https://avatars.discourse-cdn.com/v4/letter/a/34f0e0/32.png) [@adamgundry](https://discourse.haskell.org/u/adamgundry)\
**Post date:** [October 26, 2022, 7:10am UTC](https://discourse.haskell.org/t/hf-coordination-for-test-framework-ide-integration/5221/3 "2022-10-26T07:10:12Z")

</div>

This seems plausible, and might not be vastly difficult to do as a HLS plugin? (Although how nice the UI could be would depend on what support LSP provides for this kind of thing, or on adding custom IDE support.) I’m reminded of [hls-eval-plugin: Eval plugin for Haskell Language Server](https://hackage.haskell.org/package/hls-eval-plugin) and [Proof of concept: Diagrams plugin by edsko · Pull Request #3250 · haskell/haskell-language-server · GitHub](https://github.com/haskell/haskell-language-server/pull/3250). In particular, the latter looks for bindings of type `Diagram` and renders them on hover; one could imagine instead a plugin that looks for bindings whose types instantiate `IsTest`.

---

<div class="post-metadata">

**Author:** ![david-christiansen](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/david-christiansen/32/2404_2.png) [@david-christiansen](https://discourse.haskell.org/u/david-christiansen)\
**Post date:** [October 26, 2022, 7:28am UTC](https://discourse.haskell.org/t/hf-coordination-for-test-framework-ide-integration/5221/4 "2022-10-26T07:28:05Z")

</div>

I use the eval plugin all the time, just as I used its predecessor in Dante. It’s really quite nice.

I suspect that an implementation along the lines that you’re suggesting would not be astoundingly difficult.

---

<div class="post-metadata">

**Author:** ![david-christiansen](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/david-christiansen/32/2404_2.png) [@david-christiansen](https://discourse.haskell.org/u/david-christiansen)\
**Post date:** [October 26, 2022, 7:29am UTC](https://discourse.haskell.org/t/hf-coordination-for-test-framework-ide-integration/5221/5 "2022-10-26T07:29:50Z")

</div>

Come to think of it, the eval plugin basically already can run unit tests with a clunky UI, so you’re definitely on to something here.

---

<div class="post-metadata">

**Author:** ![michaelpj](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/michaelpj/32/417_2.png) [@michaelpj](https://discourse.haskell.org/u/michaelpj)\
**Post date:** [October 26, 2022, 10:22am UTC](https://discourse.haskell.org/t/hf-coordination-for-test-framework-ide-integration/5221/6 "2022-10-26T10:22:22Z")

</div>

A problem is fitting this into the LSP interaction model. I know rust-analyzer supports this, so we could look at what they do.

> Libraries like JUnit rely on Java annotations to unambiguously mark tests, but we don’t have anything like Java annotations in Haskell.

Haskell has these too: the ANN pragma. It’s not much used, and seems to be undocumented, but it is there!

---

<div class="post-metadata">

**Author:** ![david-christiansen](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/david-christiansen/32/2404_2.png) [@david-christiansen](https://discourse.haskell.org/u/david-christiansen)\
**Post date:** [October 26, 2022, 11:18am UTC](https://discourse.haskell.org/t/hf-coordination-for-test-framework-ide-integration/5221/7 "2022-10-26T11:18:02Z")

</div>

Perhaps this is mostly a philosophical question, but if it’s not used and it’s undocumented, is it _really_ there?

---

<div class="post-metadata">

**Author:** ![jaror](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/jaror/32/3271_2.png) [@jaror](https://discourse.haskell.org/u/jaror)\
**Post date:** [October 26, 2022, 11:24am UTC](https://discourse.haskell.org/t/hf-coordination-for-test-framework-ide-integration/5221/8 "2022-10-26T11:24:10Z")

</div>

It is documented:

[https://downloads.haskell.org/~ghc/9.4.2/docs/users\_guide/extending\_ghc.html#source-annotations](https://downloads.haskell.org/~ghc/9.4.2/docs/users_guide/extending_ghc.html#source-annotations)

---

<div class="post-metadata">

**Author:** ![david-christiansen](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/david-christiansen/32/2404_2.png) [@david-christiansen](https://discourse.haskell.org/u/david-christiansen)\
**Post date:** [October 26, 2022, 11:25am UTC](https://discourse.haskell.org/t/hf-coordination-for-test-framework-ide-integration/5221/9 "2022-10-26T11:25:23Z")

</div>

And here I was reading [the page in the manual about pragmas](https://ghc.gitlab.haskell.org/ghc/doc/users_guide/exts/pragmas.html). Thanks!

---

<div class="post-metadata">

**Author:** ![david-christiansen](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/david-christiansen/32/2404_2.png) [@david-christiansen](https://discourse.haskell.org/u/david-christiansen)\
**Post date:** [October 26, 2022, 11:27am UTC](https://discourse.haskell.org/t/hf-coordination-for-test-framework-ide-integration/5221/10 "2022-10-26T11:27:45Z")

</div>

It looks like something like `data Test = Test` and `{-# ANN myTest Test #-}` could work as well, though I do like how the type class mechanism would allow lightweight adaptors from the various testing libraries to be written.

---

<div class="post-metadata">

**Author:** ![santiweight](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/santiweight/32/1187_2.png) [@santiweight](https://discourse.haskell.org/u/santiweight)\
**Post date:** [November 21, 2022, 12:22am UTC](https://discourse.haskell.org/t/hf-coordination-for-test-framework-ide-integration/5221/11 "2022-11-21T00:22:10Z")

</div>

I’ve been thinking about your solution for a little bit…

Something interesting about your solution is that it treats test declarations as primitives. Initially, I was thinking about individual tests as CLI calls with some kind of filtering. In `tasty`, this might look like: `cabal test --test-options='-p "/myTest/"'`.

Your solution has a some advantages: it’s very simple; it avoids the need for any naming conventions or annotations.

However it also has some disadvantages: the main one I can think of is that many test frameworks have flags that affect the runtime; sometimes I want to filter down to a test helper that is called many times by different top-level tests.

I think your suggestion is better though, because it’s an easy solution to a complex problem.

However, we can recover some of the power of a CLI with:

```haskell
data TestOpts where
  TestOpts :: forall opts. Data opts => opts -> Test

simpleTest = Test ()

runFailedTests = Test ["--only-failed"]

```

Each test framework could add its own helpers, and one canonical implementation of `IsTest`, where `runTest` now has the type `runTest :: TestOpts -> a -> IO TestResult`.

An example:

```haskell
{-# ANN myTest simpleTest #-}
myTest = _

-- The following is during development when I want to isolate a specific test
{-# ANN myTest (filterTest "mySpecificTestCase") #-}
myTest = _

```

---

<div class="post-metadata">

**Author:** ![santiweight](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/santiweight/32/1187_2.png) [@santiweight](https://discourse.haskell.org/u/santiweight)\
**Post date:** [November 21, 2022, 12:29am UTC](https://discourse.haskell.org/t/hf-coordination-for-test-framework-ide-integration/5221/12 "2022-11-21T00:29:05Z")

</div>

Reading more, I think my small addition makes the ANN solution completely compatible with `tasty`, for example, even with all its CLI options and support for “ingredients”.

[https://hackage.haskell.org/package/tasty-1.4.2.3/docs/Test-Tasty.html#v:defaultMain](https://hackage.haskell.org/package/tasty-1.4.2.3/docs/Test-Tasty.html#v:defaultMain)  
[https://hackage.haskell.org/package/tasty-1.4.2.3/docs/Test-Tasty-Ingredients.html#t:Ingredient](https://hackage.haskell.org/package/tasty-1.4.2.3/docs/Test-Tasty-Ingredients.html#t:Ingredient)

A codebase could declare its own `TestOpts` helpers with helpful ingredients and options for tests during developer debugging…

---

<div class="post-metadata">

**Author:** ![angerman](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/angerman/32/892_2.png) [@angerman](https://discourse.haskell.org/u/angerman)\
**Post date:** [November 21, 2022, 4:11am UTC](https://discourse.haskell.org/t/hf-coordination-for-test-framework-ide-integration/5221/13 "2022-11-21T04:11:22Z")

</div>

I’ll very strongly suggest _not_ to use `ANN` if you can help it. It will trigger the TemplateHaskell pipeline, and thus increase compilation time, and may cause increased recompilation.

---

<div class="post-metadata">

**Author:** ![santiweight](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/santiweight/32/1187_2.png) [@santiweight](https://discourse.haskell.org/u/santiweight)\
**Post date:** [November 21, 2022, 7:10pm UTC](https://discourse.haskell.org/t/hf-coordination-for-test-framework-ide-integration/5221/14 "2022-11-21T19:10:21Z")

</div>

Thanks for the pointer!

I guess that finding a way to pass test options to the tests is going to be a little difficult then… I guess we can just say “this is for the 80% of use cases where it’s helpful and go from there”.

On the other hand, I worry about choosing a typeclass design that isn’t extensible.

I think the next step is to define what are the most valuable features in the design space and decide on whether a type class-driven approach can hit most of the right notes! I’ll get on this over thanksgiving I think…

---

<div class="post-metadata">

**Author:** ![david-christiansen](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/david-christiansen/32/2404_2.png) [@david-christiansen](https://discourse.haskell.org/u/david-christiansen)\
**Post date:** [November 21, 2022, 7:44pm UTC](https://discourse.haskell.org/t/hf-coordination-for-test-framework-ide-integration/5221/15 "2022-11-21T19:44:45Z")

</div>

Using a type class to indicate “click to run this test with default options” doesn’t preclude having a more detailed means of running a test suite in “expert mode”, does it?

---

<div class="post-metadata">

**Author:** ![taylorfausak](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/taylorfausak/32/54_2.png) [@taylorfausak](https://discourse.haskell.org/u/taylorfausak)\
**Post date:** [November 21, 2022, 7:48pm UTC](https://discourse.haskell.org/t/hf-coordination-for-test-framework-ide-integration/5221/16 "2022-11-21T19:48:41Z")

</div>

It might be worth looking at the `detailed-0.9` test type. Cabal has support for that, but as far as I know, very few people actually use it.

---

<div class="post-metadata">

**Author:** ![santiweight](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/santiweight/32/1187_2.png) [@santiweight](https://discourse.haskell.org/u/santiweight)\
**Post date:** [November 21, 2022, 7:50pm UTC](https://discourse.haskell.org/t/hf-coordination-for-test-framework-ide-integration/5221/17 "2022-11-21T19:50:59Z")

</div>

Definitely not, but I think it’s worth considering the possible design space for an expert mode, just so we know the design space before implementation 🙂

Fwiw, I am convincing myself that your type class design is already very good. My biggest worry was not having control over options such as “accept golden tests”, but I think these I can get these back, so I’m going to write something up for the type class design.

---

<div class="post-metadata">

**Author:** ![santiweight](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/santiweight/32/1187_2.png) [@santiweight](https://discourse.haskell.org/u/santiweight)\
**Post date:** [November 21, 2022, 8:32pm UTC](https://discourse.haskell.org/t/hf-coordination-for-test-framework-ide-integration/5221/18 "2022-11-21T20:32:49Z")

</div>

So I’ve thought on it a little more, and I think I have a better intuition on what makes Haskell’s testing situation a little unique from other languages.

Most languages use a OO method-orientated design for test layout and discovery. For example: Java uses the `@Test` annotation to indicate a test; Python’s unittest framework takes any class extending `unittest.TestCase` and assumes that all its functions are test cases.

Haskell differs in a few ways:

1. the organization of test code is handled by an eDSL where you create a value representing a test tree, typically with combinators such as `testGroup` or `testCase`, and assigning them names
2. we do not have access to location data for a test case, whereas Java and the like have more immediate access to this data. This is because combinators such as `testGroup` do not typically have access to the location they were called at.
3. this one is more conceptual: Haskell test structures are less focused around top level declarations in comparison to other languages. In other languages, it is not as easy to have nested test cases within one declaration. In Haskell a single top-level declaration can quite easily contain many different tests.

In other words, in Haskell, the unit of test is actually a `testGroup` or `testCase` call, and not top level declarations. The test tree is normally discovered (e.g. `tasty` and `HUnit`) with a flag such as `--list-tests`.

I am wondering whether it is possible to have test frameworks provide not just the test tree, but also the location of said tests. For example:

```Haskell
-- Test.hs
tests = testGroup "all" [testGroup "unit" unitTests, testGroup "property" propertyTests]
--Test.Unit.hs
unitTests = [testCase "my unit" ..., testCase "unit2" ...]

-- Current output of --list-tests
all.unit.unit1
all.unit.unit2
all.property.prop1

-- Suggested output of --list-tests-json
{"name" : "all.unit.my unit", "location" : "path/to/Test/Unit.hs"...}

```

I think this is a preferable option for a few reasons:

- We can run compiled code for tests, not interpreted, which would be the case with a GHCi solution
- It is what other test experiences appear to do
- This solution provides a very easy way to get the global test tree (see screenshot below from a sample python project)
- This is, afaict, what IntelliJ and other IDEs do for their test integration. For example, running test1 in the image below incurs the following command line call: `/.../python3.9 ~/.vscode/extensions/.../visualstudio_py_testlauncher.py --us=. --up=*test.py --uvInt=2 --result-port=53890 -ttest.MyTest.test2 --testFile=./test.py`

I think the type class oriented design addresses a use case that I also think is convincing: what if I have a function that could be run, such as an `IO ()` or a function whose arguments have `Arbitrary` instances. In some ways I think the two use-cases are very related, but I think the experience provided by most IDEs would not work with a type class solution for the reasons above…

 ![image](https://us1.discourse-cdn.com/flex002/uploads/haskell/original/2X/2/246dd0ce30356e4c67c82ad76feb28b88f68af21.png)

---

<div class="post-metadata">

**Author:** ![mfussenegger](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/mfussenegger/32/1271_2.png) [@mfussenegger](https://discourse.haskell.org/u/mfussenegger)\
**Post date:** [November 21, 2022, 9:27pm UTC](https://discourse.haskell.org/t/hf-coordination-for-test-framework-ide-integration/5221/19 "2022-11-21T21:27:41Z")

</div>

Just want to mention that at least for neovim there is some integration to run tests from the editor: [GitHub - MrcJkb/neotest-haskell: Neotest adapter for Haskell (cabal-v2 or stack) with Hspec](https://github.com/mrcjkb/neotest-haskell)  
It supports hspec and uses tree-sitter to parse the source and find test definitions and then it calls the test runner executable.  
For other languages it’s sometimes the case that the language server is used and provides off-spec extensions to locate the tests. Or they use code lenses. The client then also triggers the test runner executable

For vim something similar exists too ([GitHub - vim-test/vim-test: Run your tests at the speed of thought](https://github.com/vim-test/vim-test)). Afaik it uses regex to discover tests.

What could help development of such things is a common output format for the test results, to make the parsing easier.

---

<div class="post-metadata">

**Author:** ![santiweight](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/santiweight/32/1187_2.png) [@santiweight](https://discourse.haskell.org/u/santiweight)\
**Post date:** [November 22, 2022, 2:11am UTC](https://discourse.haskell.org/t/hf-coordination-for-test-framework-ide-integration/5221/20 "2022-11-22T02:11:29Z")

</div>

I’ve written an initial proposal for an integration that doesn’t use HLS directly, but instead provides a unified API that testing frameworks can support via the command line.

It isn’t fully fledged yet, but I’d people’s initial thoughts as to whether it makes sense, before I go ahead and try to implement this, which would require support in `tasty` and VSC’s Haskell extension.

> <https://github.com/haskellfoundation/tech-proposals/pull/46>
>
> This is a proposal for integrating various testing frameworks' output format so …that they can be used in IDEs that support Haskell. There is a design for two APIs for test frameworks: \`getTestTree\` which outlines the test tree of tests in a test suite, and \`runTest\` which allows for a test group or test case to be run from the test tree.
> 
> Feedback is very helpful, especially from testing framework maintainers, so that I can better understand what use cases are most important and we don't miss anything important.
> 
> \[Rendered\](https://github.com/santiweight/tech-proposals/blob/ide-test-integration/proposals/accepted/50-ide-test-integration.md)