Tooling improvement // Integrate test watcher (cabal)

From a developer’s perspective one of those basics I need is a watcher that reloads rapidly on any change I make on the code base and reruns all the tests. I dare to say it’s standard for many to develop in such a fashion.

Yet it was surprisingly long and difficult until I finally learned the right command:

ghcid -c 'cabal repl --enable-multi-repl test:sandwich lib:mylib' --test Main.main --warnings

Of course this is a special cabal repl call using ghcid.

Why isn’t this a builtin command within cabal?

What does it take to make it one?

I mean, there is value in language extensions but I feel what people need is stuff like this. Basic things that simplify their development experience.

I hold off from filing an issue of this before understanding the landscape properly. An issue should not be a discussion. It should be a complete requirement.

(Also, I was surprised to see that there is not really a category for this one this discourse).

For the long haul: Which other basic things bring high practical value to the common developer yet don’t exist for Haskell?

I mean, has anyone compiled a table of current languages and their tooling comparing it with Haskell’s?

20 Likes

I think it’s a good idea. The Cabal project has a Proposals process, so I’d encourage you to draft one. I’d be in favour of more integrated tooling within cabal-install, especially since it puts us less at risk of suffering from unmaintained third-party tools (like ghciwatch).

7 Likes

That would be a good candidate for an external command, too.

1 Like

Funny you should ask. I’ve been wondering the same thing.

FWIW here’s my current default invocation, suitable(?) for multi-package projects, including some cargo-culting from a plain ‘ghcid’'s invocation of ghci:

ghcid \
    -c 'cabal repl --enable-multi-repl --repl-options="-fno-break-on-exception -fno-break-on-error -ferror-spans -j -fprefer-byte-code" all' \
    -o errors.err \
    $(git ls-files '*.cabal' '*.hsc' | xargs -Iλ echo --restart λ ;)

It’s the --repl-options that are cargo-culted. I suspect some of them are wrong/unnecessary.

  • -o errors.err gives immediate integration with vim’s :cfile
  • all as target builds every component of every package
  • automatically restart on any .cabal and any .hsc change, which ghcid doesn’t notice.

What this can’t do is run tests. It’s a limitation of --enable-multi-repl, at least for the GHC versions I’ve tried it on so far.

Would love to simplify this somehow…

2 Likes

Yeah, this is one of the more embarrassing ecosystem gaps IMO. I have so many variations of horrible scripts using ghcid and ghciwatch (both of which have their own awkward bugs and limitations), for stuff that really should be pretty simple.

There is quite a big design space, and probably a lot of fixes across various parts of the stack. Plus discussions to be had about which logic belongs in Cabal versus GHCI versus an external tool. It’s all doable. It just needs someone motivated to take it on. Adding support in Cabal itself has been proposed at least once here, so that could be the place to start.

What I personally want is to be able to go in to a big multi-component Cabal project and just run cabal watch 'some compound expression' and have that expression get re-evaluated precisely when GHC detects that its dependencies have changed. (I’d previously wanted a more REPL-based approach, with a :watch command in GHCI, but the main reason for that was not wanting to wait to reload the whole session just to change the expression I want to watch, and that concern should be becoming obsolete due to the work on serialisable bytecode and making Cabal spend less time unnecessarily reconfiguring stuff.)

FWIW, I’m currently working on a file watching tool specifically for Obelisk, with separate REPL processes for the backend and the Wasm frontend. I’ll try to upstream what I can to GHCI etc.

5 Likes

Who is willing to sponsor this financially?

The other side is to get the requirements right.

See also ghc!14440

4 Likes

Haskell Foundation tech proposals are the correct place to request funding for such things (after you’ve of course cleared the details with the Cabal project)

tricorder does this by default. It’s fairly recent, nicely designed (more streamlined feel) and actively maintained.

5 Likes

Does it reload only changed parts of the code like ghci :r would do?

The difficulty with a tool of this nature is that it simply does not scale. You eventually need to build only a selection of the module graph instead of the whole thing (though with the work we’ve done on improving GHC’s memory use, we’re able to load ~30k modules in GHCi, though the GC pauses and reload times are pretty punishing when you’re at 50-100GB of RAM). This is where the --repl-no-load flag can save the day.

Eval comments let you selectively run tests, but it doesn’t scale nicely due to the same GC pauses that make reloading slow also making test running slow.

I’m extremely excited to eventually be able to open source and release all of the work we’ve been doing on the buck2 build system integration with Haskell. Then the magic invocation will simply be buck2 test //... to answer what you’re looking for here - between test caching, module-level parallelism between packages, and test execution being possible before the whole package is compiled, this will significantly outperform any cabal based workflow.

5 Likes

Are there any lessons from this work that cabal could learn? I’m quite interested in our native tooling getting inspiration from elsewhere.

3 Likes

I think the biggest one is that buck2 has a notion of a “target graph” (analogous to the package graph in Cabal terms) and “action graph” (analogous to the module graph in GHC terms). In buck2, we can say “This target has a test suite which depends on these target libraries,” and then it decomposes the test suite target into the actual actions - “build test module” and “run test module” are separate “actions” and you can start an action once its action dependencies are satisfied, even if the target dependencies are not complete yet.

This would allow you to begin running the test suite for a library before the library is done compiling. It also allows you to begin building a library before all of it’s dependent libraries are done.

In Cabal/GHC’s, case, we need cross-package module-level parallelism. I’ve proposed this before, but I’m unsure how far away that is from being realized.

3 Likes

The last Well-Typed activity report mentioned this (“Cabal modernisation” section), and the latest writeup seems to be in this GitHub issue. My reading is that it isn’t imminent, but we’re moving in the right direction. Which is exciting.

9 Likes
  1. Cabal Add
  2. Cabal Glob support in Exposed Modules
  3. Better support for Cabal in HLS
  4. Fixed Documentation Links for HS source code via HLS
  5. GoTo for external modules
  6. Clippy but for Haskell
1 Like

(2) just isn’t gonna happen idt. iirc the justification is that a cabal file in isolation should fully describe a haskell package. with globs, you need IO and the source in-hand

just use cabal-gild tbh. i love that thing.

3 Likes

cabal-gild is a new one to me - very cool! Thanks @Ambrose !

1 Like

I tried tricorder and found a few showstopping bugs, but I think they have already been fixed.

Meanwhile I had an agent choose the latest and greatest repl speed-up options available with ghc >=9.10.3, and I used it to write a cabal-project-aware ghcid wrapper:

#!/usr/bin/env bash

# -T/-r evaluate an expression, which needs actual code. Only a plain
# check-loop can get away with typecheck-only.
code_opt=-fno-code
for arg in "$@"; do
  case $arg in
    -T*|--test*|-r*|--run*) code_opt=-fbyte-code ;;
  esac
done

repl_opts=(
  "$code_opt"
  -fprefer-byte-code         # TH splices load bytecode deps, skipping codegen
  -fno-break-points          # no debugger under ghcid; skip breakpoint ticks
  -j"$(nproc)"
  -fdefer-diagnostics        # group diagnostics after the parallel-load noise
  -ferror-spans
  -fdiagnostics-color=always
  -fwrite-if-simplified-core # restart from interfaces, not source
  -fno-hide-source-paths     # ghcid's output parser needs full paths
  +RTS -A64m -RTS
)

# A separate builddir keeps repl artifacts out of the
# real build's dist-newstyle, so it doesn't compete with `cabal build`.
# THIS SHOULD BE DEFAULT.
cmd="cabal repl --builddir dist-newstyle-ghcid --enable-multi-repl all"
for opt in "${repl_opts[@]}"; do
  cmd+=" --repl-options=$opt"
done

# Restart ghcid if any of the cabal files change.
mapfile -t cabals < <(git ls-files '*.cabal')
restart_args=()
for cabal in "${cabals[@]}"; do
  restart_args+=(--restart "$cabal")
done

ghcid "${restart_args[@]}" --command "$cmd" -o errors.err "$@"

There are still bugs in GHC’s multiple-home-units that break this workflow on some projects I work on, but the future is looking better.

A single repl that has all cabal components loaded at once has been something I’ve wanted since the first month I started using Haskell professionally. Stack has provided it in a hacky form since ~2015. Excited that it might become the ecosystem-wide-default soon.

2 Likes

In the meantime, I made a somewhat less sophisticated but fully functional Buck2 integration that folk can use today. Here: GitHub - simonmar/haskell-buck2: Buck2 build system for Haskell projects · GitHub

It builds libraries using ghc --make like Cabal, so it doesn’t have the cross-package parallelism that the Tweag/Mercury Buck2 integration does. But it does support enough that I was able to build Glean with it, and it’s vastly preferable to rebuilding with Cabal.

5 Likes

If not cross package parallelism, what makes it better than cabal?

1 Like