Caching for incremental changes (including `ghc-nix`)

Learn
1

At my previous Haskell role, we had a workflow where our CI/Production builds were using Nix (using cabal2nix), but the issue is that Nix rebuilds things on the package level, so if one file in a package changes it must rebuild the entire package. This is too slow to be practical for building changes during development.

So for development, I had the following setup:

  1. A cabal.project file with quite strict warning and error options (like -Wall -Werror)
  2. A second .project file, say cabal-dev.project, which import:ed cabal.project, but with a looser set of warnings/errors, in particular, turning off warnings/errors for stylistic things, like unused imports, and also turned off optimisation (i.e. -O0) to speed up builds.
  3. A third .project file, which was called cabal-hls.project, which imported from cabal-dev.project, but had an even looser set of warnings, and did things like turn on -fdefer-typed-holes and even -fdefer-type-errors. I found the issue with the HLS language server with a standard set of build options that minor issues in some modules (like an unused import) would actually break downstream HLS processing on files that imported a file with an issue, so keeping the compile warnings very loose for HLS would allow edited interactivity to continue even if some files had minor compile issues.

So, this set up worked okay, but there’s a few issues:

  1. cabal caches the last build, but if any of the settings change, that cache is unused and overridden. I actually solved that by setting up scripts to build these various versions of cabal.project in different directories, but…
  2. If I change my branch/commit, it would reuse any cached build results for unchanged files, but rebuild anything changed. But then if I change back to the previous branch, it’s build results would then need to be rebuilt even though they were built minutes ago. I could perhaps improve this by caching the build results in different directories based on commit id also, but that would mean whenever I created a new commit I’d need to rebuild everything from scratch which is probably worse.
  3. cabal2nix doesn’t look at cabal.project files but instead just looks at individual .cabal files. So one has to replicate your project based settings in Nix and keep these in sync with the cabal.project settings. Also whilst we tried to ensure the same package versions were being used by pinning index-state: in cabal.project to a similar time/date as our flake.lock, there could sometimes still be slight mismatches.

I understand haskell.nix fixes problem 3, by building the Nix derivations directly from the cabal build plan, but that still leaves the issues with having reasonably fast builds of small incremental changes.

I quite like how Nix solves issues 1 and 2. There’s no need for scripts to put builds in separate directories/hacks based on git commit id, it just reuses what previous results are reusable, no matter how many times you’ve change build options/git branches. The only problem I see that it does this at the granularity of entire packages, not per file.

It really looks like ghc-nix is the solution I’m looking for. It just solves everything, fast incremental builds, no difference between development and CI build processes etc. But it hasn’t been updated in over three years now, and comes with a big warning “this project is still in very early days, so it’s not too easy to use… yet.”.

The other alternative I’ve found is Snack which seems to be a similar offering, but working as a cabal/stack replacement instead of a GHC wrapper. But Snack has been unmaintained for six years.

I was just wondering if there has been any recent work in this space? I quite like the benefits that Nix gives particularly regarding reproducibility and transparent caching, but it’s just too slow in a development workflow when the smallest build unit is at the package, not module level. Or should I just dive into ghc-nix even though it hasn’t had any updates for a few years? Is anyone using it for current projects on a modern version of GHC?

4 Likes
2

There has been more recent work in this space! Please check out GitHub - obsidiansystems/sandstone: Fine-grained Haskell builds with Nix's dynamic derivations (slides: sandstone/planet-nix-2025.pdf at slides · obsidiansystems/sandstone · GitHub ), a demonstration my coworkers and I made about a year ago.

It too has not seen more work in the past year, but I have been work on the unstable features of Nix that is uses behind the scenes. Progress should be continuing on that front too, until everything that Sandstone needs is quite production ready. At that point, I would like to take Sandstone and integrate it into a an alternative backend for Cabal’s new `build-type: Hooks` that works under the same principles — when Sam Derbyshire (no discourse account?) worked on that, he and I had some conversations where I specifically mentioned some things that would help with this eventually happening.

4 Likes
3

I would honestly recommend against fighting with the Haskell-Nix tooling in order to get a nix build behavior that’s ultimately sub-optimal for local development. Even if you get nix build to only recompile files that got changed and their dependencies, that will still be orders of magnitude slower than dropping down to a multiple home units ghci session that just loads the module you’re working on plus the unit test module that tests it. That setup allows you to iterate much much faster than a nix build that would recompile only the changed files (and their dependencies), because:

  1. Loading a module into ghci is MUCH faster than compiling the same module, even with -O0
  2. IME, by only recompiling what’s changed, you’ll still typically recompile half of the project on average, because you’re only pruning modules below you in the dependency tree and not those above you. This is in contrast to the app/lib module + test module :reload workflow I mentioned above, where, if you’re careful with your module dependency tree, you bypass most of the dependencies below AND above you and only reload the modules between your app/lib module and the test module.

So, my recommendation is to keep your Haskell-Nix setup boring and easy to maintain over time and instead channel your local development efficiency efforts to setting up a great REPL environment and REPL-driven workflows.

3 Likes
4

Thinking out loud here somewhat, but I both want reproducibility and really want my local dev and “prod” builds using the same package set.

I guess I could just wrap cabal build or stack build in one big nix derivation, but that has the issue that any change to the code would result in requiring a full rebuild, including of any hackage dependencies.

Perhaps I could solve this though using haskell.nix, which apparently makes the nix build plan based on a cabal/stack one, and they both have the ability to have pins so perhaps that’s solves the out of sync issues?

So going back to GHCI, one of my main issues is of course I’d like my editor to use HLS, and at least with what I’ve found with the VSCode plugin, it just calls cabal on the command line. I have previously done hacks to ensure it calls cabal with some parameters (namely, the “HLS” style .project file I mentioned above), but yes, there’s no need for HLS to actually compile the code, interpreting and typechecking is all that’s needed. Is there anyway to force cabal to use GHCI instead of GHC, or is there another way I should go about getting HLS to work quickly and responsively particularly on a significant codebase?

5

I’m not sure what sort of project size we’re talking about, but IME, HLS was doing fine in projects that had >50k LoC. I’m not sure what exactly it does under the hood, but I’m pretty sure it doesn’t just call cabal to build the project, it probably calls it to get the compiler flags and the dependency tree.

IME, HLS struggles the most when you open two files simultaneously with too may modules in between on the dependency tree. I.e. you’re working on the Utils module that almost every other module depends on, and you have the Main module open at the side, so that means HLS has to reload your entire project every time you make any change in Utils. I always try to avoid doing that. You can easily put yourself into this situation if you have a TestUtils module that all of your tests use and that TestUtils depends on a large chunk of your app/library modules. Then whenever you’re working on tests together with the module you’re testing, you’ll be waiting for the entire project to reload. Employing a little mechanical sympathy for HLS and being mindful of the shape of your module dependency graph goes a very long way.

My default choice for a Haskell project setup is to go with a cabal.project + haskell.nix. Specify your GHC version and the hackage snapshot in the cabal.project and that will make sure you get the same Haskell packages across your cabal build/repl and nix build. If your project depends on tricky system dependencies that you get from nixpkgs, you can always prepare a nix develop/shell environment that brings in just those packages and then run cabal build in that shell.

IMO, the ultimate Haskell development experience is when you set up a way to test the code you’re working on in a very fast ghciwatch loop. In your editor you have the app module on one panel and the test module in the other, with very few dependencies in between and in a terminal you run ghciwatch to run the tests whenever you save, so you get feedback within seconds of saving. Set this up from day one, make sure it stays smooth and neutralize with a taser anyone that tries to take that away from you.

2 Likes
6

So going back to GHCI, one of my main issues is of course I’d like my editor to use HLS, and at least with what I’ve found with the VSCode plugin, it just calls cabal on the command line

IIUC it calls `cabal repl` which will use GHCi under the hood.

another way I should go about getting HLS to work quickly and responsively particularly on a significant codebase

Generally making typechecking faster should help, probably also want to enable multi-repl. See Cheaper: producing a program with less developer time

really want my local dev and “prod” builds using the same package set.

You can easily get a nix-shell which will build all your dependencies with nix (exactly same derivations as your prod build) so that only your actual project, where you’ll be iterating, is built with cabal/ghci, for fast iteration.

You just need your `shell.nix` to consist of a call to `shellFor`. Throw your own local packages into `packages`, then cabal/ghcid/etc into `nativeBuildInputs`. You can also throw HLS in there, but you’ll want to run `code .` inside the shell and tell VSCode to use $PATH instead of installing HLS itself.

Example usage:

shellFor definition: