GHC build server for optimizing one-shot compilation

#1

WARNING: Dumb question alert!

I’ve seen in various places that building a whole project with ghc --make is faster than compiling files one-at-a-time with ghc -c because ghc --make can keep information in memory across modules. Isn’t one possible solution to have a long running server that keeps things in memory, and you can submit requests to compile files one-at-a-time? Is there an existing proposal for this?

Random thoughts:

  • This is separate from the new -jsem functionality, right? -jsem is for ensuring parallel builds don’t consume all the host memory, but it doesn’t help the parallel builds actually share in-memory information
  • Is this related to HLS? (I don’t use HLS, so maybe this is the whole purpose for HLS)
  • I’m semi-familiar with the gradle build system for Java projects. I believe it spins up a server when you first run a gradle command; is that build server similar to this idea?
5 Likes
Hécate's Crackpot Ideas: A compilation server for GHC
#2

The rules_haskell for bazel build has a worker implementation in experimental mode Common Haskell Build Use Cases — rules_haskell documentation .

Source: https://github.com/tweag/rules_haskell/tree/master/tools/worker

1 Like
#3

ghc -d is one of the items on my todo list :sweat_smile:

I fully agree that it would be great to have!

3 Likes
#4

@pranaysashank Cool! Thanks for sharing!

@angerman That looks promising! Are there any issues, proposals, or conversations I could track for that?

#5

@brandonchinn178 I’m afraid it’s just a bunch of “this would good to have and spend some time on” notes :frowning: there is one reason where this is tricky, getting this to work in nix across derivations could be messy and would likely require some kind of sandbox exception.

#6

…?

#7

Thanks @atravers, I should’ve mentioned ghcid too. I don’t think that’s what I’m looking for either, it looks like it’s a long running ghci server, not a long running ghc compilation service. That is, I’m not looking for an interactive “show me errors as I work” service, but rather a “run ghc on these files to cache info as I build” service

#8

GHCi can also be instructed to compile modules as usual with -fobject-code:

# cat SomeHaskellProgram.hs
main = putStrLn "Here it is!"
# ls SomeHaskellProgram.*
SomeHaskellProgram.hs
#  ghci -fobject-code SomeHaskellProgram.hs
GHCi, version 9.4.4: https://www.haskell.org/ghc/  :? for help
[1 of 2] Compiling Main             ( SomeHaskellProgram.hs, SomeHaskellProgram.o )
Ok, one module loaded.
ghci> :q
Leaving GHCi.
# ls SomeHaskellProgram.*
SomeHaskellProgram.hi  SomeHaskellProgram.hs  SomeHaskellProgram.o
#

(…adapting ghcid to work in this manner is left as an exercise.)

#9

Yes, I think this would be an excellent idea. I think it would be sensible to spawn one worker per package build but that should still save a significant amount of time on typical workloads.

3 Likes