https://well-typed.com/blog/2023/08/reducing-haskell-parallel-build-times/
21 Likes
Great work! I was wondering if the ideas for chunking up the build pipeline discussed in #14095: Improve build parallelism · Issues · Glasgow Haskell Compiler / GHC · GitLab are also things you’re investigating? The possibility of having typechecking and code generation running in parallel still seems to me like it could make a big difference even for packages with quite sequential build graphs.
2 Likes
This has been implemented for Stack by Ching Pei Yang. However, I am thinking about sensible defaults. Is there any reason not to use semaphore by default, when it is supported (GHC 9.8.1 or later with Cabal-3.12.0.0 or later)?
One remaining issue for cabal-install is https://github.com/ghc-proposals/ghc-proposals/pull/673. If GHC is built against libc (say, it’s a Debian bindist) and Cabal is build against musl (say, it’s an Alpine bindist), the underlying implementations of semaphores will be incompatible. In such scenario Cabal fails hard.
That said, I think that it’s wrong to fail hard and instead a graceful degradation to non-semaphore mode should be implemented. If Stack implements such graceful degradation, there should be no reason not to enable semaphores by default.
1 Like
Thanks! As a Windows user, can I play that back? If I understand correctly, if the ‘official’ Stack for Linux is a statically-linked binary built in an Alpine Linux Docker container (which it is), the ‘system’ semaphores that it creates will not be the ones that the GHC for non-musl Linux distributions expects? If I understand correctly, that sounds like a major problem.
Correct, it is a major problem on Linux.
1 Like
I have yet to find a way to make semaphores work on FreeBSD, so I have to disable the options for the platform in my build systems 
1 Like
I don’t think this is a good design, tbh.
I think the right solution is to develop a GHC daemon. There are some companies interested in that, so maybe it will happen some day.
3 Likes
4 Likes