# Modularizing GHC, do we want an HFTP?

**URL:** <https://discourse.haskell.org/t/modularizing-ghc-do-we-want-an-hftp/4022>\
**Category:** Uncategorized\
**Created:** [February 3, 2022, 5:51pm UTC](https://discourse.haskell.org/t/modularizing-ghc-do-we-want-an-hftp/4022 "2022-02-03T17:51:10Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ericson2314](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/ericson2314/32/1026_2.png) [@Ericson2314](https://discourse.haskell.org/u/Ericson2314)\
**Post date:** [February 3, 2022, 5:51pm UTC](https://discourse.haskell.org/t/modularizing-ghc-do-we-want-an-hftp/4022/1 "2022-02-03T17:51:10Z")

</div>

@hsyl20 and, more recently, @doyougnu have done excellent work untangling GHC, in particular banishing `DynFlags` and `HscEnv` from as many modules as possible. Many of the refactors are quite hard, not because they are necessarily _technically_ advanced, but because GHC is a labyrinth that would make Thesus blush, and so breaking down this task into manageable chunks requires great wisdom and patience.

However, once the plan is made, executing on it isn’t always so hard. I just stumbled upon a module at random and did [GHC.HsToCore.Coverage: No more HscEnv, less DynFlags (!7467) · Merge requests · Glasgow Haskell Compiler / GHC · GitLab](https://gitlab.haskell.org/ghc/ghc/-/merge_requests/7467/diffs), which I’d say is an exceptionally _unimpressive_ patch:). It is 100% rote “cinching upstream” projection from `DynFlags` and `HscEnv`, whenever it is easy to do so, in just one module. Now it may not always be this easy, but if I just stumbled into this, I bet there is a few more cases like it!

I know @hsyl20 is doing some writing to better explain what’s going on, and I don’t want to upstage that / his own planning more broadly. But due to the interest in [https://github.com/haskellfoundation/tech-proposals/pull/24](https://github.com/haskellfoundation/tech-proposals/pull/24) and [Pre-HFTP: Coordination for finishing Trees That Grow](http://discourse.haskell.org/t/pre-hftp-coordination-for-finishing-trees-that-grow/3999), I thought it would be good to mention this as the 3rd such major “Cleanup GHC incrementally” task I can think of.

---

<div class="post-metadata">

**Author:** ![rae](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/rae/32/888_2.png) [@rae](https://discourse.haskell.org/u/rae)\
**Post date:** [February 3, 2022, 9:47pm UTC](https://discourse.haskell.org/t/modularizing-ghc-do-we-want-an-hftp/4022/2 "2022-02-03T21:47:49Z")

</div>

While I would welcome modularizing GHC and would actively support such an initiative, I wonder whether our limited resources are better spent working on issues with immediate impact on users. That is, modularizing GHC has the potential to be transformative, in that it will open up the possibility of new plugins, new compiler components, a way of exploring new concrete syntax, etc. But these will take years to bring to reality in a way that will benefit users. Instead, supporting e.g. [https://github.com/haskell/error-messages/](https://github.com/haskell/error-messages/) or improving cabal’s UX or building up a knowledge library of common Haskell tasks (e.g. performance debugging best practices and techniques) would be much more helpful in the short term.

Put more simply: I would love for the HF to work on items that will deliver wins to our users in the short term, rather than focus on internal improvements that would yield fruit only in the long term.

---

<div class="post-metadata">

**Author:** ![Ericson2314](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/ericson2314/32/1026_2.png) [@Ericson2314](https://discourse.haskell.org/u/Ericson2314)\
**Post date:** [February 4, 2022, 5:30am UTC](https://discourse.haskell.org/t/modularizing-ghc-do-we-want-an-hftp/4022/3 "2022-02-04T05:30:06Z")

</div>

For sure, I think we should first do your structured errors proposal, and after that any trees-that-grow proposal we end up approving, since we do have concrete end-beneficiaries in mind with a decent time table.

However, after those two projects, I do think it makes sense to start this.

First of all, at that point, with two successful projects under our belt (if one fails, obviously we’ll reconsider), we would have significant momentum onboarding volunteers for GHC, and it would be good to channel that momentum somewhere useful.

Second of all, much of the work is extremely rote. I just rattled off another [Purge DynFlags from GHC.Stg (!7479) · Merge requests · Glasgow Haskell Compiler / GHC · GitLab](https://gitlab.haskell.org/ghc/ghc/-/merge_requests/7479) in ~2 hours, and I suspect, say, students could do the same in not more than ~ 2 days after being shown the ropes.

Improving Cabal’s UX and error messages are laudable goals that would make a lot of end users excited, but the goal is somewhat nebulous, let alone the steps to get there.

For Cabal/Hackage in particular, I think we we need to set up the equivalent of `ghc-proposals` and the GHC steering committee before doing anything. Treating the package manager / package description language as less intellectually rigorous work than the compiler / main language is, I believe, a serious mistake just about every language community makes, and how each such community got into these messes in the first place.

I mention the rote-ness / clarity of the tasks, because I think it cannot be said enough that the often the hardest part of a programing task is just knowing what to do. And so modularizing GHC, as daunting as it sounds, actually has some huge advantages. Really all 3 GHC projects do, because they continue existing lines of work where we’ve already figured out what idioms work.

Finally, because of these advantages, I think the time-line for end-user benefit should be not more than a year, because making GHC as a library not terrible to use for the first time ought to turbocharge HLS development across the board.

---

<div class="post-metadata">

**Author:** ![mpickering](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/mpickering/32/4585_2.png) [@mpickering](https://discourse.haskell.org/u/mpickering)\
**Post date:** [February 4, 2022, 9:27am UTC](https://discourse.haskell.org/t/modularizing-ghc-do-we-want-an-hftp/4022/4 "2022-02-04T09:27:12Z")

</div>

> [@Ericson2314](#):
>
> Finally, because of these advantages, I think the time-line for end-user benefit should be not more than a year, because making GHC as a library not terrible to use for the first time ought to turbocharge HLS development across the board.

It seems the assertion which powers this proposal is that this will make developing with the GHC API easier.  
However, I don’t think that’s entirely obvious, when using HLS you always have a HscEnv in scope so it is just more annoying to have to cast it to more particular data types. I agree with Richard about short-term obvious improvements are better to prioritise.

---

<div class="post-metadata">

**Author:** ![Ericson2314](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/ericson2314/32/1026_2.png) [@Ericson2314](https://discourse.haskell.org/u/Ericson2314)\
**Post date:** [February 4, 2022, 4:02pm UTC](https://discourse.haskell.org/t/modularizing-ghc-do-we-want-an-hftp/4022/5 "2022-02-04T16:02:59Z")

</div>

When you want to do “just what normal compilation would do”, just passing a `HscEnv` and `DynFlags` is easy and fine.

When you want to leverage GHC API to do something novel, like say a refactoring tool, etc. it’s hell on earth. The presence of these giant config objects + the history of GHC being an executable, means we have [configuration over composition](https://johno.com/composition-over-configuration), which is exactly backwards and wrong, and the opposite of how we normally do things in Haskell.

Really, splitting up the config is just the first step to allow _locally_ restoring these compositional principles. This can be similar to Norman’s “refunctionalizing” in that we see how various `Bool`s are used, mentally construe their elimination as two functions of the same type, and then pull out a function parameter to replace the boolean.

These sorts of refactors our more “art” than “science”, though, so I wouldn’t want to do this follow-up work as part of the HFTP. I am happy to let people like @csabahruska and @isovector doing interesting things with the GHC API (and example of with and without HLS) to shave these sort of yaks on an as-needed basis, which should become possible for the first since merely splitting the config allows not-super-painful local refactors.

* * *

Stepping back a bit, I concede right now there is enough controversy with you not yet convinced to bring down any would-be HFTP. But as I wouldn’t propose starting this until after the other tracks of GHC work are attempted, I hope there is plenty of time to sort that out.

In particular the document I mentioned in the first should cover all this in much more detail, and hopefully push the GHC-side conversation to some sort of conclusion one way or another.

---

<div class="post-metadata">

**Author:** ![doyougnu](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/doyougnu/32/2248_2.png) [@doyougnu](https://discourse.haskell.org/u/doyougnu)\
**Post date:** [February 8, 2022, 6:56pm UTC](https://discourse.haskell.org/t/modularizing-ghc-do-we-want-an-hftp/4022/6 "2022-02-08T18:56:43Z")

</div>

I view these modularization efforts _as_ performance and test improvements. In particular I think a more modular code base would:

1. lower the cost of code optimization
2. lower the barrier of entry for contributions (this is a force multiplier)
3. increase the surface area of the testsuite
4. reduce the number of possible errors in ghc development

Re: 1  
One of the big takeaways I had from the performance internship at tweag was that simple, and obvious performance wins (in the spirit of the low-hanging fruit issue #18541), such as using more sensible data structures become difficult \_because) of the anti-modularity in the code base. Ideally we would be able to work on performance improvements to only phase `Foo` and then have a single `fromList` and `toList` around the boundaries of `Foo`. Then if the perf results for phase `Foo` look good we could tackle the next phase upstream or downstream modularly. Unfortunately, due to the amount of coupling performance changes like this (very invasive) are hard because the merge request suddenly balloons to touching 50+ modules across the compiler. Furthermore, this makes it hard to detect a clear performance win because we might make phase `Foo` faster, but at the cost of offloading the deserialization to a downstream phase `Bar`, which then dwarfs the perf win even though in the long run a different data type is the right decision in the long term.

Re: 2  
If we had a more modular code base then performance optimization project plans which proceed by phase (similar to [this project plan](https://gitlab.haskell.org/ghc/ghc/-/issues/20730)) are easier to write, easier to implement, _and_ easier to contribute to. One could imagine a set of new contributor issues that are tagged `perf:data_structure_swap` and then a project plan that’s simply `remove lists in phase Foo for sequences/trees/whatever`. Thus we could elicit community contributions more easily and preserve the value `work potential` we have.

Re: 3  
Similarly, we should be able to write tests that are independent of the `DynFlags` or `HscEnv`. For example in !7325, by removing `DynFlags` and creating `StgToCmmConfig`, and isolating `Backend` information we can now write tests we previously [could not write](https://gitlab.haskell.org/ghc/ghc/-/merge_requests/7325#note_402836). If this trend continues then the test suite would catch more errors and therefore we would accumulate less technical debt as development happens. All of this means the core developers can spend less time putting out fires (hello windows CI) and more time on user facing issues.

Re: 4  
The modularization efforts _should_ make more error states obvious. Which, in turn, should mean that we can encode more constraints into the type system to make those states not representable. This is, I think, a major point missing from the current conversation and is (I think) one of the motivations behind !7442 and phase 2 of #20730. But the idea is with a more modular compiler we can have a series of handshakes that pass more information into the type system. This should make error states harder to represent and manifest, and remove the pervasive use of runtime guard checks (and thus increase performance and branch prediction!).

I think the points @rae bring up are good and I am all for the better error messages proposal (and I do think it should have priority over modularization), and I do understand the drive to preserve the ghc developer time (this is the right question to ask for a new project after all!). But I do think that the modularization efforts are worth the effort because they lead to performance improvements, better testing, more correctness and at least one possible force multiplier.

PS: new users can only have 2 links in their posts. How annoying!

---

<div class="post-metadata">

**Author:** ![romes](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/romes/32/2912_2.png) [@romes](https://discourse.haskell.org/u/romes)\
**Post date:** [February 8, 2022, 7:43pm UTC](https://discourse.haskell.org/t/modularizing-ghc-do-we-want-an-hftp/4022/7 "2022-02-08T19:43:00Z")

</div>

> 1. lower the barrier of entry for contributions (this is a force multiplier)

As a wanna-be new contributor I second this. I’ve been looking through the modularisation and e.g. the TTG paper and following the threads and am enthusiastic about it. It seems to me like there are underway many things appealing to newbies. (EDIT: this may be out of context but) Additionally, if `template-haskell` shared the AST with GHC, I believe more people would be exposed to the internals “for free”, and more readily jump onto them.

---

<div class="post-metadata">

**Author:** ![AntC2](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/antc2/32/1872_2.png) [@AntC2](https://discourse.haskell.org/u/AntC2)\
**Post date:** [May 4, 2022, 3:55am UTC](https://discourse.haskell.org/t/modularizing-ghc-do-we-want-an-hftp/4022/8 "2022-05-04T03:55:17Z")

</div>

> [@Ericson2314](#):
>
> labyrinth that would make Thesus blush

(The guy who survived the labyrinth is spelled ‘Theseus’.)

You might also want to consider the myth(?) of the [Ship of Theseus](https://en.wikipedia.org/wiki/Theseus#Ship_of_Theseus).

> To preserve the ship, any wood that wore out or rotted was replaced; it was thus unclear to philosophers how much of the original ship remained, giving rise to the philosophical question of whether it should be considered "the same" ship or not.

Applies to maintaining complex software systems. Is any part of today’s GHC actually 30 years old?

---

<div class="post-metadata">

**Author:** ![ParetoOptimalDev](https://avatars.discourse-cdn.com/v4/letter/p/b782af/32.png) [@ParetoOptimalDev](https://discourse.haskell.org/u/ParetoOptimalDev)\
**Post date:** [May 8, 2022, 9:25pm UTC](https://discourse.haskell.org/t/modularizing-ghc-do-we-want-an-hftp/4022/9 "2022-05-08T21:25:36Z")

</div>

> [@rae](#):
>
> or improving cabal’s UX

FYI I’ve stopped submitting UX issues to cabal-install because only UX issues that align with subjective interests of maintainers seem to get through.

Don’t want to derail the thread, but feel free to verify what I say by looking through the issue tracker.

_More on topic_ though might be how the friction like this and other places dictates where people contribute.

---

<div class="post-metadata">

**Author:** ![jaror](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/jaror/32/3271_2.png) [@jaror](https://discourse.haskell.org/u/jaror)\
**Post date:** [May 10, 2022, 1:12pm UTC](https://discourse.haskell.org/t/modularizing-ghc-do-we-want-an-hftp/4022/11 "2022-05-10T13:12:59Z")

</div>

5 posts were split to a new topic: [Cabal UX discussion](https://discourse.haskell.org/t/cabal-ux-discussion/4502)

---

<div class="post-metadata">

**Author:** ![Ericson2314](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/ericson2314/32/1026_2.png) [@Ericson2314](https://discourse.haskell.org/u/Ericson2314)\
**Post date:** [May 9, 2022, 2:56am UTC](https://discourse.haskell.org/t/modularizing-ghc-do-we-want-an-hftp/4022/12 "2022-05-09T02:56:05Z")

</div>

Heh I forgot it was the same Theseus who did both.

> [@AntC2](#):
>
> > To preserve the ship, any wood that wore out or rotted was replaced; it was thus unclear to philosophers how much of the original ship remained, giving rise to the philosophical question of whether it should be considered “the same” ship or not.
> 
> Applies to maintaining complex software systems. Is any part of today’s GHC actually 30 years old?

Ah, so here’s the thing, the extent to which it _is_ the same ship is in many ways precisely the problem!

Per “Domain-Driven Design” even if every part of the ship is replaced, if those renovations are too incremental and short-term-ist, the design rush the risk of becoming an incoherent hodgepodge. A lack of “ubiquitous language” is like saying “rerig” once the sales have been replaced with coal boilers, or “full steam ahead” once the coal has been replaced with diesel internal combustion engines.

Short term slap-dash renovations cannot take advantage of the knowledge that eventually all the ship will be replaced, but they _must_ deal with the fact that most of the ship is not _currently_ being replaced.

I’m describing the general old software tendencies (slap-dash is rather harsh for GHC) but you get the idea.

---

<div class="post-metadata">

**Author:** ![Ericson2314](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/ericson2314/32/1026_2.png) [@Ericson2314](https://discourse.haskell.org/u/Ericson2314)\
**Post date:** [May 21, 2022, 9:27pm UTC](https://discourse.haskell.org/t/modularizing-ghc-do-we-want-an-hftp/4022/13 "2022-05-21T21:27:42Z")

</div>

Two bits of news

1. The [GSOC project](https://summerofcode.withgoogle.com/programs/2022/projects/WWgHBTde) was accepted!

2. We made a chat room, [https://matrix.to/#/#ghc-modularity:matrix.org](https://matrix.to/#/#ghc-modularity:matrix.org), to coordinate without flooding #ghc.