The evolution of: Decoupling base and GHC

#29

…yeah, why bother with abstraction when there’s only one mainstream implementation of Haskell?

As for potential solutions:

  • if it’s a RTS primitive which is calling a base entity, there may be two options:

    • if the primitive is rarely used, you could try passing in the base entity as an extra argument;
    • if it’s widely used, another option is to replace the RTS primitive with a new base declaration via the FFI - it may still be easier to add that extra argument, and ignore the complaints…

  • if, for some reason, it’s some part of the fundamental workings of the RTS that make the call to base…let’s just say “tricky” is being diplomatic:

    • If the offending part can be easily isolated, you may be able to replace it as well using the FFI;
    • Otherwise, being lazy is probably the simplest option - log an issue report with the tracker for GHC, and make a note of the miscreant in your progress report.

…progress report? Yes: for a good example of their use, see Philip Heron’s series of reports about Rust-GCC - most other requests for updates can then be silently deleted (ideally automatically!). You can then more easily focus on this task, which will require plenty of your attention.

Those progress reports will be invaluable e.g. if this topic is still being debated in, let’s say, 2027 people can just be directed to read them to find out what happened…

#30

@AntC2 you are most welcome to elaborate on these statements. You seem to be under an impression that CLC has an articulated opinion on the matter, but I have no clue what makes you think so.

3 Likes
#31

I said: I thought that …, I could prod people to articulate the why

Because the impression I’m under is that the CLC does not have an articulated opinion on the matter, so is failing to give guidance where leadership is clearly needed. I have no clue what makes you think I’m seeing articulated opinions.

Those quotes from me you snipped are pretty close to links and observations. But in case you missed it …

And I see @Ericson2314 got there before me: “we get these massive threads”. And that was barely a third of the way through that thread. (Addit: And after plenty of discussion on the mailing list.) At the half way point the Committee decided/closed the issue. Then there were requests to re-open and more rounds of discussion – which came to no conclusion I can see.

So somebody thought it a good idea to have Principles that would guide future discussions. (yes please!) A thread of 63 comments so far. No conclusion I can see.

Guiding Principles are the sort of thing a Committee should be there for. The CLC’s Charter says there will be Controversial Decisions. (d’uh, well yes) And then … ?

Specifically where I came in on Decoupling base from GHC wired-in, there’s 42 replies on the original thread (which went off track) and 108 on the newly-spawned attempt to corral it. I see no criteria for how the Committee might evaluate any proposal. I’d expect the criteria to derive from the general Principles guiding assessing impacts of breaking changes vs expected benefits.

I get it that Discourse isn’t under the aegis of CLC; and that the bulk of comments on those GitHub Issues are not from Committee members. If the CLC had some guiding Principles, they could shape the discussion to tone down apocalyptic language and sweeping unquantifiable claims (like @chreekat’s “blast radius for changes of any sort is decreased” – of any sort?).

There’s been mutterings about the design of base all the time I’ve been following Haskell. I’ve seen (versions of) this wishlist plenty of times. Nearly every attempt at it has faltered – with the conspicuous exception of the bruising ‘FTP changes’.

@Ericson2314 is about to do what? Split out the GHC wired-in parts and their dependencies? Revive nomeata’s work from 2013? “Rip base in two” along its natural fracture points?

I get the impression @Ericson2314 is not doing (whatever it is) for his own entertainment. It sounds like a lot of work. Whatever results, is the Committee just going refuse point blank – because it’s too controversial or causes too much breakage? If there is to be a benefits vs cost assessment, what are the metrics for benefits? As opposed to chreekat’s ineffable claims.

Your interventions in these threads have generally been at the ultra-conservative (small-c) end:

It’s difficult to imagine slower than the pace of failing to make a decision about removing (/=) from Eq. For the sake of managing potential contributors’ expectations, what does “slower” mean? And is (whatever that means) the collective view of the Committee?

1 Like
#32

I am rather bemused by the whole idea a set of sourcecode (the libraries) could be insulated from the very compiler they rely on. If GHC (the compiler) changes – even with a tiny bugfix release – all the object code for those libraries must get regenerated.

Of course GHC developers are meticulous to avoid side-effects of changes; and to document everything that’s changed. Nevertheless, prudent software release practice is to trust nothing and re-run the whole test suite against the re-compiled application and libraries.

Reorganising base to minimise dependencies won’t remove the chief dependency: on the compiler. So how could changes to libraries be more conservative (small-c) than changes to the compiler?

Upgrading a compilation environment is different to upgrading an application environment, in which the asset to protect is its database content and continuing business operations that query and maintain it.

#33

I don’t agree. Changes in the compiler should only affect a tiny fraction of the language and I think that has historically been the case (at least recently), so it isn’t necessary to consider adjacent compiler versions to be completely different. Of course you should still re-run test-suites and otherwise check that the changes don’t break things, but I don’t see how that makes it impossible to split base.

4 Likes
#35

@antc2 The splitting of base is, strictly speaking, a mere implementation detail — it doesn’t even require CLC input if we were were to go with scary sounding ghc-internal-* libraries.

I do, in fact, want the split base libraries to be intended for public consumption, so I will in fact engage the CLC on what portions of the interfaces look nice stabilized separately, but I don’t anticipate this being very controversial.

In particular, even if I somehow get the split all wrong, it is very easy to turn those libraries into ones that reexport stuff defined elsewhere. Ergo, I see very little costs in terms of stability and tying our hands in the future with this stuff.

I am rather bemused by the whole idea a set of sourcecode (the libraries) could be insulated from the very compiler they rely on.

This is funny coming from someone who likes HUGS and misses the era of a diversity of implementations and perspectives!

Code is strongly linked to the interface of a compiler, but not its implementation. What we call base–GHC coupling is still just each observing the “unabridged, private” interface of the other.

Interfaces usually have some sort of subtyping / partial order structure. The goal is to fact things nicely to rely on as little private stuff as possible.

I would encourage you to check out Rust’s core vs std split, where std is far bigger but also far more normal a library, with very few wiring-in points in either direction. Nothing I am hoping for with the split isn’t pioneered “in production” over there already.

6 Likes
#36

Precisely because I value a diversity of features, and am tolerant of churn at the bottom of the tech stack (the compiler), I expect everything built over it will get wobbly from time to time. What I don’t expect is you could mix-and-match different versions of base with different releases of GHC. As if you could pull-push blocks in-out of the bottom of the Jenga tower, and keep the upper storeys somehow suspended from a sky-hook.

As I thought I made clear, I reverted to HUGS not because I’m an antedeluvian, not because GHC is moving too fast or is too unstable, but because GHC is moving sideways relative to where I want to go. And because I found I could understand and tweak the innards of HUGS.

1 Like
#37

What I don’t expect is you could mix-and-match different versions of base with different releases of GHC.

Why not? Perhaps we need an example. Let’s pick something boring from base, hmmm, Bifoldable. Suppose we began our base split by putting just Bifoldable in base and everything else in ghc-base (being the library shipped with GHC).

Now, could we build this base with different versions of GHC? I don’t see why not. It’s just a typeclass, why wouldn’t it compile with various versions? Maybe changes in GHC would make it not compile, and we solve that the way we do for any other library: ugly CPP. But it would work.

Similarly, we could change Bifoldable (add another instance, say), release a new version of base, and expect that to work with old GHCs too. Even GHCs from before we did the release! Radical!

Now: lots of stuff in base is in the same position as Bifoldable. Some stuff isn’t, for sure, and that’s why we’re going to be stuck with some kind of ghc-base thing. But that doesn’t mean it’s somehow impossible to remove anything from base and put it in a more “normal” library.

9 Likes
#38

Maybe one challenge in this conversation is the name? That is, we want a library of standard helper functions / data structures that everyone will want access to. I’ll call it standard-goodies. The key aspect of standard-goodies is that it is just a library – no particular cozy relationship with GHC. Of course, standard-goodies could be upgraded independently of GHC. The conversation in this thread is about taking stuff out of a library whose existence is tied to GHC and putting more into standard-goodies.

6 Likes
#39

We already have plenty of libraries of standard helpers such as relude and other alternative preludes. They indeed can be upgraded independently of GHC and strive to provide a stable interface across several GHC versions. This is a solved problem.

2 Likes
#40

Keep in mind the GHC base dependencies go both ways. Trivially, we do know how to write Haskell code that doesn’t depend on GHC secret interfaces – that is most Haskell! There is also the case of GHC depending on regular Haskell code, like Maybe, List, and Monad, definitions that it needs to know about for various reasons.

When we don’t hard code GHC to look up these wired-in definitions in base, but instead do something more flexible, untangling the knot gets easier.

C.f Rust’s “lang items”, some of which could be defined any library (that uses unstable features).

#41

Thank you Richard, yes that’s slowly becoming clear. I named this thread by smurshing together names from two threads, both of which seemed to lack direction or specifics. In my head, “GHC” means the compiler and language extensions it supports. So a thread ‘Evolution of GHC’ I expect to be about language and extensions. Turns out that wasn’t what was meant.

base or Prelude are (in my head) independent of any particular compiler. The Prelude is specified in the Language Report, which is compiler-agnostic. Once upon a time we could talk of " A substantial collection of library modules that are supplied with Hugs, GHC and Nhc98."

And the criteria for base was " the Prelude and its support libraries, and a large collection of useful libraries ranging from data structures to parsing combinators and debugging utilities." IOW stuff closely aligned to compiling and supporting developers running a compiler.

Thank you to @michaelpj for (at last!) providing a concrete example. So my thrashing around in these threads can boil down to one question: why on earth is Bifoldable in base? It’s not required to be from the Language Report. (For example it’s not in Hugs’ Hierarchical Libraries.) It wasn’t in GHC base until about 5 years ago. It doesn’t provide support to compiling or debugging tasks that I can see. It seems eminently suited to be in standard-goodies.

Just move Bifoldable and its ilk out of base (after the usual three-release deprecation cycle of course). Put it on Hackage. Job done. By all means create standard-goodies as an empty shell that just imports stuff that’s been moved out of base.

#42

And yet … The run-of-the-mill trickle of proposals on the libraries mailing list seems to consist only of adding things to base: extra/specialised methods, optimising constraints on existing classes, tweaks to help performance. Progressively making base more tangled and interdependent. Until somebody calls a halt.

What’s special about 'adding to base’ that people are so keen on? To contra @michaelpj’s “Why Not?”: If you want a Bifoldable to work across different releases of GHC-the-compiler

  • Why not copy the source to MyBifoldable and use import/module naming to insulate yourself from base's Bifoldable? Or
  • Why not put on Hackage as NotbaseBifoldable and ditto ditto.
  • Why not help yourself rather than waiting around wringing your hands that base is a mess (allegedly)?

OTOH, Bifoldable hasn’t changed in the five years since it was added to base. It hasn’t (so far) been a source of instability – despite its import Control.Applicative (amongst others) and that’s import GHC.Prim etc that (presumably) could never be moved out of base or ghc-base or ghc-internal-*, depending how we skin the cat.

Control.Applicative and GHC.Prim have changed; if those changes have been a source of instability for Bifoldable, no amount of reorganising base will distance you from that import.

So do you expect that by being in a blessed library named base, it’s the GHC team’s responsibility to make Bifoldable in release 9.10 compile and run unchanged under GHC-the-compiler release 9.12? Has anybody asked the already-overworked GHC team whether they have capacity to regression-test every release of GHC-the-compiler against the last three base releases?

Or do you expect it to ‘just work’ without testing? Because why?

I’d call it “magical”: test a compiler release against a release of a library that hasn’t been developed yet. Please tell me when GHC produces that time machine. Oh! you mean when the library is developed, ask the GHC team to cast their mind back to the compiler-as-was-9-months-ago and regression-test.

#43

why on earth is Bifoldable in base? It’s not required to be from the Language Report

The report doesn’t mention base, or packages at all does it? At least, I couldn’t find any such mention after a quick browse.

#44

Or alternatively, test the new library release against compiler releases that have been developed.

2 Likes
#45

While we do have those alternatives already, it isn’t really a solved problem, b/c when we onboard new users, they don’t use those alternatives, they use what GHC and the community “ship”. This is also why the stack/cabal duplication is still a problem for onboarding - newbs don’t know how to make that choice effectively, and there’s no authoritative guidance from an authoritative voice in haskell (haskell.org for example) specifying what we should do.

2 Likes
#46

Maybe we can use a few possibilities as example inputs for a set of thought experiments (guided by the particulars of the situation in the example)?

Caveat: For each of these, I would guess there is some disagreement that there is even a change to make - I assume they would have already been addressed otherwise. But either way, I think it helps to map out what a transition would actually look like, what problems there are to overcome and smooth out.

Here are a few I see discussed from time to time:

  • Fixing bad default functions which lead to prominent mistakes and flaws in production software, foldl, head, lack of strictness, etc.

  • Adding new types to replace bad defaults - vector type to replace list as a default, for example.

  • Better default guidance and support in base WRT String / Text / ByteString (or list/vector, etc).

  • Introduction of and transition to Dependent Haskell or a Dependently Typed Haskell, or in some way running the Haskell community (hackage/package ecosystem) through the DT simulation game to figure out what that might all look like.

For each of these types of secenarios, what would the deprecation/transition/migration look like?

And how might we reduce the human/time cost, and result with significant improvements, with more correctness overall for the majority of production use cases?

Another observation from these discussions: the community already has many examples or attempts of fixing these issues and “doing it right” - alternative preludes, documentation about problems with the “standard library” and how to work with those, packages with functions/types we need, but integrating these solutions into the “core / shared” domain has not been allowed, accepted, or followed-thru with to completion. While I support the idea of decoupling base/GHC, I also don’t see how this decoupling will address any of the issues which lead to the failure of those previous attempts to improve the common domain.

Lastly, and on a similar note, how do we break the cycle so we don’t continue repeating ourselves in this circle (insert broken record reference).

3 Likes
#47

I’m probably being naive here, but if we really want GHC to be decoupled base, i.e the ecosystem and compiler can evolve mostly independently, isn’t the solution to just total isolate GHC from the rest of the Haskell ecosystem?

If GHC had been implemented in another language, it wouldn’t have been able to import other libs from hackage, and other packages wouldn’t be able to import GHC code.

It seems to me like if we really want GHC to be decoupled, we need to duplicate all the code it depends on from the ecosystem, and keep/maintain its own copies of that code internally. Then we need to make it not available to the rest of the ecosystem so normal libs can’t depend on GHC’s internals at all. And vice-versa, duplicate base out of GHC into its own repo, and duplicate any dependencies on GHC’s internals to be maintained separately of GHC.

I know this is a super naive perspective here, likely riddled with many issues that I’m totally ignorant of and probably way easier said than done, but it feels to me like this is the basic gist of how to fix the issue. For as naive as it is, it feels the simplest to me too :sweat_smile::innocent:

Summing it up it one phrase:
Nobody should know GHC is implemented in Haskell, not even GHC itself!

2 Likes
#48

@vanceism7 GHC being implemented in Haskell does have some costs (and benefits), but that is largely unrelated to why GHC and base are so coupled.

Rather, it’s because the code that GHC generates.

  1. base is implemented with knowledge of all sorts of secret things about what GHC does. does is the key word, this is stuff like secret interfaces like what the primops are, etc. etc. If GHC was written in another langauge all those same secret interfaces would exist. They do for any language that has primitive definitions.
  2. Generated code assumes many random things are found in base and base alone. Again this doesn’t matter how GHC is written, for compiler in another langauge could also generate code assuming stuff is in base. Indeed, the RTS is in another langauge (C) and also assumes the existence of some base stuff.

Most language implementations have grown organically very gnarly implementation—standard library interactions. Libc is nasty too. C++ STL library in freestanding mode is also sort of CPP hack. Rust however proved that you can actually layer this stuff nicely and indeed rustc is written in Rust.

(They in fact insist the compiler is always built against the same standard library, so they have long done some CPP to make the standard library work with the current and previous versions.
I think that last bit is silly: the compiler shouldn’t require bleeding edge standard library features. But the principle of layering and decoupling is very good!)

Rust doesn’t do any magic to accomplish this. In fact, defects in the design of Cargo, and the lack of anything like Backpack make it harder for them than for us! It’s simply a matter of desiring the same elegance of abstraction and division of labor in the standard library as every other library, and spending the time to make it happen.

I believe in addition to standard libraries exposing top-quality beautiful interfaces, they should also be implemented in top-quality beautiful ways. The standard library often is the user’s first impression of the language ecosystem, and so it should make a good first impression. But the user impressed and enamored with the interfaces and thus enticed to peak behind the current shouldn’t have their honeymoon wrecked by seeing a rats nest.

Beautiful turtles all the way down alone impresses the user that the principles of the community are in fact sustainable design principles, and not merely a mirage from shifting pain around.

6 Likes
#49

Haskell Foundation Stability Working Group - #21 by Ericson2314 :

We have enough threads (2!) on ghc-base-splitting at the moment […]

Agreed. Since my attempt to bring the perpetual nattering on the topic towards something resembling a useful conclusion has been about as useful as putting out an aluminium fire with water, I’m going to take a break from all this for several weeks - right now, I’m out of ideas, and the last time my patience was this depleted…regrettable things happened D-:

The significant problems of our time cannot be solved by the same level of thinking that created them.

Albert Einstein.

1 Like