The evolution of: Decoupling base and GHC

#42

And yet … The run-of-the-mill trickle of proposals on the libraries mailing list seems to consist only of adding things to base: extra/specialised methods, optimising constraints on existing classes, tweaks to help performance. Progressively making base more tangled and interdependent. Until somebody calls a halt.

What’s special about 'adding to base’ that people are so keen on? To contra @michaelpj’s “Why Not?”: If you want a Bifoldable to work across different releases of GHC-the-compiler

  • Why not copy the source to MyBifoldable and use import/module naming to insulate yourself from base's Bifoldable? Or
  • Why not put on Hackage as NotbaseBifoldable and ditto ditto.
  • Why not help yourself rather than waiting around wringing your hands that base is a mess (allegedly)?

OTOH, Bifoldable hasn’t changed in the five years since it was added to base. It hasn’t (so far) been a source of instability – despite its import Control.Applicative (amongst others) and that’s import GHC.Prim etc that (presumably) could never be moved out of base or ghc-base or ghc-internal-*, depending how we skin the cat.

Control.Applicative and GHC.Prim have changed; if those changes have been a source of instability for Bifoldable, no amount of reorganising base will distance you from that import.

So do you expect that by being in a blessed library named base, it’s the GHC team’s responsibility to make Bifoldable in release 9.10 compile and run unchanged under GHC-the-compiler release 9.12? Has anybody asked the already-overworked GHC team whether they have capacity to regression-test every release of GHC-the-compiler against the last three base releases?

Or do you expect it to ‘just work’ without testing? Because why?

I’d call it “magical”: test a compiler release against a release of a library that hasn’t been developed yet. Please tell me when GHC produces that time machine. Oh! you mean when the library is developed, ask the GHC team to cast their mind back to the compiler-as-was-9-months-ago and regression-test.

#43

why on earth is Bifoldable in base? It’s not required to be from the Language Report

The report doesn’t mention base, or packages at all does it? At least, I couldn’t find any such mention after a quick browse.

#44

Or alternatively, test the new library release against compiler releases that have been developed.

2 Likes
#45

While we do have those alternatives already, it isn’t really a solved problem, b/c when we onboard new users, they don’t use those alternatives, they use what GHC and the community “ship”. This is also why the stack/cabal duplication is still a problem for onboarding - newbs don’t know how to make that choice effectively, and there’s no authoritative guidance from an authoritative voice in haskell (haskell.org for example) specifying what we should do.

2 Likes
#46

Maybe we can use a few possibilities as example inputs for a set of thought experiments (guided by the particulars of the situation in the example)?

Caveat: For each of these, I would guess there is some disagreement that there is even a change to make - I assume they would have already been addressed otherwise. But either way, I think it helps to map out what a transition would actually look like, what problems there are to overcome and smooth out.

Here are a few I see discussed from time to time:

  • Fixing bad default functions which lead to prominent mistakes and flaws in production software, foldl, head, lack of strictness, etc.

  • Adding new types to replace bad defaults - vector type to replace list as a default, for example.

  • Better default guidance and support in base WRT String / Text / ByteString (or list/vector, etc).

  • Introduction of and transition to Dependent Haskell or a Dependently Typed Haskell, or in some way running the Haskell community (hackage/package ecosystem) through the DT simulation game to figure out what that might all look like.

For each of these types of secenarios, what would the deprecation/transition/migration look like?

And how might we reduce the human/time cost, and result with significant improvements, with more correctness overall for the majority of production use cases?

Another observation from these discussions: the community already has many examples or attempts of fixing these issues and “doing it right” - alternative preludes, documentation about problems with the “standard library” and how to work with those, packages with functions/types we need, but integrating these solutions into the “core / shared” domain has not been allowed, accepted, or followed-thru with to completion. While I support the idea of decoupling base/GHC, I also don’t see how this decoupling will address any of the issues which lead to the failure of those previous attempts to improve the common domain.

Lastly, and on a similar note, how do we break the cycle so we don’t continue repeating ourselves in this circle (insert broken record reference).

3 Likes
#47

I’m probably being naive here, but if we really want GHC to be decoupled base, i.e the ecosystem and compiler can evolve mostly independently, isn’t the solution to just total isolate GHC from the rest of the Haskell ecosystem?

If GHC had been implemented in another language, it wouldn’t have been able to import other libs from hackage, and other packages wouldn’t be able to import GHC code.

It seems to me like if we really want GHC to be decoupled, we need to duplicate all the code it depends on from the ecosystem, and keep/maintain its own copies of that code internally. Then we need to make it not available to the rest of the ecosystem so normal libs can’t depend on GHC’s internals at all. And vice-versa, duplicate base out of GHC into its own repo, and duplicate any dependencies on GHC’s internals to be maintained separately of GHC.

I know this is a super naive perspective here, likely riddled with many issues that I’m totally ignorant of and probably way easier said than done, but it feels to me like this is the basic gist of how to fix the issue. For as naive as it is, it feels the simplest to me too :sweat_smile::innocent:

Summing it up it one phrase:
Nobody should know GHC is implemented in Haskell, not even GHC itself!

2 Likes
#48

@vanceism7 GHC being implemented in Haskell does have some costs (and benefits), but that is largely unrelated to why GHC and base are so coupled.

Rather, it’s because the code that GHC generates.

  1. base is implemented with knowledge of all sorts of secret things about what GHC does. does is the key word, this is stuff like secret interfaces like what the primops are, etc. etc. If GHC was written in another langauge all those same secret interfaces would exist. They do for any language that has primitive definitions.
  2. Generated code assumes many random things are found in base and base alone. Again this doesn’t matter how GHC is written, for compiler in another langauge could also generate code assuming stuff is in base. Indeed, the RTS is in another langauge (C) and also assumes the existence of some base stuff.

Most language implementations have grown organically very gnarly implementation—standard library interactions. Libc is nasty too. C++ STL library in freestanding mode is also sort of CPP hack. Rust however proved that you can actually layer this stuff nicely and indeed rustc is written in Rust.

(They in fact insist the compiler is always built against the same standard library, so they have long done some CPP to make the standard library work with the current and previous versions.
I think that last bit is silly: the compiler shouldn’t require bleeding edge standard library features. But the principle of layering and decoupling is very good!)

Rust doesn’t do any magic to accomplish this. In fact, defects in the design of Cargo, and the lack of anything like Backpack make it harder for them than for us! It’s simply a matter of desiring the same elegance of abstraction and division of labor in the standard library as every other library, and spending the time to make it happen.

I believe in addition to standard libraries exposing top-quality beautiful interfaces, they should also be implemented in top-quality beautiful ways. The standard library often is the user’s first impression of the language ecosystem, and so it should make a good first impression. But the user impressed and enamored with the interfaces and thus enticed to peak behind the current shouldn’t have their honeymoon wrecked by seeing a rats nest.

Beautiful turtles all the way down alone impresses the user that the principles of the community are in fact sustainable design principles, and not merely a mirage from shifting pain around.

6 Likes
#49

Haskell Foundation Stability Working Group - #21 by Ericson2314 :

We have enough threads (2!) on ghc-base-splitting at the moment […]

Agreed. Since my attempt to bring the perpetual nattering on the topic towards something resembling a useful conclusion has been about as useful as putting out an aluminium fire with water, I’m going to take a break from all this for several weeks - right now, I’m out of ideas, and the last time my patience was this depleted…regrettable things happened D-:

The significant problems of our time cannot be solved by the same level of thinking that created them.

Albert Einstein.

1 Like