The evolution of: Decoupling base and GHC

#18

OK so counting from after the ‘FTP changes’ brouhaha, which of those breaking changes has base perpetrated? And how would a reorganised base have avoided the breakages, or at least minimised the impact?

@Ericson2314 seems to be throwing in the kitchen sink. To pick one point as an example:

  1. Classes like Num with no clear laws.

Has base made breaking changes to Num? Indeed has base made any visible changes to Num? I rather thought Num is the same now as Haskell 98. Then don’t mix up complaints about breaking changes with complaints about (what turned out to be with a great deal of hindsight) poor design. In early 1990’s when the Prelude was developed, who’d have expected typeclasses would be so wildly powerful, and would have connected to Category Theory?

As I mentioned, Num (and Integral, Fractional) are pretty much baked into the definition and syntax of Haskell. Nothing’s stopping you creating a whole bunch of other Numerical classes and operators with all the nice properties.

  1. Partiality, especially in methods like *1 in Foldable and with Enum

Has base changed methods/functions to be more partial? Again don’t mix up complaints about breakages with complaints about a design you don’t like. Again there’s nothing stopping you creating safeHead, etc. (I’m not going to defend design choices in Foldable; but any more changes to it had better have an enormous benefit/cost ratio.)

A usage of head or (!!) is not necessarily unsafe: it may be surrounded by checks to avoid calling it unsafely. Yes it’s unfortunate those aren’t type-safe checks, in a language which vaunts the benefits of type safety. “Well-typed programs can’t go wrong” is bunkum.

I think a better approach is through education: stop teaching newbies so much about Lists (including String) and so little about appropriate datatype design/including especially other off-the-shelf recursive data structures.

So is it this (below) what you want? And is this the opinion of you all:

  • Reorganise base so a program can (for example) exclude Num and all its Pomps; then
  • instead import theoretically-pure ShinyNum using all the same class and operator ids; and
  • otherwise use standard Prelude.

How about GHC wired-in modules for implementation stuff like arithmetic on pointers and indexes? Is that also to use ShinyNum? How about modules like Vector with Int indexes; or Data.Set with a size :: Int embedded in every node and (Num.+) to calculate it? Checking for numeric over/underflow or index-out-of-array comes at computational cost. Those modules are already making limited checks, enough to avoid IllMemRefs. ShinyNum will duplicate work for programs that already don’t throw those exceptions. Who’s then responsible for addressing the performance degradation?

#19

@antc2 you have misunderstand my post in the way I feared would happen.

Right now, all these changes to base we can’t even seriously debate because there would be far to much breakage simply because users are stuck with the version of base GHC ships with. With the decoupling, we at least expand the overton window on things the CLC can consider, and I think that is very good ----- even if none of those no longer beyond the pale changes end up being accepted.

Also, see what @atravers linked in Pre-Pre-HFTP: Decoupling base and GHC - #43 by atravers ; there is better motivation there than what I wrote.

Speaking of String, here’s a very good example. I actually have little problem with [Char] being easily obtained from literals — for if beginners need to eliminate strings, this is gentlest way there is. The real problem with String is not that it exists, or even that interfaces that use it exist, but that type classes do ridiculous shit to support it.

For example, the Read class is a disaster:

class Read a where
  readsPrec :: Int   -> ReadS a
  readList :: ReadS [a]
  readPrec :: ReadPrec a
  readListPrec :: ReadPrec [a]

the bottom two are proposed to be replaced…we should have a deprecated cycle to move them outside the class then, just like return = pure should become mandatory. So:

class Read a where
  readPrec :: ReadPrec a
  readListPrec :: ReadPrec [a]

But then readListPrec is a hack for String. If we had

newtype String = String [Char]

that could have the string literate instance, and then we can delete readListPrec

readListPrec and friends are, in my view, overloaded instances laundered as extra methods. Not good!

1 Like
#20

D’oh please stop using apocalyptic language. You’re just undermining your case. (Are there more than two type classes with specific provision for String?) Read has been doing its job happily for 30 years. It’s not broken/don’t fix it/don’t use it as a pretext for breaking all sorts of other stuff.

Show uses exactly the same ruse for [Char] vs arbitrary [a]. So you’ll have to change that at the same time. But there might be good reasons I want to show a [MyType] in a special format and be able to read back in that format. Yes, it’s a cheat to avoid overlapping instances. But since overlapping instances still aren’t blessed, it’s the lesser of two evils.

(The class gives default definitions for the two methods you want to move out, so I guess moving them will be mostly invisible. Unless somebody’s giving some custom overloading somewhere. I’m still not seeing any motivation for tinkering.)

There’s a thread discussing the threshold to clear for a breaking change. That seems to have fizzled out without conclusion. Then isn’t the Read proposal going to go the same way as the moving out (/=) proposal? (Unusually for me – some would say – I didn’t comment/just couldn’t even … I do regret the community uses up so many cycles on such small issues.)

#21

So let’s all now get out of @Ericson2314’s way and see what he can do with @nomeata’s previous work - unless I’ve missed something (again!), we’re not being asked to join in. @Ericson2314 may be able to apply fresh insights to the problem, which @nomeata wasn’t aware of in 2013.

#22

Thanks for the explanation, I think I get it now. The instance tree only is forced to only branch near the root, so most of the tree are just linear lists.

I do think that is rather limiting. For example you wouldn’t be able to write the popular data types a la carte subtyping class:

data (f :+: g) e = Inl (f e) | Inr (g e)

class (Functor sub, Functor sup) => (:<:) sub sup where
  inj :: sub a -> sup a

instance Functor f => (:<:) f f where
  inj = id

instance (Functor f, Functor g) => (:<:) f (f :+: g) where
  inj = Inl

instance (Functor f, Functor g, Functor h, (:<:) f h) => (:<:) f (g :+: h) where
  inj = Inr . inj

But that should probably be a closed type class anyway, so perhaps there are other ways to work around that limitation. I would love to see an analysis of what instance trees look like in Hackage packages.

#23

Sure you could. It needs trickery to make it appear instances are in a strict substitution ordering. So you end up with difficult-to-follow code (untested):

instance {-# OVERLAPPING #-} Functor f => (:<:) f f where
  inj = id

instance {-# OVERLAPPABLE #-} (Functor f, Functor g', SubT' f g') => (:<:) f g' where
  inj = inj'

class (Functor sub, Functor sup) => SubT' sub sup  where
  inj' :: sub a -> sup a

instance {-# OVERLAPPING #-} (Functor f, Functor g) => SubT' f (f :+: g) where
  inj = Inl

instance {-# OVERLAPPABLE #-} (Functor f, Functor g, Functor h, (:<:) f h) => SubT' f (g :+: h) where
  inj' = Inr . inj

(Needs UndecidableInstances.)

One of the limitations with Swiestra’s approach is that the tree is right-biased. (The instances don’t try to recurse to the left of (:+:) like (f :+: f') :+: (g :+: g'). [2008 JFP paper, middle of 2nd page of Section 4 “may fail to find an injection”.]) This technique can be expanded to cope with that. (You need two auxiliary SubT classes; and you need to choose which direction to prefer first. The instances get pretty ugly.)

You mean that feature that we do not have? And do not need?

Yes. In particular are there any ‘partially overlapping’ instances?:

instance C a Bool  ...
instance C Int b  ...

Allowing that really hampers GHC sharpening up its act on overlaps. And I suspect nobody actually wants it. (Or if they do, the above technique can handle it, at cost of forcing a choice.)

1 Like
#24

(It was your comment that drew me in.) @Ericson2314 doesn’t have to use up his time responding to me. His investigations are going to come back with the costs side, in terms of effort to reorganise base and breakages liable in client code.

The benefits side will still be unevaluated. So all this discussion will happen again. (Which seems to be the pattern with CLC.)

I thought that as I’m disinterested and largely uninterested, I could prod people to articulate the why – which would both build the evaluation for benefits and motivate where to put the split(s) in base.

#25

I admit I’m in favor of the split, but I don’t understand how juicy, interesting, breaking changes to base are a benefit that the split would provide for us. Split or no split, everybody uses base, so breaking changes are just as hard to swallow either way.

Here, on the other hand, are concrete benefits to the split:

  1. Decreased cognitive load.
    1. Project maintenance gets more tractable. Building, releasing, packaging, etc.
    2. The blast radius for changes of any sort is decreased — both damage done by breaking changes, as well as reasoning about the breakiness of changes in the first place.
  2. Improve backward compatibility. base currently includes a bunch of GHC internals. Since they change with each version of GHC, they trigger a breaking change for all of base. If those changes were constrained to a different library altogether, GHC could stay backward compatible with older bases.
5 Likes
#26

Can those out-of-place “GHC internal” definitions in base be replaced using e.g. Haskell 2010 FFI declarations and some extra C modules?

#27

Can they? I thought they were mostly Haskell code in the first place. The GHC.* module hierarchy in base.

1 Like
#28

Yeah some of the more tricky stuff is when the RTS calls a Haskell function in base! We would arguably need cyclic backpack to really do this sort of thing right.

#29

…yeah, why bother with abstraction when there’s only one mainstream implementation of Haskell?

As for potential solutions:

  • if it’s a RTS primitive which is calling a base entity, there may be two options:

    • if the primitive is rarely used, you could try passing in the base entity as an extra argument;
    • if it’s widely used, another option is to replace the RTS primitive with a new base declaration via the FFI - it may still be easier to add that extra argument, and ignore the complaints…

  • if, for some reason, it’s some part of the fundamental workings of the RTS that make the call to base…let’s just say “tricky” is being diplomatic:

    • If the offending part can be easily isolated, you may be able to replace it as well using the FFI;
    • Otherwise, being lazy is probably the simplest option - log an issue report with the tracker for GHC, and make a note of the miscreant in your progress report.

…progress report? Yes: for a good example of their use, see Philip Heron’s series of reports about Rust-GCC - most other requests for updates can then be silently deleted (ideally automatically!). You can then more easily focus on this task, which will require plenty of your attention.

Those progress reports will be invaluable e.g. if this topic is still being debated in, let’s say, 2027 people can just be directed to read them to find out what happened…

#30

@AntC2 you are most welcome to elaborate on these statements. You seem to be under an impression that CLC has an articulated opinion on the matter, but I have no clue what makes you think so.

3 Likes
#31

I said: I thought that …, I could prod people to articulate the why

Because the impression I’m under is that the CLC does not have an articulated opinion on the matter, so is failing to give guidance where leadership is clearly needed. I have no clue what makes you think I’m seeing articulated opinions.

Those quotes from me you snipped are pretty close to links and observations. But in case you missed it …

And I see @Ericson2314 got there before me: “we get these massive threads”. And that was barely a third of the way through that thread. (Addit: And after plenty of discussion on the mailing list.) At the half way point the Committee decided/closed the issue. Then there were requests to re-open and more rounds of discussion – which came to no conclusion I can see.

So somebody thought it a good idea to have Principles that would guide future discussions. (yes please!) A thread of 63 comments so far. No conclusion I can see.

Guiding Principles are the sort of thing a Committee should be there for. The CLC’s Charter says there will be Controversial Decisions. (d’uh, well yes) And then … ?

Specifically where I came in on Decoupling base from GHC wired-in, there’s 42 replies on the original thread (which went off track) and 108 on the newly-spawned attempt to corral it. I see no criteria for how the Committee might evaluate any proposal. I’d expect the criteria to derive from the general Principles guiding assessing impacts of breaking changes vs expected benefits.

I get it that Discourse isn’t under the aegis of CLC; and that the bulk of comments on those GitHub Issues are not from Committee members. If the CLC had some guiding Principles, they could shape the discussion to tone down apocalyptic language and sweeping unquantifiable claims (like @chreekat’s “blast radius for changes of any sort is decreased” – of any sort?).

There’s been mutterings about the design of base all the time I’ve been following Haskell. I’ve seen (versions of) this wishlist plenty of times. Nearly every attempt at it has faltered – with the conspicuous exception of the bruising ‘FTP changes’.

@Ericson2314 is about to do what? Split out the GHC wired-in parts and their dependencies? Revive nomeata’s work from 2013? “Rip base in two” along its natural fracture points?

I get the impression @Ericson2314 is not doing (whatever it is) for his own entertainment. It sounds like a lot of work. Whatever results, is the Committee just going refuse point blank – because it’s too controversial or causes too much breakage? If there is to be a benefits vs cost assessment, what are the metrics for benefits? As opposed to chreekat’s ineffable claims.

Your interventions in these threads have generally been at the ultra-conservative (small-c) end:

It’s difficult to imagine slower than the pace of failing to make a decision about removing (/=) from Eq. For the sake of managing potential contributors’ expectations, what does “slower” mean? And is (whatever that means) the collective view of the Committee?

1 Like
#32

I am rather bemused by the whole idea a set of sourcecode (the libraries) could be insulated from the very compiler they rely on. If GHC (the compiler) changes – even with a tiny bugfix release – all the object code for those libraries must get regenerated.

Of course GHC developers are meticulous to avoid side-effects of changes; and to document everything that’s changed. Nevertheless, prudent software release practice is to trust nothing and re-run the whole test suite against the re-compiled application and libraries.

Reorganising base to minimise dependencies won’t remove the chief dependency: on the compiler. So how could changes to libraries be more conservative (small-c) than changes to the compiler?

Upgrading a compilation environment is different to upgrading an application environment, in which the asset to protect is its database content and continuing business operations that query and maintain it.

#33

I don’t agree. Changes in the compiler should only affect a tiny fraction of the language and I think that has historically been the case (at least recently), so it isn’t necessary to consider adjacent compiler versions to be completely different. Of course you should still re-run test-suites and otherwise check that the changes don’t break things, but I don’t see how that makes it impossible to split base.

4 Likes
#35

@antc2 The splitting of base is, strictly speaking, a mere implementation detail — it doesn’t even require CLC input if we were were to go with scary sounding ghc-internal-* libraries.

I do, in fact, want the split base libraries to be intended for public consumption, so I will in fact engage the CLC on what portions of the interfaces look nice stabilized separately, but I don’t anticipate this being very controversial.

In particular, even if I somehow get the split all wrong, it is very easy to turn those libraries into ones that reexport stuff defined elsewhere. Ergo, I see very little costs in terms of stability and tying our hands in the future with this stuff.

I am rather bemused by the whole idea a set of sourcecode (the libraries) could be insulated from the very compiler they rely on.

This is funny coming from someone who likes HUGS and misses the era of a diversity of implementations and perspectives!

Code is strongly linked to the interface of a compiler, but not its implementation. What we call base–GHC coupling is still just each observing the “unabridged, private” interface of the other.

Interfaces usually have some sort of subtyping / partial order structure. The goal is to fact things nicely to rely on as little private stuff as possible.

I would encourage you to check out Rust’s core vs std split, where std is far bigger but also far more normal a library, with very few wiring-in points in either direction. Nothing I am hoping for with the split isn’t pioneered “in production” over there already.

6 Likes
#36

Precisely because I value a diversity of features, and am tolerant of churn at the bottom of the tech stack (the compiler), I expect everything built over it will get wobbly from time to time. What I don’t expect is you could mix-and-match different versions of base with different releases of GHC. As if you could pull-push blocks in-out of the bottom of the Jenga tower, and keep the upper storeys somehow suspended from a sky-hook.

As I thought I made clear, I reverted to HUGS not because I’m an antedeluvian, not because GHC is moving too fast or is too unstable, but because GHC is moving sideways relative to where I want to go. And because I found I could understand and tweak the innards of HUGS.

1 Like
#37

What I don’t expect is you could mix-and-match different versions of base with different releases of GHC.

Why not? Perhaps we need an example. Let’s pick something boring from base, hmmm, Bifoldable. Suppose we began our base split by putting just Bifoldable in base and everything else in ghc-base (being the library shipped with GHC).

Now, could we build this base with different versions of GHC? I don’t see why not. It’s just a typeclass, why wouldn’t it compile with various versions? Maybe changes in GHC would make it not compile, and we solve that the way we do for any other library: ugly CPP. But it would work.

Similarly, we could change Bifoldable (add another instance, say), release a new version of base, and expect that to work with old GHCs too. Even GHCs from before we did the release! Radical!

Now: lots of stuff in base is in the same position as Bifoldable. Some stuff isn’t, for sure, and that’s why we’re going to be stuck with some kind of ghc-base thing. But that doesn’t mean it’s somehow impossible to remove anything from base and put it in a more “normal” library.

9 Likes
#38

Maybe one challenge in this conversation is the name? That is, we want a library of standard helper functions / data structures that everyone will want access to. I’ll call it standard-goodies. The key aspect of standard-goodies is that it is just a library – no particular cozy relationship with GHC. Of course, standard-goodies could be upgraded independently of GHC. The conversation in this thread is about taking stuff out of a library whose existence is tied to GHC and putting more into standard-goodies.

6 Likes