GHC Medium-Term Priorities

Haskell Foundation
#20

There are indeed incompatible changes connected with GHC releases. I think they come in three categories:

  • Changes to the ordinary, “public” part of base and other standard libraries like mtl or bytestring - this is really a topic that the CLC needs your feedback on, and the GHC team is bound by their decisions
  • Changes to the part of base that are really more “exposed GHC guts” (e.g. GHC.Exts)- this is the most difficult thing, in my opinion, and we don’t have a good story about it. It would be good at the very least to make it more clear to users when they’re depending on this kind of thing, but we’ll likely still all end up transitively depending on things like the details of word types. See also CLC#105.
  • Changes to the language itself - these should all go through the GHC Steering Committee. While these certainly are problems for compatibility sometimes (I’m looking at you, simplified subsumption), my own anecdotal Haskell experience points to changes in libraries more often causing incompatibilities.

A consequence of the way that the Haskell community is organized today is that there really isn’t a single group or person where the compatibility buck stops. Given this organization, I think it’s unfair for the GHC developers to get the full pummeling when things change incompatibly, even though we do typically encounter the pain upon upgrading (because new GHC means new base, which means all the changes come as a bundle).

6 Likes
#21

I explicitly did not call out the GHC HQ directly here. This is a community issue, however, I’m not ok with deflection. It is as much the GHC HQ’s problem as it is the CLCs and the Steering Committees.

The GHC HQ is ultimately the entity that produce the releases. As such they are the gateway as to what goes into the compiler or not. And this power has been exercised before.

What I’d like to see is the GHC HQ taking ownership of the compiler, including what source code it accepts, and which not. And then I’d like the following two policies set forward:

  1. GHC will not include any library that abruptly changes its interface, without a proper deprecation story (with warnings, …) over a specified timeframe. (e.g. 2 years).
  2. GHC will not include language features that conflict with past behaviour unless they are behind a language pragma (we have language pragmas, so let’s use them).

Today we have the situation that you have a thousands to millions of lines of codebase, and try to migrate to a new compiler (Q: why is this even a thing? migrating to a new compiler?). If you are extremely lucky the surface language didn’t change, and your code as it is still compiles. Most likely not though. (you probably need something like allow-newer for base and other libs), but I guess we can dream.

The knock on effects are, your dependency tree is massive. So you can’t really upgrade to a new compiler until your complete dependency tree is compatible with that new compiler. But maybe you didn’t upgrade all your dependencies to their lates major version all the time. Maybe you are just happy with the old libraries interface? Well tough luck, the probably that that old major version is being made compatible with that new compiler is even less likely, so now you have to deal with interface breakage throughout the whole dependency tree. And why? Because GHC doesn’t accept the old code it did accept.

So, best is probably to just stick to that one compiler you know that works, and not spend major resources on the migration. (Note that you’ll need to have everything compatible with the old and new compiler; otherwise what are you going to do should the new compiler regress in any form?).

Fine, ignore GHC development outright, and stick with the old compiler, but ohh wait, you still live on a moving planet, and things change, new architectures show up, operating systems change. So now the old compiler becomes a liability. And patches you write for that compiler have zero probability of ending up in what ever is upstream. So you patch a virtually dead compiler. And getting those patches into master is likely a major undertaking. Again, it just costs enormous resources.

The problem is not that, for a single module, I can’t fix that up for compatibility, it’s the complexity of the whole dependency chain.

Do I want to use GHC 9.6, try it out, see what regressions we see? Absolutely. Do I have any realistic chance of doing so. No. So I can hope to use GHC 9.2 soon. But writing patches against 9.2 is going to equally frustrating and not going to end up in master, as it is on 8.10 today.

Hence even from a contribution to GHC, and a regression testing of GHC perspective, the GHC HQ should want GHC to keep accepting old code just fine.

if GHC was able to perfectly fine accept old code we wouldn’t even need to upgrade base, mtl and other libraries. Now, I know there are interaction with the RTS that make this a bit harder; so it’s probably not completely realistic to expect to just compile completely old base with the new compiler and ignore newer base and other CLC libraries.

Let’s assume we have deprecation warnings, and language macros then. I could most likely still compile code that was perfectly fine accepted by GHC 8.10 (assuming that didn’t have lots of deprecation warning already), with GHC 9.2. Also the surface language wouldn’t change, because any change to the surface language would require a language pragma. (you called out simplified subsumptions here).

If we wouldn’t be actively contributing to GHC, we’d probably just stay on 8.10 forever. As we are actively contributing (and thus contributions go into new releases, not 8.10), we somehow have to migrate forward to use our own contributions. I absolutely do not want to hack features into 8.10; and the times I’ve contemplated back porting the aarch64 ncg to 8.10–which ironically is the compiler in which that NCG started–make me very sad.

David, if you can make sure that the relevant parties are informed of this, that would be great!

6 Likes
#22

Regardless, I’d be interested in a more concrete report that explains the specific challenges, so that we can untangle how much of this is surface language breakage, base breakage or dependency issues. Because these may require different solutions/approaches.

#23

Absolutely.

I have prior experiences (though with a much smaller codebase, and no in-house GHC team) of very similar struggles, and the problems you’re talking about are very real. In our case, we had a DSL implementation with an LSP server, and updating base caused us to become incompatible with one of the LSP library’s dependencies (due to new interfaces to words, IIRC), which meant updating it, which meant adapting our server to a whole new interface in the new LSP library. It was decidedly non-trivial, but we needed that aarch64 NCG :slight_smile:

This is definitely a thing with other programming languages. I’ve got friends whose jobs have been migration from older versions of Java to newer ones, and for some larger corporate systems, this can be a big effort. I think Haskell requires migration work more often, and it’s harder, but we shouldn’t kid ourselves into thinking that we’re the only language in which work is required to upgrade.

There is, of course, a point where quantitative differences become qualitative differences. Perhaps we’ve crossed that precipice.

This is a thread about a reply from GHC HQ, and GHC was the system that you mentioned in your prior comment. I think that reading this as an explicit reply to them in particular was a reasonable interpretation of the words as written.

I also don’t think this is deflection: the GHC team have expressed to me numerous times that they in fact do feel bound by what the GHC Steering Committee and CLC decide. Likewise, these two committees feel like the GHC team is bound by their decisions. Perhaps we, the Haskell community, should find a new way to organize these technical discussions and decisions, or do a better job representing the interests of people who don’t want migration costs vs. the interests of those who want to clean up mistakes of the past and have a beautiful, coherent interface. Perhaps our current way of making decisions is neglecting important stakeholders. That’s an important discussion to have, but having that discussion is not deflection. Thank you for your input to that discussion, by the way - it’s important.

Regardless of details, your overall point is correct: people who want to use Haskell shouldn’t have to (and mostly won’t) care which organization is imposing migration costs on them, and frequent migration costs are a problem for all the reasons you point out. From time to time, I dust off Racket code that I wrote a decade ago (including things like GUIs, uses of fancy delimited control operators, and complicated compile-time metaprograms), and it essentially always just works, even though they’ve innovated a great deal on the language and compiler in the meantime, including completely replacing their bytecode compiler and RTS with a native compiler based on Chez Scheme.

Thank you again for your feedback, it is really valuable.

8 Likes
#24

The problem is not that, for a single module, I can’t fix that up for compatibility, it’s the complexity of the whole dependency chain.

When we release more frequently there will simply be fewer emergent gotchas that crop up, culpability for problems that do easier to assign, and mitigations in the form of e.g. holding back versions easier to perform.

I am working on the beginning of technical solutions in https://github.com/haskellfoundation/tech-proposals/pull/47 (which I should finish up!) but it can’t be emphasized enough that the social solution of releasing frequently (say every 6 weeks like Rust) will ultimately force us to fix issues and wring problems out of the system.

For example, bumping base every time is bad, my proposal gets into separating concerns so it is possible to have a stable layer, and frequent releases expose the ludicrously of regular major version bumps.

The latter two things fundamentally go hand in hand: frequent releases take our existing dull-pain bad habits and crank them up to excruciating pain, and this forces us to fix them.

4 Likes
#25

[…] frequent releases take our existing dull-pain bad habits and crank them up to excruciating pain, and this forces us to fix them.

(original image here.)

…would switching to a Rust-style release schedule require switching to something like a Rust-style development infrastructure first, in order to cope with that cadence?

#26

@atravers I think it would require doing some things differently, and that’s why I am working on my proposal, but if we just tried without planning and then failed I think that would also be healthy and useful too. Failure means empirical evidence about what exactly the bottlenecks / weak points are!

2 Likes
#27

That’s odd - I already thought we knew what the problem was: too much work for too few people.

#28

There should be very few costs that are linear in the number of releases. Releasing more should mainly be a chance to amortize the same amount of work, leaving people no busier than before.

1 Like
#29

Aren’t the relative few maintaining Haskell already too busy now? If so, how does “leaving [those] people no busier than before” actually help them?

#30

There’s research showing frequent releases improve system quality (although I believe they’re wrong and missing confounders).

However, those findings are based on the assumption that there is a feedback loop and that the increased cadence yields better and earlier feedback.

This may work well for customer facing platforms, where releases don’t incur any real cost.

As for GHC (or even core libraries), I’m questioning whether the need for feedback is really the driving factor and also want to point out that there are other solutions to getting at least some of that feedback without a proper release. Additionaly, more releases do incur a cost on the entire ecosystem, even when they don’t include breaking changes.

So I urge you to reconsider whether more releases = better is really true. I believe it’s a mantra that emerged out of tooling and social dynamics and less out of technological necessity.

2 Likes
#31

They are definitely “missing” confounders. That’s the point! The process of shifting to more frequent releases forces other quality improvements to happen at the same time. For example, as @Ericson2314 said above

the social solution of releasing frequently (say every 6 weeks like Rust) will ultimately force us to fix issues and wring problems out of the system

Now, that’s much easier said than done. You can’t simply “release more frequently” and expect everything else to get better. Rather, unless you actually work to improve everything else too, everything else will just get worse!

Some things are very difficult to improve unless release frequency is increased at the same time. Maintaining high quality at high frequency forces effort spent improving other areas too, such as communication, automation of what should be automated, deautomation of what should be done manually, calibration with upstream (“suppliers”) and downstream (“customers”), …

Some background reading for those who are interested in this area, which is much studied in industrial engineering:

  • The Goal, Eli Goldratt
  • The Phoenix Project, Gene Kim et. al
  • The Principles of Product Development Flow, Donald G. Reinertsen
  • Beyond that, anything good on Lean, the Toyota Production System, W. Edwards Deming. (Unfortunately a lot of this stuff has been cargo culted, like Agile, so there’s a lot of dross amongst the gold.)
8 Likes
#32

Does it really? I doubt that applies generally. E.g. I’ve never seen frequent releases improve the design and stability of APIs. Instead these projects seem to follow very “flexible” patterns instead or constantly evolve existing APIs.

Usually it takes time to think and reflect and carefully do release management, when your priority is well thought through API/design. Also: constant user feedback is not always useful here.

#33

So if all this information is already available, let’s apply it first before increasing the cadence - right now, there seems to be a belief that increasing the cadence “straight up” will somehow deliver all these benefits…hence my response.

Who knows: just by making those changes alone, things may improve to the point where an increased cadence won’t be necessary…

#34

I doubt that applies generally.

Your doubt is correct. Very little applies generally. It’s hard to explain the benefits of more frequent releases, and how achieve them whilst not making anything else worse, in a discussion thread. I refer interested parties to the references I posted above. I don’t know any substitute for doing the reading on this, unfortunately.

1 Like
#35

Yes, imagine bash going “agile” and breaking interface and behavior every couple of months and then go “it’s documented and we bumped the major version”. You almost never want user feedback post-release for such a critical piece of software… you want it pre-release (in form of RFCs and discussions and during betas). And you don’t want many releases at all.

One coping mechanism with such things is distributions simply skipping certain releases, both to keep the maintenance cost lower and also exposing users to less disruption and possibly bugs. This is very common. But I’m not sure we can do that in GHCup or HLS. But I’m sure we can’t do that in GHCup or HLS. I mean, stack sort of does that already.

1 Like
#36

“More frequent releases” to me does not mean “more frequent breakage”. I can understand why you are puzzled, if that’s what you thought I meant. As I said above, I strongly recommend that people try to disentangle in their minds the concept of frequency of release from the concept of breakage per unit time.

Personally I would prefer GHC to release on every MR and break never. In the real world, this is impossible, but I think we can get sufficiently close that it would astonish people who believe the status quo is reasonable.

7 Likes
#37

I’m totally aware and I wrote that above. But even without breakage, there is a cost to frequent releases, if they are to be adopted and supported (bindists, tools, version bounds, …).

1 Like
#38

But even without breakage, there is a cost to frequent releases, if they are to be adopted and supported (bindists, tools, version bounds, …).

Perhaps, but my motivation for proposing more frequent releases is to decrease costs on the rest of the ecosystem. I can appreciate that sounds paradoxical, for those who are used to the status quo.

1 Like
#39

That is to say, I don’t want more frequent releases for their own sake. I want them because I think it will make a lot of other stuff much easier (or could make other stuff much easier, if done in the right way – but that’s easier said than done of course).

1 Like