Language, library, and compiler stability (moved from GHC 9.6 Migration guide)

There seems to be some absolute disconnect here. Suppose I want to make sure that the compiler only introduces features that don’t abruptly break a lot of code. (Good idea, right?). So how do I know if a feature abruptly breaks a lot of code? Well, I try to compile a lot of code! But whoops, the compiler breaks one thing in bytestring. Now, I can’t compile any code, because bytestring is broken. So how can I even begin to test if a compiler does or does not break packages if I do not have a head.hackage type thing to unstick me?

You might say, “ok, then don’t break bytestring”, which I think is what you mean when you say “can we make the compiler more resilient to accept existing code”. But – the whole point is, suppose that bytestring is the only thing that breaks, well, we’d like to at least know that, especially if we can just ship a new bytestring along the way and be done with it. But if we do not have a head.hackage, we cannot possibly know that. You might say “who cares, it cannot possibly matter the scope of breakage, any breakage at all without a full cycle is too much!” But first, we have established that some is unavoidable (such as fixing primitive types for architecture support reasons).

But more importantly, I think the process you advocate would make progress unbearably slow. It would discourage contributions to the compiler, and it would also, frankly, enrage the sizable contingent of end-users who are always clamoring for features.

Let me drag out again my haskell users quadrant (from A tick-tock release cycle for GHC by bgamari · Pull Request #34 · haskellfoundation/tech-proposals · GitHub)

       Developer Maintainer
      ┌─────────┬─────────┐
    L │         │         │
    i │         │         │
    b │         │         │
      │         │         │
      ├─────────┼─────────┤
    A │         │         │
    p │         │         │
    p │         │         │
      │         │         │
      └─────────┴─────────┘

This would go straight down the middle. App and Lib maintainers would of course be thrilled. They could make changes at an incredibly leisurely pace. App and Lib developers would be much less happy. The features they want and fixes to improve their experience would be terribly far away. And bear in mind that as a relatively small language that continues to grow, we really do need to promote end-user adoption, which means keeping developers happy, since the work new users getting into the language will do is definitionally new development. I am already imagining the enraged threads on the discourse explaining how Haskell is a stagnant language that refuses to fix known issues in a timely fashion for the sake of compatibility, and etc.

So again, what I want to underline is that you have very real concerns. But they must be balanced against other concerns, and to do so means having a concrete assessment of potential breakage. And at the moment, it seems to remain the case that the best way to have that is to have a solid head.hackage process which is widely used. With that, we can concretely navigate tradeoffs. Without it, we are left with only extremes which will necessarily make some group or another feel very unrepresented and unheard.

3 Likes

Am I correct in understanding that head.hackage is widely misunderstood? :smiley: It seems to me that there is a useful community service lurking that just needs to get better understood and developed into something more broadly usable.

I myself don’t think I know it perfectly, so please correct me if I’m wrong:

It sounds like head.hackage serves two distinct purposes: one is to notice breakages (intentional or otherwise) caused by GHC development, and the other is to update libraries in advance when there are (intentional) breaking changes in GHC. Let’s call those “Canary Hackage” and “Prerelease Hackage”, if you will.

Here are some examples of Canary Hackage:

  1. An unintentional breakage was noticed and fixed: Core lint error on head.hackage lint build (#22357) · Issues · Glasgow Haskell Compiler / GHC · GitLab
  2. Another one, an instance where fixing one bug caused another one: head.hackage: package "what4" no longer compiles (#22519) · Issues · Glasgow Haskell Compiler / GHC · GitLab
  3. Something totally different: tracking GHC memory usage over time as it compiles head.hackage packages

And an example of Prerelease Hackage:

  1. Adding the Functor superclass on Bifunctor broke packages: head.hackage: packages failing due to new Functor superclass on Bifunctor (#22567) · Issues · Glasgow Haskell Compiler / GHC · GitLab

As people have highlighted up thread, GHC devs absolutely need Prerelease Hackage, but it’s Canary Hackage that may have broader appeal among maintainers (using @sclv’s distinction between maintainers and developers). I just went back and read the summary of Introduce GHC.X.hackage, and perhaps it got off on the wrong foot by only mentioning Prerelease Hackage?

3 Likes

Me too. So all of what follows may be total nonsense…


Over in this thread angerman made a simple suggestion: put all breaking changes under flags. This could be combined with santiweight’s suggestion from this “tick-tock release” thread as follows:

  • in ghc-v.k.m (v constant) - breaking changes can only be enabled by using their respective flags

  • in ghc-v+1.k.m (new major release) - breaking changes (those which have proven to be reliable and widely-acceptable) from version v are then enabled by default, but can still be disabled.

  • in ghc-v+2.k.m (next major release) - assuming no further problems, breaking changes from version v are made permanent (they cannot be disabled).

Haskell developers can then use breaking features by enabling the appropriate flags (before the new major release arrives), and Haskell maintainers only have to be concerned with widespread breakage upon the arrival of each new major release.

2 Likes

Without judgement implied in any form, I find your comment quite interesting @sclv

We arrive at the same conclusion but with almost opposite assumptions, I consider the current state to be very discouraging towards contributions. We basically contribute despite all this to GHC, and consider a much more stable (non-breaking without warning) situation much more welcoming to contributions. But then I’ll add that the contributions I care about are almost always orthogonal to language changes (linker improvements, better system compatibilities, new codegen backends, new compilation targets, …).

You and I also seem to be frequenting very different circles of end users.

You are however right in the assumption that I would want warnings prior to breakage, ideally accompanied with a note as to how to migrate.

You bring up two points in here. One is base libraries shipped with GHC. The other is a limit on how many packages you are willing to patch to compile a larger set of code. Is the limit 1, 5, 10? Is it constrained to only base libraries?

To avoid any misunderstanding, if you need head.hackage during development, I don’t see an issue with this at all. If head.hackage is required when the compiler is released, then, yes I do see an issue with that. head.hackage should at best be a GHC internal development crutch. An end-user would under most circumstances not notice this bytestring breakage (if that’s the only one) or really any to the shipped packages with GHC, especially those that cabal considers non-reinstallable. If releases for those compatible with the new compiler are on hackage prior to the compiler release, there would be no need of head.hackage (outside of ghc development).

I have in the past proposed different other strategies:

  • have a stable and a development branch in GHC.
  • have a tick-tock cycle (only every other release can break code).

We used to have one release per year, which mean you had breakage once a year, now we have two. We now expect maintainers to adapt their code to a new compiler every 6mo.

2 Likes

The first port of call is indeed “make the compiler more resilient”, but sometimes that is very difficult. Suppose a change to the signature of a function in base has been changed, after due deliberation by CLC and with deprecation cycles. But now it has finally landed. Alas some package maintainer has failed to update. That blocks the entire dependency tree of that package.

That is the brittleness that I referred to. I don’t want to have to rely on the 100.0% cooperation of thousands of library authors, some of whom will have legitmate reasons for being out of action. And it’s extremely difficult to make the compiler compile both programs that expect f :: Int -> Int and programs that expect the same f :: Int -> Int -> Int.

@chreekat’s analysis is spot on. I want both Canary Hackage and Prerelease Hackage.

I think perhaps @angerman fears that head.hackage will be a mechanism for saying “oh well, breaking changes don’t matter”. That is, if you crack open the door, someone will drive a lorry though it. But it doesn’t have to be that way – we can simply be vigilant.

There are also times when there is simply no good option. We want to fix an outright bug that users have reported in production. But alas other users are inadvertently relying on that same bug. We can’t satisfy both! Sometimes it is possible to have -XContinueWithBug, but sometimes that is really, really difficult to do. (And of course it slows down the entire bug-fixing process by adding friction. Yes, breakage is also friction; I’m just saying that there is a balance to be struck here.)

Another example: we do not currently have a well-defined API for GHC itself. So clients who depend on GHC as a library reach deep into the belly of GHC and use functions that were never inteneded for external use. The only way to achieve zero breakage is to never change the signature or semantics of any function defined in any GHC module, ever. That’s clearly not sensible. So we should work on defining a decent API for GHC, a project that David Christiansen is leading. But that will take time, and meanwhile we want to be able to patch the packages that break.

My main point is that it’s a bit more nuanced than it might at first appear. head.hackage is a fantastic tool for adding resilience. It should not (and I think is not) be used as a way to smuggle in breaking changes without careful thought.

Does that help at all? This is a useful debate, I think.

(For what it’s worth, the GHC team spends a lot of time on back-compat issues. We really do try hard. We may make mistakes, but I think we are getting better. Canary Hackage is part of that getting better.)

3 Likes

Hi everyone.

This is the problem that horizon-haskell intends to solve. I’m quite confident this will be more comprehensive than head.hackage, serve more usecases, and be easier for package maintainers to integrate ahead of time.

It is my opinion, (based on having done a lot of migration work spanning 8.8 to 9.4 over the last several years), that the reason open source libraries took so long to upgrade to 9.0 and 9.2 is that they were specifically waiting on stackage to upgrade the compiler version, since they use stack files to provide the build plan locally and not cabal’s constraint solver, in which case, head.hackage would not have helped the end user here to prepare for the compiler bump. stackage in turn, refused to bump the compiler in nightly until a majority of the package set’s reverse dependencies built with the new versions of lens and aeson. This is clearly a chicken and egg problem, which may have more than one approach to it, but in my mind the simplest path forward was to try to remove the centralized dependency on stackage’s reverse dependency policy and allow people to define their own.

For me, when managing a package set it should not matter from a technical perspective whether a new compiler kicks out all reverse dependencies due to a new compiler feature, or whether the compiler upgrade is delayed to allow reverse dependencies to catch up, but should only matter that such behaviour is configurable and decidable by the parties interested. In some cases you want one behaviour, and in some cases you want the other. This is why I have opted for an approach where the package set can be self-hosted, and so you as a package set maintainer can decide the reverse dependency policy most suitable.

For example, horizon-core has the policy that ghc should track master as closely as possible, except that the horizon toolchain (brick, dhall and cabal2nix) must stay in the set.

horizon-platform has a much wider set (~1000 packages) at 9.4, which has a policy that approximately beam, persistent and servant have to build before ghc can advance or before it will consider kicking out other non-building reverse dependencies.

This reverse dependency policy behaviour isn’t formalised, but I believe it can be well-formed and encoded into logic. So, this is my take on the situation and my 2c here, that really the problem is more about clear and configurable stable package set management, so we can have build plans that are intelligible to different kinds of consumers, without either having to hamstring ghc or other kinds of unsatisfying solutions.

(But yes it’s completely nix based, is the only thing.)

edit: fixed the link

2 Likes

Ohh, and we used a dependency of which there is already a new major version. So most likely the old version we were depending on (could even be one of our own), won’t be updated, and we are urged to also upgrade that dependency. But this may now require changes to consumers of those libraries,

I would like to call out this part of what @angerman is saying. which I think had gone unnoticed. This is a very bad situation where to fix one sort of breakage one has to take on new source of breakage. That prevents the upgrade process from converging nearly as rapidly as it should.

Let’s leave aside the medium of head.hackage and just focus on its content. Fixes in head.hackage are never bundled with misc other breaking changes to libraries, and so they don’t have this problem.

We want to arrive at a situation where one is either upgrading the compiler, with possible breaking changes, or libraries, but never both at the same time. That is a necessary part of any solution. The question is how we get there. A few answer are:

  1. Encourage regular users to use head.hackage and ghc.xxxx.hackage.
  2. Get head.hackage fixes onto hackage as minor version releases the library author might otherwise not bother making.
  3. Release GHC more often so that we are less likely to “accumulate” other sources of breakage (such as library major versions) in the meantime.

I think they are all good and we should pursue them all!

3 Likes

Sorry. I shouldn’t have picked bytestring as an example because it ships with ghc, which confuses things. If you read my above post but pick say “aeson” or “lens” instead of bytestring, perhaps that will clarify things!

I realized I should add by the way that I mainly am not clamoring for new features (and also am not overly put out by the work I have to do as an open-source maintainer or on work projects to upgrade when newer compilers come out – though at work we tend to trail behind a bit, to stick close to what is provided by nix). I just try to notice and listen to all sides, and remember what sorts of things make lots of people frustrated.

I honestly still don’t think I understand what you have thought the purpose of head.hackage is for, because it seems like you think it is being promoted as a thing end-users should use. My sense is that it has always been intended only for upstream maintainers – both as “canary” and also as “prerelease”. It is just the case that the “unstick some package that has not been updated to continue testing more things” is a use-case that applies both to ghc developers and clc maintainers assessing stability and breakage, and also to library maintainers who want to do the same – just from the opposite standpoint!

Not necessarily end-users; and I also see why it is necessary given the fact that there can occationally be hard breakage, which prevents code from simply being compiled.

This assessment is correct. I do not see policies around

  • a limit on what an acceptable number of patches for packages for a compiler is.
  • that patches are being automatically removed after a given decay time. If we were unable to get the hackage package ready for a new compiler, why is this?
  • a deadline when head.hackage should not be necessary anymore? Should the set of packages we care the compiler to be compatible with on hackage, compile with -rc1 without head.hackage? How many weeks prior to the public X.Y.1 release should a subset (how large?) of hackage compile with the compiler without the need for head.hackage?

My primary issue really is that to me head.hackage is a (necessary–because we have frequent breakage) crutch, and it is not explicitly called out as something we ideally wouldn’t want, but see as necessary evil to balance necessary breakage during development, and the reality that maintainers can’t be expected to patch packages at a moments notice.

A very interesting question to me, is how this brittleness comes up? Do we provide enough warning time for packages (provide assistance with patches) when things break? I find it hard to believe that there would be 2+ GHC versions warning about a change (12+mo), and during this time no patch being apply, and release being made. If someone can provide me with concrete instances, where packages did not manage to get updated releases out in a reasonable time, despite warnings, I’d be more than happy to have dialogue trying to find out what the constraints are that result in no updates. This of course doesn’t address what @Ericson2314 highlighted, that it usually only leads to the most recent version being patched (thus forcing consumers to upgrade).

If someone can provide concrete examples of these brittle packages that broke despite warnings and migration assistance, I’d be quite curious to know.

2 Likes

I see it as a necesary evil. Every patch in head.hackage represents a problem. But it’s a very helpful canary: if we end up with a lot of patches in head.hackage that is tangible, concrete, quantifiable evidence of breakage that we can draw on to say “can we un-break this before release”. Perhaps we need a head.hackage manifesto?

A very interesting question to me, is how this brittleness comes up?

I gave a copule of examples above.

  1. There are also times when there is simply no good option. We want to fix an outright bug that users have reported in production. But alas other users are inadvertently relying on that same bug. We can’t satisfy both! Sometimes it is possible to have -XContinueWithBug, but sometimes that is really, really difficult to do. (And of course it slows down the entire bug-fixing process by adding friction. Yes, breakage is also friction; I’m just saying that there is a balance to be struck here.)

  2. Another example: we do not currently have a well-defined API for GHC itself. So clients who depend on GHC as a library reach deep into the belly of GHC and use functions that were never inteneded for external use. The only way to achieve zero breakage is to never change the signature or semantics of any function defined in any GHC module, ever. That’s clearly not sensible. So we should work on defining a decent API for GHC, a project that David Christiansen is leading. But that will take time, and meanwhile we want to be able to patch the packages that break.

I’ll add one more

  1. Template Haskell is a frequent cause of breakage. On the one hand, we want to extend TH’s data type of Haskell source code, so that it can reflect the source langauge that GHC compiles. On the other hand, doing so risks breaking packages that examine this source tree. Once again, it’s difficult/impossible to satisfy both clients with a single data type. There is a built-in tension here that is not simply due to thoughtlessness.

My point is that it is simply too brittle to insist on head.hackage being empty. My point is not that we should be blase about adding patches to head.hackage and consider problem solved.

2 Likes

I had this thought as well when thinking about this topic, and I’ve created an issue on the stability working group’s repo to coordinate efforts to improve template-haskell's stability: Stabilising template-haskell · Issue #16 · haskellfoundation/stability · GitHub

4 Likes