Language, library, and compiler stability (moved from GHC 9.6 Migration guide)

Not quite! Here is my thought process

  • I want to test GHC against 100x more packages than GHC acutally depends on, becuause I want do dicover early if I have inadvaertently introduced a breakage.
  • The only certain way I can do that is to actually compile all those 1000’s of packages.
  • But if just one of those packages fails to compile (for reasons discused upthread) I’m totally blocked; that makes me brittle
  • head.hackage means I can test GHC against a big package set, without being being totally blocked if (for good reasons or bad) one of those packages has not yet been updated in Hackage.

This is not theoretical. We have experienced extensive frustration with slow package updates. I am not blaming the authors for that – there can be legitimate reasons. I just want a resiliency mechanism, that allows us to do extensive testing without relying on 100.0% timely responses from package authors: that’s head.hackage.

So it very much isn’t just the packages that GHC itself depends on.

Moreover, this same mechanism would be useful to you too. You kindly offered

I wouldn’t mind testing nightly releases against large codebases and reporting bugs, regressions, …

But again, you can only do so if the nightly build comes with packages that actually compile with it.

Does that make sense?

I hope it’s clear that I am not arguing that breakage should be regarded as unimportant, or swept under the carpet. I just want our testing and deployment infrastructure to be resilient to the churn that (however well advertised) I think we are all committed to dealing with as the price of innovation.

4 Likes

I feel like at the core of this entire issue is that building the software commons that we’d like to see requires a lot of work. Sometimes it’s highly specialized work like creating a refactoring tool that can help adapt to breaking changes. Sometimes it’s communicating planned changes, or figuring out if a change can cause breakage. Sometimes it’s less specialized work like submitting one-line MRs to fix a changed import.

But at the end of the day it’s all work and often it’s work that people are doing for free.

I think when thinking about these things these considerations should be brought to the fore.
I think we should be asking ourselves things like:

  • how we can support each other as a community to lower each others load
  • how can we make each other feel appreciated and valued
  • how can we make communication as easy as possible
  • how can we lower barriers to get involved

I feel like too often we focus on technical things and erase the human component to our detriment.

11 Likes

I think it’s good to indeed clarify that we have all various vantage points. It’s very easy for a web application (which is my domain of expertise) to directly depend on eighty packages (and I’m not exaggerating) from Hackage, and three hundred indirectly.

I count 35 dependencies in the libraries directory of GHC, and so I understand a bit more that there can be a disconnect between the GHC team and downstream users of the Haskell ecosystem on the (sometimes) disproportionate effect that dependencies have on projects.

Edit: Our main product at work as 531 indirect dependencies, for a bit less than 700 modules.

1 Like

I think there’s an interesting question about how warnings help. Having warnings is no good unless you

  1. Build against a recent compiler where it is a warning
  2. Have the desire or time to fix them aggressively

Often this is not true for maintainers of packages. But Moritz points out that sometimes it is true for downstream. So that raises a question: could we find a way to encourage downstream consumers to help update packages that they depend on to fix compatibility issues? That might look something like: build some decent subset of Hackage with a recent GHC and -Werror=compat or some other way of sucking out the compat errors, make a big list, and make it available for people to pick up and submit patches for.

In some ways this is similar to what we might have hoped for with GHC.X.Hackage, except that the idea there was to have a focussed burst of fixing between an alpha release of a GHC version and its full release. Doing the same process based on compat warnings gives you much more time to handle things, but could still be amenable to community effort.

3 Likes

Let’s assume for a moment GHC 9.6 would accept code that 9.2 accepted warnings-free. We could put 9.6 alphas into CI today. And see which packages are a liability, and we’d have a serious incentive either (helping to) fix those packages. Or if that is not feasible, find alternatives instead.

We could also very proactively report regressions (it’s the same code between two compilers, if the new one exhibits bugs the old one did, or regresses, we could inform about this). As it is today, I can maybe tell you if 9.2 regresses against (a patched version) of 8.10. But I can’t tell this for the battle tested code against 8.10, as the migration to 9.2 required significant changes to the dependency tree. So I may be able to say if the new dependencies regressed against the old ones. And I may be able to say if 9.2 regresses against a patched 9.2; but what do I do with this information? 9.2 is pretty much end of line already. We have 9.4, 9.6 and master now.

As it is today, I have very little faith that putting 9.6 into CI today would yield anything that even remotely compiles. I’d love to be surprised, and I may perform this experiment next week if time permits.

I want to stress that downstream consumers of GHC (and the haskell ecosystem) have a vested interest in keeping their code working. This can either be by freezing it for all eternity, or by proactively helping along. But a compiler that doesn’t compile existing code, and requires significant surgery just to make existing (warning free!) code compatible, is not very useful.

4 Likes

I think @TeofilC’s observation is perhaps the most important observation in the whole thread.

3 Likes

@TeofilC I want to highlight these two points from your comment:

We rely on a lot of volunteer work. Especially a lot in the ecosystem of maintainers diligently looking after packages. Often in their spare time. And even if you are being paid to work on Haskell, package maintenance might not be on your job description.

In my opinion, being cautious and mindful of the downstream impact changes have goes a long way towards this. Getting a warning or deprecation notices that informs the user of the necessary changes reduces the burden to figure out what

Often times the breakage is not obvious. Taking the example I posted, the error messages are barely helpful.

Error 1:

Broken.hs:21:21: error: Operator applied to too few arguments: !
   |
21 | data X a = X { s :: ! a }
   |                     ^

:confused: what arguments are missing? I guess I could try to just remove the space? But the error doesn’t say: “did you intent to use a strictness annotation !a?”. After all BangPatterns is enabled.

Error 2:

Broken.hs:31:10: error:
    • Non type-variable argument in the constraint: MonadError D m
      (Use FlexibleContexts to permit this)
    • In the instance declaration for ‘C m T’
   |
31 | instance MonadError D m => C m T where
   |          ^^^^^^^^^^^^^^^^^^^^^^^

this one is actually quite good. It tells you that you need FlexibleContexts here. It doesn’t say that this used to be implied by UndecidableInstances though, which would have been helpful. (Fun fact: this error will not happen if you don’t have Haskell2010 enabled, something you might just have in your codebase because the stock cabal file contained the language stanza).

Error 3:

Broken.hs:38:10: error:
    Variable not in scope: (@) :: (String -> a0) -> t0 -> t1
   |
38 | h = read @ Int "5"
   |          ^

Broken.hs:38:12: error:
    • Illegal term-level use of the type constructor ‘Int’
        imported from ‘Prelude’ at Broken.hs:14:8-13
        (and originally defined in ‘GHC.Types’)
    • In the second argument of ‘(@)’, namely ‘Int "5"’
      In the expression: read @ Int "5"
      In an equation for ‘h’: h = read @ Int "5"
   |
38 | h = read @ Int "5"
   | 

Pretty much as unhelpful as (1).

Error 4:

Broken.hs:36:5: error:
    • No instance for (Data.Bifunctor.Bifunctor Y)
        arising from a use of ‘bimap’
    • In the expression: bimap id id Y
      In an equation for ‘g’: g = bimap id id Y
   |
36 | g = bimap id id Y
   |     ^^^^^

This one is so-so. It claims no instances was found, though the code to derive the instance is clearly there. It does not give any hint that there are now new staging/ordering restrictions.

The point I’m trying to highlight here (to exemplify @TeofilC’s first point) is that on a codebase with lots of code, you might not have written and are not intimately familiar with (someone else wrote that code, and maybe yet someone else reviewed it, and now you are spending your time looking at errors …), it is increasingly hard to decipher what’s wrong, and what the appropriate fix is. Especially if the file is multiple hundreds of lines.

While this thread might look like I’m complaining that GHC keeps breaking a production codebase, I don’t even think that’s the worst. There are people who are paid to look at this. What about all those poor folks who maintain packages in their spare time. Is their time best spent on dealing with continuous churn? I’d love to help out! If you are paid to work on a haskell codebase, and can trivially test that codebase against a recent compiler, instead of waiting until everyone else has done the work for you, you could proactively help migrating packages you rely on. This all however can only work if you can just swap out the compiler trivially.

As I’ve alluded to before, experience over the last years has been that there is little point in trying a compiler for any serious project before we haven’t hit at least .4, too much is simply broken hard, and simply won’t compile. Without using the compiler, no one will know about all the migrations necessary.

3 Likes

An idea floated around before is to gate a new release of ghc until haskell-language-server has caught up and ready to be used with it. Even without making this a hard rule, I think it would be useful to the GHC team to take haskell-language-server as a benchmark, it is becoming a central piece of the ecosystem and it has enough depedencies to give an assessment of the changes required to support a new GHC (yes, intentionally, including the GHC API itself).

8 Likes

Thank you for clarifying this @simonpj!

Why does this lead to “let’s patch the packages” instead of, “can we make the compiler more resilient to accept existing code”?

No, I could do that if the compiler accepted existing code (potentially with warnings), and not abruptly stops accepting existing code.

To come back to an earlier idea around a -DEXPERIMENTAL compiler, which could break arbitrarily and without any notice. Having a experimental.hackage overlay for this seems sensible to me.

2 Likes

There seems to be some absolute disconnect here. Suppose I want to make sure that the compiler only introduces features that don’t abruptly break a lot of code. (Good idea, right?). So how do I know if a feature abruptly breaks a lot of code? Well, I try to compile a lot of code! But whoops, the compiler breaks one thing in bytestring. Now, I can’t compile any code, because bytestring is broken. So how can I even begin to test if a compiler does or does not break packages if I do not have a head.hackage type thing to unstick me?

You might say, “ok, then don’t break bytestring”, which I think is what you mean when you say “can we make the compiler more resilient to accept existing code”. But – the whole point is, suppose that bytestring is the only thing that breaks, well, we’d like to at least know that, especially if we can just ship a new bytestring along the way and be done with it. But if we do not have a head.hackage, we cannot possibly know that. You might say “who cares, it cannot possibly matter the scope of breakage, any breakage at all without a full cycle is too much!” But first, we have established that some is unavoidable (such as fixing primitive types for architecture support reasons).

But more importantly, I think the process you advocate would make progress unbearably slow. It would discourage contributions to the compiler, and it would also, frankly, enrage the sizable contingent of end-users who are always clamoring for features.

Let me drag out again my haskell users quadrant (from A tick-tock release cycle for GHC by bgamari · Pull Request #34 · haskellfoundation/tech-proposals · GitHub)

       Developer Maintainer
      ┌─────────┬─────────┐
    L │         │         │
    i │         │         │
    b │         │         │
      │         │         │
      ├─────────┼─────────┤
    A │         │         │
    p │         │         │
    p │         │         │
      │         │         │
      └─────────┴─────────┘

This would go straight down the middle. App and Lib maintainers would of course be thrilled. They could make changes at an incredibly leisurely pace. App and Lib developers would be much less happy. The features they want and fixes to improve their experience would be terribly far away. And bear in mind that as a relatively small language that continues to grow, we really do need to promote end-user adoption, which means keeping developers happy, since the work new users getting into the language will do is definitionally new development. I am already imagining the enraged threads on the discourse explaining how Haskell is a stagnant language that refuses to fix known issues in a timely fashion for the sake of compatibility, and etc.

So again, what I want to underline is that you have very real concerns. But they must be balanced against other concerns, and to do so means having a concrete assessment of potential breakage. And at the moment, it seems to remain the case that the best way to have that is to have a solid head.hackage process which is widely used. With that, we can concretely navigate tradeoffs. Without it, we are left with only extremes which will necessarily make some group or another feel very unrepresented and unheard.

3 Likes

Am I correct in understanding that head.hackage is widely misunderstood? :smiley: It seems to me that there is a useful community service lurking that just needs to get better understood and developed into something more broadly usable.

I myself don’t think I know it perfectly, so please correct me if I’m wrong:

It sounds like head.hackage serves two distinct purposes: one is to notice breakages (intentional or otherwise) caused by GHC development, and the other is to update libraries in advance when there are (intentional) breaking changes in GHC. Let’s call those “Canary Hackage” and “Prerelease Hackage”, if you will.

Here are some examples of Canary Hackage:

  1. An unintentional breakage was noticed and fixed: Core lint error on head.hackage lint build (#22357) · Issues · Glasgow Haskell Compiler / GHC · GitLab
  2. Another one, an instance where fixing one bug caused another one: head.hackage: package "what4" no longer compiles (#22519) · Issues · Glasgow Haskell Compiler / GHC · GitLab
  3. Something totally different: tracking GHC memory usage over time as it compiles head.hackage packages

And an example of Prerelease Hackage:

  1. Adding the Functor superclass on Bifunctor broke packages: head.hackage: packages failing due to new Functor superclass on Bifunctor (#22567) · Issues · Glasgow Haskell Compiler / GHC · GitLab

As people have highlighted up thread, GHC devs absolutely need Prerelease Hackage, but it’s Canary Hackage that may have broader appeal among maintainers (using @sclv’s distinction between maintainers and developers). I just went back and read the summary of Introduce GHC.X.hackage, and perhaps it got off on the wrong foot by only mentioning Prerelease Hackage?

3 Likes

Me too. So all of what follows may be total nonsense…


Over in this thread angerman made a simple suggestion: put all breaking changes under flags. This could be combined with santiweight’s suggestion from this “tick-tock release” thread as follows:

  • in ghc-v.k.m (v constant) - breaking changes can only be enabled by using their respective flags

  • in ghc-v+1.k.m (new major release) - breaking changes (those which have proven to be reliable and widely-acceptable) from version v are then enabled by default, but can still be disabled.

  • in ghc-v+2.k.m (next major release) - assuming no further problems, breaking changes from version v are made permanent (they cannot be disabled).

Haskell developers can then use breaking features by enabling the appropriate flags (before the new major release arrives), and Haskell maintainers only have to be concerned with widespread breakage upon the arrival of each new major release.

2 Likes

Without judgement implied in any form, I find your comment quite interesting @sclv

We arrive at the same conclusion but with almost opposite assumptions, I consider the current state to be very discouraging towards contributions. We basically contribute despite all this to GHC, and consider a much more stable (non-breaking without warning) situation much more welcoming to contributions. But then I’ll add that the contributions I care about are almost always orthogonal to language changes (linker improvements, better system compatibilities, new codegen backends, new compilation targets, …).

You and I also seem to be frequenting very different circles of end users.

You are however right in the assumption that I would want warnings prior to breakage, ideally accompanied with a note as to how to migrate.

You bring up two points in here. One is base libraries shipped with GHC. The other is a limit on how many packages you are willing to patch to compile a larger set of code. Is the limit 1, 5, 10? Is it constrained to only base libraries?

To avoid any misunderstanding, if you need head.hackage during development, I don’t see an issue with this at all. If head.hackage is required when the compiler is released, then, yes I do see an issue with that. head.hackage should at best be a GHC internal development crutch. An end-user would under most circumstances not notice this bytestring breakage (if that’s the only one) or really any to the shipped packages with GHC, especially those that cabal considers non-reinstallable. If releases for those compatible with the new compiler are on hackage prior to the compiler release, there would be no need of head.hackage (outside of ghc development).

I have in the past proposed different other strategies:

  • have a stable and a development branch in GHC.
  • have a tick-tock cycle (only every other release can break code).

We used to have one release per year, which mean you had breakage once a year, now we have two. We now expect maintainers to adapt their code to a new compiler every 6mo.

2 Likes

The first port of call is indeed “make the compiler more resilient”, but sometimes that is very difficult. Suppose a change to the signature of a function in base has been changed, after due deliberation by CLC and with deprecation cycles. But now it has finally landed. Alas some package maintainer has failed to update. That blocks the entire dependency tree of that package.

That is the brittleness that I referred to. I don’t want to have to rely on the 100.0% cooperation of thousands of library authors, some of whom will have legitmate reasons for being out of action. And it’s extremely difficult to make the compiler compile both programs that expect f :: Int -> Int and programs that expect the same f :: Int -> Int -> Int.

@chreekat’s analysis is spot on. I want both Canary Hackage and Prerelease Hackage.

I think perhaps @angerman fears that head.hackage will be a mechanism for saying “oh well, breaking changes don’t matter”. That is, if you crack open the door, someone will drive a lorry though it. But it doesn’t have to be that way – we can simply be vigilant.

There are also times when there is simply no good option. We want to fix an outright bug that users have reported in production. But alas other users are inadvertently relying on that same bug. We can’t satisfy both! Sometimes it is possible to have -XContinueWithBug, but sometimes that is really, really difficult to do. (And of course it slows down the entire bug-fixing process by adding friction. Yes, breakage is also friction; I’m just saying that there is a balance to be struck here.)

Another example: we do not currently have a well-defined API for GHC itself. So clients who depend on GHC as a library reach deep into the belly of GHC and use functions that were never inteneded for external use. The only way to achieve zero breakage is to never change the signature or semantics of any function defined in any GHC module, ever. That’s clearly not sensible. So we should work on defining a decent API for GHC, a project that David Christiansen is leading. But that will take time, and meanwhile we want to be able to patch the packages that break.

My main point is that it’s a bit more nuanced than it might at first appear. head.hackage is a fantastic tool for adding resilience. It should not (and I think is not) be used as a way to smuggle in breaking changes without careful thought.

Does that help at all? This is a useful debate, I think.

(For what it’s worth, the GHC team spends a lot of time on back-compat issues. We really do try hard. We may make mistakes, but I think we are getting better. Canary Hackage is part of that getting better.)

3 Likes

Hi everyone.

This is the problem that horizon-haskell intends to solve. I’m quite confident this will be more comprehensive than head.hackage, serve more usecases, and be easier for package maintainers to integrate ahead of time.

It is my opinion, (based on having done a lot of migration work spanning 8.8 to 9.4 over the last several years), that the reason open source libraries took so long to upgrade to 9.0 and 9.2 is that they were specifically waiting on stackage to upgrade the compiler version, since they use stack files to provide the build plan locally and not cabal’s constraint solver, in which case, head.hackage would not have helped the end user here to prepare for the compiler bump. stackage in turn, refused to bump the compiler in nightly until a majority of the package set’s reverse dependencies built with the new versions of lens and aeson. This is clearly a chicken and egg problem, which may have more than one approach to it, but in my mind the simplest path forward was to try to remove the centralized dependency on stackage’s reverse dependency policy and allow people to define their own.

For me, when managing a package set it should not matter from a technical perspective whether a new compiler kicks out all reverse dependencies due to a new compiler feature, or whether the compiler upgrade is delayed to allow reverse dependencies to catch up, but should only matter that such behaviour is configurable and decidable by the parties interested. In some cases you want one behaviour, and in some cases you want the other. This is why I have opted for an approach where the package set can be self-hosted, and so you as a package set maintainer can decide the reverse dependency policy most suitable.

For example, horizon-core has the policy that ghc should track master as closely as possible, except that the horizon toolchain (brick, dhall and cabal2nix) must stay in the set.

horizon-platform has a much wider set (~1000 packages) at 9.4, which has a policy that approximately beam, persistent and servant have to build before ghc can advance or before it will consider kicking out other non-building reverse dependencies.

This reverse dependency policy behaviour isn’t formalised, but I believe it can be well-formed and encoded into logic. So, this is my take on the situation and my 2c here, that really the problem is more about clear and configurable stable package set management, so we can have build plans that are intelligible to different kinds of consumers, without either having to hamstring ghc or other kinds of unsatisfying solutions.

(But yes it’s completely nix based, is the only thing.)

edit: fixed the link

2 Likes

Ohh, and we used a dependency of which there is already a new major version. So most likely the old version we were depending on (could even be one of our own), won’t be updated, and we are urged to also upgrade that dependency. But this may now require changes to consumers of those libraries,

I would like to call out this part of what @angerman is saying. which I think had gone unnoticed. This is a very bad situation where to fix one sort of breakage one has to take on new source of breakage. That prevents the upgrade process from converging nearly as rapidly as it should.

Let’s leave aside the medium of head.hackage and just focus on its content. Fixes in head.hackage are never bundled with misc other breaking changes to libraries, and so they don’t have this problem.

We want to arrive at a situation where one is either upgrading the compiler, with possible breaking changes, or libraries, but never both at the same time. That is a necessary part of any solution. The question is how we get there. A few answer are:

  1. Encourage regular users to use head.hackage and ghc.xxxx.hackage.
  2. Get head.hackage fixes onto hackage as minor version releases the library author might otherwise not bother making.
  3. Release GHC more often so that we are less likely to “accumulate” other sources of breakage (such as library major versions) in the meantime.

I think they are all good and we should pursue them all!

3 Likes

Sorry. I shouldn’t have picked bytestring as an example because it ships with ghc, which confuses things. If you read my above post but pick say “aeson” or “lens” instead of bytestring, perhaps that will clarify things!

I realized I should add by the way that I mainly am not clamoring for new features (and also am not overly put out by the work I have to do as an open-source maintainer or on work projects to upgrade when newer compilers come out – though at work we tend to trail behind a bit, to stick close to what is provided by nix). I just try to notice and listen to all sides, and remember what sorts of things make lots of people frustrated.

I honestly still don’t think I understand what you have thought the purpose of head.hackage is for, because it seems like you think it is being promoted as a thing end-users should use. My sense is that it has always been intended only for upstream maintainers – both as “canary” and also as “prerelease”. It is just the case that the “unstick some package that has not been updated to continue testing more things” is a use-case that applies both to ghc developers and clc maintainers assessing stability and breakage, and also to library maintainers who want to do the same – just from the opposite standpoint!

Not necessarily end-users; and I also see why it is necessary given the fact that there can occationally be hard breakage, which prevents code from simply being compiled.

This assessment is correct. I do not see policies around

  • a limit on what an acceptable number of patches for packages for a compiler is.
  • that patches are being automatically removed after a given decay time. If we were unable to get the hackage package ready for a new compiler, why is this?
  • a deadline when head.hackage should not be necessary anymore? Should the set of packages we care the compiler to be compatible with on hackage, compile with -rc1 without head.hackage? How many weeks prior to the public X.Y.1 release should a subset (how large?) of hackage compile with the compiler without the need for head.hackage?

My primary issue really is that to me head.hackage is a (necessary–because we have frequent breakage) crutch, and it is not explicitly called out as something we ideally wouldn’t want, but see as necessary evil to balance necessary breakage during development, and the reality that maintainers can’t be expected to patch packages at a moments notice.

A very interesting question to me, is how this brittleness comes up? Do we provide enough warning time for packages (provide assistance with patches) when things break? I find it hard to believe that there would be 2+ GHC versions warning about a change (12+mo), and during this time no patch being apply, and release being made. If someone can provide me with concrete instances, where packages did not manage to get updated releases out in a reasonable time, despite warnings, I’d be more than happy to have dialogue trying to find out what the constraints are that result in no updates. This of course doesn’t address what @Ericson2314 highlighted, that it usually only leads to the most recent version being patched (thus forcing consumers to upgrade).

If someone can provide concrete examples of these brittle packages that broke despite warnings and migration assistance, I’d be quite curious to know.

2 Likes

I see it as a necesary evil. Every patch in head.hackage represents a problem. But it’s a very helpful canary: if we end up with a lot of patches in head.hackage that is tangible, concrete, quantifiable evidence of breakage that we can draw on to say “can we un-break this before release”. Perhaps we need a head.hackage manifesto?

A very interesting question to me, is how this brittleness comes up?

I gave a copule of examples above.

  1. There are also times when there is simply no good option. We want to fix an outright bug that users have reported in production. But alas other users are inadvertently relying on that same bug. We can’t satisfy both! Sometimes it is possible to have -XContinueWithBug, but sometimes that is really, really difficult to do. (And of course it slows down the entire bug-fixing process by adding friction. Yes, breakage is also friction; I’m just saying that there is a balance to be struck here.)

  2. Another example: we do not currently have a well-defined API for GHC itself. So clients who depend on GHC as a library reach deep into the belly of GHC and use functions that were never inteneded for external use. The only way to achieve zero breakage is to never change the signature or semantics of any function defined in any GHC module, ever. That’s clearly not sensible. So we should work on defining a decent API for GHC, a project that David Christiansen is leading. But that will take time, and meanwhile we want to be able to patch the packages that break.

I’ll add one more

  1. Template Haskell is a frequent cause of breakage. On the one hand, we want to extend TH’s data type of Haskell source code, so that it can reflect the source langauge that GHC compiles. On the other hand, doing so risks breaking packages that examine this source tree. Once again, it’s difficult/impossible to satisfy both clients with a single data type. There is a built-in tension here that is not simply due to thoughtlessness.

My point is that it is simply too brittle to insist on head.hackage being empty. My point is not that we should be blase about adding patches to head.hackage and consider problem solved.

2 Likes

I had this thought as well when thinking about this topic, and I’ve created an issue on the stability working group’s repo to coordinate efforts to improve template-haskell's stability: Stabilising template-haskell · Issue #16 · haskellfoundation/stability · GitHub

4 Likes