Priorities for upcoming GHC releases

Links
#21

Just to be clear: The GHC Project officially supports FreeBSD as a Tier2 platform.
(See platforms · Wiki · Glasgow Haskell Compiler / GHC · GitLab)

Are you saying contributions to fix less supported platforms are not welcome? I’m not sure that’s what I got from GHC devs.

We absolutely welcome fixes and improvements that improve GHC on all these platforms, including FreeBSD.

However because of lack of resources GHC HQ can’t support FreeBSD as a Tier 1 platform. And therefore just like for other Tier 2 platforms we do not provide official distribution for these platforms, generally don’t prioritize fixing bugs which only happen on these platforms, and don’t consider bugs on these platforms as release critical.

4 Likes
Monthly DevOps Logs
#22

As a GHC developer I of course want every release to be good enough to be a recommended release. Even if that isn’t always possible.

I agree with @chreekat that there is cost associated with recommending older versions but I can understand that sometimes it won’t be possible to recommend newer versions due to issues with GHC itself, or issues in the wider ecosystem.

Personally I think there is a lot of value in the recommended version being one that is still seeing releases. Primarily because it allows for far quicker feedback loops between bugs being discovered and users being able to use a fixed version of GHC.

Ultimately it’s up to GHCup to make a choice for their project by weighing the costs and benefits of any particular pick. Even when the GHCup project chooses a version older than the one I would have preferred I really appreciate the project making newer version easily accessible to a lot of people.

5 Likes
#23

I’ll have to be blunt. I’m not sure GHC HQ is feeling the pulse of which GHC version is:

  • popular
  • used in industry
  • widely supported by the ecosystem
  • widely supported by distributors

Their release schedule is catered towards GHC contributors and getting changes in without dragging them through long running branches.

I understand that, because they’re dealing with resource constraints.

But at the same time it’s incredibly disconnected from the community.

GHC 8.10.7 used to be popular for a very long time and way past its EOL, not receiving any further patches.

So the reality is: the priorities of distributors and GHC HQ may never align. Please look at the GHC versions packaged by mainstream distros like ubuntu.

It would be great if I could always recommend a GHC version that is marked as receiving bugfix releases. But it’s very unrealistic.

4 Likes
#24

That’s understandable. I don’t want GHCup to recommend an active branch just because it’s active or anything like that. But looking at the RFC thread about recommending 9.6 [RFC][GHCup] Should GHC 9.6.4 be 'recommended'? - #3 by juhp it doesn’t seem like the list of things preventing a 9.6 recommendation is that long, so I hope that 9.6 being the recommendation and 9.6 still being maintained will overlap in the future.

#25

Perhaps you are right.

But to be constructive, what alternative policy would you advocate, within the current resource budget? Perhaps you are advocating that we should actively maintain one or more older release(s)? But which ones specifically? And in a world of finite resources which other releases would you advocate not maintaining?

I think you are right to say that there are some built-in tensions between stability one on the one hand, and dynamism on the other. But these are tensions I am happy for us to grapple with together. I don’t think anyone wants GHC to cease development and become bug-fixes only! And on the other hand, as you know, everyone is taking stability very seriously these days, so things are much better now than in the past. Not perfect, but better.

One real tension is that it’s more expensive to maintain an old branch than a newer one, because backporting fixes becomes harder, sometimes much harder, as HEAD diverges from that branch. So there’s a resource implication there too.

So this is a good conversation to have. What release policy would you like to see? Is there a consensus around such a policy? If there was a consensus from our users about what different policy we should follow, I think we’d take that very seriously.

15 Likes
#26

I’d like to see LTS releases. So not just “the last 3 releases”.

But to be fair: my post maybe sounded like I’m expecting WT to do this. No, I think it’s fair that other companies and volunteers can step up to do that.

But GHC HQ has to be welcoming to such an effort. Because it is not trivial.

I don’t have someone specific in mind who could do that, but I think it’s clear that distributors and end users would be excited about that (I’m deriving that lazily from past Haskell survey results and anecdotal evidence).

As to how to decide what becomes an LTS version, I think that should roughly follow the patterns that stack, ghcup and other distributors use:

  • a branch with few known bugs, defects and regressions
  • UX across platforms is mostly consistent (8.10.7 was in fact a problematic release, because it was rock stable on most platforms, but extremely experimental on aarch64 darwin)
  • large ecosystem support
  • opinions of distributors
  • maybe a yearly survey

I don’t really believe in making a branch LTS up-front or based on version numbers/cadence. It should be something that crystallizes.

I’m curious to see what the Haskell Foundation and the Stability Working Group thinks about this.

It would be much easier for distributors to make such decisions about default versions if there were LTS releases.

12 Likes
#27

Can you be more specific? Perhaps you are saying:

  • Identify certain releases as Long Term Support
  • Maintain LTS releases for much longer

But then I wonder

  • How many actively maintained LTS releases should there be?
  • How long should they be maintained?
  • How can we do that within the same resource envelope? For each actively maintained LTS release we’d have to stop supporting something else. But what?

A possible response to the resource question might be, as you say:

I think it’s fair that other companies and volunteers can step up to do that. But GHC HQ has to be welcoming to such an effort. Because it is not trivial.

I’d be thrilled if a group stepped up to the challenge of supporting LTS releases. I, for one, would definitely welcome such an effort. (Assuming it was sustained, and doesn’t flicker out after a few months. This is a tricky challenge with volunteers – I have myself encouraged/supported efforts that I have subsequently felt unable to sustain, to my embarrassment.)

10 Likes
#28

Correct.

At the moment I don’t see how the 3-release cadence is useful for end users. It’s useful for GHC developers and contributors when planning large changes and keeping backport madness to a minimum as you said.

But the end user doesn’t update their GHC version every half year, do they? Companies sometimes wait 1-2 years before considering a new version.

So I feel there’s a gap.

I think we have to start small and say: one.

I’d like to see a span of 2 additional years (that is… the version is considered EOL, but so popular that it’s promoted to LTS).

They should receive bugfixes and performance improvements.

I’m not sure we can do that within the same resource envelope.

It would probably need some refinement of the release process as a whole.

I don’t want to pretend to know what the daily pain points of GHC development are and where time is spent the most.

But I think we wouldn’t really need 3 supported branches if we had:

  • properly working nightlies (things go to master fast and people and companies can test new GHC features)
  • LTS branches for popular versions

Wouldn’t that satisfy both parties?

At the moment I feel the release model is somewhere in between and neither satisfies those that want to push through large changes quickly, nor those that don’t care about the new stuff.

Maybe I missed something though.

3 Likes
#29

[…] properly working nightlies […]

Hrm:

Unless of course you’re putting together a taskforce to get the nightlies “properly working” ; then go for it!

1 Like
#30

Wouldn’t that amount to more than 3 supported branches? (Just some of them would be labeled “LTS”).

And if we didn’t know up front what branches would “crystalize” into LTS at what point would we decide?

Let me spitball an idea here: GHC HQ supports the latest two branches, and some third branch. The third branch, rather than being just the “prior” branch, is whatever a regular industry survey alights on as the consensus most important branch (subject to GHCHQ saying “lol, you picked 8.4, we’re not supporting that, our window will have to be more recent”). Whatever the survey picks remains the supported branch for a year or so, thus giving a bit more stability. It may well be that the industry survey does alight on the third-most-recent branch and this policy works out the same – but it gives the flexibility to choose otherwise.

2 Likes
#31

May I ask, how do GHC major version numbers work? Does a major bump always mean “something that used to work (in a previous major version) no longer works” or could it mean only “we have added something new” (and existing functionality is intended to be unaffected)?

EDIT: Is it that each version of GHC specifies one version of the base library and, so, a major bump in the latter requires a major bump in the former?

1 Like
#32

I remember the discussion about GHC 8.12, which became GHC 9.0;
Basically, people felt that linear types were a headline feature, so the versioning was changed to reflect this.

1 Like
#33

I really meant from, say, GHC 9.8 to 9.10 rather than first number version bumps.

EDIT: To elaborate, I mean ‘GHC as a tool’, as opposed to ghc as a library. When I read, say, 2.1. Version 9.10.1 — Glasgow Haskell Compiler 9.10.1 User's Guide the ‘breaking changes’ from GHC 9.8, as a tool, do not jump out at me.

#34

I really meant from, say, GHC 9.8 to 9.10 rather than first number version bumps.

The policy as I understand it is this:

  • Each six-monthly release of GHC uses a 0.2 version bump, thus from 9.8 to 9.10, or 9.10 to 9.12.
  • “Patch-level releases” change only the minor version. Thus 9.8.1, 9.8.2, 9.8.3 etc. These patch level releases fix bugs but are intended to change absolutely nothing else, not features, not APIs, nothing.
  • The odd-level major versions, 9.7.X, 9.9.X, etc are used for development builds, never for releases.
  • We do not have a formal policy for moving from 8.X to 9.X, or from 9.X to 10.X.

Would it be helpful to write down these four points? For example here: releases · Wiki · Glasgow Haskell Compiler / GHC · GitLab

4 Likes
#35

Thanks! What is on my mind is this:

‘industry’ has to think about the ‘costs’ of change as well as the ‘benefits’. However, if change actually ‘breaks’ nothing (or little), only ‘adds’, the ‘costs’ of change might be low. I am wondering if we (as a community) could do more to communicate what is actually ‘broken’ by change, making it easier for ‘industry’ to identify and assess that ‘cost’ and - perhaps - lowering the perceived barrier to moving forward.

(EDIT1: Taking this opportunity to acknowledge @tomjaguarpaw’s Upgrading from GHC 8.10 to GHC 9.6: an experience report)

(EDIT2: I may have underestimated what the GHC project already does, given the migration guides for each of GHC 7.8 to GHC 9.12.)

2 Likes
#36

That suggestion sounds resaonable, but it’s surprisingly hard to execute.

  • New features are (increasingly) not a source of breakage; even when they are, you often get a deprecation cycle or two. See GHC Steering committee stability principles

  • Fixing bugs, on the other hand, really can cause breakage – and does! It turns out that people sometimes unwittingly rely on bugs. It’s really really hard to predict this.

  • Library changes are a major source of breakage. This is silly; just because you change from GHC X to GHC X+1 doesn’t mean you should have to change libraries. But currently you do – See GHC stability state of play

It’s hard to know how to communicate all this better, but if there are simple things we can do that would help our users, we’re all ears. Perhaps you can float some ideas? (Remembering the opportunity cost: doing X means not doing Y.)

8 Likes
#37

We could have:

  1. latest stable branch (mind you, this one can be a little slower too if we have nightlies… there’s less pressure to get releases out “just because”)
  2. previous stable branch
  3. LTS branch

Then we always have two candidates for LTS.

But that might mean to choose a different release cycle (more than half a year for a new major version). Otherwise the versions “run” too fast. I believe that’s already the case, but it will be worse when there’s only two short term supported branches.

The way I see it is that the current “stable” releases are (ab)used to get experimental features out. This can all be delivered with well working nightlies support instead.

I find it hard to foresee the involved workload for GHC devs though. But this makes sense for end users imo.

#38

So instead of nightly builds, what about monthly or “fortnightly” (i.e. twice a month) releases? If that can be made to work…then weekly builds could be possible. But the impression I’m getting is that 28-31 builds per month is too large an opportunity cost, amongst others:

#39

Could you outline what you mean by “latest/previous stable branch”? What goes on those branches?

We are about to release 9.12. Perhaps you mean something like the 9.10 (latest) and 9.8 (previous) series? When 9.12 is released, then “nightly” will follow 9.13 (what will become 9.14), “latest” will follow 9.12 and “previous” will follow 9.10. Correct?

This means we can afford one additional LTS branch. Even that is a stretch, because the cost of backporting fixes rises as the LTS branch diverges more from master, so maintaining a 2.5 year old LTS release is more costly than additionally maintaining 1 year old 9.6.

Moreover, how do you pick the new LTS release? That is, after we have used GHC 8.10 for 3 years (1 year regular support + 2 years LTS), what is the next LTS version? Should it be 9.8 or 9.10? What if 9.8 is (sadly) known to be bug-ridden, 9.10 is better but it’s too fresh to judge? What is the policy here? Keep maintaining 8.10 for another half year (but the maintenance cost!) or embrace 9.10 instead? I’m uncertain how industry users respond to immediately having to migrate from 8.10 to 9.10 because the former lost its LTS status in favour of the latter. I appears that we need to notify users at least half a year before we make the switch from 8.10 to 9.10, so that they may migrate from one supported version to another. So, in practice 3 year LTS means that LTS releases are 2.5 years apart to have overlapping support windows.

3 Likes
#40

My preference is for our limited capacity to be spent working towards making it easier to upgrade GHC (better stability guarantees, tooling for upgrades, etc) rather than maintaining old versions.

Maintaining an LTS allows you to delay upgrading your GHC. But you will still eventually have to do it.

Upgrading to a new version of GHC can be broken down into the cost of upgrading your own code, and upgrading your dependencies. The former is a fixed cost. The second is more variable. When a new GHC is released many libraries will need version bumps, etc, to be compatible with the new version. An early adopter would have to do all this work themselves. But as time goes on the community will do more of this work. In my experience, most patches get made and released relatively quickly, but there’s often a long-tail of packages that take longer to be made compatible.

Waiting a bit longer before trying to upgrade is therefore helpful because you hope that other people will have done the work to upgrade libraries in the ecosystem. But I think you eventually get diminishing returns. But if everyone waits, then no one writes the patches(!). I think we often see this dynamic. For instance, when Stackage nightly gets bumped there are a flurry of patches, but it would be great if people started this work sooner.

I’m quite excited for more core libraries becoming reinstallable (template-haskell, base) with coming releases of GHC. I think this will make upgrading orders of magnitude easier. With that and other stability improvements we’ve already had, I’m really hoping that more people will upgrade sooner, and we can all reap the rewards from that.

10 Likes