That’s understandable. I don’t want GHCup to recommend an active branch just because it’s active or anything like that. But looking at the RFC thread about recommending 9.6 [RFC][GHCup] Should GHC 9.6.4 be 'recommended'? - #3 by juhp it doesn’t seem like the list of things preventing a 9.6 recommendation is that long, so I hope that 9.6 being the recommendation and 9.6 still being maintained will overlap in the future.
Perhaps you are right.
But to be constructive, what alternative policy would you advocate, within the current resource budget? Perhaps you are advocating that we should actively maintain one or more older release(s)? But which ones specifically? And in a world of finite resources which other releases would you advocate not maintaining?
I think you are right to say that there are some built-in tensions between stability one on the one hand, and dynamism on the other. But these are tensions I am happy for us to grapple with together. I don’t think anyone wants GHC to cease development and become bug-fixes only! And on the other hand, as you know, everyone is taking stability very seriously these days, so things are much better now than in the past. Not perfect, but better.
One real tension is that it’s more expensive to maintain an old branch than a newer one, because backporting fixes becomes harder, sometimes much harder, as HEAD diverges from that branch. So there’s a resource implication there too.
So this is a good conversation to have. What release policy would you like to see? Is there a consensus around such a policy? If there was a consensus from our users about what different policy we should follow, I think we’d take that very seriously.
15 Likes
I’d like to see LTS releases. So not just “the last 3 releases”.
But to be fair: my post maybe sounded like I’m expecting WT to do this. No, I think it’s fair that other companies and volunteers can step up to do that.
But GHC HQ has to be welcoming to such an effort. Because it is not trivial.
I don’t have someone specific in mind who could do that, but I think it’s clear that distributors and end users would be excited about that (I’m deriving that lazily from past Haskell survey results and anecdotal evidence).
As to how to decide what becomes an LTS version, I think that should roughly follow the patterns that stack, ghcup and other distributors use:
- a branch with few known bugs, defects and regressions
- UX across platforms is mostly consistent (8.10.7 was in fact a problematic release, because it was rock stable on most platforms, but extremely experimental on aarch64 darwin)
- large ecosystem support
- opinions of distributors
- maybe a yearly survey
I don’t really believe in making a branch LTS up-front or based on version numbers/cadence. It should be something that crystallizes.
I’m curious to see what the Haskell Foundation and the Stability Working Group thinks about this.
It would be much easier for distributors to make such decisions about default versions if there were LTS releases.
11 Likes
Can you be more specific? Perhaps you are saying:
- Identify certain releases as Long Term Support
- Maintain LTS releases for much longer
But then I wonder
- How many actively maintained LTS releases should there be?
- How long should they be maintained?
- How can we do that within the same resource envelope? For each actively maintained LTS release we’d have to stop supporting something else. But what?
A possible response to the resource question might be, as you say:
I think it’s fair that other companies and volunteers can step up to do that. But GHC HQ has to be welcoming to such an effort. Because it is not trivial.
I’d be thrilled if a group stepped up to the challenge of supporting LTS releases. I, for one, would definitely welcome such an effort. (Assuming it was sustained, and doesn’t flicker out after a few months. This is a tricky challenge with volunteers – I have myself encouraged/supported efforts that I have subsequently felt unable to sustain, to my embarrassment.)
10 Likes
Correct.
At the moment I don’t see how the 3-release cadence is useful for end users. It’s useful for GHC developers and contributors when planning large changes and keeping backport madness to a minimum as you said.
But the end user doesn’t update their GHC version every half year, do they? Companies sometimes wait 1-2 years before considering a new version.
So I feel there’s a gap.
I think we have to start small and say: one.
I’d like to see a span of 2 additional years (that is… the version is considered EOL, but so popular that it’s promoted to LTS).
They should receive bugfixes and performance improvements.
I’m not sure we can do that within the same resource envelope.
It would probably need some refinement of the release process as a whole.
I don’t want to pretend to know what the daily pain points of GHC development are and where time is spent the most.
But I think we wouldn’t really need 3 supported branches if we had:
- properly working nightlies (things go to master fast and people and companies can test new GHC features)
- LTS branches for popular versions
Wouldn’t that satisfy both parties?
At the moment I feel the release model is somewhere in between and neither satisfies those that want to push through large changes quickly, nor those that don’t care about the new stuff.
Maybe I missed something though.
3 Likes
[…] properly working nightlies […]
Hrm:
Unless of course you’re putting together a taskforce to get the nightlies “properly working” ; then go for it!
1 Like
Wouldn’t that amount to more than 3 supported branches? (Just some of them would be labeled “LTS”).
And if we didn’t know up front what branches would “crystalize” into LTS at what point would we decide?
Let me spitball an idea here: GHC HQ supports the latest two branches, and some third branch. The third branch, rather than being just the “prior” branch, is whatever a regular industry survey alights on as the consensus most important branch (subject to GHCHQ saying “lol, you picked 8.4, we’re not supporting that, our window will have to be more recent”). Whatever the survey picks remains the supported branch for a year or so, thus giving a bit more stability. It may well be that the industry survey does alight on the third-most-recent branch and this policy works out the same – but it gives the flexibility to choose otherwise.
2 Likes
May I ask, how do GHC major version numbers work? Does a major bump always mean “something that used to work (in a previous major version) no longer works” or could it mean only “we have added something new” (and existing functionality is intended to be unaffected)?
EDIT: Is it that each version of GHC specifies one version of the base library and, so, a major bump in the latter requires a major bump in the former?
1 Like
I remember the discussion about GHC 8.12, which became GHC 9.0;
Basically, people felt that linear types were a headline feature, so the versioning was changed to reflect this.
1 Like
I really meant from, say, GHC 9.8 to 9.10 rather than first number version bumps.
EDIT: To elaborate, I mean ‘GHC as a tool’, as opposed to ghc as a library. When I read, say, 2.1. Version 9.10.1 — Glasgow Haskell Compiler 9.10.1 User's Guide the ‘breaking changes’ from GHC 9.8, as a tool, do not jump out at me.
I really meant from, say, GHC 9.8 to 9.10 rather than first number version bumps.
The policy as I understand it is this:
- Each six-monthly release of GHC uses a 0.2 version bump, thus from 9.8 to 9.10, or 9.10 to 9.12.
- “Patch-level releases” change only the minor version. Thus 9.8.1, 9.8.2, 9.8.3 etc. These patch level releases fix bugs but are intended to change absolutely nothing else, not features, not APIs, nothing.
- The odd-level major versions, 9.7.X, 9.9.X, etc are used for development builds, never for releases.
- We do not have a formal policy for moving from 8.X to 9.X, or from 9.X to 10.X.
Would it be helpful to write down these four points? For example here: releases · Wiki · Glasgow Haskell Compiler / GHC · GitLab
4 Likes
Thanks! What is on my mind is this:
‘industry’ has to think about the ‘costs’ of change as well as the ‘benefits’. However, if change actually ‘breaks’ nothing (or little), only ‘adds’, the ‘costs’ of change might be low. I am wondering if we (as a community) could do more to communicate what is actually ‘broken’ by change, making it easier for ‘industry’ to identify and assess that ‘cost’ and - perhaps - lowering the perceived barrier to moving forward.
(EDIT1: Taking this opportunity to acknowledge @tomjaguarpaw’s Upgrading from GHC 8.10 to GHC 9.6: an experience report)
(EDIT2: I may have underestimated what the GHC project already does, given the migration guides for each of GHC 7.8 to GHC 9.12.)
2 Likes
That suggestion sounds resaonable, but it’s surprisingly hard to execute.
-
New features are (increasingly) not a source of breakage; even when they are, you often get a deprecation cycle or two. See GHC Steering committee stability principles
-
Fixing bugs, on the other hand, really can cause breakage – and does! It turns out that people sometimes unwittingly rely on bugs. It’s really really hard to predict this.
-
Library changes are a major source of breakage. This is silly; just because you change from GHC X to GHC X+1 doesn’t mean you should have to change libraries. But currently you do – See GHC stability state of play
It’s hard to know how to communicate all this better, but if there are simple things we can do that would help our users, we’re all ears. Perhaps you can float some ideas? (Remembering the opportunity cost: doing X means not doing Y.)
8 Likes
We could have:
- latest stable branch (mind you, this one can be a little slower too if we have nightlies… there’s less pressure to get releases out “just because”)
- previous stable branch
- LTS branch
Then we always have two candidates for LTS.
But that might mean to choose a different release cycle (more than half a year for a new major version). Otherwise the versions “run” too fast. I believe that’s already the case, but it will be worse when there’s only two short term supported branches.
The way I see it is that the current “stable” releases are (ab)used to get experimental features out. This can all be delivered with well working nightlies support instead.
I find it hard to foresee the involved workload for GHC devs though. But this makes sense for end users imo.
So instead of nightly builds, what about monthly or “fortnightly” (i.e. twice a month) releases? If that can be made to work…then weekly builds could be possible. But the impression I’m getting is that 28-31 builds per month is too large an opportunity cost, amongst others:
Could you outline what you mean by “latest/previous stable branch”? What goes on those branches?
We are about to release 9.12. Perhaps you mean something like the 9.10 (latest) and 9.8 (previous) series? When 9.12 is released, then “nightly” will follow 9.13 (what will become 9.14), “latest” will follow 9.12 and “previous” will follow 9.10. Correct?
This means we can afford one additional LTS branch. Even that is a stretch, because the cost of backporting fixes rises as the LTS branch diverges more from master, so maintaining a 2.5 year old LTS release is more costly than additionally maintaining 1 year old 9.6.
Moreover, how do you pick the new LTS release? That is, after we have used GHC 8.10 for 3 years (1 year regular support + 2 years LTS), what is the next LTS version? Should it be 9.8 or 9.10? What if 9.8 is (sadly) known to be bug-ridden, 9.10 is better but it’s too fresh to judge? What is the policy here? Keep maintaining 8.10 for another half year (but the maintenance cost!) or embrace 9.10 instead? I’m uncertain how industry users respond to immediately having to migrate from 8.10 to 9.10 because the former lost its LTS status in favour of the latter. I appears that we need to notify users at least half a year before we make the switch from 8.10 to 9.10, so that they may migrate from one supported version to another. So, in practice 3 year LTS means that LTS releases are 2.5 years apart to have overlapping support windows.
3 Likes
My preference is for our limited capacity to be spent working towards making it easier to upgrade GHC (better stability guarantees, tooling for upgrades, etc) rather than maintaining old versions.
Maintaining an LTS allows you to delay upgrading your GHC. But you will still eventually have to do it.
Upgrading to a new version of GHC can be broken down into the cost of upgrading your own code, and upgrading your dependencies. The former is a fixed cost. The second is more variable. When a new GHC is released many libraries will need version bumps, etc, to be compatible with the new version. An early adopter would have to do all this work themselves. But as time goes on the community will do more of this work. In my experience, most patches get made and released relatively quickly, but there’s often a long-tail of packages that take longer to be made compatible.
Waiting a bit longer before trying to upgrade is therefore helpful because you hope that other people will have done the work to upgrade libraries in the ecosystem. But I think you eventually get diminishing returns. But if everyone waits, then no one writes the patches(!). I think we often see this dynamic. For instance, when Stackage nightly gets bumped there are a flurry of patches, but it would be great if people started this work sooner.
I’m quite excited for more core libraries becoming reinstallable (template-haskell, base) with coming releases of GHC. I think this will make upgrading orders of magnitude easier. With that and other stability improvements we’ve already had, I’m really hoping that more people will upgrade sooner, and we can all reap the rewards from that.
10 Likes
This sounds reasonable. Currently:
Current branch, prior branch, prior prior branch.
Proposed:
Current branch, prior branch, some earlier branch.
And periodically, each when the “current branch” shifts upwards by one and becomes the prior branch, we evaluate if the now prior prior branch should become the new LTS branch, or just fall out of maintenance and we preserve the existing LTS branch.
So, maintenance-wise, the only difference from now is that if the LTS branch lasts longer, patches will typically become harder to backport. But that seems unavoidable for longer-lived branches, no matter how structured. And who knows how much older that branch will get – maybe only a year more, maybe two.
The one other thing I’d note is that earlier it was proposed that LTS branches also get performance improvements. I think this is probably mainly off the table – performance work in my experience is often pervasive, involving many small changes, and only makes a noticeable difference in combination with other performance work – the hardest stuff to backport. (As opposed to bugfixes which ideally are localized and individually impactful). I would think that typically such stuff should not be backported – only bug fixes.
5 Likes
I hope not. Because it is more work for me and for every single distributor.
If we had LTS releases, I could focus more complex and time-intensive bindist work on those, e.g. Build fresh versions of old bindists · Issue #903 · haskell/ghcup-hs · GitHub
The same goes for tier 2 platform support. It’s going to suffer (it already is). And it seems it’s becoming a habit to increasingly outsource certain work onto distributors (e.g. me). I’m not pleased.
It’s similar for HLS. They can focus on supporting less versions, so as long as the latest LTS is supported.
New releases are not for free.
3 Likes
Upgrading to a new version of GHC can be broken down into the cost of upgrading your own code, and upgrading your dependencies.
If I may, this is a tad simplistic: you forget the opportunity cost. Few organisations have people dedicated to taking care of GHC, and so for most of us, upgrading GHC comes with the cost of not doing anything else.
For reference, at work we have made the main monolith compile with GHC 9.6.6 last week. Unless there are critical bugs that will never get fixed for the release family we use, we have little interest in migrating every time a new GHC version comes out, because it simply does not bring anything to the business.
6 Likes