[Pre-HFTP] GHC.X.hackage: A tool for easing migrations to new GHC versions

For a few years now the GHC team has maintained head.hackage a set of patches to a subset of Hackage allowing Hackage packages to be built against pre-release compilers. While head.hackage has been an invaluable resource for testing GHC, its user-base has been limited to GHC developers.

In the past few months, there have been a few proposals seeking to lower the cost of migrations to new GHC releases. A common theme in these discussions has been the long lead-time for adoption of new releases due to the largely serial nature of library migration. For instance consider a package A, which depends on B which depends on C. Migration to a new GHC release will currently go like this:

  1. GHC version X is released.
  2. The maintainer of C must wake up, fix their package, and make a Hackage release. The maintainers of dependent packages A and B can do nothing at this point.
  3. The maintainer of B must wake up, fix their package, and make a Hackage release. The maintainer of A, which depends upon B can still do nothing until this happens,
  4. The maintainer of A can finally fix their package and make a Hackage release.

In the past this has meant that a significant fraction of GHC’s release cycle will pass before a new GHC release sees any significant adoption.

We propose to extend head.hackage as a tool to make this process less serial, helping users migrate and test their libraries more quickly on new GHC versions. We describe this proposal in this HFTP proposal. Please do let us know what you think.

20 Likes

I very much like the emphasis on trying to parallelize the care-taking of our library ecosystem!

Cutting down on critical path length, and being blocked by tangential things one isn’t even using, are key objectives I would like to keep in mind not only to motivate this project, but other projects in the future.

My main concern is strictly procedural. This seems like a great idea, but there is no concrete “ask” of the HF. I thus am not sure this is currently eligible to be an HF proposal!

I think it would be good to see GitHub - haskell/ecosystem-proposals: Proposals for the Haskell Ecosystem resurrected for these sorts of things, namely technical plans that are cross-cutting and don’t fall into ghc-proposals or a similar per-project bucket.

If/when the project needs extra funding, a HFTT proposal could be written, but frankly I doubt that is necessary, because as far as I am concerned @bgamari and others are within rights to just decide GHC’s CI needs this / includes a superior head.hackage.

(In fact, I think they could even do that without the “ecosystem proposal” — it is also means to the existing end, and we shouldn’t be micromanaging the “how” ! But it can be still be nice to do the ecosystem proposal to gauge excitement / gauge whether people seem likely to help contribute to so the goal of parallelized / crowd-sourced upkeep is realized.)

2 Likes

I think the HFTT is perfect for this. HFTT proposals don’t necessarily require funding by the HF. Rather, the HF just offers an umbrella under which cross-cutting proposals can be debated and refined by the community.

I’d forgotten about the Haskell Ecosystem proposal idea. It’s a good one! But it was created five years ago, has never been used, and has now effectively been subsumed by the Haskell Foundation. And that’s an accolade! It was such a good idea that the community got together and built the Foundation!

3 Likes

I see the confusion here around the role of the HFTT. But I see the HF more broadly as the place to make community-wide policy decisions. For example, if the community wanted to propose and discuss a tweak to the PVP, where would that discussion happen? It’s unclear without the HF, but discussing it within the HF seems at least very plausible. And where within the HF? The HFTT, of course! :slight_smile:

As to the proposal itself: I’m in support of this plan to ease the burden of open-source maintainers by making updating to a new GHC version less dependent on when other volunteers do their part.

1 Like

So in our first meetings as the HFTT, we discussed this sort of stuff. tech-proposals/PROPOSALS.md at main · haskellfoundation/tech-proposals · GitHub and especially tech-proposals/TEMPLATE.md at main · haskellfoundation/tech-proposals · GitHub emphasize resourcing over technical design, and ironically enough, I was one of the people questioning why this was so! But the answer I got that the HFTT proposals were indeed for resourcing, and the technical decisions themselves were largely out of our purview!

So I am not actually opposed to changing that, in the slightest. But, given the very different interpretations of the status quo, I think this is a conversation that should be had. Also, we will need a modified or additional template: this proposal had to completely buck the template because it was inappropriate for such a proposal, but if we want proposals of this sort going forward, we should have a template that encourages them!

1 Like

Also let me encourage other HFTT members to to chime in here (e.g. @sclv ?), because I don’t want to play telephone possibly misrepresenting others’ views.

Under “implementation” there’s a fairly concrete ask for resources, so it makes sense to me as an HFTT.

Vis a vis where one would discuss the pvp, at the bottom there’s a link to github issues: Issues · haskell/pvp · GitHub

In the main repo the “proposing changes” section reads:

The PVP evolves over time to adapt to changing requirements of the community. Everyone is invited to propose and discuss changes to the policy.

Formally, the PVP is maintained by the Core Libraries Committee together with the Hackage Trustees.

Please review the guidelines for contributing.

@sclv I see

There’s a lot of flexibility in filling these roles. They could be

  • Occupied by the same person, or different ones
  • Rotating per release, or few releases
  • Filled by a keen community member, or a paid professional

Currently, the GHC.X.hackage maintainer role is effectively filled by the GHC developers who maintain head.hackage. However, if the scope of the role expands, this is likely to be unsustainable. Potentially this is somewhere where the HF could provide assistance.

But this doesn’t seem “fairly concrete” to me. Conversely I see the proposal is as really 3 proposals in 1:

1. Concrete Technical plan

I’ll paraphrase as

Fork head.hackage a few times and/or add more branches

This is trivial and doesn’t need HFTT authorization.

2. Concrete Social plan

I’ll paraphrase as

Promote the the use of head.hackage to assuage community concerns about GHC upgrades being too difficult / there being too much unforeseen breakage.

This is very interesting to me! But it is a question of trying to steer the ecosystem write large (GHC, libraries) towards different migration norms and practices, not of resourcing in and of itself. It is still unclear to me whether you (@sclv) think the proposal is “valid” as a whole because:

  • the HFTT is supposed to deliberate on social/community policy

or

  • the authors of this policy (@bgamari, simonpj) have the authority to make this decision on their own, so any residual resourcing questions are all that remains to deliberate.

(Relatedly, I recall it was you who first mentioned resurrecting GitHub - haskell/ecosystem-proposals: Proposals for the Haskell Ecosystem for these sorts of things, versus @simonpj above saying that repo is obviated by HFTT proposals.)

3. Nebulous resourcing promise

I’ll paraphrase as

for now us GHC devs are probably good to go, but FYI if the social plan succeeds in making GHC-specific package wildly very popular, we’re going to need some help maintaining them

I don’t think that resourcing ask is concrete at all. I think it is instead asking the HF to promise to have the GHC dev’s backs and relieve them of the burdens of success, so they aren’t punished for their good deed.

Now, that is still a resourcing question of sorts, and a good thing for the HF to deliberate on. And as member of the community, I think

Yes! This is great, the HF should absolutely have the GHC dev’s backs here!

But as a mere engineer on the HFTT not familiar with the ins and outs of hiring, I feel a little awkward approving making a vague promise that I am not responsible for keeping. The as-yet unknown future ED will be responsible for making good on this promise, with an unclear time line, and they are not here to discuss it with the rest of the HFTT.


My bias is still:

the HFTT as a board of engineers is ill-suited to make resourcing decisions that almost invariably are flexible matters anyways (like this) and one the HF executives should be empowered to make asynchronously from agreeing upon the object to be resourced.

but I am not trying to promote that so much as just ask for clarity.

Right now each proposal we get seems to need its own “rubric”, and I just want some to know what I am supposed to do as an HFTT member and as someone that would like to see more HFTT proposals written.

Nor am I trying to be a process weenie for rigidity’s sake!

If we do want each proposal to be judged somewhat freely, I am actually OK with that! But it’s a bit at odds with the current documents for the HFTT, which seem (to me at least) to imply a rigid constitution of what the HFTT is for and what we do. I don’t think that rigidity is reflecting what we are actually doing terribly well, or even more so any consensus among the various folks in this thread over what we should be doing. It is that gap that concerns me.

1 Like

I see proposals as being a combination of “we want to do X and we need Y resources to do it from HF.” If either X or Y is missing or ill defined then that’s not a good proposal. I think the resourcing could be better spelled out in this case. However, given that we have a devops person coming on, then this in a sense is specifying further where we would hope their work would go. And since the HF in part directs their work, then this makes sense to be an HFTT proposal.

In the course of evaluating proposals, we may say either the resourcing is not well enough defined, or the X we want to do is not well enough defined, or the X is not a good idea, but a related X may be a good idea, or maybe just X is well defined and fine but not worth resource expenditure.

So one cannot draw a “bright line” between these discussions. A proposal that is mainly “let’s figure out how to do X” is not a good proposal. A proposal that’s “X needs to happen, so we should allocate a resource to figure it out” is fine, if risky. Etc. We’ll have to work this stuff out as we go.

In this case, I may as well ask my question here – as compared to what we have now, what’s the difference? If it is multiple head.hackage instances, what are the cases where such instances might serve us? If it is more about promoting head.hackage, then how should that be done? If it is about more resources to patches to head.hackage, then that seems vague in terms of how that happens.

I can imagine that the multiple versions thing seems the most concrete, and at the moment I can, I suppose see a purpose in that head.hackage reflects ghc pre-releases, but even a recently released ghc could also benefit from this, so having at least two (current release, genuine head) could be a win. Is that about it?

2 Likes

OK thanks @sclv, that does clarify your position to me. I still think that leaves some would-be ecosystems-proposals that don’t involve resourcing at all in the lurch, but we can ignore that for now as it doesn’t pertain to proposal.

In this case, I may as well ask my question here – as compared to what we have now, what’s the difference? If it is multiple head.hackage instances[…]? […] even a recently released ghc could also benefit from this, so having at least two (current release, genuine head) could be a win. Is that about it?

I took this to be one about overlay per release, and one for HEAD. When ever a release is cut, the overlay is also cut to make a new GHC.X.hackage.

The big questions to me are thus:

  1. Do GHC.X.hackages need to modified post release?
  2. Are changes ever worth applying to multiple releases?

I think the answer is “yes” to both, because it is possible (if a bit tragic) for a library to release a new version that doesn’t yet support the latest released GHC.

In that case, it probably makes sense put the corresponding patch in the oldest GHC.X.Hackage that needs it (i.e. the one for newest GHC it doesn’t support), and then merge that into all the new ones since?

This makes me think that multiple branches is better than multiple repos.

1 Like