Head.hackage usage, rationale, and problems

#1

There aren’t really strict criteria at the moment, which is something that I think the README should clarify.

One informal criterion that has developed over time is “don’t submit a patch unless it’s needed”. For example, if a package foo-0.1 fails to build with GHC HEAD, but foo-0.2 does, then it’s usually not worth submitting a head.hackage patch for foo-0.1, given that you can just build foo-0.2. This isn’t a hard-and-fast rule, however—it can sometimes be worthwhile to have head.hackage patches for old versions of a library (e.g., if it takes the Haskell ecosystem a long time to upgrade to foo-0.2).

The main head.hackage use case is making it easier to build Hackage libraries with pre-release or HEAD versions of GHC. Some secondary use cases are:

  • Acting as an “extended GHC test suite” by making it easier to run the test suites for packages like bytestring, which are difficult to run in the standard GHC test suite as-is
  • Making it easier to test in-flight GHC changes against a subset of Hackage libraries to find possible regressions

It occurs to me that this discussion, while useful, sidetracks a bit from the original discussion in this thread. I’d be happy to continue this discussion elsewhere if it’s deemed too off-topic.

3 Likes
GHC 9.10.1-alpha3 is now available
#2

Because you can’t do any real testing of GHC without putting head.hackage into your cabal.project, but we want people to actually trial alphas, to get enough feedback about regressions and other issues, that could be addressed leading up to the proper release.

Adding head.hackage to your cabal.project is easy, removing it requires someone to notice and be bothered enough to actually go and remove it. The next person sees head.hackage fixing some build issues and keeps cargo culting it along.

What breaks it, I guess it’s often template-haskell. It could be in-flight changes in GHC (perfectly fine), or GSC / CLC changes.

@RyanGlScott highlights some good usecases.

The main head.hackage use case is making it easier to build Hackage libraries with pre-release or HEAD versions of GHC.

My argument is it should be exclusively required for HEAD versions of GHC. And there it’s perfectly fine.

If we end up putting head.hackage into all kinds of cabal.projects people will end up depending on it, cargo cult it further, and any changes to it will then lead to surprising behaviour (because people shouldn’t have been using head.hackage in the first place).

1 Like
#3

I second the request for splitting the thread about “head.hackage maintenance”.

3 Likes
#4

@tomjaguarpaw regarding proliferation: Release a version compatible with GHC 9.10 · Issue #45 · haskell-hvr/text-short · GitHub

It is actively encouraged by now to use head.hackage for pre-releases. I consider this wrong. Why release alphas if they are not usable without in-flight development tools? At the point it’s also not head.hackage anymore. It’s 9.10-alpha.hackage or something.

2 Likes
#5

Which if a specific temporary such hackage existed I might be less concerned about. But it has to be deleted when the release is out. It can not be some indefinite running thing like head.hackage.

2 Likes
#6

A post was merged into an existing topic: GHC 9.10.1-alpha3 is now available~

#7

For what it’s worth, I completely agree with this. This was one of the reasons why I proposed ghc.X.hackage, making this explicit. However, I also understand (and agree) that ideally we would not have as much need for head.hackage.

On the other hand, I can’t wave a magic wand and make the need disappear. Core library (e.g. text), base, and template-haskell changes happen. In some of these cases we can try to minimize/mitigate changes (e.g. #24021: Making Template Haskell (more) stable across GHC releases · Issues · Glasgow Haskell Compiler / GHC · GitLab) but ultimately some amount is (practically) unavoidable.

I think it is defensible to say that we shouldn’t actively encourage head.hackage usage by end-users (as ghc.X.hackage would have done). However, in the short term I struggle to see a world where we could reasonably release an alpha which could build a significant fraction of Hackage without something like head.hackage.

4 Likes
#8

To be clear, that’s because it’s unrealistic for GHC to wait for patched versions of all relevant packages to be available on Hackage before the release happens, right? I.e. head.hackage allows for loose coupling between the release of GHC and its dependencies. Without it, GHC would have to wait for all its dependencies to be patched and released before GHC could make an official release.

If so, I understand. And yet, I believe in Moritz’ vision of a world where this is not necessary. I don’t know what it looks like yet, but it seems like a good problem to eventually solve.

1 Like
#9

I’d even argue more vehemently against this, for another reason.

If I as a library maintainer get a pull request saying “this will fix compatibility with upcoming GHC release”… I will want proof and I want to be able to install said GHC version, before I accept said patch. For that to work, you need to release a pre-release. So we have a chicken and egg problem here.

So in my opinion:

  • pre-releases are for the ecosystem to catch up (libraries in particular), not for application developers to conveniently test their whole stack and expect the ecosystem has already caught up
  • there should be a longer period between pre-release and proper release (for new major versions at least), allowing for said catch up
  • X.Y.1 will then naturally be the phase where application developers can start bringing in feedback
  • GHCup will always avoid recommending X.Y.1 versions and I believe the community is aware of what this release means… I don’t see the point of doing two phases of pre-releases. It’s already hard enough to tell people to use any pre-release, so we shouldn’t make it more complicated.
7 Likes
#10

The main reason I don’t like head.hackage it not even that.

head.hackage is a pool of patches from different authors and I as an end-user have no easy way of knowing whether those patches are:

  • correct
  • have been submitted upstream
  • have been approved upstream

You don’t even write such information in the patch header: patches/lens-5.2.3.patch · master · Glasgow Haskell Compiler / head.hackage · GitLab

To me that is enough reason to prefer going through all the PRs and issue trackers and cooking up source-repository stanzas instead.

4 Likes
#11

I don’t think this is quite right. None of the examples given are dependencies of GHC. It’s a set of patches for ecosystem packages broken by the new version of GHC.

2 Likes
#12

Ah, you are right. Rereading Ryan’s post at the top of this discussion makes that clear:

But to check my understanding, does that mean that most patches are added by (or at the behest of) Hackage package maintainers?

1 Like
#13

One thing that’s a shame is that the source-repo stanza strategy requires quite a bit of duplicated work for each early user (building, gathering failures, searching/making PRs, repeat). It would be good if we could get head.hackage or an alternative into a state where it was able to fulfil this role.

Perhaps it would be good to make some issues on the head.hackage repo?

2 Likes
#14

AFAIK patches are mostly written by GHC devs (@RyanGlScott does the lion’s share of this) or other users of head.hackage, which can overlap with maintainers but mostly doesn’t I think.

#15

Isn’t it the complete opposite? If Hackage package maintainers wanted changes to their packages, they’d just make them! It seems to me that the only people want to incorporate something into head.hackage are those who can’t make a change to a package directly.

2 Likes
#16

Yes, I think the existence of head.hackage is an indication that we might need better collaboration mechanisms for small patches. E.g. a way in hackage to submit such a patch, that is then relayed to either a github PR or git email/etc.

It’s also an indication we might need a better mechanism for release candidates on hackage (GHC compat patches for unreleased GHC versions should naturally go there). Their ergonomics currently is rather awful. An end user should be able to tell cabal “consider all release candidates”.

1 Like
#17

I have to imagine you’re right! But then, who are these non-maintainer, non-GHC-developer users of head.hackage? I feel like we’re missing their voice.

In other words, the only uses of head.hackage that I am actually familiar with are the “secondary” use cases Ryan mentions. I know head.hackage is built against GHC every night (sometimes it even succeeds! But not usually), and I know that GHC tests that require non-boot packages have been bolted on to it.

Who else uses it?

#18

Who else uses it?

As I understand from @angerman, it’s used by developers of either libraries or applications who want to bring their codebase up-to-date with a new GHC release, but who depend on packages that require changes in head.hackage to build correctly.

#19

I can’t tell from that if anybody is actually using it in the way Moritz describes. Yes, we want people to trial alphas… but from listening in on GHC calls, it sounds like that isn’t happening.

#20

Ben’s proposal gives a pretty good summary of the end user use case: tech-proposals/proposals/001-ghc-x-hackage.md at ghc-x-hackage · bgamari/tech-proposals · GitHub. But as Ben said on the other thread one reason why this proposal was parked was because folks seemed to not do this too much in practice

1 Like