GHC 9.10.1-alpha3 is now available

[TeofilC]

I think trying to upstream patches more from head.hackage before a release and keeping track of the status of patches both sound like good ideas. Though I’m somewhat sceptical that it would be worth blocking a release over this stuff.

At the end of the day, stuff like this boils down to capacity. Redundant patches get removed when someone realises and makes an MR. head.hackage could always benefit from more people writing patches, from more people using them, and from more people upstreaming and tracking the status of upstreamed patches.

Though it doesn’t always make sense to expect upstream patches to be released before a GHC release:

• some maintainers have a policy of waiting until after the GHC release, which is quite understandable and their prerogative.
• upstreaming too early can lead to churn, eg, #24557: Support invisible type patterns in Template Haskell quotes (e.g., [t| @a |]) · Issues · Glasgow Haskell Compiler / GHC · GitLab where we realised that a feature should be implemented a different way, but the previous version’s changes had already been applied to head.hackage.
• an upstreamable patch often takes more effort to write than one that merely works (does this patch conform to the code style of the repo?, import style?, etc)

[tomjaguarpaw]

@angerman

I do not want to see any proliferation of head.hackage outside of GHC development

A couple of questions:

1. In what way do you anticipate head.hackage proliferating outside of GHC development?
2. What are the changes to GHC that are causing these packages to need to be patched?

[RyanGlScott]

Several packages with head.hackage patches are no longer maintained (e.g., FPretty and critbit). As such, their head.hackage patches have never been removed, as there haven’t been new Hackage releases that would render the patches obsolete.

Arguably, we should remove these patches after a certain amount of time has passed, although I haven’t found the time to do so. Contributions would be welcome here.

[hasufell]

So what makes a patch/package go into head.hackage? Are there strict criteria? What is the main use case precisely?

[hasufell]

Unofficial GHC JS cross bindists have been built by me for this release (provisional platform/distro support):

• Index of /ghcup/unofficial-bindists/ghc/javascript-unknown-ghcjs-9.10.0.20240413/

And they’ve been added to the ghcup-cross-0.0.8.yaml channel: Add javascript-unknown-ghcjs-9.10.0.20240413 by hasufell · Pull Request #203 · haskell/ghcup-metadata · GitHub

To try, you first need emscripten:

git clone https://github.com/emscripten-core/emsdk.git
cd emsdk
./emsdk install latest
./emsdk activate latest
source ./emsdk_env.sh

Then install the GHC cross via ghcup:

ghcup config add-release-channel https://raw.githubusercontent.com/haskell/ghcup-metadata/master/ghcup-cross-0.0.8.yaml
emconfigure ghcup install ghc --set javascript-unknown-ghcjs-9.10.0.20240413

Then do some hello-world:

echo 'main = putStrLn "hello world"' > hello.hs
javascript-unknown-ghcjs-ghc -fforce-recomp hello.hs
./hello

Also see https://www.haskell.org/ghcup/guide/#cross-support

[bgamari]

@hasufell

What is the main use case precisely?

The main use-case of head.hackage as far as I’m concerned is to facilitate testing of GHC. For better or worse, source changes sometimes are needed to build projects with GHC HEAD. head.hackage exists to collect these patches in one place so that we (and possibly our users) can readily test their projects against the compiler.

I proposed to extend the mandate of head.hackage to address end-user migration explicitly in the ghc.X.hackage proposal although there was little appetite in the community for this. Ultimately I think this is fine; we all agree that it would be better if we rather tried to minimize the need for head.hackage.

I agree with @hasufell that we cannot create a dependency between the GHC release process and the updating of the ecosystem. Managing a GHC release is already very tricky, with far too many “known unknowns” which make adhering to a concrete schedule a challenge. Adding yet more such constraints to the process would make the process significantly more costly than it already is.

[bgamari]

@Bodigrim

May I suggest a wider gap between a release candidate and the final release? Many people will only starting testing 9.10 once RC1 is out, and it’s unlikely we get much building just in a week.

The primary motivation for the short gap between release candidate and final is to maintain the original final release date despite having pushed back alpha2 by a week. In principle the alpha series exists released specifically to allow for a wide range of testing early, giving us plenty of time to address issues before the final release. However, I recognize that we often don’t see much vigorous testing until later in the series. Consequently, perhaps it would be preferable to err on the side of giving more time to the late alphas and release candidates.

All of this is to say, I would be fine with moving the final release back by a week to allow more time to testing.

[mpilgrem]

Like alpha2, if you want to use with Stack:

1. Upgrade to the master branch version of Stack (can be used only if you are not using GHCup to manage versions of Stack): stack upgrade --source-only --git.

2A. If you are not using GHCup to manage versions of GHC, augment Stack’s default setup-info dictionary in a configuration file (needed only until Stack has fetched the compiler once). For example, on Windows:

setup-info:
  ghc:
    windows64:
      9.10.0.20240413:
        url: https://downloads.haskell.org/ghc/9.10.1-alpha3/ghc-9.10.0.20240413-x86_64-unknown-mingw32.tar.xz
        # Can be extended with SHA protections etc: see https://docs.haskellstack.org/en/stable/yaml_configuration/#setup-info

2B. If you are using GHCup to manage versions of GHC, augment ~/.ghcup/config.yaml. For example:

url-source:
  - StackSetupURL
  - setup-info:
      ghc:
        windows:
          9.10.0.20240413:
            url: https://downloads.haskell.org/ghc/9.10.1-alpha3/ghc-9.10.0.20240413-x86_64-unknown-mingw32.tar.xz
            # Can be extended with SHA protections etc: see https://docs.haskellstack.org

3. Specify the compiler in a Stack configuration file (eg stack.yaml):

compiler: ghc-9.10.0.20240413

[hasufell]

@mpilgrem

If you are using GHCup to manage versions of GHC, augment ~/.ghcup/config.yaml. For example:

Fascinatingly, GHCup also allows to mix the stack metadata with any GHCup metadata channel.

E.g. if you want the stack logic for bindists + ghcup logic for prereleases, you can do:

url-source:
  - StackSetupURL 
  - https://raw.githubusercontent.com/haskell/ghcup-metadata/master/ghcup-prereleases-0.0.8.yaml

No need to write your own setup-info dictionary.

[Bodigrim]

@bgamari

The primary motivation for the short gap between release candidate and final is to maintain the original final release date despite having pushed back alpha2 by a week.

I’m afraid you are tired of my usual rant, but community at large does not care whether a release is a week or even a month later. There are significant costs to pay for each minor GHC release: ghcup has to support it, parsers and exact printers have to be updated, hls has to support it and make a new release, Docker images, Stackage, Stack, Haskell-CI, haskell-actions/setup, etc. There is no point to incur all these costs for the sake of an artificial deadline.

@bgamari

In principle the alpha series exists released specifically to allow for a wide range of testing early, giving us plenty of time to address issues before the final release. However, I recognize that we often don’t see much vigorous testing until later in the series.

Since alpha releases do not guarantee to be feature-full and more breakage can be introduced before release, maintainers should not relax build-depends bounds or make new releases of their packages until RC is out. It is expected that almost no testing outside of root packages happens until RC1.

Maybe you want to adjust the nomenclature of GHC releases and mark, say, alpha3 as RC1? This is assuming you can guarantee that no further breakage is happening afterwards.

@bgamari

All of this is to say, I would be fine with moving the final release back by a week to allow more time to testing.

A gap of two weeks between RC1 and 9.10.1 means that maintainers have essentially one weekend to test, adjust and release everything. I’d rather release 9.10.1 as RC2 and then make 9.10.1 in time when you’d normally do 9.10.2.

[Kleidukos]

@Bodigrim

A gap of two weeks between RC1 and 9.10.1 means that maintainers have essentially one weekend to test, adjust and release everything. I’d rather release 9.10.1 as RC2 and then make 9.10.1 in time when you’d normally do 9.10.2.

This is in line with what I tell people when new major GHC releases are published: “the .1 release is basically an RC that serves to catch bugs from downstream codebases, you’re better off with the .2 release”.

[chreekat]

As a somewhat-neutral observer, maybe I can head off a miscommunication that seems to be brewing here.

The GHC team feel they are under pressure from the community to make releases more quickly. Now they are getting pressure to make releases more slowly. If we go back to GHC Medium-Term Priorities from a year ago, “we received feedback that our release cadence is too fast, and other feedback that it is too slow.”

Maybe those two types of feedback are talking about different aspects of a release? Maybe “faster” means “more versions released per year” and “slower” means “a longer period of time between RC1 and the final release”. But then again, maybe different quarters of the ecosystem simply want different things. Without more data it will be hard for the GHC team to make adjustments.

[Kleidukos]

Certainly, it’s important to realise that the crux of the issue is the lack of compartmentalisation between improvements that could be shipped faster as minor releases, and the necessary breaking changes to TH and syntax, that ought to happen with more time between each, so that third-party maintainers don’t have to work around the clock to get their projects and libraries up-to-date.

Faster, non-breaking, minor releases mean that the tooling and methodology for making releases improves and the process becomes less painful. That’s a net win for the release engineering team.

(keyword is “tick-tock releases”)

[hasufell]

@chreekat

The GHC team feel they are under pressure from the community to make releases more quickly.

I have no idea where this comes from and my experience as GHCup maintainer indicates the opposite is true: people long for less releases, but more high quality and with less breakage.

I wonder where this difference in perception comes from.

[michaelpj]

I have no idea where this comes from and my experience as GHCup maintainer indicates the opposite is true: people long for less releases, but more high quality and with less breakage.

My guess would be that people mostly long for high quality and less breakage, and the release cadence primarily affects how salient those things are.

My impression from the rest of the industry is that the usual belief is that you can’t get better quality and less breakage by releasing less often. If you have a process that allows you to produce low quality releases with lots of breakage, then slow releases will just accumulate large amounts of quality issues and breakage. Whereas if you have a good process, then you can release as often as you like. And pushing for a faster cadence puts pressure on the process, hopefully leading to improvement. As the saying goes “if it hurts, do it more often”.

(Plus, faster releases mean smaller batches, shorter queues, shorter cycle times, faster feedback, etc. etc. Lots of good stuff if you can get it.)

Which is to say, I’m not sure that slower releases would actually help. I think the only thing that will help is continuing to work on making the GHC development and release process produce releases with fewer bugs and less breakage.

[hasufell]

@michaelpj

My impression from the rest of the industry is that the usual belief is that you can’t get better quality and less breakage by releasing less often. If you have a process that allows you to produce low quality releases with lots of breakage, then slow releases will just accumulate large amounts of quality issues and breakage. Whereas if you have a good process, then you can release as often as you like. And pushing for a faster cadence puts pressure on the process, hopefully leading to improvement. As the saying goes “if it hurts, do it more often”.

This duality can be achieved with nightlies (see rust), while maintaining a very slow cadence on the stable channel. It may be more work for the compiler team, but nothing comes for free. It is about priorities in the end.

[hasufell]

@michaelpj

My impression from the rest of the industry is that the usual belief is that you can’t get better quality and less breakage by releasing less often.

This study doesn’t seem to support that rapid releases cause higher quality either: https://seal-queensu.github.io/publications/pdf/EMSE-Foutse-2015.pdf

If you find a more recent study, that would be interesting.

In my own experience, the major benefits of rapid releases are for the project and less so for the end users, because you’re increasingly utilizing end users for QA purposes, but in a low impact way. That still causes churn for everyone involved (including open source supply chain), but that’s not work the project maintainers do and as such is easy to neglect.

With nightlies (and prereleases), this becomes opt-in and is more transparent, imo.

[tomjaguarpaw]

@chreekat

The GHC team feel they are under pressure from the community to make releases more quickly. Now they are getting pressure to make releases more slowly.

I believe this seeming paradox can be resolved by the hypothesis that what both groups want is less busywork per unit time.

I suspect that the group that wants GHC to release faster thinks they will have less busywork if they can address breaking changes in smaller increments, and that the group that wants GHC to release slower thinks they will have less busywork if they can address more breaking changes in one go, to avoid frequent context switches.

The simplest way to please both groups is, I think, to cause less busywork by making fewer breaking changes.

[Bodigrim]

@michaelpj

My impression from the rest of the industry is that the usual belief is that you can’t get better quality and less breakage by releasing less often.

This is an argument to increase throughput, while my argument above was to increase latency.