GHC 9.12.2 is now available

The GHC developers are happy to announce the release of GHC 9.12.2.
Binary distributions, source distributions, and documentation are available at
downloads.haskell.org.

We hope to have this release available via ghcup shortly. This is a small
release fixing a critical code generation bug, #25653, affecting some subword
division operations.

As always, GHC’s release status, including planned future releases, can
be found on the GHC Wiki status.

We would like to thank IOG, the Zw3rk stake pool, Well-Typed, Tweag I/O,
Serokell, Equinix, SimSpace, the Haskell Foundation, and other anonymous
contributors whose on-going financial and in-kind support has
facilitated GHC maintenance and release management over the years.
Finally, this release would not have been possible without the hundreds
of open-source contributors who contribute their code, tickets, and
energy to the GHC project.

As always, do give this release a try and open a ticket if you see
anything amiss.

29 Likes

I have updated Stack’s default setup-info dictionary for these binary distributions, and added GHC 9.12.2’s global hints to global-hints.yaml.

6 Likes

Thanks @bgamari

I wonder though why it took 3 months to deliver a critical bugfix. From what I see the offending patch was simply reverted?

What needs to be done to deliver such critical bugfixes faster?

Edit: to be precise, the exposure time was 3 months, the time from bug report to fixed release was 2 months.

5 Likes

@hasufell, it has been eight weeks since the issue was filed. As soon as we were made aware of the bug we advised users to avoid the release, noting that a follow-up release would be forthcoming.

In this case, characterisation of the issue and its cause took only a few days. We determined fairly early in the process that a revert would be the most appropriate course of action. However, we did not want to make the situation worse by blindly reverting and potentially producing out another release that was affected by similar issues. Consequently we took a more holistic view, leaving time to assess the surrounding logic for related pathologies and improving our testing infrastructure to ensure that these issues would be reliably caught.

With this work done, we turned to preparing the release. While 9.12 is minimal, even a minimal release takes time. In particular, there were other issues with 9.12.1 which we wanted to ensure did not persist in 9.12.2 (e.g. #25687, various minor submodule inconsistencies). Such issues require time to resolve and test. This is especially true for GHC’s release pipeline, which takes the better part of a day to run, severely hampering iteration cadence. In practice this means that one might get a perhaps three to five attempts per work-week at a viable release pipeline with no other major obligations.

This iteration time hits particularly hard when one encounters new or otherwise unexpected failures. One such case in this cycle was a set of unexpected failures on Darwin platforms due to Rosetta (#25793). Identifying this failure as reproducible, isolating it, identifying the cause, and proposing a fix stalled the release by several days, despite not even warranting a patch in the final release branch.

Another source of delay in this cycle has been on-going infrastructure work. As you may know, GHC is currently in the process of moving away from our primary infrastructure hosting provider, Equinix. The preparation for this migration has both taken engineering effort and has had unfortunate side-effects on our infrastructure (specifically, necessitating changing and rebuilding a number of Docker images used by the release pipeline, a process which is not always straightforward).

Finally, all of this happens concurrent to the usual GHC maintenance workload. Ticket triage, review, preparations on other release series, work on other issues, and life outside of GHC all compete for engineering time. While in principle we could drop all other work in favor of release engineering, doing so would represent an enormous cost. Release work is very much a “hurry-up-and-wait” sort of activity and consequently is best undertaken concurrently with other (often, by necessity, shallow) work.

In sum, all of this meant that six weeks passed from the opening of the ticket (and public communication warning users of the issue) to the release candidate. In my opinion it would have been better if this had been closer to four weeks. However, in light of everything currently in flight, I think this release’s time-frame is not unreasonable.

Of course, I do hope that we can further streamline the release process. In particular, recent work on 9.6.7 has made it clear that we need to further decentralize and perhaps automate some steps of the process.

44 Likes

Thanks for the explanations.

Just to be clear… my comment wasn’t an attempt to discredit the release work, but to reflect on the incident as a whole.

From the distributors point of view, my concerns are as follows:

  • the exposure time was 3 months (for everyone installing latest GHC)
    • did end users actually receive the warning?
  • how can distributors deal with these situations best
    • there’s a lot to be desired on the GHCup side of this

My perception of the issue was that it could have been fixed faster. I think one of the reasons that it wasn’t is that GHC does not have a process to expedite bugfixes/releases. It is hard to do a swift critical bugfix release if there is no process for it and it has to be squeezed into the rest of the daily work, other releases etc.

An expedited process is very standard in high-risk industry. I can’t really say what the best way would be to implement something like that for GHC, but a few common patterns I’ve seen are:

  • all other release work stops (at least if resources are shared)
  • no other patches are considered… the scope of the bugfix release is only that one fix (that seems to have been the case here)
  • normal lead times (e.g. for release candidates) don’t apply
  • certain parts of the release process may be skipped or accelerated

It was also discussed that the perceived exposure was low, because it was a “.1” release, not warranting an excessive expedited process. While this makes sense for power users, it’s a bit problematic to make such assumptions in general… we could go so far to exclude .1 releases entirely from GHCup, if they are deemed experimental from upstream itself (or expose them only from the vanilla channel).

I think in the past there have been cases where very critical bugfixes would have benefited from an expedited process (such as correctness issues on the new M1 backend). So I want to kick off this discussion gently.

7 Likes

I think that for a project as large and complex as GHC, with CI turnaround times of a better part of a day, a turnaround time of six weeks on a bug is reasonable. If I think that such turnaround times are unacceptable for my business, I can always try to pay for a service-level agreement (SLA). On the other hand, if I am a hobby user, I’m perfectly happy to make my packages compatible with the latest GHC and leave the choice to downstream users.

Assuming that GHC turnaround times are what they are, I think that this is worth improving — I can’t mitigate a critical bug that I don’t know exists.

4 Likes

The existence and significance of the GHC 9.12.1 bug was announced:

  • Here, 561 people viewed the announcement: [PSA]: Correctness issue in GHC-9.12 ;
  • that was picked up, and reproduced by, issue 455 of the Haskell Weekly on 16 January 2025; and
  • GHC download now routes through a wiki page covering also GHC status (albeit that is a relatively recent development). That page warned about the bug and advised that GHC 9.12.1 was not suitable for use.

Users of Stackage (which underpins Stack) were likely unaffected because there is no Stackage snapshot for GHC 9.12.

1 Like

Well, that’s for anyone who regularly uses discourse. I don’t think that’s really sufficient.

GHCup has a way to:

  • print pre-install warnings
  • print post-install info

And yet, that doesn’t (obviously) trigger for already installed GHC versions. And we actually missed adding it…I just did now.

There’s an attempt at providing something like ghcup news, but even that will require user interaction.

So no, I don’t think that was sufficient all in all.

3 Likes

FWIW, I think that even the above announcement is more than many open-source compilers will provide. LLVM, for instance, has hundreds of miscompilation issues open, many of which are years old, affect common programs, remain unfixed, and yet rarely are they even mentioned in release notes, much less proactively advertised pre-release.

Regardless, I agree that it reasonable to ponder how we might do better; in particular, how can we broadly advertise this sort of issue in the future. ghcup news sounds like a good step. I also wonder what other projects have done in this regard. Perhaps there are other communications channels that we should be taking advantage of?

7 Likes

Stack’s online documentation uses Material for MkDocs, and that has an ‘announcement bar’ functionality.

GHC 9.12.1’s bug may not have met that threshold, but if the GHC project ever needs to get a message out urgently and widely, I am happy to use Stack’s web site to help do that.

2 Likes

After a bit of thinking, I came to the following conclusion:

  • every compiler correctness issue should trigger a Haskell security advisory
  • an HSEC item has a well-defined procedure
    • public communication
    • a database that can be consumed by tooling

That means, ultimately, both GHCup and cabal-install (or cabal-audit) could warn when they detect a borked compiler that is in use through the HSEC process/database.

Why should every compiler correctness issue trigger a security advisory?

The most prominent definition of “software insecurity” that I know from my research in automotive industry is that from LANGSEC: “unexpected computation”.

Any unexpected computation can potentially lead to an exploit:

  • DoS: a web handler suddenly crashes due to an unexpected numerical result
  • data integrity: code that branches on numerical ranges may suddenly behave different and compromise a database
  • etc.

Just because we don’t know a concrete exploitation scenario doesn’t mean it’s not security relevant. LLVM being a bad citizen is not a particularly good argument to me. But I’m interested to hear what the SRT has to say: @frasertweedale @gdifolco @tristanC

7 Likes

In principle, the Stack tool could maintain a list of GHC versions that were deprecated (in a ‘DO NOT USE!’ sense), and warn by default if a user built specifying such a version (and have a Stack configuration option to allow such warnings to be muted). However, to implement that, I would need advice about the versions on the DO NOT USE! list.

2 Likes

Raising HSEC advisories on some high-impact compiler issues (e.g. the issue under discussion here) is something that we could consider. However, I think we should be careful not to put in place an expectation that any issue plausibly affecting correctness will have an advisory. Looking at the incorrect runtime result label reveals over 100 issues with a wide range of severities and scopes, most being quite minor. We cannot, and should not, insist that these all incur the cost of an advisory.

LLVM being a bad citizen is not a particularly good argument to me.

To be clear, I’m not claiming that this is an argument against improving. I’m merely observing that this is a hard problem which even projects with considerably larger resource budgets than GHC also struggle with.

7 Likes

I agree that we should try to minimize the impact of bugs like this. That’s why I pushed for limiting the scope of 9.12.2.

I’m less convinced that a rigid process for expedited releases would help. There are always many decisions to make depending on the release(s) affected and the bug in question. Combined with the need for such measures thankfully being somewhat rare I’m afraid it would quickly turn into a very fuzzy thing not meaningfully different from what we do now.

I think we should evaluate if there are things that we could have done better. I don’t think anyone, the maintainers included, enjoys the calendar-time a typical release requires. And 9.12.2 might be a uniquely informative release to look back on. But if we want to avoid mistakes slipping into releases then I’m not sure how much can actually be done away with. Both for expedited releases and in general.

Personally I suspect the intersection between people who were able to avoid all discussion and announcements about the 9.12.1 bug, but would have seen a HSEC entry to be very close to, if not, zero.

That is not to say the idea is without merit. But it seems like the wrong tool if reaching end users is the primary goal.

Reading between the lines I imagine what you want is primarily some machine readable source of warnings that ghcup can show to users if necessary. HSEC primarily being interesting because it’s the closest to such a thing we currently have?

2 Likes

The HSRT already added support for compiler bugs to the advisory database (see PR#213) and we proposed to issue some HSECs as discussed in issue#210 according to the following Common Weakness Enumeration:

I agree with @bgamari that not every compiler bug should automatically warrants an advisory, though it might be helpful in case the fix is not readily available too (e.g. that was not obvious for ghc#23034). However cabal-audit doesn’t seem to support the new GHC ComponentIdentifier yet, but I think that would be a nice way to warn the end user, at least for known vulnerabilities.

Hope that helps,
-Tristan

6 Likes

@mpilgrem GHC musl image glcr.b-data.ch/ghc/ghc-musl:9.12.2 (linux/amd64, linux/arm64/v8) is available now.

The image is also published on Quay and Docker Hub:

  • quay.io/benz0li/ghc-musl:9.12.2
  • docker.io/benz0li/ghc-musl:9.12.2

The separate linux/riscv64 image glcr.b-data.ch/ghc/ghc-musl:9.12.2-linux-riscv64 was built using --flavour=quick+llvm instead of --flavour=perf+llvm+split_sections.

Cross reference:

5 Likes

That’s not what I was getting at.

I want to avoid introducing special casing to GHCup for compiler unsoundness bugs. However, dealing with security advisories:

  • has a technical interface
  • a well defined format and process
  • can be implemented by all tooling, including GHCup and cabal

“The compiler is broken, look at the GHC issue and try to figure out what the impact is” is not very convenient, as compared to a proper HSEC, with links to CWE, CVE and CAPEC, including scoring, mitigation information etc.

Does that make it clearer?

cabal-audit can refer to a version of GHC-the-package as vulnerable.

An incorrect runtime result per se might not be a vulnerability, but a silent regression in this area likely is.

4 Likes

I filed an advisory: GHC 9.12.1 sub-word division · Issue #265 · haskell/security-advisories · GitHub

8 Likes

The Fedora Linux ghc9.12 package (for Fedora 41 and later) has been updated to 9.12.2: the builds are all stable now.

Just run sudo dnf install ghc9.12 to install.

4 Likes