Charting a course toward a stable API for GHC

The Haskell ecosystem has accumulated a wealth of developer tools over the years. Some of them, such as Hoogle and HLint, have existed for nearly two decades, while others, such as the Haskell Language Server and Retrie, are much more recent. GHC has changed substantially in that time, and this presents a significant challenge to tooling authors saddled with the burden of keeping their tools up to date. Too often, a promising new Haskell utility simply disappears because it’s too difficult to maintain. This is unfortunate, as programmers’ expectations for language tooling have only continued to rise; for many prospective users of Haskell, robust tools are table stakes.

At the heart of all Haskell tooling is, of course, GHC itself, which has become the de facto source of truth for both the definition of the Haskell language and many key packaging concerns. In theory, GHC has precisely the information many tooling authors need: it knows how to parse, typecheck, and transform Haskell code, and it knows a great deal about its structure. Sadly, initiatives to expose this information to tool authors have been challenging. Though attempts have been made to equip GHC with APIs for doing more than just building Haskell code, such as the plugin API and the API to GHCi, the APIs are complex, expose users to GHC internals, and are not particularly stable. More restricted avenues for providing information to tool authors, such as HIE files, have proven more successful, but they are a far cry from a cohesive story for how tool authors ought to get at information about Haskell programs.

It is for the above reasons that I am pleased to announce I will be leading the Haskell Foundation’s new GHC API stability initiative. As the project’s coordinator, my aim is explicitly not to dictate a design to be implemented but to facilitate the necessary conversations and distill whatever conclusions result into a concrete plan. This is no small task: any workable design must weigh the needs of tool authors, the maintenance burden placed on GHC developers, and the fundamental challenges of designing stable APIs in Haskell more broadly, among other things. Our plan of attack is to start small. We cannot hope to address all the complex needs of, say, typechecker plugins or HLS right out of the gate, but we believe we can provide real, immediate utility to many existing tools. This requires being deliberate about what we choose to support, starting from the needs of tooling authors rather than from the set of features that are easy to expose from the GHC codebase. Our goal is not to produce a totalizing model for how all GHC functionality can be or even ought to be exposed but merely a useful kernel we can commit to that can be grown and extended over time.

In the coming weeks, I will be reaching out to potential project stakeholders—mainly GHC developers and tool authors—to both collect perspectives and assemble a minimal working group. I do not expect this group’s work to be terribly glamorous: our immediate objective is to simply catalog existing needs and assess which ones we might practically address. Once the landscape is clear, we will begin the laborious process of exploring and weighing the design space of potential solutions with as pragmatic and realistic an eye as we can manage. I do not expect this to happen particularly quickly, but a major part of my role is to make sure discussions stay on track. Throughout the process, I’ll be posting updates to the community here on Haskell Discourse to keep you informed about our progress and to provide opportunities for a broader discussion.

If you have experience in this area and are interested in participating, by all means reach out to me at alexis@haskell.foundation. Your knowledge and wisdom are invaluable, and understanding your needs is crucial to this project’s success. Please keep in mind that drafting a grand vision for the future is an explicit non-goal of this project, so what we need most are case studies and stakeholders. If this applies to you (and you can spare a little time), don’t hesitate to drop me a line.

97 Likes

This is a project that’s been much demanded from the Haskell community, promised by the HF both at the 2022 Haskell Symposium and in the GHC project priorities plan earlier this year. Too many of our tools have bitrotted and disappeared; it’s important for all Haskell users that we can find a way to make it affordable to maintain what our community produces. I’m very happy that Alexis is able to lead this effort - I greatly trust her technical knowledge, communication skills, and organizational abilities, and her experiences and background allow her to easily take the perspectives of both GHC developers and GHC API customers. I look forward to all the extra ways that my Haskell development environment will be able to help me in a few years!

23 Likes

This is great!

I wonder what you consider API though? Is the places where GHC puts files, for example, API too? It affects distros, stack, ghcup, etc. This also includes things like the settings file.

Until now, GHC devs have tried to communicate when they make changes to those things, but it’s hard for them to foresee what people actually rely on. E.g. the new short hashes appended to GHC library dirs have caused problems for Fedora maintainers.

3 Likes

@lexi.lambda is running the project, and part of the job is to determine the scope of what the API should and should not cover. I’ll defer to her for specifics as the discussions progress.

That said, a successful project cannot serve all interests and be completed in a reasonable timeframe, and the primary beneficiaries of this project in particular are intended to be GHC developers and the developers of tools like linters, formatters, static analyzers, and automated refactorings, rather than software distributors. Making it easier to distribute Haskell is important; it’s just not the focus of this initiative in particular.

5 Likes

Too many of our tools have bitrotted and disappeared

This point was the opening point to the 2022 keynote of the Haskell Symposium!

The IOG GHC team would love to participate in this, it aligns with our modularity work. If and when the time comes please do drop us a line and dispatch some action items :slight_smile:

9 Likes

I’d love to see this work cover the exact-printing APIs!

3 Likes

Looking forward to hopefully participating in this too!

I hope Split out AST and Parser libraries from GHC by Ericson2314 · Pull Request #56 · haskellfoundation/tech-proposals · GitHub can help with this too. By pulling out and isolating some functionality that is already in use by many tools, I hope we can have a good “kernel” to keep work focused and the design space tractable as @lexi.lambda says. Ideally, that model of carving off and isolating some functionality so “stable via decoupling” (less likely to be disturbed by unrelated changes to distant part of the codebase) is very repeatable.

5 Likes

Did you perhaps mean another link? That one seems deleted an irrelevant. Maybe you meant Split out AST and Parser libraries from GHC by Ericson2314 · Pull Request #56 · haskellfoundation/tech-proposals · GitHub

2 Likes

Yup; lost the last character of the URL; thanks @tomjaguarpaw!

1 Like

I’m sure this is redundant information, but in case anyone missed it, Haskell for all: A GHC plugin for OpenTelemetry build metrics is a cool example of using GHC and even has feature requests, e.g.

This plugin was surprisingly difficult for me to implement because GHC’s Plugin interface is so constrained.

For example, the hs-opentelemetry-sdk package asks you to finalize any TracerProvider that you acquire, but there’s no good way (that I know of) to run finalization logic at the end of a ghc build using the Plugin interface.

6 Likes

@jmct is the HF still engaged in this project?

6 Likes

The HF still intends to continue with this project, but for reasons outside of our control, the staffing of the project has been an issue. Until we can find a qualified and available contractor the project is on hold from the HF’s perspective.

3 Likes

Everyone: please read this as an invitation!

Everyone would like a most stable and better-documented GHC API. Doing this takes work, including talking to lots of people that use the GHC API. It just needs someone who cares, to lead the effort. I bet there’d be quite a few people willing to contribute … but they need leadership and coordination.

If anyone is interested in helping, talk to Jose :slight_smile:

11 Likes

The initiative continues!

3 Likes