State of the Core Libraries Committee

[emilypi]

Hello Haskell!

The Core Libraries Committee is undergoing some changes at the moment, and I’d like to fill everyone in about the state of the CLC, what’s happening behind the scenes, and what we hope to accomplish in the future. It’s no secret that the CLC hasn’t been as active as of late, and hopefully this post will serve to clear that up and invite new interest in our plans going forward.

History of the CLC

Originally, the CLC was created to steward the base library and review changes to it, which no one was really empowered to do before its inception. In particular, this was extremely helpful in stewarding large changesets such as the FTP and AMP proposals. With the success of these proposal implementations, the responsibilities and headcount of the CLC grew to the point where they were asked to steward not just base, but also a collection of “mission critical” libraries that GHC relied upon in order to function. Slowly, the number of these packages grew to 13 including base. The previous authors and maintainers of the core libraries slowly exited the community for various reasons, and due to lack of ability to find new maintainers, responsibility devolved to the CLC taking control.

The CLC, however, did not grow along with it. CLC members viewed these packages as stable, and therefore not in need of consistent code-changes modulo a version bump here and there, or the odd update for new GHC features.

CLC as of Today

Today, we realize that the perception of a stable set of core libraries is not true to reality. In fact, this was realized about the time I joined in 2019, but it had not quite sunk in: our core libraries are in need of constant maintenance. In fact, many of the core libraries we do have are not stable because they are in a stable, performant and well-maintained state, but are stable because nothing has been done with them. It is a dire error that the two were confused. In many cases, we were the ones blocking the changes! This is not to say that each of the core libraries suffers, but many do, and this is a problem.

For example, recently orders of magnitude changes in performance and usability came in the form of pull requests to random and bytestring, where before they were considered to be completely stable. In the case of random, a CLC member stood in the way of that progress. In the case of other libraries, progress stalled because the CLC was not equipped to maintain the libraries it was charged with stewarding, and new maintainers could not be found.

However, this is not to heap blame on anyone in particular. The creeping expansion of our responsibilities on the CLC caused two major problems:

1. We had too many responsibilities, and scope kept creeping. What started out as base turned into base and stewarding core libraries. Stewarding the core libraries turned into maintaining the core libraries. Maintaining turned into architecting new feature changes etc.
2. Despite life generally being balanced with CLC duties, the last few years have also added the weight of a general malaise of stress induced by our respective sociopolitical and economic climates.

As a result, despite being 8-strong on paper, the CLC has actually been operating with only 2.5 active people since 2019 (the .5 coming from 2 sporadically active CLC members). This is not enough to keep up with the general workload, nor enough to decide quorum. Behind the scenes, we’ve been feeling this pressure and working with it since the COVID-19 pandemic started, but at this point in time, we’re looking to disband the current iteration of the CLC and work towards a new structure that will be more maintainable in the future.

CLC in the Future

Three weeks ago, I sent out an email to the CLC asking for a role call, advertising the will to restructure. With some gentle prodding, we have 3 members who raised their hand (Chessai, myself, and gwils), and 1 who will help us rebuild and reassess their involvement pending the outcome (Cale Gibbard). The 4 of us will meet on Thursday of next week to discuss a restructuring of our responsibilities and what to do moving forward.

I expect that the structure of the CLC will scale back to maintaining base as a main focus, with a new proposal process in the vein of the Haskell Foundation or GHC tech proposals process, and the designation of new roles to help shepherd issues to us and manage contributors. This will effectively be reasserting the CLC’s original mission statement, with some better process for onboarding community-led contributions to base, and a clearer notion of what it means to maintain core libraries. Then, finding new maintainers for some of the core libraries will be the final hurdle for the new CLC.

I think this is a very necessary change that is possibly 5 years too late, but c’est la vie. I’m happy that the CLC are amenable to this change, and I hope that we see alot of very fruitful discussion amongst the community on how best to structure in a way that meets your demands more effectively. I’m putting a time box on a finished process by mid to late September (give us a month+ to sort it out), and we’ll be back and more effective than ever. Also, since the email sent out served as a means of figuring out which members were actively, we will be adding as many or more CLC members to help move things forward.

Cheers, and I hope this gives enough detail for everyone.

Emily

[Kleidukos]

Thank you for the transparency, this was quite needed by the community at this point. I believe we can give back our trust to the CLC, now that we are more certain that it will able to better assume its duties.

[simonpj]

This all sounds great to me, thank you Emily.

Here is one thought in response. The reason that the CLC’s role expanded to 13 libraries is that they were perceived as particularly critical to the ecosystem; and more concretely, GHC itself depended on them. So, for example, a new release of GHC depends completely on a timely release of updated versions of these libraries. It wasn’t entirely accidental growth.

And these dependencies remain. Your post suggests, I think, that (say) bytestring or containers are no longer in the purview of the CLC, but instead are solely the responsibility of their maintainers. But I wonder if the CLC might still have a couple of contributions it could make to a larger set of libraries than just base, without supplanting the primacy of the maintainers? I’m thinking of two things in particular:

1. Offer an proposal process and infrastructure for discussing API changes. API changes in these key packages affect many people and — like changes to base — deserve broad discussion, perhaps broader than might happen on the individual packages’ GitHub repos. Note I say “offer” rather than “require” – this should be a support to the maintainers, not an imposition.

2. Provide a last-resort recovery mechanism in the event that maintainers become unresponsive. This can happen for a variety of perfectly understandable reasons, but in the rare occasions when it happens, it’s really, really helpful to have a recovery mechanism.

   It’s a kind of resiliency mechanism. If I go offline, or otherwise behave badly, others can take over GHC – it is not mine. In the same way, our most mission-critical libraries have in a way become – through their very success – public property, rather than owned by one person. They have become utilities, like gas or electricity, rather than nice-to-have with many suppliers like soap.

   Quite which libraries have become utilities in this way is a matter of judgement, of course. But it should be a badge of honour!

Lastly, a small thing: libraries “below” base, notably ghc-prim should be directly under CLC. ghc-prim is effectively part of GHC.

[emilypi]

Now I’m sad that I didn’t go into personal thoughts on the details of what we’d propose, but I agree 100% with your thoughts on this, and have voiced similar things! The idea of a proposal process for API changes to core libraries is not some thing we’ve spoken about though, and would be good for helping prospective maintainers stay on track with the rest of the community’s needs. That’s an excellent point. Point #2 is almost verbatim what we were thinking our function would be with respect to these libraries: recovery and stewardship, with a strict non-moderation policy except in emergencies.