State of the Cabal - Q1+Q2/2021

Announcements
#1

Hello Haskell!

Cabal has been going through some changes over the past 6 months. COVID-19 left a major impact on the status of maintenance for Cabal and cabal-install. Due to various life events, routine maintenance fell on the shoulders of a single individual, resulting in a Cabal-3.4.0.0 and cabal-install-3.4.0.0 release that took a human toll. As a result, the library and the tool were left without a primary maintainer as of its release. I mention these things, because they’re a nasty part of the OSS process, and it’s good to be aware of them, but I’d also like to highlight in particular why Cabal (resp. cabal-install) has had such a rough time over the past few years. Cabal (resp. cabal-install) is a very difficult library to maintain, and carries with it a deep burden in terms of legacy code (the breadth of GHC’s supported by the library stretches back to GHC 7.6, released in April, 2013). It also, historically, has had a very colorful history in terms of the ease of contribution processes and transparency, as well as working with other tooling. Without trying to erase anything in particular that has happened, the Cabal team would like to make some announcements and see if we can begin to establish a better dialogue with modern tooling, and help make Cabal (resp. cabal-install) a more community demand-driven project. This announcement will detail some updates to the team, as well as the direction we want to take in the future to make Cabal (resp. cabal-install) a friendlier library and tooling system to work with. We submit this to the community as an RFC, so we can take your feedback into account moving forward.

Changes in Management

  • The current (active) maintainer team consists of Mikolaj Konarski (Well-Typed), Francesco Gazzetta (Independent), and Emily Pillmore (Haskell Foundation).

  • Oleg Grenrus is no longer a maintainer of Cabal or cabal-install, and has moved into a more advisory capacity. You may see him commenting on merge requests (MRs) from time to time, helping out.

  • Mikhael Glushenkov and Herbert Valerio Riedel are still maintainers, but life events during the pandemic have made it hard for them to continue their work, so you may not see them for a while. We wish them well, and hope things get better soon.

  • Gershom Bazerman (gbaz) is an advisor of the project

Current Needs

One of the things that makes Cabal (resp. cabal-install) difficult to work with is the lack of component-wise exact-printing and associated data structures that allow one to manipulate .cabal and cabal.project files. We would like to change this. Workflows such as formatting, doc testing, build planning, local doc hosting, and env file manipulation are commonplace in modern Haskell. The reasons these things live in places like Oleg’s excellent cabal-extras repository, are twofold:

  1. Cabal does not have a round-tripping parser and pretty printer. A knock-on effect of this is that we don’t have a coherent story for talking about individual component-wise manipulation of entries into the .cabal or cabal.project files exposed at the API level. Historically, attempts at this have been made: hpack, for example, is a dependency of Stack that provides ergonomic ways of working with a .cabal file.
  2. Commands in the cabal-install tool are largely ad-hoc, and are difficult manipulate, accumulating cruft almost immediately due to the lack of a coherent API. For example, cabal gen-bounds is a fantastic command, which generates reasonable bounds for a .cabal file by analyzing its dependencies. However, it’s missing half of the story: it will not update the .cabal file for you, because it has no reasonable means of manipulating dependencies in a given stanza! Additionally, consider the recent cabal init work, where, due to not being able to represent stanzas or package descriptions in a reasonable format, resulted in many issues over its lifetime, resulting in a near-complete rewrite over the past 6 months.

An exact printer is a byte-for-byte round tripping parser/printer combo. In order to begin clearing out the cruft, and to make working with Cabal’s constructs as friendly to tool builders as possible, we need an exact-printer, and its associated data structures. This leads me into the direction we’d like to take in the future.

Future Plans

Modern Haskell programmers have a few workflows that are now so common place that even GHC has started to incorporate them. Namely, doctesting, file formatting (both source and .cabal), env file manipulation, and build planning. To meet this demand, Oleg has done a great job with cabal-extras (see above), but juggling as many binaries as there are extras is cumbersome, and there is no real reason they do not exist in the tool. In addition to incorporating extra conveniences, with the introduction of an exact-printer we can start providing a means for cleaning up the legacy cruft. As a result, we expect the timeline for the next 3.8 release to be as follows:

For 3.8:

  • Write the exact-printer and its associated data structures
  • Finish the cabal-env work
  • Incorporate cabal-plan and --show-build-info into the Cabal library as a first class feature. This will have the additional effect of improving peripheral tooling such as Haskell Language Server (HLS), and provide the ability to parallelize builds more easily.
  • Incorporate a selection of Oleg’s cabal-extras work into the cabal-install tool for convenience

Nice to have:

  • Open season on cruft and removing the rest of the v1 legacy commands in accordance with Oleg’s plan: https://mail.haskell.org/pipermail/cabal-devel/2020-September/010488.html. A note on this that can be addressed as part of the RFC: I believe ripping off the bandaid is better than a slow and confusing sunset. Would people prefer if we just removed them by 3.8, or is there interest in keeping them alive according to schedule?

Better Contributor Onboarding

Cabal hasn’t been the most friendly place to submit a pull request due to various hurdles and attitudes. We’d like to change that. Please keep in mind that Francesco, Mikolaj and I are not the previous maintainers, and that we’re trying to start fresh here. We want your feedback, your contributions, and your ideas! After all, what is a build tool for, if not for helping users build? We’d like to make contributing to the library and tool a much more pleasant experience, and we welcome anyone and everyone to contribute. We’re working on going through the backlog and finding tickets for people to work on. Don’t hesitate to chat us up in the #hackage channel on irc.libera.chat. If you see a reference to Freenode, please alert us so that we can update it to Cofreenode and point other people to the correct network :wink:

Conclusion

It’s an exciting time to be a part of Cabal, and thanks for reading.

Cheers,

The Cabal Team

56 Likes
Haskell Foundation May Update
#2

Thank you, Emily, for this important and encouraging update, this is good news indeed!

One thing I really want to encourage is better version coordination between ghc, Cabal, and cabal-install: in my opinion each cabal-install release should target the actual Cabal version in the current (or next imminent) ghc release, and also Stackage LTS as far as possible. Personally I have long believed that cabal-install should be included in ghc to enforce this (like cargo and rust) - well of course the code is actually included but not built…

The lack of sync has been problematic for the last few ghc and Stackage major releases, eg cabal-install has not been in Stackage since LTS 14 with 2.4.1.0!! Supporting back to ghc7 is great, but if it can’t even build with current Haskell distributions that is a bit moot (yes orthogonal things but still).

Really want to thank you people for stepping up to lead the maintenance of cabal. :purple_heart:

9 Likes
#3

Great analysis and great news for the haskell ecosystem and its tooling.
I would like to have more time to help this new effort in make cabal an even better tool. All contributions will be welcomed, including documentation improvements or triaging the big issue tracker.
If you are interesting in the second point i would recommend subscribe to https://www.codetriage.com/haskell/cabal. It will send you an email with some issues/prs uniformly distributed by creating date to take a look and know a little bit more about the project and its needs. After some time you could even try to make the issue progress and help maintainers in its handling.

8 Likes
#4

Those v1-* commands need to go, it’s been years.

6 Likes
#5

Unfortunately there are still quite a few users using them, partly cause they tried to switch to v2 and the new behaviour did not fit their workflow and/or bugs pushed them back (the issue tracker tell us that sad history). So i think we should be kind and cautious with the deprecation, specially the main ones: v1-configure, v1-build and v1-install.

5 Likes
#6

The main culprit is the replacement for cabal install v1-install for libraries had to have been cabal v2-install --lib. Users were used to install libs globally with cabal and then work with ghc standalone, in a non project centric workflow.
But cabal v2-install -lib does not have an appropiate name and a ergonomic interface. Iirc the plan is merge @phadej cabal-env to cover those workflows.
So at least i would wait to that change to deprecate cabal v1-install
//cc @emilypi

4 Likes
#7

A couple thoughts regarding long term:

Questions:

  1. Why is cabal upgrade necessary for (major) GHC upgrades?

  2. cabal-install has a lot of code besides the executable.
    Some of the code in cabal-install could be useful, be referenced by tools - if this code were made available as a library.

Do these goals seem worthwile:

  1. extract a stable interface, which GHC would expose to cabal.
    Move / keep all GHC-specific code necessary for cabal to function, to GHC.
    This would allow cabal-install to grow at its own pace. A new release of cabal-install would not be required with each (major) GHC release.

  2. keep as much of the code as possible in cabal-install-library.
    Export / expose as many modules and functions as possible.
    The cabal-install executable could check command line args and call the relevant API in cabal-install-library.

?

2 Likes
#8

Thanks, this is great news! I’m really looking forward to see what improvements and new ideas this unlocks.

#9

Yes, this is great news!

Has there been any discussion (with or without the stack team) about the possibility of merging with stack, so there is one build tool we know and love and work on together?

While the two provide slightly different workflows, it seems like a merge would result in not only a better workflow for onboarding, but also allow us to better maintain both of those general workflows. For example, stack is a bit too structured for more experienced haskellers, or those who have stuck with cabal over the years, but it seems like a small gap to fill. And while cabal has attempted to learn from stack, there’s significant gap there.

Whereas on the other hand, maintaining this split in the community tools seems silly and unproductive.

Why would it make sense to continue maintaining the split?

3 Likes
#10

Choice and competition are good. Let users decide.

3 Likes
#11
  1. different philosophy
  2. different maintainers
  3. different codebases (and as such different classes of bugs)

Cabal has always followed unix principles. That means you can combine command line tools and use them in unforeseen ways, because they’re built to be modular.

Stack follows the principle of being a holistic solution. Something you tell your client to use, so you don’t need to do annoying babysitting.

While in practice there’s little difference for power users, since you can bend the tool’s behavior in one or the other direction through configuration (such as system-ghc in stack), the main thing is expectations.

So although one or the other use case may be currently supported, you don’t know if it will be in x years, when the tool has evolved. The interface may be completely different, but what will likely remain is the underlying philosophy (unless there’s a major shift in maintainership).

5 Likes
#12

Sure, that’s all understandable, but I think there’s much more to consider than that:

  • stack came about (in large part) b/c improvements and changes to cabal were constrained and restricted enough to inhibit exploring options
  • we’ve already had the choice between cabal and stack for quite a few years, and as a result, we’ve been able to explore options
  • with the existence of stack and the exploration of those options, there was more pressure on improving and making cabal more powerful and correct.
  • our community is pretty small, we don’t have a lot of resources, combining resources and working together gets us further. Especially now that we’ve had the stack/cabal split for some time.
  • one tool can support different philosophies, especially if the various perspectives work together on that tool.
  • the continued split between cabal/stack maintains unnecessary and burdensome fractures in our community, tooling, and related aspects of the interactions in our community.
  • onboarding is way complicated b/c of these fractures and the burden on new users is avoidable.
  • at some point, this is more about egos than what’s best, reasonable, or sensible.

Especially with the existence of the HF, I think it’s reasonable to push back on this fracture, and to ask why we’re continuing to maintain it over exploring what it would be like to merge or otherwise combine efforts.

5 Likes
#13

I question this narrative. I believe there were disagreements about “philosophy” to the point that both sides gave up trying. And this has continued into the present. I see no change here.

Yes and users have pretty much decided that it’s a split: https://taylor.fausak.me/2020/11/22/haskell-survey-results/#s3q0
There seems to be no significant steering into either direction.

Maybe.

Right. But you seem to forget that this is open source. We’re not run by a company that tells us what to work on and I hope that won’t happen.

That depends on the philosophy. Under the unix philosophy of things you can build a holistic solution that consists of many small well-defined parts.

I don’t think it’s possible the other way around. If you want to satisfy both unix and holistic… you have to start with unix.

Well, if you fork/re-invent a solution, you should be aware that you’re contributing to fragmentation. That may be the lesser of two evils, sure, but that doesn’t change it.

Until now I haven’t seen any reasonable suggestion on how to remove the fragmentation.

But fragmentation isn’t the worst of our problems, really. Open source lives through fragmentation, to a degree. But I agree that it isn’t strictly necessary in this case.

My suggestion is to use a proper stack.yaml and let developers decide what they want and use stack2cabal to convert.

I prefer to not have ad-hominem arguments in public, but I think this isn’t even the main issue.

I think we need to be very careful about what we wish for here otherwise it may be the cause of more fragmentation.

I see that the HF is very engaged in improving things (and I actively participate in some parts of it), but there’s also the danger that it becomes a top-down approach, which I hope it will not. It would be detrimental to the success of the HF.

HF should connect people, empower maintainers, mediate in discussions, advise in technical issues. But should they decide about the course of the ecosystem? I don’t think so.

5 Likes
#14 
cabal-install has a lot of code besides the executable.
Some of the code in cabal-install could be useful, be referenced by tools - if this code were made available as a library.

There is a library that exposes a ton of this code, to be referenced by tools. That library is called Cabal! All the ghc-specific code necessary for cabal to function is in this library, and it ships as a build library of GHC. This is in fact precisely why the cabal-install tool needs to be shipped often with major GHC upgrades – so it can link against the latest Cabal library that ships with GHC.

Most of what is in cabal-install and not the Cabal library is code specific to Cabal install, and also with a larger dep footprint than we would want to import fully into the GHC bootstrap deps. That said, there are certainly a few parts of cabal-install that are not now exposed as libraries, and we would like to see exposed moving forward – I’m particularly interested in the solver (and ideally in such a way that there’s a “modular” interface to the solver so on can experiment with plugging in other solvers as well).

In general there’s been a lot of attempts over the years to reduce the friction and coupling with GHC and cabal-install releases, each with only limited or partial success. One thing that people are hoping to move towards is decoupling “big feature” cabal releases from “compiler compat” cabal releases – having more active and proactive release managers should help a great deal in this respect.

2 Likes
#15

I think its fair to say that if there are particular stack-inspire features that people would like to contribute to cabal (say, better integration with snapshots) then its worth trying to see if that can be done with a streamlined process of discussion and review as compared to the past. If that goes well (who knows?) then it opens up some future possibilities for discussion. Better to make small, concrete steps that are in themselves worthwhile, and go from there…

5 Likes
#16

Yeah sure. However, I believe some things have no place in cabal, such as automatic installation of GHC.

Instead, the cabal (and unix way) would be to provide the ability of simple scriptable hooks (much like git hooks), that are executed at different stages of cabal. One such stage could be pre-configure and a user hook script could invoke, say, ghcup to install the GHC version that’s requested by cabal.

This is also what certain distro package managers do. Hooks is a common unix design pattern.

2 Likes
#17

Exactly, and so that’s why I ask to what extent the cabal team has considered the possibilities for addressing the split.

I’m not suggesting they should, I simply pointed out that with the existence of the HF, it’s a great time/opportunity to explore these options.

I’m curious why there’s resistance to even discussing the possibilities.

2 Likes
#18

We’re doing that right now and I have been questioning that you can unify the philosophies, unless you start at the unix philosophy and build your way up.

#19

I too hope that one day stack can be deprecated in favour of cabal (I’m currently a stack user). This would unify the ecosystem on one tool, which has a lot of benefits in my opinion.

I would start by adding cabal support for stackage. I do not think that cabal installing ghc is necessary as a short-medium term goal. I think many would switch back to cabal just with direct stackage support.

(I would suggest that cabal installing ghc is an eventual goal, primarily so beginners can get started as easily as possible).

2 Likes
#20

We (haskell ide team) know what is the some of the costs of having two build tools: we have to maintain one cabal.project and upt to six stack.yaml’s, our install script support both tools, we have builds in ci for both, etc. We have lot of issues labeled with one or another tool. Many hours of maintaining effort.
But still i personally think there is value in to have both:

  • you can have only one default set of config options per tool, if we merge both we’ll have to choose one of them. But it reveals that we only can have one design philosophy (or exponential complexity) per tool.
  • both tools has improved being influenced by the other one
  • to have more than one tool is a normal situation in some ecosystems (java with ant, maven, gradle and others, javascript with npm, grunt, gulp, etc). There are other ones with one predominant tool like Rust but i am not sure if it should be the “standard”

If both tools slowly and organically converge is a thing we will have to see. Maybe other ones can born, who knows.

6 Likes