Convenience in Haskell: Ergonomics of Cabal

#1

A sort of branch-off discussion of Convenience in the Haskell ecosystem:

TL;DR:
Should we be trying to create some type of wrapper around hpack/cabal that focuses on user experience, avoiding the work of creating a cabal exact-printer, not burdening cabal maintainers with thinking about perfect UX, and leaving cabal alone to be a good packaging tool/power user tool?

Background Context

So I was trying out Bodigrim’s cabal-add tool the other day, and it worked great. Admittedly I didn’t push it very hard, I just ran through the standard user-story for it but it worked basically how I wanted it to. Thanks Bodigrim for making that!

But as I was working through one of my projects, I was reminded of another major annoyance when working with cabal: other-modules and exposed-modules. Forgetting to add a module to one of these lists causes HLS to get angry at you, and the error reporting for it can be a bit vague. So I got to thinking, it sure would be nice if these two module types could be detected automatically. I started researching past discussion on this and ran into State of Cabal Q1/Q2 2021 and this github issue

I thought that Matt Parsons’ glob syntax was a great way to solve this issue, but Oleg was against the change:

I’d be 100% against allowing files with globs to be uploded to Hackage, as ”which package exposes XYZ module” will be impossible to answer only considering the index.

I’d like to have solution to this, but losing the declarativeness is huge concern, and if it’s not addressed, this will be wontdo.

This comment also really resonated with me:

I agree with @cdsmith .

I think cabal should decide to either be:

  • A low level build tool designed to be used by high level build tools (ie. stack) and IDEs.
  • A fully featured build tool that has everything you need to build + maintain your project in a convenient package.

I don’t take issue with Oleg’s opposition to this change, he has a valid reason for not wanting the glob pattern since hackage relies on explicit module lists to properly discover things. But then, it sounds like cabal’s primary target is not the user, it’s the packaging system.

My thoughts

It seems like its fundamentally a mismatch to try and make cabal a user facing tool since it has strict requirements to adhere to a machine based specification. I’ve also seen the problem recurring for a while now that a big part of what makes this difficult is we don’t have an exact-printer for cabal files. It seems to me that cabal should be a tool for power users. Cabal files use a proprietary format that is meant to provide exact descriptions for hackage, but also gives power users a lot of strength to express themselves.

I’ve also seen some other comments where people have said, the solution is to rely more on generating cabal files (rather than directly editing them?).

So if that’s the case, why not use a config format that’s easy to edit both for humans and programmatically-wise, like yaml? Then I found hpack, which seems to satisfy the requirement for a less exotic config format.

Potential Solution

We already have the easier to use yaml version of cabal files, and it seems like its fairly popular already! At this point, it feels like it would make more sense to write some wrapper around hpack/cabal which focuses on a user-friendly experience.

I could imagine some tool hacker with cli options like

  • hacker add <component> <package>
    Automatically adds packages to the dependencies section of an hpack file

  • hacker remove <component> <package>
    Automatically removes the package from dependencies

  • hacker build

  • hacker test

  • hacker run

and setting exposed-modules and other-modules to file system paths like

otherModules: "./src/internal/*"
exposedModules: "./src/public/*"

And then this tool would browse those paths to create an explicit package list when generating the cabal file.

Im sure there are other convenience commands people could think of that programmatically alter the hpack file from the cli. And each time one of these commands is run, hacker automatically regenerates the cabal file.

This tool would be the “user-facing cabal”, free to focus on an ideal user experience, without having to worry about breaking the cabal spec, and without having to create a cabal exact-printer/parser. And it’s not really the same as stack since it’s not focused on package sets, it would just be the normal hackage use-case.

I dunno! What do you all think?

8 Likes
#2

To be honest, I think the effort would be better spent on actually improving the aspects of cabal that need improving rather than (re)inventing another package description format/tool. I think having such an additional package description file (let alone having another format) is confusing [1]. Furthermore, I think it hampers adaptation of newer features in the ecosystem. For example, one of the reasons why backpack is not used all that much is because one of our two build tools (stack) does not support it (or has that changed recently? At least it did not support it for a very long while). I think by creating yet another tool/format you are again creating a situation where one needs double work to add/support new features, or those features will not get used at all.

Finally, especially initially, this new format will again be contented, and not everyone will use it. Why not spend your time on actually improving cabal, so that all haskellers can benefit from your work.

[1] at some point I lost 2 hours trying to figure out why I could not compile my package using stack even though I added everything to the cabal file, just to discover that there was also some hpack file and stack was overwriting my cabal file all the time.

(as a minor remark: the cabal file format is not my favorite format, and certainly cabal has places that can use improvement/polishing, but I really don’t get that so many people seem to fall over having to edit a text file to add a dependency, or explicitly specify that some file is part of your package. I actually really like both of these aspects, and even if I didn’t they seem to be just minor inconveniences at worst).

16 Likes
#3

I think it’s fine to experiment yourself! Build the tool as you go and use it on your own projects.

There’s really no harm in that. And tbh, it’s necessary. As far as standardization here, there are three options:

  1. Extend cabal
  2. Standardize on a new tool
  3. Have a proliferation of tools

You can’t do 1 or 2 with a nascent tool, so you have to go experiment. And if many people experiment, 3 happens.

So long as you focus on compatibility with cabal, I think you’ll be fine in the same way hpack is fine.

Forking hpack while maintaining compat with it might be a good option btw. Nixpkgs already supports package.yaml and hpack, so if you alias your tool, you can already get support there.

I actually started using hpack’s advanced features (includes) and immediately ran into issues. [1] [2] So I think there’s definitely room for innovation in this area.

#4

Just to be clear, I’m not advocating for another packaging format. I’m advocating to use hpacks format since it already exists. Stack uses hpack under the hood so I think that testifies to how popular it’s become. I’m advocating for a tool that simply wraps hpack and cabal to make a more user friendly experience. I was thinking of experimenting, but I figured I’d solicit opinions first

1 Like
#5

OP and people reading this thread might be interested in #7548.

4 Likes
#6

From an early version of the Cabal page on the Haskell site:

…but no mention of actually compiling packaged code - perhaps “Capal” would have been a better name. To me, it seems the intent here was more on packaging rather than compiling, which is only a consequence (or side effect) of Haskell currently being a compiled language - if Haskell was only interpreted, then packaging is all that’s required, with the interpreter eliminating the need for a compiling feature.

So while the ambiguous term “building” is thought of as both:

  • bundling the separate modules of a Haskell project into a conveniently-distributable format,

  • and the compilation process required to make that packaged code work for some specific Haskell implementation,

…the confusion over what is and isn’t C4641 intended purpose will probably continue! Since Hackage is append-only the only way forward seems to be this suggestion:

[…] support both formats indefinitely (but will not advance the old format) […]

(thank you, @f-a!) That new format should not care about the minutiae of how the Haskell sources have to be “prepared ahead-of-time” for use with some particular implementation - that is a detail for that implementation to deal with, not the package format. But if you’re still not convinced, then just look at OS-packaging tools like apk or apt - if they have a "download-source-then-compile-locally" option at all, it’s usually standardised or compartmentalised on an entirely-separate “third-party” build tool e.g. make:

K.I.S.S.

1 Like
#7

Yea agreed - I think that’s the point I’m trying to make. Cabal files are meant to describe packages for Hackage, and cabal-install uses that file to inform its operations. Maybe we’re asking too much by trying to make cabal cater to both package registration and users - a separate tool and project format that generated cabal files would alleviate the needs of the user without putting the burden on the cabal maintainers. Leave the cabal format alone and use a format easier to work with thats meant for users specifically.

This is the general sentiment I’ve seen from #7548 as well. “It’s not cabal’s job”, “Cabal needs things this way for hackage”, “This change is too disruptive”. We don’t have to disrupt it if we just build on top of it

#8

I’d put a glob as a comment in Cabal file:

  exposed-modules:
    Data.Foo
    Data.Bar
    -- EXPAND FROM src/*/*.hs

and write a stupid tool, which watches folders and updates exposed-modules accordingly, keeping the marker EXPAND FROM in place.

6 Likes
#9

I think cabal-fmt either has this or something similar

1 Like
#10

@Ambrose looking at Oleg's gists - ANN: cabal-fmt, it seems you are right. AFAIU cabal-fmt is opinionated and does many things at once, so an enthusiatic developer might be interested to extract this particular piece of functionality. But otherwise I don’t see a strong reason to go any further than this approach.

1 Like
#11

cabal-expand when :cowboy_hat_face:

#12

I agree with this. I’d much rather improve the tooling for the already-standardised Cabal format than switch to another format. I’ve had the same problems you had with hpack.

This was exactly my point in creating the parent thread. These inconveniences are minor, so they don’t get fixed. But the many inconveniences add up, and especially for beginners, they create a bad experience.

3 Likes
#13

Yea, I agree that it’d be better to see cabal fixed than to see another competing format; standardization is really important. But sometimes it seems to me that we’re often at odds with balancing the technical needs vs the user needs. My thought is that maybe we shouldn’t try to shoehorn both needs into one tool. Maybe we need one set of tools to address the first need (e.g: cabal), and another set of tools to target the 2nd need, and let the user-friendly tool bridge that gap between the two ends of it.

But ultimately, I personally don’t care how we get to the place where the UX is better, I just want to see it happen. I want to help out, but I also think its worth just contributing to discussion before pouring a lot of efforts towards things. I’m just chewing over my own thoughts and asking questions to help clarify things for myself. Thanks for the discussion everyone!

#14

I think there’s a good case for .cabal to be a flat file with nothing fancy. Easy to parse and analyze purely, for instance. I might -1 adding wildcards to :cabal myself for that reason.

Which is why package.yaml is the format you’re looking for :grin:

#15

I do often wish that the cabal team would have just said “Oh! Wonderful! stack does all this cool stuff so we can keep cabal simple!”

But instead, cabal team realized they were losing marketshare hard, and got salty, and decided they need to reimplement the killer features that were drawing folks to stack. So now we have cabal.project to replicate stack.yaml (but with fewer features, worse UX, and an awkward file format), cabal-file includes to sort of do what package.yaml and YAML includes natively do (but without the ability to do cross-file referencing, an important feature for multi-package repositories).

There’s some obvious things that cabal can add to bridge the gap. Support package.yaml natively. It’s easy! Hell, support stack.yaml, too. Add a --file-watch flag. Load multiple targets in GHCi (sure, print a warning if it borks, but at least let me do the thing that works fine in stack and is a huge improvement for working on multi-package projects).

But why is cabal's goal "catching up to stack"? Why can’t we just have two tools at different levels of abstraction?

4 Likes
#16

I shouldn’t get drawn into these discussions which relitigate a very frustrating past, but I’ll just say this is not an accurate historical account of when, how and why cabal.project files were developed.

More generally, cabal’s goal has never been “catching up to stack”. It is certainly the case that if there’s a feature a lot of people like in another tool, then that’s a feature that is sometimes worth considering adding to a tool – but that’s how all tools in a common space interact, always, and presenting it negatively is strange.

In fact, the fact that cabal has no plans or desire to add package.yaml or stack.yaml support, cross-file referencing of cabal files, or a filewatch flag for that matter (all issues which have been discussed), is strong evidence that cabal is not motivated by “catching up to stack”.

So what this comment presents is basically a strawman claim that cabal has a certain goal that it does not, then it complains that cabal isn’t doing a good enough job at this goal – when in fact the evidence should lead one to conclude that cabal isn’t doing those things because that is not the goal.

On multi-package support, btw, was under active development for some time but required changes to ghc to do properly, and is now merged: Add support for loading multiple components into one repl session by mpickering · Pull Request #8726 · haskell/cabal · GitHub

19 Likes
#17

Are there docs about cabal’s design goals and non-goals?

It sounds like the main missing features are specialized for industrial use, which ofc isn’t the only driving faction of Haskell development. I’m sure the cabal contributors are balancing more varied concerns.

So I’d be interested in reading more about these decisions and the tradeoffs at play. I find Haskell design decisions usually have good reasons :grin:

#18

Here’s the original goals back when Cabal was first proposed and introduced in 2005, almost 20 years ago:

The Haskell Package System (Cabal) has the following main goal: to specify a standard way in which a Haskell tool can be packaged, so that it is easy for consumers to use it, or re-package it, regardless of the Haskell implementation or installation platform.

The Cabal also supports tool authors by providing an infrastructure that automates the process of building and packaging simple tools. It is not necessary to use this code—indeed complex libraries may exceed its abilities—but it should handle many cases with no trouble.

see: The Haskell Cabal

The above describes the goals of Cabal the library, and cabal packages.

We don’t have anything quite so pithy describing cabal-install the binary, though its description in its own cabal file perhaps is a start:

The ‘cabal’ command-line program simplifies the process of managing Haskell software by automating the fetching, configuration, compilation and installation of Haskell libraries and programs.

Since that was written, I think it it would be worth adding that it is a tool for automating common related tasks in developing such things – i.e. the test and benchmark facilities, etc.

The main thing these goals clarify is why some things do not make sense for cabal files – they interfere with their ability to be used as modular units of package distribution.

Regarding features of cabal-install the considerations are often a lot more loose, and just involve trying to avoid feature and code-bloat where external tools will suffice. The proposal for an external command system will help with this to some degree, as has the work that has been done in opening up more of cabal-install as a library that other tools can make use of: An external command system for cabal: what would you do with it?

3 Likes
#19

On the specific issue of glob expansion, there is a ticket about it, and a proposal on how to handle it that everyone seems to agree is at least fine, and some think is actively good. It was even in the issue linked in the initial post on this thread!

So this entire discussion is motivated by a false premise – that cabal is hostile to this feature, or must be. Instead, there was a technical architectural concern raised, and a good solution proposed. That solution, like other things we would like, remains blocked on an exact-printer.

So what we see is not any issue with the design goals of various systems or the need for new systems, or anything else, although reaching and maintaining a common understanding is of course important. Everything there is basically fine and good. What we see is somebody needs to write an exact-printer.

And again, there is a ticket for that (linked in the above ticket), which has, within it, a perfectly good design that just needs someone to implement it.

9 Likes
#20

Sorry for bringing up any conflict with this post. I did read through that github issue you mentioned but I suppose I didn’t understand the conversation enough to realize an acceptable solution had already been proposed. That there was already a satisfying solution on a proposed design of a cabal exact-printer also eluded me during my readings.

As far as I had read, it seemed that cabal exact-printer was pretty complicated and still pretty far out of reach to be implemented. That’s why I had the idea of just trying to wrap around cabal/hpack to make something that was more easily achievable now. But I’ll have to go back and re-read the issues to see where it landed.

Thanks for clarifying!