Convenience in the Haskell ecosystem

#1

I read this comment in the NeoHaskell thread, on convenience in the Haskell ecosystem:

As it happens, I got a taste of some of these issues just yesterday. I thought I’d write up my experiences here, in the hope of spurring some discussion on these points.

My motivation yesterday was wanting to do a spot of data analysis. Nothing complicated: essentially, I just needed to plot some fields from a JSON file.

I first tried to do this in Haskell. The steps involved were, as I recall:

  1. cabal init
  2. Try to recall precisely which 10 packages I needed, and add them all to the Cabal file
  3. Start up Emacs… whoops, my set GHC version doesn’t have HLS, better change that…
  4. Add a cabal.project with a more appropriate GHC version, remembering of course to fix the base version
  5. Start up Emacs again… wait, why is HLS using the wrong compiler version?
  6. ghcup set ghc 9.2.7 just to be completely sure
  7. Finally start writing code… oh wait, I forgot to add a package to do HTTP requests…
  8. Add wreq to the Cabal file, restart HLS
  9. Start writing code for the second time… wait, why isn’t HLS reporting errors properly?
  10. Give up on HLS, use cabal build instead to find type errors
  11. Finally make good progress, but find Charts a little too restrictive for my needs

At this point, I gave up on Haskell and reluctantly decided to try Python instead, where it went like this:

  1. Create a virtualenv
  2. Activate the virtualenv
  3. pip install jupyter matplotlib
  4. jupyter notebook
  5. Make good progress.

Now, I want to emphasise at this point that Haskell’s solutions are actually technically superior. Virtualenvs are a monstrous hack which should never have been necessary, and Jupyter is, let us say, not a great environment for editing. Meanwhile, Cabal and Emacs are both great pieces of software. Even comparing Charts vs matplotlib, it feels safe to say the former has the better design, even if matplotlib is more featureful.

But, when it comes to convenience, Python wins hands down. HLS, for instance, is… not actually unstable, as such, but certainly requires a bit of care every now and then. (I still have no idea why it suddenly stopped working properly last night, though I’m sure I’ll figure it out eventually.) Having to remember to add the right libraries for your usecase — aeson, wreq, lens, Chart, Chart-gtk, text, … — is a bit of a pain compared to Python’s ‘batteries-included’ standard library and ecosystem. And various things are incompatible in subtle ways which are only obvious to someone with more experience. (GHC version requires HLS version and base version which affects other versions…)

So what’s the solution? I’m not entirely sure, but what I want is a workflow which looks something like this:

  1. cabal init creates a sane Cabal file, with a good set of default packages
  2. cabal add Charts Charts-gtk wreq for the extra packages
  3. Open up Emacs, and HLS works immediately with my compiler version

It would require a lot of coordination between community members, but I hope that something like this should one day be possible.

EDIT: On reflection, I’ve realised that most of the papercuts I mentioned are to do with HLS specifically. And, in particular, the fact that maintaining it for different GHC versions is a major pain. Perhaps we should direct more efforts towards solving this problem (though I know there’s work being done on the GHC API already).

EDIT2: Figured out the HLS error messages problem I mentioned earlier. It was because HLS was for some reason not detecting that I had a Cabal file, so it couldn’t find modules from my dependencies. Adding an hie.yaml fixed the problem.

This, let me note, is yet another problem which requires a lot of experience to diagnose and solve. Ideally, beginners shouldn’t even need to know what an ‘HIE cradle’ is, let alone need to write one manually…

33 Likes
Introducing NeoHaskell: A beacon of joy in a greyed tech world
[Dream] Towards standard source code formatting
Convenience in Haskell: Ergonomics of Cabal
#2

I’ve been thinking that it would be great if HLS (or perhaps a separate project) would make a much more minimal language server which only does error reporting (and perhaps some other low hanging things). That could just be implemented with a parser of GHC’s error messages (possibly using GHC’s JSON mode) without tight coupling to specific GHC versions. Then you can’t have things like “go to definition”, but it would be useful to use that as a fallback in case the HLS version does not match the GHC version.

That’s basically what summoner was for. Or at least it allowed you to scaffold new projects with a custom prelude. I think the approach was a bit limited, for example many versions had to be hard coded in the summer itself instead of relying on external sources of truth and as far as I know it did not really allow you to add a default set of useful packages (other than the custom prelude).

I’d love to help build a better replacement, but I don’t think I have enough time to do it on my own.

6 Likes
#3

That’s pretty much what dante is. (And Intero before it, which is now unmaintained.) But it’s true that integrating something like that with HLS would be fairly satisfactory as a fallback experience. I wonder what would be involved in implementing it?

Sounds like a great idea! I’d definitely be willing to help: right now I have a Master’s thesis to finish, but in a month or so I should have more time.

5 Likes
#4

Not fan of this. Cabal should not default to any package other than base for one simple reason. Now the complaint is “It bothers me to add every single package”, if cabal added by default some sensible packages (ex: containers), the complaint would be “It bothers me to remove package I don’t need”.

I’d prefer something like cabal init --with-template <template-name> where templates are either stored in a local folder or internet. (Actually stack has something like this IIRC, and I don’t know if cabal actually has this functionality)

hell yes!! this is needed. How would the interface be for a multiple-package project?? (honest question) example: cabal add Chart -p <all | local-package-name>?. Isn’t it a little bit confusing?

hls is a little bit clumsy to use. Generally, I use gen-hie to generate the hie.yaml file, but this is a no solution as I depend on another tool. I wonder if hls could handle the “no cradle” error by asking “do you want to create a hie.yaml based on your cabal file?”

To be fair, jupyter has less power than a language server (ex: pyrigth or pylance).

I think this is kind of a big point. Python has great tooling if you are bounded to the python ecosystem. Sure, jupyter notebooks are great tool for beginners. I work as a data scientist and notebooks are kind of the go-to tool for many of us… My problem is that I am the guy who has to translate sh***y notebook code into actual python executable, and oh boy, that is a mess!!: PYTHONPATH issues, broken dependencies everywhere when building docker containers with python code. Do you use vscode/vim/emacs? Sorry, but your co-worker uses Pycharm which uses all sort of magic to create a great user experience at the cost of breaking every other editor configuration… and a large etc (conda vs pipenv vs virtualenv vs poetry vs …)

Ok, sure that haskell ecosystem could do better but I think it is mostly about providing good error message. As an example (IIRC) cabal fails with a dependency error when base version missmatches with compiler. Wouldn’t it be better if the error was “this cabal file needs ghc-XYZ but found ghc-YYY. Please, install a compatible compiler version”. Same with the cradle thing in hls…

9 Likes
#5

Fair enough on cabal init. But I really like the idea for Cabal templates! Stack templates were great; in my personal opinion, their main problem was that it was unclear which template to use. There weren’t many ‘official’ ones, IIRC.

My immediate thought is to pass it the location of a Cabal file, defaulting to a *.cabal file in the current directory if it exists and is unique.

That’s essentially what it does already, to my understanding. If there’s no hie.yaml, it calls gen-hie to create one on-the-fly. It seems to work pretty well, except for when it doesn’t. In my case, I think the problem was that it was looking in the wrong directory, so it couldn’t find the Cabal file.

Ooh, that would be really great! It doesn’t even sound like a big change, though I’m sure someone will prove me wrong there…

2 Likes
#6

Nice. cabal add Charts --file <my-folder>/<package.cabal>. I also had this proposal in other thread

wait what? I has never done it to me… hahaha maybe I’ve configured hls wrongly all this time

yes, It is always more complex than I looks. But I don’t know abotu the cabal code base.

2 Likes
#7

Oh, I didn’t mean that it creates a file hie.yaml. It just generates the cradle as needed, if there’s no hie.yaml.

2 Likes
#8

For a Stack/Stackage user, the problem of package versions that work well together is solved, but still leaves the problems of: (a) finding what packages you can depend on (and chosing between packages, when you have alternatives); and (b) specifying them.

In respect of (a), that is an ‘opinionated’ thing. I understood packages such as rio to be examples of people collecting together other underlying packages to solve certain types of task (in the case of rio, the task of ‘writing production software’). I do not know if there is a ‘data analysis’ equivalent of rio.

In respect of (b), when you look at the potential complexities, I am not convinced that a ‘specify it at the command line’ solution is a better one than simply listing them under the dependencies: key of a package’s package.yaml file (assuming you are using Hpack).

3 Likes
#9

It’s solved even for Cabal: I was just able to list packages under build-depends, without versions, and Cabal found working versions for me without a problem. My concern is more the tooling around that working core of the package manager.

True. But I wasn’t talking about an all-in-one framework like rio: I was just thinking about a template for an initial Cabal file which includes — let’s say — text, bytestring, containers, aeson alongside base. (Or something along those lines; don’t take that list too seriously.) And then you can just remove whichever ones you don’t need, if you want to do so.

This is a similar story to (a): cabal add would merely add the package to the Cabal file. It’s nothing different to manually editing the Cabal (or hpack) file, just a bit more convenience in certain cases.

True, that’s not a very important thing to do… but in a way that’s my point in this thread: on its own that change adds merely a little bit of convenience, but as you make many such improvements, the little bits add up.

2 Likes
#10

For Stack users, the problem of ‘templates’ is also solved. stack new takes the template as an argument (the default being new-template). stack templates provides information on the use of templates with Stack.

4 Likes
#11

Yes, Stack does solve some of these issues. On the other hand, it exacerbates others: most importantly. its HLS integration isn’t nearly as good (and was the main reason I eventually switched from Stack back to Cabal a couple of years ago).

2 Likes
#12

Incidentally, I’d like to be clear that I don’t mean this to turn into yet another Stack-vs-Cabal argument. I hope we can all agree that Stack is better at some things and Cabal is better at other things. My main argument here is: Haskell tooling in general has lots of papercuts, and we should figure out what these are and fix them so that everyone has a better experience.

7 Likes
#13

I think HLS may have moved on in recent iterations. I think, with one possible exception, HLS today works with Stack like it works with Cabal (the tool). I say that because I use Visual Studio Code with the Haskell extension with the code of the Stack project (which is reasonably complex), and I use Stack to build, and it works fine.

The exception is internal libraries and this Stack issue: stack ghci/repl with internal sublibraries only works after a build · Issue #4148 · commercialhaskell/stack · GitHub. That is an issue for HLS because, currently, it makes use (I think) of stack ghci to obtain the information it needs. That may change for projects that use versions of GHC that come with more modern versions of Cabal (the library). There is a discussion about that here: Improve stack support for HLS · Issue #6154 · commercialhaskell/stack · GitHub.

4 Likes
#14

That is my interest too: what do people experience as ‘papercuts’ in the currently available tools (my own focus is what people experience when they try to use Stack) and, having identified them, can they be ‘fixed’. EDIT: Often the ‘fix’ is better documentation, including tutorials.

One thing I have observed with Stack is people ask ‘Can Stack be more configurable, to allow me to configure it do to this my preferred way?’. The answer is often ‘yes’. However, the consequence of increased configurability is that the tool appears to newcomers to be ever more complex. The online help on configuration grows ever longer: Environment variables - The Haskell Tool Stack, Configuration (project and global) - The Haskell Tool Stack and Global flags and options - The Haskell Tool Stack ; as does the documentation of flags and options for individual Stack commands - stack build and its synoyms being the poster child: test command - The Haskell Tool Stack.

Stack’s response to that has been to try to carve out a ‘tutorial-like’ help for the most common commands/options (User's guide (introductory) - The Haskell Tool Stack).

7 Likes
#15

Ah, great to hear!

I think there’s three we’ve established so far in this thread:

  1. cabal add (or stack add or hpack add, as the case may be)
  2. A better story around templates, especially for Cabal (but I feel stack-templates could do with some curation too)
  3. A fallback for HLS when it can‘t find tooling

Yep, I’ve always liked that page! It looks like Cabal has a ‘getting started’ page too, but it’s much less detailed.

Documentation only goes so far, though… it’s tooling I’m more interested in here.

3 Likes
#16

The history of the Stack project’s approach to templates and ‘curation’ at a central location is that is how it started (in about 2015). However, puttng it bluntly, there were not enough people willing to spend enough of their time on that activity - nobody wanted to be a ‘template manager’. So, by 2020, Stack had ‘frozen’ the additon of new templates ‘centrally’ and had made template provision a ‘distributed’ activity (if it happens at all; I don’t see any people announcing they have added to the universe of public Stack templates).

I do look after the ‘central’ ‘frozen’ collection of about 25 templates, at GitHub - commercialhaskell/stack-templates: Project templates for stack new.

6 Likes
#17

A little off-topic, but: one of the challenges of a potential stack add (and it arises in other areas of Stack too) is this:

If you have a *.yaml file that is ‘nicely formatted, with comments’ (from the perspective a human user of the file), and a machine then wants to add (or remove) some element of it, and you want the end result to still be ‘nicely formatted’ (as before) - that is a non-trivial task.

If anybody reads this and knows of an existing Haskell library that can acheive it - please post here! EDIT: By the way, ChatGPT can do it, if you ask it nicely!

4 Likes
#18

Just wanted to say that I’ve been a big stack user since day dot, thank you @mpilgrem for picking up its maintenance :raised_hands:

5 Likes
#19

The task is not just non-trivial, Cabal has certain operations that do similar things, like cabal gen-bounds, and the issue has been sitting unresolved for years (#7544). It’s not even all that hard to do, just that it requires a lot of commitment and everyone else has better things to do.

5 Likes
#20

Yes, I’m aware of the history.

If I may be blunt for a moment, my own view of the reason that curated Stack templates were unsuccessful is that they just aren’t very useful for most users. If I look through the stack-templates repo, they mostly fall into two groups:

  1. Templates focussed around a specific library: foundation, hakyll-template, servant…
  2. Templates associated with a specific person: chrisdone, franklinchen, kurt…

Group (1) are useful if you’re using that specific library — but if you’re not, they’re irrelevant. Group (2) is even worse: how am I, as a beginner, supposed to know if I should follow Chris Done’s or Frank Linchen’s template? And I’m not even sure what value curation has for such idiosyncratic templates.

I suspect curated templates would offer the most value if they were just the opposite: there should be a small choice, each template should be of general applicability, and it should be obvious which one to use when. I’m thinking it could work better to have a much more limited choice between, say, minimal-lib, minimal-exe, large-exe and batteries-included, or something along those lines. Further choice could be provided by a mechanism like that offered by the current Stack template support.

Indeed, this requires writing an exact-printing parser, which is difficult. As it happens, I believe @Liamzy is currently working on one for Cabal, but I’m not sure of its current status.

4 Likes