Proposal: unified installer

Haskell Foundation
#3

(This is probably going off on a tangent, so apologies for that. Feel free to direct me elsewhere.)

Has anyone considered going kind of the opposite way and provide all of Stackage as pre-built binary distro packages so that we don’t need to rebuild everything again and again in CI and on each and every dev’s laptop? I’d really love to be able to do something like:

apt-add-repository ppa:haskell/lts-16
apt install libghc-pandoc-dev

or even better:

apt-add-repository ppa:haskell/stackage
apt install libghc-pandoc-dev-lts16

(I’m reasonably confident that with careful file placement and some use of metapackages and virtual package provides/requires, having one repo with multiple co-installable LTS version should be possible.)

Now why would I want such a thing? One recent example: we updated the https://xmonad.org/ website recently. The first prototype was done using Hakyll, but the GitHub Action to rebuild the website took 45 minutes. Sure, we could get it down to a couple minutes (which is still orders of magnitude more than what we get with GitHub Pages’ Jekyll, btw) with a cache, but this (several hundred megabytes large) cache gets evicted quite soon, so the next time we’d get to updating the website, we’d need to wait another hour. And every contributor who forked the repo and wanted to see what their changes look like would need to wait as well. We ended up using Jekyll instead as all this waste of time and resources isn’t worth it.

We could’ve used Hakyll from Debian, but that only solves our problem. Anyone who needs a different version of Hakyll or Pandoc for any reason is screwed. See Hakyll’s own CI: https://github.com/jaspervdj/hakyll/actions. All builds take something between 30 to 60 minutes. That’s completely unacceptable.

I may be mixing things together. What I’m really proposing is probably two separate things:

  1. pre-built LTS binaries
  2. deb packages for those with carefully selected file placement and interdependencies to make them co-installable and easy to use

Alternatively, ghc could be faster, but I don’t think that’s a realistic goal. Most compilers I’ve worked with in my career were slow, and using pre-built dependencies was the only way to make CI reasonably fast. Rust is being mentioned as an example a lot recently, and from my limited experience with it, it’s still slow enough for pre-built binaries to make sense.

Thoughts?

3 Likes
#4

Is cargo really not backwards compatible with older rustc’s?

That I’m not sure of. I just meant that in the default usage of rustup, I end up with a new version of cargo for each new version of rustc.

The other way around where every rustc gets a new cargo is kinda what we have today? Most major GHC releases get an updated Cabal library I think.

That’s about the library, not about the binary executable. Both cabal-install and Stack are intended to be installed separately from GHC and to support multiple GHC versions.

Has anyone considered going kind of the opposite way and provide all of Stackage as pre-built binary distro packages so that we don’t need to rebuild everything again and again in CI and on each and every dev’s laptop?

There has been some conversation of this kind of thing in the past, but there are a lot of obstacles to overcome along the way. If someone wants to look into this and try to take ownership of something like this, it’s certainly something I’d be happy to advise on.

#5

This is indeed a bit of a tangent, but I just thought I should mention that Tikhon Jelvis and I just implemented a Github Action for building the haskell.org Hakyll executable. We used Nix and it takes only three minutes. I’m a sceptic who hasn’t really used Nix before, but if it saved 45 minutes then it sounds really worthwhile!

2 Likes
#6

As the ghcup team wasn’t invited to that meeting, I can reply here:

What you’re describing pretty much sounds like a re-invention of ghcup, which already exposes a (probably brittle and imperfect) API: https://hackage.haskell.org/package/ghcup-0.1.14.1/docs/GHCup.html

Adding more exectuables to ghcup is rather easy. As was done with HLS recently.

8 Likes
GHCUp button on www.haskell.org
#7

I think the proximate problem that really needs to be solved (I don’t care how) is that (some) people on windows don’t want to use a command line tool. They want a traditional windows installer (MSI) that looks and works like all the others.

A solution that lets them point and click and double-click to a bunch of stuff on their system is the key thing. Any libraries or whatever that are reusable or not to get there – eh, that’s a detail.

1 Like
#8

I think the proximate problem that really needs to be solved (I don’t care how) is that (some) people on windows don’t want to use a command line tool. They want a traditional windows installer (MSI) that looks and works like all the others.

Sure. As I explained in the other haskell foundation thread there are long standing plans to add windows support to ghcup.

First of all… ghcup is an API (that was the idea… but it will probably not prove to be a good API until someone actually tries to use it). Second, it’s very well possible to build an installer around it. So all the installer would really do would be to expose the ghcup options as checkboxes and then run ghcup in the background. The user doesn’t really have to understand what’s going on.

4 Likes
#9

Oh, so if I understand it correctly Nix (or Cachix?) provides those pre-built binaries? That’s nice. I suspect it might insist on also installing its own versions of libc and other shared C libraries, which is not 100% what I want, but it’s probably good enough for the CI use cases. I should probably learn more about Nix sooner or later, thanks for pointing me that way.

2 Likes
#10

For the record, this was the standard weekly tech track meeting. The way things typically work there is an issue is raised, and if there’s some proposal to address it we put it out publicly. It’s easier to discuss things asynchronously on Slack or Discourse once addressed rather than try to wrangle everyone into a meeting.

What you’re describing pretty much sounds like a re-invention of ghcup

I wasn’t aware of ghcup’s API, thanks for sharing. But keep in mind that the Stack codebase for this also already exists, has been in use for years, and already has full Windows support.

(From the linked update thread) During the process of investigating this course, I got stuck in the abstract filepath proposal 6, which I consider a must for porting ghcup to windows.

As one of the cosigners on that proposal, and well aware that it’s been stalled for years, and also aware that you can write Windows installation tools without it, this gives me concern.

3 Likes
#11

Nix cachix provides that experience. It’s quite nice.

At least for CI, properly configured caching will get you a whole lot of the way there.

(sorry to add to the OT)

1 Like
#12

Nix (or Cachix?) provides those pre-built binaries?

Yes exactly. I think it was just Nix, not Cachix, but I am not familiar with these things.

I should probably learn more about Nix sooner or later

That may be the fate of all of us.

2 Likes
#13

My end-game plan is

  1. Nix on Windows
  2. GHC with and without MinGW.

My admittedly biased take is that (1) has no substitute, and the needs of GHC this isn’t a Haskell-specific issue. As to (2), this will give us better windows support, because LLVM’s windows support is much more polished than GCC’s, and also match Rust which supports both methods. https://awson.github.io/ghc-nw/ is a currently closed source prototype of (2) which would be a good starting point.

In the short term, I reject the implicit premise that installing GHC need be as complex as it is. @hsyl20 and I have been working on changes so that we can have a single, portable “sdist”. That means cabal install ghc to build ghc the library should just work. No caviats, no Setup.hs dark whole, etc. Building the raw executable is a trivial next step.

Now, an actual working toolchain has libraries, and wrapper scripts that pass e.g. -B flags. But even the libraries one should be able to build with plan cabal/stack — we should be able to give rts a configure script just like base.

What remains with the -B flags, juggling different toolchains, etc. is only marginally more work than what cabal-install and stack do already. I would rather just add slightly more features to them and develop and installer from scratch.

3 Likes
#14

You absolutely do not need -B on most sane platforms, if bin and lib are next to each other. Windows hasn’t used the wrapper scripts for a very long time. And the ones we ship with Linux and darwin should just go away and only be installed if someone decides to install bin and lib into non-sibling locations.

1 Like
#15

That’s true. I guess more broadly, with or without -B, there would be a little bit of work putting things in the right places, getting the settings file in place, etc. But this should be kept to a minimum, which is much less work and more surgical work whatever I think these installers do today.

#16

I wasn’t aware of ghcup’s API, thanks for sharing. But keep in mind that the Stack codebase for this also already exists, has been in use for years, and already has full Windows support.

Sure, but stack isn’t just an installer, it’s bigger in scope and doesn’t expose an API that would let you build on it.

I’m not exactly sure what you’re proposing so far. And I don’t see how this will contribute to less fragmentation of installation methods.

I’d be interested to know what you think is missing from ghcup to be that “unified installer”. IMO, it is already that and supports alpine linux, freebsd, mac and even ARM and AARCH64. The reason that it doesn’t have windows support is pretty much that I don’t know how to do msys2 correctly and haven’t had anyone explain it to me, since I’m a windows novice.

As one of the cosigners on that proposal, and well aware that it’s been stalled for years, and also aware that you can write Windows installation tools without it, this gives me concern.

It isn’t a big showstopper. I’m aware that FilePath somewhat works.

3 Likes
#17

Just for the record: I’m not the genesis of the request for a Windows installer based on the Stack codebase. I’d really appreciate if those people who have the need for it would speak up. I have no such need. I regularly use Stack on Windows quite happily.

Sure, but stack isn’t just an installer, it’s bigger in scope and doesn’t expose an API that would let you build on it.

That’s simply not true, the blog post mentions this directly. You can use Stack.Setup right now, and as a proof of concept I already did.

I’m not exactly sure what you’re proposing so far. And I don’t see how this will contribute to less fragmentation of installation methods.

I think my blog post was clear on my proposal. I’d prefer there was one installer codebase that handles all operating systems. But I’m fine sticking with Stack as-is for now, since I don’t have a gap. Others, however, have asked me to use the fact that Stack has already solved the Windows problem to help non-Stack users.

I’d be interested to know what you think is missing from ghcup to be that “unified installer”.

For one, a Windows installer.

The reason that it doesn’t have windows support is pretty much that I don’t know how to do msys2 correctly and haven’t had anyone explain it to me, since I’m a windows novice.

Right, and I worked at this problem about 7 years ago and figured out a method for installing the tools together that works. I also was a Windows novice.

I’d be interested to know what you think is missing from ghcup to be that “unified installer”.

I guess I could turn this around to you. What was missing in Stack’s installation features that you felt it necessary to create a new codebase for installing GHC?

7 Likes
#18

Sure, but your blog post doesn’t mention what you envision that doesn’t already exist. So I’m not sure what new concepts or ideas it brings to the table.

If that’s the only thing then I think there are no technical controversies here and I’m happy to work on it.

Have you written down somewhere what the correct way of dealing with msys2 is?

For one, cabal-install support.

I wouldn’t mind if there existed a library that both ghcup and stack could use, but I don’t think either of the codebases are currently doing really the same thing. This would need a proper discussion on how such a thing would look like, what the exact goals are etc. The blog post doesn’t clear up those questions for me. That’s why I’m not sure how we are going to solve the installer fragmentation.

5 Likes
#19

Is the punchline here that a little GUI wrapper around windows suport for ghcup might be a great idea for widening accessibility of windows users ?

1 Like
#20

I wouldn’t mind if there existed a library that both ghcup and stack could use, but I don’t think either of the codebases are currently doing really the same thing. This would need a proper discussion on how such a thing would look like, what the exact goals are etc. The blog post doesn’t clear up those questions for me. That’s why I’m not sure how we are going to solve the installer fragmentation.

What I’m getting it with my response is that installing and even building GHC should be so easy that there isn’t much overlap in a stack or cabal-install installer anyways.

Secondarily, tying everything together, I think attempts to shove complexity under the rug for sake of new users will not succeed because that complexity will leak back out, and the best plan, even if it seems a bit more ambitious / slower, is just remove the extra complexity where it exists.

4 Likes
#21

A few random thoughts on this:

  • ghcup’s “installer” doc/website - https://www.haskell.org/ghcup/ - inspired by rustup, is quite nice. Regardless of which codebase serves as the foundation, it would be nice to keep this part.
  • While it’s not an issue for most users, it’s worth noting NixOS 20.09 doesn’t include ghcup. Maybe some more skilled nixos people can easily integrate that, but anyone on stock 20.09 has to put in work. Stack “just works” there (I wanted to see what ghcup was like or provided as a workflow).
  • Perhaps we can enumerate the feature that ghcup provide, and which are missing in stack? Maybe we could merge those into the unified installer.
  • Regardless of what codebase is used, stack should be included in the installed toolchain. I could not tell if ghcup did this (from the docs I read, it seemed like no).
3 Likes
#22

I’m very much in support of this. It sounds like both ghcup and stack are mostly there, modulo:

  • For ghcup: Windows support, installation of stack
  • For stack: Direct use of GHC, installation of cabal-install

I’m a little less sure why one would want to wrap ghc in this way. I actually am quite happy with how ghcup does this, where it does its job and then gets out of the way so that I can run the tools it installed. I can run ghcup set ghc 8.10.4 to set the default ghc in the path to a specific version, or I can just run ghc-8.10.4 if I don’t want to change the system state. My main concern with wrapping is the conceptual simplicity, but I also know that others will be unhappy with a state of affairs where running ghc might actually invoke something that downloads and runs software from the internet instead of just a locally installed GHC. I can definitely see that a management tool like stack or cabal might want to install a new GHC version in addition to libraries, but I think it’s the wrong choice for GHC.

3 Likes