[Call for Ideas] Forming a Technical Agenda

Haskell Foundation
#42

Yes Eta is a long shot, but I hope the success of a GHCJS merge (which I think would be enormous) could build some momentum. I left a comment in https://github.com/typelead/eta/issues/984#issuecomment-761232613 saying basically that the Haskell Foundation’s existence should address the stated issues why Eta forked in the first place.

I would not at all be opposed HF funding @rahulmutt to get the the backend resurrected and merged upstream. I know there is a bad taste in many of our mouths over the fork happening in the first place, but IMO getting ETA merged would be awesome and make me feel “all’s well that ends well”.

(Lastly, GHCJS, Asterius, and Eta I believe all work from STG, so I think there could be some good synergy underneath the hood really fleshing out “backends from STG” abstractions, etc.)

1 Like
#43

Something that came up years ago was the idea of abstracting hackage to support multiple ways of working with the whole set of haskell packages. Right now, Cabal + PVP + trustee revisions is the blessed way to interact with Hackage, but it’s not the only one.

Perhaps this is still too hot of a topic. I just remember thinking at the time that we could have our cake and eat it too if we treated the blessed model as a distribution of Haskell, like the Ubuntu of Haskell, in an ecosystem that supported e.g. Trisquel and Kubuntu as well. Hackage revisions make perfect sense in such a model.

#44

Building projects on air-gapped or merely firewalled environments continues to be a huge PITA. Reproducible builds that depend on someone’s server somewhere and network availability - good luck with that.

There were efforts for completely offline builds, alas none reached maturity.

And I would say nothing of GHC bootstrapping…

#45

I’m surprised that you have found GHC bootstrapping to be problematic. Under the make build system GHC can be built with only alex, happy, and ghc. What specific issues did you encounter?

1 Like
#46

Hackage security model :eyes:

#47

@chreekat, can you elaborate? I’m not sure whether you are mentioning Hackage security in reference to @wiz’s comment or suggesting work on Hackage as part of the technical agenda.

#48

re airgapped, Nix in principle can tell what you need to download and then guarantee you offline working, but waiting for https://github.com/commercialhaskell/all-cabal-hashes to sync with the latest Hackage it still a huge pain. I would love to see Hackage provided as big git repo in addition to the big tarball. That will prevent weird file metadata and other things that can end up in tarballs, and also give us nice normative hashes all the way down.

1 Like
#49

Backpack
It seems like backpack has an important role in Haskell’s future. I believe that a team should be formed to push the improvement of backpack’s documentation and its ability to be used from cabal and stack. I think this is something that can proceed in parallel with other activities. The immediate goal is the determination of the priority of working on backpack for achievement of all other library rationalizations.

2 Likes
#50

I believe you know about https://github.com/input-output-hk/hackage.nix, and https://github.com/input-output-hk/stackage.nix? While we do not carry the hackage index, there is logic (and nix hashes) for daily hackage snapshots.

1 Like
#51

I would like to raise the issue of abandoned packages and package “takeover”, as it is something that periodically causes frustration.
I think the current mechanism (toss a request on the mailing list in the hope that the maintainer sees it) is vastly inefficient [1,2].
It should be easy for all to see whether a package needs to be brought up to date, whether the maintainer is unresponsive, etc., without having to read long email chains. I’m proposing to introduce a ticketing system (similar in spirit to Stackage in fact), where interested parties can report issues and track progress. In extreme cases, “takeover” would be automatic and uncontroversial, e.g. by establishing upfront a cutoff period of 6 months.

See the recent handover attempts for toml and cryptonite :
[1] : https://mail.haskell.org/pipermail/haskell-cafe/2021-March/133562.html
[2] : https://mail.haskell.org/pipermail/haskell-cafe/2021-March/133636.html

cryptonite is worth mentioning because it started as a one-person project and became the cornerstone of our networking and security stack ( https://packdeps.haskellers.com/reverse/cryptonite ). It is unthinkable that its issues could go unresolved for months.

6 Likes
#52

Thanks for reminding me they exist as part of the haskell.nix ecosystem, but that is still a periodic scrape, right? I was hoping for something Nix (and other non-haskell-specific tools) would be able to use directly so no scraping is needed, and one can use a library as soon as it’s uploaded.

#53

It’s updated once a day or so automatically.

#54

Yeah, sorry for being cryptic. :slight_smile:

The first time I went to upload a package, I made a mistake and accidentally overwrote somebody else’s package. Somebody noticed, but still!

That was a long time ago, however: 2013. Maybe this is no longer possible?

I’m not trying to pick on Cabal or Hackage, by the way. It’s just my personal interest to see them improved where improvements are possible, reasonable, and benign. I.e. if I had the resources, I’d love to help out on those projects in particular!

#55

Under the make build system GHC can be built with only alex, happy, and ghc.

And building those tools needs GHC :dizzy:

#56

That stopped being possible years ago when hackage 2 landed. The whole new process is an artifact of it now being secure and requiring an auditable process that’s visible.

3 Likes
#57

I believe that one reason why many of the core libraries get so little love is because the necessary performance work is so terribly tedious:

Benchmarking requires a lot of time and a silent machine. To produce a Core diff I have to add the right OPTIONS_GHC in the relevant module at two different commits, rebuild, then wait for meld to render the huge diffs.

Obviously better tooling and automation could reduce this tedium by a lot!

In particular, I’d like to have some GitHub bots that I can ask to run the benchmarks, produce Core diffs, etc. Something like the rust-timer bot that helps with improving the rust compiler: You simply summon it with “@rust-timer queue” and a bit later you get a link to a nice summary of the benchmark results.

Setting up such a bot would obviously require a bit of infrastructure and also some decisions and rules on who gets to use the bot. A good fit for the HF, I hope! :slight_smile:

5 Likes
#58

Yes! This is something that a few of us have been thinking about. I have written a proposal for consideration by the HF here. Let me know what you think.

2 Likes
#59

The mechanism isn’t hoping the maintainer sees it. The mechanism is – you ask the maintainer and try to resolve it otherwise. The message is a last step to ensure that a “public notification” has been posted so the maintainer, who has apparently been otherwise unreachable, is documented to have been made aware of the plan as best possible.

(here’s a page I just googled up with some history of the principle of public notice: https://www.pnrc.net/about-2/about-public-notice/)

The way to update packages absent a takeover is through the nmu process.

Certainly some aspects could be more standardized… I’m not sure how much it is processes that can make this more efficient, and how much it is just coordinating contributor effort from people who are not maintainers and who wish to update things…

I’m not sure what you mean by takeover being automatic. Someone surely would be taking it over as a new maintainer, and they would need to state that intent, no?

1 Like
#60

That’s a great proposal, Ben! It would be great to have a website where I could watch compile time and runtime benchmarks improve over time.

It’s not so obvious from the proposal how the new infrastructure could be used to support ongoing performance work though. Could I use it to get the numbers for an open GitLab MR to GHC?

What I also like about the idea for a GitHub bot (and possibly a GitLab bot) is that I don’t have to look up the benchmark results myself, hoping that they have already been measured. Instead I can just ping the bot, lean back, and wait for the notification that the benchmark results are ready.

#61

Yes, all builds (including MR validations) currently record performance metrics. Consequently, the numbers we need are available even today. The problem is that we have no good way of exposing them to the user. Having a bot that after every pipeline posts a summary of the metric changes would already be a good start. Another option would be to merely post a link to a Gipeda-like interface which compares the MR head against its base commit.

1 Like