[Call for Ideas] Forming a Technical Agenda

Haskell Foundation
#19

To me the big question is whether to plow right into that, or first focus on CI to free up more capacity for a new research project. I would normally say yes, CI first — capital projects to cut operating costs before capital projects for other purposes — but the solution for CI I trust most — doubling down on Nix everywhere — is highly politically fraught, perhaps more so than greenfield research about orphan instances with radical implications for how libraries are organized!

#20

Compiler performance. I’m not sure if there is need here for tooling, or it’s just a matter of lots of work. Possibly this could also be tools to profile our own builds to speed those up (eg identify bottlenecks in parallelism).

Anything HLS needs from GHC’s side seems pretty high leverage. Especially anything that helps HLS on large codebases

Better documentation for libraries would be great. Not sure if there there is a facilitation role here or it’s pure work.

Mac support, whenever issues come up. Not sure what the facilitation approach is here—maybe giving GHC developers macs or more Mac CI.

2 Likes
#21

Both of them actually :confused:

#22

I often find myself in the situation of having to recommend books and introductory material on Haskell to students. On the Haskell website https://www.haskell.org/documentation/ there is a list of beginner books. A quick check shows that among these books only three, “Real World Haskell”, “Learn you a Haskell” and “Developing Web Applications with Haskell and Yesod” are freely available. (In a HTML version, and not in the typesetted pdf form).

While we have a well-stocked university library which our students can use, I still prefer to recommend material that is freely available on the internet. Maybe the Haskell foundation could try to contact the authors and publishers to see whether it is possible to make some of these books available as PDFs.
Some of these books have been available for several years, and these books are usually not particularily profitable for the publishers or authors anymore.

Several of the books have been published by Cambridge University Press, a non-profit publisher which is open to allowing freely available PDF versions of their books. (Cf. the preface to Tom Leinster’s book on Basic Category Theory: https://arxiv.org/pdf/1612.09375.pdf).
There is also precedence for companies sponsoring the open-access availability of books published by for-profit publishers, and several other programming languages have books that are simultaneously available in print and for free online.

Many thanks again to everyone responsible for setting up the Haskell foundation and making the joy of programming in Haskell available to more people :slight_smile:

11 Likes
[RFC] Documentation overhaul on haskell.org
#23

I also agree that there’s a lot of work that can be done on documentation, learning resources and the haskell.org website. But I’m under the impression that this thread is on more technical issues? Maybe it’s better to have a different thread on this when the foundation is ready to tackle it.

2 Likes
#24

Hey all,

I wanted to capture some initial thoughts around pain points we have been observing in the past several months (years, perhaps) through our industrial use of Haskell (fairly extensively) at Well and Soostone. I’ll post some high level descriptions here, but can certainly elaborate further if helpful. As a disclaimer, I feel strongly only on the first couple of points - I’m capturing the rest here for discussion’s sake, but readily admit I couldn’t claim they’re the most important priorities at the level of the ecosystem.

Finally, please note some of these comments come from an environment where we champion the use of Haskell in competition with more mainstream alternatives like JavaScript/NodeJS/TypeScript/etc and where practical results are what drive/justify the use of Haskell.

Faster compilation

To set a bold target, we need to reduce our compile times to something like 10% of what it is
today. The amount of productivity lost to this one particular pain point is astronomical, to put it mildly. That said, any and all improvements would be most welcome, even if just a small percentage.

Perhaps a better way to say this is to make GHC compilation speed one of our primary, top-level objectives for Haskell adoption.

To illustrate the point further by offering a trade-off, I would much rather delay progress on developments in advanced typing features for the sake of compiler speedups or at least to ensure that they don’t cause further speed regressions. This is given, of course, that Haskell already provides for an amazingly expressive programming model unmatched elsewhere.

If helpful, here’s another mental model: When GHC compile times slow down by 10% for the benefit of a new advanced feature, industrial users immediately feel a multi-million dollar hit to their productivity/happiness/QoL in return for a modest offsetting gain but only in use cases where they benefit from the new advanced feature.

Lower memory usage

This is less important than compilation speed, but still worth mentioning. We’re currently unable to work on our 500+ module project on a 16GB RAM laptop. The immediately solution is of course to get larger machines, which is what we’re having to do, but it isn’t doing us any favors in arguments against, say, JavaScript.

Add to this HLS, ghcid, repl, etc. and you really need a heavyweight
machine to work effectively on a growing commercial codebase.

Mac support needs to get better

We’re still running into mach-o linker errors on Mac on large projects and having to
jump through hoops to avoid it. In some corporate setups (especially under regulation), developers are forced into a single platform (e.g. Mac) and ensuring proper “industrial” compatibility is pretty important.

An answer for security scanning / static analysis needs

Big corps are increasingly mandating that their vendors must use static
analysis tools for cybersecurity and other compliance reasons. We should think about improving our answer here.

A very clean, extensible Nix project setup/skeleton

I know this is a contentious point, as we have a multitude of preferences in our ecosystem, but I’m of the opinion that reducing time-to-prototype for a full-stack application setup would be a big win for increasing Haskell industrial adoption.

  • Reduce energy needed to spin up a new project to 0
  • Have clear path to transforming into a serious project
  • Must be easily extensible to add non-Haskell dependencies; python,
    databases, tensorflow, whatever.

A clear, easy path to modern DevOps style deployments

Manual management of VMs with SSH access is a non-starter in serious corp setups. They immediately fail compliance audits and a variety of needed certifications. Having an easy answer for productionalization via something like Terraform, containers, Gitlab runner style CI/CD deployments, etc, would go a long way so folks don’t have to reinvent the wheel themselves.

13 Likes
#25

tl;dr I think the Haskell Foundation should focus on low-hanging fruit (namely documentation and guidelines) for most of the problems outlined in @ozataman’s original post, rather than focusing its effort primarily on technical improvements to GHC.

Additionally I believe that the Haskell Foundation should strive to be as unopinionated as possible in its focus and recommendations, and that separate “Working Groups” should be established to address concerns for specific areas (e.g. DevOps, Nix, etc.).

I want to preface my response by saying that I think all of the Haskell-specific points in this post are worth thinking on and improving in GHC; I’m taking the time to write this up for two reasons:

  1. So that other potential industrial Haskell users can potentially chart a path for themselves that avoids these problems without having to wait for GHC to implement technical improvements
  2. To encourage the folks working on a technical agenda to focus on what I believe to be the “lowest hanging fruit” for us as a community: establishing a set of best practices, design patterns, and general frameworks for architecting large Haskell applications that avoid these sorts of pitfalls

Faster Compilation

tl;dr We should always strive to improve GHC’s performance, but for most users I posit that the slowdowns people have experienced can be immediately mitigated by a proper set of best practices

I’m going to make the, perhaps bold, assertion that GHC’s performance as a compiler is actually mostly fine, and that the vast majority of compile-time regression which users encounter is a result of writing code and pulling in dependencies which rely on the increasingly advanced language features that GHC incorporates.

I’m not suggesting that compilation time isn’t a good metric to set for improvement (regression-testing compiler performance between releases is a good thing), but that for most codebases that I have experience with the issue is more often that Haskell developers tend to reach for extremely baroque language features to accomplish common tasks.

In other words: Haskell programmers should learn from C++ developers and establish some guidelines for:

  • practical “novelty budgets” within common classes of Haskell codebases
  • the performance implications of certain language features which often tend to make up these novelty budgets
  • playbooks for recovering from situations where unmitigated technical debt has “blown” a project’s novelty budget
    • I imagine there are plenty of industrial users who have ended up in this situation and might be able to offer some insight

Lower Memory Usage

tl;dr As with compilation speed issues, GHC’s resource consumption is very likely to be “adequate” so long as developers avoid features that are known to trip the compiler up unless they are absolutely necessary.

I feel the same way about this as I do about the section above: there are probably many areas where GHC can be improved on a technical level, but the vast majority of memory bloat in most industrial codebases will almost definitely come from people using advanced/complex language features.

I feel fairly safe in making the claim that GHC (and much of the associated tooling) should be fine dealing with projects upwards of 1000 modules and 100k LoC without much issue as long as the project is well-architected.

Likewise (although you didn’t mention it), the runtime footprint of Haskell applications can be extremely small yet many industrial codebases can often find themselves in a situation where they consume vast amounts of memory or burn countless CPU cycles.

The key point, in my mind, is for us to understand what sorts of constructs/behaviors cause GHC’s resource consumption to balloon (both at compile-time and at runtime), and to avoid them as much as possible.

Mac Support Needs to Get Better

As far as I’m aware, linker errors on macOS these days are wholly confined to Nix build setups.

If there are mitigations on the GHC side of things which can improve this, we should certainly pursue them, however I don’t think it’s worth prioritizing this as I’ve not heard of any from stack or cabal-install users in recent memory.

A Very Clean, Extensible, Nix Project Setup/Skeleton

tl;dr Industrial users of Haskell and Nix should establish a “Nix Working Group”, but the Haskell Foundation (in general) should not concern itself with advocating for/addressing problems with Nix in any official capacity.

I don’t think that the Haskell community should get into the habit of evangelizing Nix, especially in industrial environments.

Nix is extremely complex, arguably moreso than Haskell itself; Haskell users are already much more likely to recommend Nix than users of other languages, and I worry that this gives the impression that Nix is the One True Way to develop and deploy Haskell applications.

A Clear, Easy Path to Modern DevOps-Style Deployments

tl;dr I feel very strongly that the Haskell Foundation should strive to be as unopinionated as possible in advocating for build/deployment/ops solutions so that we can maximize Haskell’s potential for industrial adoption.

Much like with Nix, I think that while it might be useful to establish a “DevOps Working Group” it shouldn’t be something that the Haskell Foundation should directly concern itself with beyond making sure that Haskell tooling conforms with whatever the state of industrial “best practices” is for artifact generation.

Realistically, I think that there are orders of magnitude more resources available for DevOps best practices than for Haskell these days; we shouldn’t expend the effort in an area where any competent company should be expected to educate themselves accordingly.

I think, as I’ve probably expressed in some of my other comments, the Haskell community does itself a disservice by tying itself so strongly to Nix. I get the impression that many community members feel that the “ideal” solution is something involving nix-copy-closure and a bunch of NixOS machines. While this may be nice for an organization that’s bought into Nix/NixOS, it’s extremely non-standard in the realm of industrial application deployment.

7 Likes
#26

Mac support needs to get better

We’re still running into mach-o linker errors on Mac on large projects and having to
jump through hoops to avoid it. In some corporate setups (especially under regulation), developers are forced into a single platform (e.g. Mac) and ensuring proper “industrial” compatibility is pretty important.

Are you using GHC 8.10.3 or later? These issues should be largely solved thanks to @angerman’s RPATH mangling logic. If you are still seeing them then please open a ticket.

1 Like
#27

Not sure if this is ‘techincal’?

Most of the people I recommend Haskell to get bogged down trying to get started with it. I think this is mostly because the content on www.haskell.org .

I would add an opinionated getting started guide. The two links from the home page are ‘Downloads’ and ‘Documentation’. Both are very information overload and don’t provide a clear ‘what to do next’ if you want to dabble with Haskell.

This would mean telling the user to just use stack or cabal. I would just pick one and give the steps needed, then perhaps at the end say ‘oh there is also this other tool you may want to check out’. A beginner doesn’t care about which tool to use, they just want to get the fun.

I would suggest it should include:

  • Installing ghc + stack / cabal
  • Running some haskell in ghci
  • Setting up a project
  • Installing a dependency
  • An example app that does something ie. hits twitter api. The user could tweak it in some simple ways?

The ultimate goal is to get the user to have clear instructions to follow to get a feel for Haskell and to have fun.

11 Likes
#28

Are you using GHC 8.10.3 or later? These issues should be largely solved thanks to @angerman’s RPATH mangling logic. If you are still seeing them then please open a ticket.

Good tip, thank you! This was just brought up yesterday based on your response here. We’ll give it a shot and report back as needed.

#29

There have been a bunch of really great suggestions, so I’ll let my list overlap to add voices and votes to those:

  • Clean up base and other core libraries, especially with laziness, partial functions, correctness, improved types and data structures, less Stringly-typed junk in core.
  • Drastic improvements to compilation speeds and memory use: big projects are difficult to work on, and advanced, FP-ninja libraries make real-world projects of significant complexity unusable (just the build times and build system requirements alone).
  • Drastic improvements to GHC errors and warnings.
  • Drastic improvements to GHC memory handling and GC characteristics, we leak too much and have to think too hard (lots of reasoning about what happens at runtime, not very declarative…)
  • Protecting/sandboxing against evil Template Haskell, seems like a really smart thing to do… and maybe before it winds us up on the front page news.
  • Improve onboarding, work with doc team to scope out best-practices for an authoritative tutorial demonstrating how to do common, real-world tasks (work with cli, files, HTTP services). Help define the recommended way for onboarding, and how to find / follow tracks thru to advanced topics.
  • Probably a long shot, but unifying hackage/stackage would be nice… can we find a way forward without being blocked on the PVP? Also, considering what’s in Stackage and how it works, seeing Rebecca’s call for improved hackage security makes me sad. Changing an already published package for the sake of PVP idealism is not a hackage feature, it undermines security and violates the trust package publishers extend to the package infrastructure admins.
3 Likes
#30

Cloud Haskell is dying, revive Cloud Haskell.

2 Likes
#31

I think the number one thing could be to lower friction in finding related information. Sure there is google, but some topics they take a long time to investigate and then there is no place to write about the findings (for example which library to choose A, B or C). You can put it on a personal blog sure. That would be one more place on scattered resources over the internet. Would be cool to start thinking of a platform (= website) that make it easier to discuss / add information. The way realworld haskell does it with inline comments is really cool. But also just general wikipedia style is useful.

Some concrete ideas how this could materialize:

  • Try to unify user accounts to become one “haskell” account. gitlab, wiki, hackage, discourse … as much as possible. Also helps moderating of content.

  • Open up new (forum) threads in the documentation on hackage.org on a: package, module, function etc … This would show as a not annoying button (again: like realworld haskell does it: “17 comments”).

  • Would be nice if comments can get likes (upvote). And then just show the comment by default when it gets enough upvotes (deemed to be important enough).

  • Also nice if packages, modules and functions can get likes, in a subtle way like reddit marks upvotes next to posts. The we can sort these things by likes so people know what others are using.

  • Allow people to open up guides (the kind of guide that they would put on a personal blog post). Does not need to relate to a single package or so. (either write protected or as wiki page, configurable). This could be as easy as having an official git repository where people can PR new markdown files which then will be generated on html page.

  • Have a way to link things together. Thing is if you just paste an http link in a comment to some other function it doesn’t link back automatically. Would be nice to have something like github “this function has been mentioned in …”. This would likely require some analysis (of code as well) + database (possibly graph structure).

I think this idea is worthy of haskell foundation because it spans over multiple concerns and working groups, which will be too difficult to carry out for a single person or company even. It’s technically a bit boring to implement, but very feasible. I like it because it will compound value over time as people are more discussing and contributing. Maybe this is not obvious to GHC maintainers or library authors … but the barrier to change some documentation is very high. Which is good in principle (to keep up quality), but next to that there is no low friction way available (or it’s in a corner of the internet where it is hard to find).

Second thing it would be nice to have a way to stay backwards compatible but not to be stuck with legacy decisions that are hard to change. For example the situation with prelude. There are alternative preludes sure, but no sight in community concensus or a way to move this forward. What rust did with the 2018 edition of the compiler was very interesting, it allowed users a choice to stay backwards compatible or flick the switch deprecating old functionality. I’m not sure how to solve this though. Having some people inventing yet another alternative prelude does not seem the way to go as this has been tried before many times. I guess the best way to start would be to have a good comparison between existing preludes, and possible even on a fine grained level (like a function), so things could reach concensus one stone at a time. The concensus thing people have been doing among each other in discussion threads on github. But it would need a bigger support with both moderation of the discussion and also a technical solution how to choose prelude out of the ghc box.

Personally i find these issues more interesting than improving a tool or getting % of compiler performance. People can (and did) take it onto themselves to make a tool or fix a performance bug. I also like a faster GHC, but there is something to be said for starting with things that can not be solved without haskell foundation.

2 Likes
#32

edit I have moved this post to a new thread since it opens a can of worms which is somewhat orthogonal to the present discussion.

The matter of the amending or adding to the base library has come up a fair amount in this discussion. While such changes are of course possible, I do wonder whether this is the best solution to the problems that users feel.

…

Towards a prevalent alternative prelude?
#33

{Moved to new thread}

#34

@hgolden, great points; could we move this discussion to the new thread?

#36

There’s no “move” mechanism as far as I know. I think it’s best just to repost it on the new thread and edit your existing post with a link to the next post.

#37

tl;dr I think the Haskell Foundation should focus on low-hanging fruit (namely documentation and guidelines) for most of the problems outlined in @ozataman’s original post, rather than focusing its effort primarily on technical improvements to GHC.

Definitely this!! Sure we can improve GHC in many ways but I don’t believe it will improve Haskell adoption.
My opinion is that the priority to lowering the entry barrier is by adding guidelines, documentation and ease the setup process (with HLS for example).

It takes too many efforts at the moment to both:

  • setup a Haskell environment
  • start playing with the language or create a toy project (We lack guidelines and great documentation)
3 Likes
#38

Personally, I think if you want more use of commercial Haskell then it needs to be easier to get into Haskell.

It took me about 10 years to finally write some Haskell that actually did anything useful. From an outsider perspective – particularly if you’ve not done CompSci – Haskell just seems incredibly intimidating. Which is a shame, as it’s a really lovely language to use.

I think my biggest stumbling blocks over the last four years of really getting into Haskell:

  • Documentation: it’s hard to find documentation. I would like Laravel-quality documentation in one easy to find place. So much existing documentation relies on understanding the types for it to make any sense, which took me a good year to get my head around.

  • (As mentioned elsewhere) The use of String, so you spend ages getting that working – only to discover a year later that Text is actually the way to go.

  • Laziness: haven’t really needed to fully understand it yet

  • All the category theory stuff: no idea – but I love writing Haskell and can get it to do the things I want

  • Build tools: what’s the right one?

Learn You A Haskell is easy reading, but I think quite dated now. The Haskell Book is incredibly detailed and fascinating, but it scared the pants off me – I still don’t follow most of it.

What’s my point? Good question. Something (anything) that makes Haskell more approachable gets my vote.

PS – Happy to do what I can to help with this (just finished a five year stint teaching on a programming bootcamp).

8 Likes
#39

so one thing that I think is long overdue and definitely needs money to make it happen:

merging the js (ghcjs) and jvm (eta) backends into ghc.

They’re not gonna survive and remain viable otherwise, and both (if done seamlessly) create lots of great new growth opportunties for haskell as a tool, as well as being interesting pressure points for revisiting the standard library problem :slight_smile:

3 Likes