[Call for Ideas] Forming a Technical Agenda

Haskell Foundation
#62

A huge barrier to entry for people new to haskell is the bad error messages generated by GHC. For new users this a lot more important than e.g. faster compile times. When users get stuck a couple of times on a error message they don’t understand, they will think haskell is too complicated for them and leave the ecosystem.

I think we can find lot’s of inspiration by looking into the elm ecosystem (sorry, I have to break the links up as there’s a link limit for new users):
https:// elm-lang .org/news/compiler-errors-for-humans
https:// github .com/elm/error-message-catalog
https:// mobile.twitter .com/jyothsnasrin/status/1037703436043603968

In the IHP community we started a repo to keep track on them. You can find it here: https://github.com/digitallyinduced/haskell-ux

Maybe we could have a central haskell-ux repo on GitHub by the foundation. We could encourage haskells to report bad error messages and after a while start working on the low hanging fruits :slight_smile: This could provide huge leverage for the longterm adoption of haskell. :slight_smile:

4 Likes
#63

There is a ton of engineering work that needs to be done before we can improve the UX of error messages, the most important of which is Alfredo Di Napoli’s efforts on the Errors as Values Proposal. I fear that the momentum we may be able to generate around better UX would be wasted if there is no direct way to act on it at the moment.

1 Like
#64

Alfredo’s work is indeed important.

But @mpscholten’s point about the crucial nature of error messages is well made. If anyone is motivated to help with this, I don’t think they need wait for Alfredo to be done – we can parallelise!

A helpful thing would be

  • To build a curated list of the most egregiously bad error messages, with concrete examples and some analysis. It’s far, far more useful to say “here is a small program X with a simple error, but we get this useless or misleading message” than it is to say “GHC’s error messages are poor”.

  • Think about how to improve them. This would probably not be a simple cosmetic exercise. It’s genuinely difficult to give insightful error messages, especially in a type system as complicated as GHC’s. But there is plenty of inspiration in other systems, of which Elm is one.

The first of these tasks is something anyone, or a group of people, could do.

4 Likes
#65

My understanding is that Elm has a dedicated bug-tracker that accepts only bugs around the phrasing of error messages. I immediately thought this would be a good idea to port to GHC, but never got around to executing on this.

The problem is that bandwidth from the current core contributors is very limited. I/we can certainly help setting up the repo in the “official” gitlab.haskell.org/ghc space (maybe gitlab.haskell.org/ghc/error-messages ?), but I would want an individual (or perhaps a small group) to stand up to be in charge of such a project.

The expectation wouldn’t be that these folks would fix the error messages, but instead give quick responses to posters and guide discussions to useful conclusions. Knowing that e.g. “expected”/“actual” should be replaced by “I thought I was going to get”/“The expression actually has type” is easy to execute on, but quite hard to reach consensus on. Of course, maybe we need even more fundamental changes (I’m sure we do), but surfacing exactly what changes we need takes real work – work that the current core team simply does not have the capacity for.

So, for anyone out there who is organized and can commit to offering sustained attention: this is a fantastic, high-profile volunteer opportunity that has a chance to make a real difference to GHC.

2 Likes
#66

Part of the problem is that experience changes how we read the error messages from GHC.

As a newb, the vast majority of the error messages you read (say 80%) are utterly meaningless, confusing in their own right, and demoralizing (if you spend un-mentored time trying to understand the error). The few that make sense are mostly obvious, missing a package, missed an import, etc, typo/misspelled function/value. The error messages are telling us what the compiler thinks is wrong, but we’re supposed to derive the fix. This is a bit hard to intuit and do correctly as a newb. Depending on what it is, we can also get stuck on type alignment issues that only a mentor can help us out of (but which I believe improved error messages could help with).

As an intermediate haskeller, you’ve begun to develop a sixth sense on GHC errors and make changes you don’t fully understand - you can “sense” what GHC is asking you to change and you make that change, sometimes to your surprise.

As an experienced haskeller, you’ve refined that sixth sense so much you don’t realize when it happens much. This also makes it significantly harder for experienced haskellers to help derive that list (not that it can’t be done, but it’s significantly hard) with improvements that actually help newbs.

Like Elm, Stack does this well by making meaningful suggested fixes where it can. Those suggestions are super helpful to newbs and experienced users alike, as if you don’t know what you did wrong, you have a thing to look into, and a reminder if you did understand what you did wrong. It would be nice if GHC’s errors ruthlessly followed this principal all the time, with correct/useful suggestions.

It makes a lot of sense to build such a list, but it might also depend on a concerted effort to gather that feedback from real-world use.

5 Likes
#67

This is a fantastic way for us to collect the feedback we’re missing. I would be willing to help with carrying out the plan, what did you have in mind for seeing that thru?

2 Likes
#68

Not much, really. Just a stated commitment to give the new bug tracker continued love and attention. This means, for example:

  • Handling routine sysadmin issues (e.g. granting permissions if necessary)
  • Advertising the tracker’s existence
  • Making sure that posts get responded to in a reasonable timeframe (we want contributors to know they’re valued)
  • Helping to connect different tickets that are really about the same error message
  • Noticing overarching trends among tickets and trying to come up with design principles behind good messages
  • Moderating conversations to try to find commonality between opposing viewpoints of what makes a good error message

One problem with opening a tracker of this sort is we might end up with, say, 500 reports of people unhappy with an error message… but that information isn’t actionable. And, if no one has responded to these reports, we have 500 people who took their time to inform us of a problem and got no love in return. So, to turn passersby into active, proud contributors and to devise actionable plans for improving messages, we need dedicated volunteer(s) to help avoid that bad fate and strive toward a better one.

In the end, I don’t think this job takes much time – maybe 1-2 hours/week is enough. But it takes continued attention over months/years, a resource that can be hard to find in an online community of volunteers. However, if you say you can provide that resource, I’ll trust you.

To be clear: these views are my own, and we would need buy-in from other core contributors to GHC to have support for creating this tracker in an official place. I’m in strong support, but I don’t speak for others. Nothing stops anyone from creating this tracker unofficially, but that’s not nearly as good as an official one.

5 Likes
#69

How can we get the input from other core contributors? Is there any process to follow or is there a list of people we need to contact?

@ketzacoatl once the issue tracker is up, I’ll add our existing error message suggestions and will spread the word in the IHP community. This should give us some initial traction to get started :slight_smile:

Let’s make this happen :slight_smile:

2 Likes
#70

This is gold! There are better chances of success if we follow this.

#71

Here is another suggestion:

revive the Haskell wikibook.
There aren’t many resources on advanced Haskell programming, and the wikibook touches upon some topics. However, many chapters are incomplete, and activity seems to have stalled.
Perhaps it could even find a new home under the haskell.org umbrella?

4 Likes
#72

@taylorfausak What do you think of the wikibook comment above, in the context of the Haskell KB we have been chatting about lately? I think it would be good to unify efforts on this front, whilst ensuring that the “KB” and “learning” materials (roughly equivalent to tutorial/guide vs ref/explanation in https://diataxis.fr/ ) are kept separate (albeit in one location to encourage contributions).

#73

I think the Wikibook looks like a fantastic resource!

I would say that I’m still testing the waters with the KB. But it is different from the Wikibook in two key ways:

  1. It is not a wiki. https://www.yesodweb.com/blog/2015/08/thoughts-on-documentation
  2. It’s not necessarily linear like a book.

I still need to decide (or discover) if the KB should be more of a tutorial or more of a resource. https://www.writethedocs.org/videos/eu/2017/the-four-kinds-of-documentation-and-why-you-need-to-understand-what-they-are-daniele-procida/

1 Like
#74

The writethedocs link is similar to the Diátaxis Framework that I recently picked up for project docs. Both are by the same author (Daniele Procida), incidentally.

image
image1920×1080 141 KB

For Haskell KB, I think reference + explanation would be a good base. But we can also have tutorials / how-to guides (that link to the ref/exp). Unlike wiki, the Zettelkasten (which Obsidian uses) enables whatever “structure” we want to build in a bottom-up fashion, rather than top-down. Technically it is called a heterarchy (wiki has hierarchy).

3 Likes
#75

A warp-tls alternative based on a widely used and tested tls library would be nice.

1 Like
#76

A lot of people trash on GHC’s errors, but I have never got a good sense of exactly what is being complained about.

The thing that bugs me most about type errors is when my own user-written variables and GHC-created unification variables have similar names and are hard to tell apart, but fixing this isn’t a small matter of the presentation of existing errors but a deeper matter of rethinking the way the type checker works.

I have a feeling that other criticism similarly is either missing the mark, or going to involve shaving a lot more yaks than might be first thought.

3 Likes
#77

But I have never got a good sense of exactly what is being complained about.

Let me enlighten you then: GHC errors are perfect for people who know very well how Haskell/GHC works. This talk by George Wilson explains it very well. The rest of the Haskell developers, and especially newcomers, seems to be left on the side of road.

I have a feeling that other criticism similarly is either missing the mark, or going to involve shaving a lot more yaks than might be first thought.

Yes, there is no “quick win”. But it’s only because it’s been postponed for so long.

5 Likes
#78

It sucks that talking about this comes across as trashing, but maybe we can look over these things together and build up more understanding of the different experiences people have. Thank you also, for your work on GHC, I hope you do not hear these discussions of error messages as personally offensive, we just want to help improve GHC as well! :slight_smile:

GHC’s errors are really great if you understand enough about GHC and what it’s telling you. However, if you do not know what the message is telling you, GHC’s error messages are also a very real source of frustration. As a result, learning haskell on your own is a not fun experience, and learning haskell efficiently requires at least a couple mentors.

As you build your experience with Haskell, you learn to interpret the feedback from GHC, but it is more of a sixth sense than anything concrete. This isn’t simply an issue of “not enough experience”. I’ve even specifically asked various mentors of mine about that (“how did you know you needed X here?”, “oh, I kinda guessed, but it also sorta fits the situation”). There’s some stink here to tend to.

This is especially true as an intermediate novice, where you are more adept at getting yourself into code that is a bit over your head.

For example, I’m still not all that great with the details of monad and applicative-do, and a bunch of the errors I’ve seen recently made little sense, though I was able to intuit that I needed to prefix a few lines with pure $ (I can see why, but I don’t yet understand why). I was not “consciously understanding” what ghc was telling me, I was only able to experiment and then see that using pure helped GHC figure out how to string together what I was doing (and I only thought to use pure because of that sixth sense and a vague "I think my mentors have used this like this before).

If GHC’s error messages were improved in those situations, we would all have more confidence in the changes we needed to make, and there would be less guess work in appeasing the compiler.

That said, it’s an incredibly difficult problem, and that is why we don’t mean to just complain, but to discuss and iterate on solutions to see thru these improvements.

2 Likes
#79

That was a nice talk. I was in fact taught with (excerpts of) How To Design Programs, and I would very much like to see that model and those sorts of errors too. But I think that requires bonafide language levels, whereas our unenabled language extensions are barely disabled. I am very skeptical of trying to fake it with custom type errors as the presenter indicates a possible approach might be.

It would also be good for research to make it more easy to swap out phases like the parser and type checker, so I do think this would be good for more than education. (Implementing the languages Racket style with e.g. macros that produce Core would also be really cool.) But all this is still super involved work.

2 Likes
#80

I want to chime in with agreement that GHC’s errors need to be made more consumable. They are currently phrased in a way that makes great sense to people who know how to read typing rules and have studied type inference – I can often extract just what I need from the error messages. But I should not be the target audience! This is a hard problem (I don’t think anyone is disagreeing here), but one that, I think, we can address over time, with a dedicated group of people (or one strong leader) who will continue to see a solution take form and be implemented.

2 Likes
#81

I am acutely aware that GHC’s error messages are often not good, and that it’s a real problem.

But it’s not simply negligence. I’ve spent ages on error message engineering. But it’s hard:

  • Hard to know what precisely the goal is.
  • How do you measure when an error message is “good”.
  • Significantly affected by the user – one person’s “good” may not be the same as another’s.

Nevertheless, systems like Elm show that if you make it a priority you can do better. I think what would really make a difference is a strong leader: someone who was interested in making a sustained commitment to better error messages, and was willing to engage both with users (on the one side), and the technical details of how type inference happens (on the other) to make things better. It’s easy to make helpful suggestions, much harder to carry them through.

But I want to advertise my own willingness to engage with someone who wants to step up to the challenge. I think that person would get quite a bit of support to execute, if we had a clear plan.

10 Likes