Proposal: An issue tracker for better error messages

#1

Over the years, many have groused about the quality of GHC’s error messages. While quality of error messages is, actually, a major driver of the design and implementation of GHC, many users are still left unsatisfied. @ketzacoatl and I have been scheming about doing something about this, leading to this proposal.

The main idea: create a new repo on GitHub, https://github.com/haskell/errors, which will be used only to track suggestions around improvement of error messages. Details:

  • The content of the repo would initially only be a README, laying out the goals of the repo and referring to the Guidelines for Respectful Communication for all participants. Perhaps someday, the repo would also contain agreed-upon high-level guidelines for how to construct good errors, but that’s a stretch goal.
  • The main event will be the associated issue tracker. This issue tracker will invite the larger Haskell community – explicitly including people on their first day of Haskell – to post about confusing error messages. There will be some amount of help offered to posters in resolving their errors, but the primary goal here will be to decide on concrete improvements to the error text. Once debate subsides and a clear, concrete direction forward (with specific new suggested wording for a message or messages) has been found, then someone will post a bug on the issue tracker for the relevant project.
  • @ketzacoatl and I envision this issue tracker to encompass a wide range of Haskell tools (hence the rather central location), though it seems starting small and growing with experience is good. The scope will thus be loosely restricted to GHC errors and not, say, stack or cabal errors, at first. By “loosely restricted”, I mean that posts about cabal or stack errors will be greeted warmly (some users may not know the difference between a GHC error and an install-tool error), but perhaps the community will not spend as much time debating a concrete improvement – at least until we have buy-in from the respective developers of cabal and stack (and perhaps other tools, too).
  • The goal of this effort is to increase the bandwidth and quality of information flowing from users to developers. Right now, a GHC user who is frustrated with an error message can only post a GHC bug. I claim without evidence that most GHC error messages are factually accurate, and so GHC developers might write on the bug report that the error is correct as GHC dishes it out. While this statement is true, it’s not necessarily helpful to the poster, who was likely befuddled. Even if the responding GHC dev is interested in improving the understandability and actionability of the message, it is hard to source a diversity of opinions and settle on a path forward that we can be confident in. With this new issue tracker, we hope that the lower barrier-to-entry (the GHC tracker is surely intimidating to many!) will be able to get the voices we need – routine and novice Haskell users with a diversity of backgrounds – to share their thoughts and help craft better messages. Only when a concrete path forward is identified does the (often busy and easily distracted) GHC dev step in to implement the change.
  • The main barrier to this effort is simply time and dedication. That is, success will depend on an individual or smallish group of individuals to continue to give this effort love and attention over a period of time, creating a welcoming culture of helpfulness and guiding discussions to useful, actionable, concrete suggestions for tool developers. I am pleased to say that @ketzacoatl has volunteered to be this initial leader, though they will likely need help to pull this off. My role in this is as midwife, ardent supporter, and sympathetic GHC dev. I will likely comment on the repo, etc. (I care deeply about error messages and learnability), but will not be running the show or doing broad organization.
  • Prior art: Elm does this (https://github.com/elm/error-message-catalog), and Evan Czaplicki (the leader behind Elm) thinks the approach here has worked well for Elm.

So, what do we think? Is this a good idea? Should we / can we put this in the haskell GitHub org? (An alternative is to host at, say, gitlab.haskell.org/ux/errors in a new, tantalizingly-named ux group. But perhaps we get more contributors at GitHub.)

And, most importantly: are you interested in helping out in community building, curation, and organization?

(Caveat: though @ketzacoatl and I have indeed been scheming together there, they have not reviewed this proposal text and may differ with me on some finer points.)

21 Likes
#2

Love the idea! And I’d like to see hlint message issues comprehended together as well.

1 Like
#3

Yes, I love the idea!
I also have a few general points that still confuse me: The lines with in the context are not just iseless fluff and unnecessarily fill the error.
Also for unexpected type error (probably the most common one) it is never clear to me which of two expected vs actual is the one that the highlighted piece of code has, so making that clearer would be good.

3 Likes
#4

That’s a great write up, thank you @rae! Very happy to see our next steps coming together too :slight_smile:

I’d agree, and I’d be happy to help with fielding feedback for hlint. It’ll take time and effort to connect with the dev teams for each of the different projects, so as Rae noted, it may take a little more time to get a smooth process in place. I believe it’s worth targeting and working towards, and interested in making that happen.

Support from other contributors, who are interested in improving this part of the Haskell experience, is very welcome - we’ll need it to make this worthwhile and successful.

#5

I really love this idea! It will be super helpful for lowering the barrier for entry and giving more information about how well the error messages work in practice for different people and ideas for improvement.

2 Likes
#6

Could this be integrated into the LSP? Have a suggestion on each error message that will create an issue with the concrete example and some user specified context?

1 Like
#7

Yes it certainly would

#8

Excellent idea! Having “field data” on real errors from real people is the key to having a truly helpful system.

#9

A question for everyone: would there be broader value and interest in a tool which specifically triggers as many GHC/tooling errors as we can write tests/code for?

1 Like
#10

I also really love this idea. I already have a list of error messages in mind that I would like to report in that tracker.

2 Likes
#11

I would want such a tool to be integrated with the GHC testsuite somehow (or, as necessary, the testsuites of other tools). The GHC testsuite currently has a few thousand test cases where GHC is expected to reject the program, and the error message generated is checked against an expectation. But this segment of the testsuite is not exhaustive, and it is not organized in a way to be educational – that is, it would be great if error message were paired with a simple-as-possible program that demonstrates the error.

I guess what I’m saying is that, instead of building a separate repository of erroneous programs that would have to be maintained separately, I think it’s probably best to somehow incorporate this in the GHC testsuite. This would keep the programs and their errors up-to-date. The existing GHC testsuite may also provide a nice seed for such a database. A downside to integrating this with GHC is that GHC devs aren’t necessarily concerned with keeping the examples educational, so some extra maintenance from outside might still be helpful.

3 Likes
#12

I did not realize GHC already had something we could build on there, that is great news!

#13

This posting really speaks to me. I’m new to Haskell, and only started learning it because the only code I could find for a certain purpose was in Haskell.

Most of my struggles, besides learning Haskell itself, have been with stack/cabal, though I’ve also had trouble with GHC error messages. I’m impressed by how helpful a lot of the error output /tries/ to be, but . … it’s still not quite as newbie-friendly as I think it could be. And it can lead to despair.

At times, I’ve thought: just keep going until I thoroughly understand the code I’ve adopted, then translate that code to some language with a more congenial environment and a more conventional culture. (Environment matters too – I’ve recently given up on debugging in VScode under Windows 10 – Haskell GHCi Debug Adapter Phoityne – because I just couldn’t get it to work.)

I think the worst thing the Haskell community could do is make newbies feel dumb. It doesn’t even matter that you don’t want to. For me, every day has been a struggle against, “Am I just not smart enough”? I don’t think that’s the problem. In college, I was good enough in computing theory that the CS department hired me to grade homework, after I took courses mostly attended by CS grad students. (They were prelim exam subjects.) I’m a fair hand with C++, Erlang and Prolog. Those languages are not for dummies. But Haskell, wow. So powerful. So . . . hard to get started. :frowning:

More newbie-friendly error messaging would help a lot. Ideally, the error messages point to web pages that don’t just say, “Here are some things to try.” The error message documentation instead says, Here are things to /check/, and if your error case matches one of those things, here’s how to fix that one thing.

4 Likes
#14

I completely agree with your insight here. It is a challenge to design messaging this way, but it’s I believe the payoff is worth the investment.

#15

Recently I wondered why playing with Haskell was starting to seem less stressful. Was I actually getting much better? No, that wasn’t it.

It finally hit me: I was ignoring everything about error messages except line numbers. I wasn’t trying to figure out the error messages anymore. I’d just go back and look at the code and think of what I knew about Haskell code being well-formed. If it didn’t come to me, I’d look at surrounding code. Ah, that’s how you say it. (facepalm) If that didn’t work, I’d try a few things. If those didn’t work, I’d go back to the code I’d copied from somewhere and changed, to see if there were some clues.

Literally, it would have been easier so far, for me, if, instead of detailed error messages, all error messages read: “Something’s wrong around line ###. We’d go into more detail, except that our heads are so deep into Haskell that we don’t know how to write informative error messages for newbies. Sorry. Maybe you can help us fix this?”

Perhaps for stack build and stack ghci, there could be a flag “–noob” or something?

Two possibilities:

(1) I’m a moron
(2) I’m not. The “avoid success at all costs” strategy has just come back to bite Haskell.

I’m leaning toward #2. After all, I’ve been able to learn a number of programming languages. But it’s been a while. Hm. Maybe it’s early-onset Alzheimers? Well, it must be setting in awfully fast. My last publication (for a space conference in 2018) was “A rotating, tapered, balanced sling launcher on the Moon made of lunar regolith basalt fiber” which you might find interesting if you’re a space geek like me. I went through the differential equation derivation used in two of the papers I cite, and checked the calculation results against other papers. Hey, I can still do math. Huh. Waddya know.

Please, think about this.

“Error of some kind at line ###. Sorry we don’t know how to write error messages that are more helpful. Compile with the --helphaskellexperts flag if you want to see the error messages we DO know how to write.”

That would actually be an improvement for newbies.

THINK ABOUT THIS.

THINK.
ABOUT.
THIS.

#16

Avoid success at all costs was never about error messages, it is about not compromising on language features.

Can you give examples of error messages that you think are bad? I think error messages like: “Variable not in scope: x” are pretty helpful and understandable for beginners, so I am strongly against removing all error messages.

Repeating “THINK ABOUT THIS” is not helping your point.

1 Like
#17

:rofl: I too have been through this process

[EDIT: well, not actually through … I still only actually read the error message if I have a hard time figuring it out from the line number alone.]

#18

“so I am strongly against removing all error messages.”

If you thought I was seriously suggesting that, when in fact I was pointing out that it would actually help newbies, ironically enough, then you’re not really thinking about this. I thought I was clear enough, but to restate it in different words:

I’m saying “do the thought experiment – maybe it will jar you out of your complacency that the error messages really aren’t all that bad, because of amnesia about what you yourself went through as a newbie.”

“Repeating “THINK ABOUT THIS” is not helping your point.”

Really? When you didn’t even really think about my point? That’s pretty clearly demonstrated by your conclusion that the error messages I proposed for a thought experiment were some kind of serious proposal.

Maybe I didn’t repeat “THINK ABOUT THIS” enough.

#19

I doubt anyone thinks that Haskell error messages are flawless. The fact that an issue tracker is being proposed shows that there is action being taken to identify problematic error messages. Internally in GHC there is also restructuring of error massages underway, so the GHC developers also acknowledge that there is a problem. Where is the complacency? Other aspects of Haskell might get more attention from contributors, but I think we cannot expect volunteers to focus all of their efforts on error messages.

I took your message seriously because I honestly think that having a separate --noob mode is a reasonable idea. Some examples I was immediately thinking of was that a lot of error messages use technical vocabulary like ‘scope’ even in that simple example, but worse are ‘wobbly’ and ‘rigid’ types. It would be nice if we could have a separate mode that uses less jargon like that or at least link to some resource where the terms are explained.

#20

““Variable not in scope: x” are pretty helpful and understandable for beginners”

The first time I got that message I was in ghci. I was deeply puzzled. It took a couple minutes for it come to me: Oh, ghci’s REPL is saying that the variable is not in ITS scope.

So I’d rewrite it something like this:

“Variable not in scope on the ghci command line: x. If it exists in some module, declare it in the appropriate module declaration.”

Better yet, though of course it means more diagnostic code before emitting the error message:

Offer a list of modules in the modules being developed (not the code being loaded from other sources) where that variable may exist. If it doesn’t exist under that variable name in those modules that would be a good thing to say.

You may respond, “Well, it was just a few minutes out of your life. So what?” But when people are learning a language that already challenges their existing programmer intuitions on a number of levels, challenging it on the level of mere error messages is making it worse than if they weren’t learning such an unusual language. In some sense, the “user experience” of Haskell for the newbie is inherently hard, it’s in the language itself, and there’s not much to be done about the language aspect. But that means error messages are an important focus, because that where the UX leverage is to be had. Even if it would be overkill of obviousness for, say, Typescript, it wouldn’t be for Haskell.

As for “avoid success at all costs” being unrelated to error messages, I think it is related. If you drift off into your own nomenclature, and everyone involved in the discussion is in a community where everyone else knows what terms mean, you’ll write error messages using that nomenclature and it will stump the newbies. It may not be an intended consequence of “avoid success at all costs”. But the world is full of unintended negative consequences caused by working with good intentions.

1 Like