Thanks for the feedback! I have added tasty-bench as a benchmarking tool: Add tasty-bench as a benchmarking tool · haskellweekly/haskellweekly@2541deb · GitHub
Is there a feedback page somewhere?
I sent the survey to some friends who are not regular contributors yet, and they were discouraged by the huge list of language extensions, not knowing most of them. I get why the survey asks about them, but it might make sense to hide them or add a huge disclaimer that newcomers are encouraged to skip this list of language extensions.
Encouraging relative newcomers is important in my opinion, since otherwise we mostly are an echo chamber of advanced users.
It was discouraging to me as well. In previous years I’ve taken the time to fill it in but this year I skipped it.
Every year, I write in Common Lisp, Racket, and Emacs Lisp as other languages that I regularly program in. It seems likely to me that I’m not the only one who does both Haskell and Lisp. Should they (or at least Racket, which I suspect is most popular of the three) have checkboxes?
This thread works fine for feedback, or you can open an issue on GitHub.
I used to have the language extensions in a scrolling list, but some people thought it wasn’t obvious that it was scrollable. I decided to just list them all out instead.
The country question has a similar problem, but respondents are only making one selection there.
Would it suffice to add some instructions to the question explaining that you only have to vote for or against extensions that you feel strongly about? I do see how it’s an imposing question. I don’t want to encourage newcomers to skip it because they may be more annoyed than experts by repeatedly enabling the same extensions.
For the questions that allow an “other” answer, I normally try to promote popular responses into regular answer choices. Similarly, I remove answer choices that weren’t popular enough (like, say, Pijul for version control). Perhaps I miscounted the popularity of “other” languages. I’ll be sure to re-tally them for next year’s survey.
The extensions lists could probably be more accessible by curating it a bit. For example remove everything on by default already maybe (GHC2021, maybe with a open question if you’d want something removed from it). And maybe remove some extensions that are not meant to be used by default (Safe, Trustworthy, CPP come to mind).
It’s also very plausible that I’m less like other Haskellers than I think 
Thanks!
Honestly the list of language extensions is exhausting to maintain. I’m almost certainly missing some extensions that were added since the last survey. And as chessai pointed out to me on Slack, some of the extensions are deprecated. For example Rank2Types is deprecated in favor of RankNTypes, and DatatypeContexts is deprecated with no replacement.
I try to err on the side of including too many extensions. Sure, we probably won’t enable Safe by default, but what if 80% of respondents wanted to? That would be interesting at least.
Perhaps for future surveys I’ll change the question to apply more specifically to GHC20xx. That way it could ask “Which extensions would you like to be included in GHC20xx?” All the current ones from GHC2021 could be pre-selected.
Honestly the list of language extensions is exhausting to maintain.
This might come handy then. I believe — as someone suggested last year — that two questions regarding language extensions (with checkboxes, one for “enabled by default” and the other for ”not enabled by default”) would mitigate survey fatigue. (Along of course with a big “you can safely skip those two questions if you don’t have strong feeling” disclaimer.)
My personal gripes:
- Likert scale for “Feelings” section is a 5-point one (“Strongly agree, Agree, Neutral, Disagree, Strongly disagree”). The dreaded Neutral is not sound for psychological measurements and creates a number of problems regarding statistical analysis, mainly excess kurtosis as it attracts more answers departing from normality. Sources for these claims (and an excellent review of best practices on choosing likert scales): one, two.
A very simple solution is to all the above mentioned problems that does not make the survey any more challenging for the survey taker is to adopt a six graded scale (“Strongly agree, Agree, Slightly agree, Slightly disagree, Disagree, Strongly disagree”). - I am quite miffed with demographics question, or the lack of them. E.g. “age” and “gender” questions are missing, the “Do you consider yourself a member of an underrepresented or marginalized group in technology?” being a poor (inexact, subjective, possibly anglocentric) substitute.
The trove of data would be much more useful to draw meaningful inference if it included such industry-standard questions.
Thanks for doing this survey every year, it is always interesting to see the trends of the Haskell community and I am sure hard data like this is very useful for upstream, maintainers and HF alike!
I have looked for a single authoritative source of language extensions before and been unable to find one. Yes, Cabal has a list of language extensions it supports. But it’s actually a different set than what GHC supports! For example Cabal knows about the XmlSyntax extension but GHC doesn’t support that one.
The ones that GHC supports can be found in the GHC.LanguageExtensions.Type module from the ghc-boot-th package. But that’s not an exhaustive list either! For example note that Safe and friends are missing, even though clearly GHC supports them.
two questions regarding language extensions […] would mitigate survey fatigue
I’m sure that I’m being dense, but I don’t see how two questions is any different than one question where you can answer “yes” or “no” to each extension.
The dreaded Neutral is not sound for psychological measurements
I am not an expert in this field. I initially chose a five-point scale because that’s what I had used in the past and it seemed like a common choice. I could be convinced that a six-point scale is better, but I’m not equipped to evaluate research papers about this.
I am quite miffed with demographics question, or the lack of them.
I feel like these are no-win questions for me. I initially didn’t have demographic questions at all. People complained about that, so I added some. People complained about the questions that were or weren’t being asked, so I changed them. I suspect it’s impossible to make everyone happy here.
Let me ask this: What would you do with this age and gender data? Which choices would it inform? Do you expect the data this year to be different than 2020, 2019, or 2018?
I have looked for a single authoritative source of language extensions before and been unable to find one. Yes, Cabal has a list of language extensions it supports. But it’s actually a different set than what GHC supports! For example Cabal knows about the XmlSyntax extension but GHC doesn’t support that one.
Ha, I see the look-for-every-extension quest is more difficult than I thought. Maybe ghc --supported-languages would do? I see Safe listed here, but I cannot vouch for comprehensiveness.
I’m sure that I’m being dense, but I don’t see how two questions is any different than one question where you can answer “yes” or “no” to each extension.
Regarding the survey question about extension: forget what I wrote (which clearly makes little sense)! Rereading @fendor’s friend feedback: “I sent the survey to some friends who are not regular contributors yet, and they were discouraged by the huge list of language extensions, not knowing most of them.”
It seems to me the pain points are two: a) “the list is long”; and b) “I don’t know what I am reading”. And indeed
- say I know some Haskell and I select YES near
ScopedTypeVariables. - Then I somewhat feel a responsibility to go through all the list and pick NO to obviously hazardous extensions like
IncoherentInstances(because not selecting that make it seems I am ambivalent towards incoherent instances). - So I patiently go through 100+ items, getting frustrated and discouraged when I meet something like
TransformListCompthat I have no idea what it is or does.
A couple of (this time hopefully useful) suggestions on how to solve this:
- drop the NOs, the user just selects what they like. By only allowing YES there is no nudge to scan each item individually, I could just pick the three extension I always put at the top of each
.hsI write and forget about the rest.
Alas this loses some (interesting) data but looking at 2021 results, the first important negative score I see (a −10%) is forApplicativeDo, which has only a +17% on its side. There does not seem to be any extension where the community is split. - add a very big “If you are beginning with Haskell, you can safely skip this!” disclaimer alongside the question. I know each question has an “(optional)” near it, but when everything is marked “(optional)”, probably something bolder is needed to have it stand out.
- If we want to go even further (risky), just have a textbox, ask the user to write them down by hand (comma separated), correct common mistakes with Levenshtein distance.
I feel like [demographic questions] are no-win questions for me. I initially didn’t have demographic questions at all. People complained about that, so I added some. People complained about the questions that were or weren’t being asked, so I changed them. I suspect it’s impossible to make everyone happy here.
Were the complains against the demographic questions public? If not, do you think you can summarise them here? I am curious to see why people are against it; the survey is anonymous and every single answer is optional, so there is no risk of — e.g. — getting your answers ID’d back to you. Are there any other concerns?
Let me ask this: What would you do with this age and gender data? Which choices would it inform?
Off the top of my head, examples of what I would like to do:
- see questions like “Where do you interact with the Haskell community?” broken down by age. This would help to see where the new folks are gathering, to ease onboarding, to make call-for-actions more specific, to check — e.g. useful for the HF — that we are not missing a channel of communication with a chunk of users that will constitue the Haskell community of tomorrow.
- similar strats for “I feel welcome in the Haskell community” for age and gender, to see if there are pain points we can overcome together.
Do you expect the data this year to be different than 2020, 2019, or 2018?
I smile: only God knows, but for sure it is going to be interesting to see and document where we are going as a community.
Did you forget to post the results?
Didn’t forget, just hadn’t done it yet. But the results are now available: