Pre-HFTP: DevOps Proposal

Haskell Foundation
#1

Introduction

This proposal is for hiring a full-time DevOps engineer for GHC and related Haskell projects and tooling. This is necessary to relieve the drain on time of skilled core developers on DevOps work. From acceptance to hiring, this proposal should hopefully take two to four months. One key risk to be aware of is that transition of DevOps could cause temporary disruption to stability.

Background

Currently there is a significant amount of time being spent collectively by GHC Operations Contributors keeping systems working. Estimates given in the recent past suggest between Ben Gamari and Moritz Angermann they were spending 20 hours per week on systems.

Gitlab takes a moderate amount of attention and issues with it affect a large pool of contributors. Gitlab has had (unplanned) outages on 19 March, and 18 October of 2021. Both were caused by being in a reactive instead of proactive mode since this is no one’s job.

In addition the CI systems are highly varied and numerous. Among them are around 30 runners, across at least 4 kernels - with more user spaces on top. Further we’re missing some CI targets due to a lack of available time and energy from the GHC Operations Contributors to set up and maintain them. GHC CI can not use most hosted CI reasonably because it is already tested on a wider variety of platforms than most provide, and some of its CI jobs far exceed resource limits due to its size.

GHC Operations Contributors include but are not limited to:

  • Ben Gamari who is the primary maintainer of our GitLab nix configuration, and both the gitlab and associated gitlab-storage physical boxes. Additionally they run marge-bot and are a primary maintainer of the CI scripts, and several CI boxes.
  • Moritz Angermann handles much of Mac CI, and some CI management in general.
  • davean does some physical Mac cluster management, and supports some of the GitLab servers on a primarily emergency basis.

Operational needs extend beyond these core issues. Further basic developer experience tools like the GHC dashboard proposal are stalled for lack of time.

Motivation

Hiring a dedicated person to maintain and improve the infrastructure, CI, and related tooling would alleviate the burden on some of GHC’s core contributors. This would free these core contributors to make more optimal use of their time instead of spending it on devops - a position which we are more prepared to hire for and requires less onboarding. Further being preemptive in avoiding issues would improve the flow for every contributor.

Supporting GHC’s process is directly in line with the community glue and adoption directives of the Haskell Foundation – it connects a wider audience of Haskellers than any other. Additionally it is a challenge for volunteers to reliably sustain this necessary and vital infrastructure. And further, this project should be as far upstream as possible, as GHC development infrastructure is the commons of the Haskell community.

With a full-time steward, the infrastructure could provide benefits beyond just GHC. Already, CI machinery is shared by cabal and ghcup, and a paid maintainer could further scale this out. In turn, this enhances the returns on other related Haskell Foundation-funded projects.

Responsibilities of a maintainer should include:

  • Maintain gitlab & CI.
  • Improve GHC CI execution.
  • Expand CI coverage to the base ecosystem.
  • Support related Haskell project CI.

Further responsibilities could include:

  • Support additional Haskell server infrastructure (help to the admin team).
  • Support OS packag(ing/ers) of Haskell (i.e. distros, such as debian, alpine, etc.).
  • CI for additional distros and architectures.
  • Documenting the infrastructure
  • GHC performance dashboard

Goals

Hire a dedicated devops person to support Haskell Foundation’s interest in GHC and Haskell infrastructure.

People

This section should detail the following people:

  • Performers:
    Andrew Boardman: Personnel management, Leading search
    Ben Gamari: Technical management

  • Reviewers:
    Andrew Boardman
    Ben Gamari
    Moritz Angermann

Most affected people:

  • Ben Gamari
  • Moritz Angermann
  • The Haskell Infrastructure team
  • All GHC developers, as well as cabal and ghcup
  • Hopefully developers on other major Haskell projects

Resources

The HF would provide the people management side via Andew Boardman in the CEO role, while the technical direction side of management would come from the GHC Operations Contributors - primarily Ben Gamari.

In addition the Haskell Foundation would provide a budget currently estimated to be up to US$125k/year.

Finally HF, via the CEO role, would lead the candidate search in coordination with GHC Operations Community and other stakeholders.

Timeline

Job listing writing: 1 week.
Candidate search period: up to 2 months.
Hiring process: up to 2 months, hopefully less.

Lifecycle:

After the initial hiring process, this is expected to be an ongoing position.

Deliverables

Deliver one hired developer operative within approximately 2-4 months.

Outcomes

We expect more time available from core ghc contributors currently bogged down in operations work, as well as more stability to the CI infrastructure, and ultimately ongoing improvements in features and coverage.

Risks

  • Hiring is hard. Especially since we’d prefer a wide mix of skills such as Nix, operations, and ideally Haskell. The bar is raised for hiring for a security sensitive position. Requires a budget sufficient to hire someone who suits the qualifications.

  • In the short term, time spent in terms of mentoring and bringing a new hire up to speed will be required from the GHC Ops team.

  • We may require a more senior and with a more varied skill set than is feasible

    • We need someone who shows up and figures out what needs to be done independently.

  • Hire competency will affect the stability of services.

  • Hiring a new person is a significant financial burden to take on for the HF.

  • Documentation and knowledge must be maintained so this person does not become a critical failure point.

14 Likes
#2

Thanks very much for writing this up Davean. I have a few questions on the details.

One key risk to be aware of is that transition of DevOps could cause temporary disruption to stability.

Stability of what? Perhaps “stability of the services that support development of GHC”? Could you please clarify and explain who would be directly affected by the disruption to stability? GHC developers only? Or the wider community? (I suspect the former but good to have this spelled out precisely either way.)

After the initial hiring process, this is expected to be an ongoing position.

Do you mean this is intended to be a permanent employed position, rather than a contractor position with fixed duration?

Outcomes

We expect more time available from core ghc contributors currently bogged down in operations work, as well as more stability to the CI infrastructure, and ultimately ongoing improvements in features and coverage.

and

Further basic developer experience tools like the GHC dashboard proposal

Those speak to the direct impact on those performing GHC development, but what is the indirect impact on the wider Haskell ecosystem? Specifically, could you elaborate on what current ecosystem problem this proposal solves? Ben and Moritz could spend the additional time on many different things, including advanced type system features, performance of the compiler, performance of the generated code, improving the GHC API, … . There’s basically an infinite list of things to do. Do we have a clear idea of what they (or others freed up by this work) would actually do and what the impact on the community would be?

Hiring is hard. Especially since we’d prefer a wide mix of skills such as Nix, operations, and ideally Haskell. The bar is raised for hiring for a security sensitive position. Requires a budget sufficient to hire someone who suits the qualifications.

Do you have data on the costs of an FTE with the skills required? Do we feel confident that $125k/year covers the fully-loaded costs (i.e. including not just salary but all other benefits and costs)?

1 Like
#3

Thanks so much @davean for writing this up! Details always matter (I will think more about them when this lands on GitHub), but I’m in strong support of this direction of travel. I’d be very excited to see this become reality.

1 Like
#4

I’ve consulted a bit on the pre-pre discussion so let me try to answer with my views:

Well, just stability in the sense that onboarding a new hire and transitioning responsibilities always makes things a bit harder at first – a new person is learning the ropes, and is more likely to do things slightly differently or make a few missteps as they get a handle on a complex system. Nothing special. But certainly, since this affects CI only, the very worst case is that it causes difficulties for the GHC dev process for a day or two as things get worked out.

Precisely.

Whatever the core ghc dev team and related people (including you!) decides the priorities are. You already listed a bunch of very significant priorities, any of which would make me and many others happy to see more attention. If we can agree that core contributor time is in short supply and there’s a lot of useful places to direct effort, then for the purposes of this proposal in particular, dictating which particular place ends up with that effort is somewhat out of scope.

Its an estimate so there’s some confidence, but its just an estimate. This has been based, afaik, on some discussions with various folks around the globe about the sorts of skillsets and salary ranges typical in their area. It gives a range to start the search. If it turns out to be infeasible, then the executive team in consultation with the board will have to consider the range again and weigh things out.

1 Like
#6

Thanks @sclv.

just stability in the sense that onboarding a new hire and transitioning responsibilities always makes things a bit harder at first … causes difficulties for the GHC dev process for a day or two

I see, that’s good to know. In that case I suggest deleting the sentence “One key risk to be aware of is that transition of DevOps could cause temporary disruption to stability.”.

If the transition affects GHC development only, and then only for a couple of days, then it doesn’t even seem worth mentioning. Having “disruption to stability” in the fourth sentence makes it sound very alarming when it’s really not!

Do you mean this is intended to be a permanent employed position

Precisely.

Thanks. If there are reasons to rule out a contractor position then I suggested stating what those reasons are in the proposal. Otherwise I suspect one of the first questions asked by reviewers will be “why not a contractor position?”.

This has been based, afaik, on some discussions with various folks around the globe about the sorts of skillsets and salary ranges typical in their area

That’s great! I think the proposal would be stronger if these details were mentioned.

1 Like
#7

I think that is also worth stating explicitly on the proposal, perhaps something like “there are many useful ends to which the newly freed-up time of the core GHC contributors could be put to, but the choice of those ends is out of scope for this proposal”.

2 Likes
#8

In the short term, I think this is a a very good idea. Ben and Mortiz spend too much time fighting fires, and we don’t want them to burn out, but do want them to have more time more interesting things that will benefit all of us. Also, I think it nice that this first full-time higher could begin largely managed by Ben/Mortiz/GHC people to burden Andrew and Emily less while they are still getting stuff up and running.

All that said, In the longer term, I am concerned that we are tacitly accepting “20 hours per week” as a reasonable amount of for keeping this stuff running. I think it is in fact unreasonable — too much time — and indicative of somewhat rough infrastructure that folks have been too busy fighting fires to properly shore up. And if I am right in that with an extra person handling ops that happens, though, then we would need to find more work for our full-time hire to do.

I think these two points are very reconcilable:

  1. In return for this extra labor covering day-to-day operations, the GHC folks should within a few months come up with a plan for how lower the maintenance burden for CI, gitlab, etc. Once we have that plan, some of the newly freed up time of existing core contributors should be used to execute it — doing the capital expenditure the operations previously left no time for.

  2. We make clear we are hiring for a role that will evolve. Ideally the person is interested in Haskell, and eager to take up new tasks / learn new skills.

  3. As the GHC maintenance burden winds down, the new hire devotes and increasing portion of their time to other HF priorities. Concurrently, other HFTT proposals will make their way through the process, and lay claim to those freed up t hours per week as queued-up labor capacity we have all ready to go.

Ultimately, I think hiring FTEs for specific tasks is inefficient — we instead want a number HF employees who can flexibly rotate through tasks with fine-grained budgets. Otherwise, we will be forced to “round up” to multiples of full time workloads, wasting money on excess hired labor, wasting time managing more people, and skipping on low hanging fruit that ought to serve as easy victories for the HF.

But of course, it’s unreasonable to say, hire a bunch of people at once, hurriedly make a backlog of approved proposals (with budgets), and try to dive right into managing the team and these tasks. We need to start somewhere. So this proposal, which shares the managerial burden with GHC, and is about more ongoing grunt work rather than a high-intensity temporary project, I think is good place to begin. I just don’t want it to set the standard of how the HF does things even after it’s larger and more established,

2 Likes
#9

Agreed.

Conversely, if we don’t think this person will be able to work on other tasks per what I proposed above, then I am also wondering why don’t go with contractor. To me the best reason to get a full time person is to avoid going through the hiring process again for future HFTPs that would lay claim to this person’s time.

Also agreed, an ind particular even if we don’t believe my “freed up core GHC contributor time can be used to shrink ops burden” proposal, we should still make clear there should be freed up core contributor time doing something as deliverable.

Put differently, if we are higher another person, and after a reasonable on-boarding period it seems we just have more busy firefighters, with Ben and Angerman just as busy putting out fires as before as before just now with more help, this proposal has failed.

1 Like
#10

DevOps is a specific job, and in my experience it always requires someone running it, and in my experience most large groups have at least one person full time focused largely or entirely on that task, which is a very specific and particular skillset. Even if you think you have a “stable” system, it quickly becomes destabilized by the needs of new machines, the deprecation of parts of the stack over time, new OSes, new architectures, new versions of OSes, etc. In the past we’ve been bitten by the optimistic viewpoint you express here – that a single concerted push will get everything in a situation such that it requires minimal future attention. I think its fair to say that this optimism was proven sadly wrong, repeatedly. At a certain point one has to learn the lesson.

The reason for full time rather than contractor is thus that simple – this is an area of concern that requires continual attention in work, and which, no matter how good the work done in the short term, does not appear to be going anywhere in the future. It isn’t a time-boxed task. Its ongoing development and maintenance.

Depending on who’s hired, and their interests, then different bits of more haskelly work could develop or not, I imagine. But that’s for the future. I think future hires would also probably have an initial area of focus, and expertise, with the understanding that things might develop over time and their responsibilities might evolve.

That said, I absolutely don’t understand the following:

Of course – but why/how would that even be possible? Only if there were actually more fires to fight (in which case the help would be welcome) or if the new person didn’t actually provide any help in alleviating the burden, which would be a failure of the hiring process.

And just to be clear – the proposal already does make clear that the deliverable is freed up core contributor time doing something – it literally says that is the goal.

3 Likes
#11

What about starting with a part-time engagement, say half a working week on average, and see where it goes? I’m sure it’s legal in many jurisdictions.

1 Like
#12

I was thinking of those, or something more Fred Brooks-style failure where we made things more complex with more cooks in the kitchen and that just made more problems.

I agree none of these problems should happen — I don’t think failure is likely. :slight_smile:

Yes, I see it in “outcomes”. Glad that’s there!

There is a specific skill here, I agree, but I would still want this persons’ responsibilities to expand.

Even if they are a “pure devops person” that doesn’t feel comfortable getting into random tasks needing to know a decent amount of Haskell, they could also grow to handle some haskell.org things, the Haskell Foundation’s own website, etc. etc.

Some amount of work is ongoing, yes. I want it to be less than a full time job. If we got the residual task down really low maybe it could revert back to Ben et all, but yes I don’t really want that outcome anyways.

I suppose that means I do want full time hire too :slight_smile: whether or not the ops burden goes down as I think it should.

Initial focus + that understanding is basically what I had in mind, so it sounds like we are in agreement.

But that’s for the future.

That said, I would caution that our hiring decisions would stick with us through the future. We might not know what other work there would be for the person to do, but we can reasonably expect/hope (unless other things go quite wrong!) that this first employee is going to be with the HF a good while past our runway of plans at job search time.

(To be clear with all this, I don’t want to engage in that sort of rhetoric that hiring is a minefield, an initial “bad hire” early will screw everything up for ever, etc. etc. I am generally optimistic in the vast majority of people’s ability to learn and cope with an evolving role if they find the work meaningful and motivating.

I just want us to be the clear-eyed about the fundamental uncertainty of a new organization with neither a long or stable roadmap planned out, and that we neither can or should seek to thoroughly insulate newly hired employees from that dynamic.)

This is not at all clear to me.

There will of course be some amount of ongoing work, but I think we’ve continued to use tools which we know are crummy and cause frequent outages:

  • Marge bot inexplicably gets stuck all the time. Lots of random uploading and download issues with docker mirrors and other things.

  • Build machines fill up with space because I guess there is no reliable garbage collection of temporary storage.

  • Supporting Hadrian and Make side-by-side has been a huge time sink, and yet Hadrian is far more complex (certainly as measured by lines of code, and it’s supposedly in the better more concise language!) and still doesn’t get us safely incremental builds.

I don’t think this is due to malfeasance, but because Ben et all have been so overworked there simply has been no time to actually make the big capital not operational pushes necessary. Only once we staff up would I expect us to have the extra capacity to make any progress on this stuff.

Conversely, once we do staff up, if we just accept the current situation (modulo GHC core contributors being overworked) as unavoidable I would be disappointed. As some one who contributes to a wide variety of open source projects (more breadth than depth), I can report with I hope some significance that GHC’s CI causes far more grief getting stuff merged than anything else I’ve seen, by a large margin. This needs to change.

I don’t think we need to a plan right out the gate. In fact there are some developments that I would want to finish first, for example a “reinstallable” ghc itself and RTS both of which will help us reexamine to what extent GHC needs bespoke infrastructure (though it’s good getting more projects to share the gitlab.haskell.org setup between them is already hinted at in this proposal). I just want some good introspection and planning once everyone is working normal hours thanks to the new higher, and the adrenaline and cortisol has subsided.

#13

I’m in favour of having a proper HF employee looking after CI and related jobs. CI is mostly incidental to all of use and often sends us down rabbit holes, debugging liking and other issues specific to some platform. Not all do us are experts on all platforms. Lastly we have few official Haskell build systems and lots of donated resources with sometimes other obligations as well. Navigating this will be interesting. With a small set of trusted people (who often know each other in person), this is just workable, but not great.

The person doing this job would have to:

  • ensure the systems are working and the gitlab instances don’t fail. This is mostly solved and requires little attention these days.
  • manage our gitlab instance (this is more involved)
  • deal with marge (or find a better replacement)
  • look at and assist/manage our CI pipelines.
  • build/unify dashboards across our CI infra. (Average time per job per machine)
  • help split the building phase from the test phase
  • assist in fixing hadrian.
  • assist in adding new os/arch’s set up.
  • deal with strange failures in CI. Why did darwin start acting up all of a sudden after an OS upgrade?
    (This list is not exhaustive, but should give a good idea what kind of work keeps people busy that is only incidental to what they are actually working on).

Finally let’s be honest. Very few of us are paid to actually work on this. Most of us do this on a voluntary basis (except for Ben I believe). That also means if things break people have to carve out some time for this (often on fairly short notice).

It then takes time away from important work like release management and focus on other more compiler related issues instead of dealing with operations and automation.

Running the cardano stake pool makes it possible for me not only to provide quite a bit of performant hardware for GHCs CI, but also help justify spending time on dealing with CI related issues (as listed above).

I don’t think a full time position would any time soon be freeing up everyone who has been involved with CI. But just the option to delegate ownership to someone who’s on payroll and not just a volunteer means we can probably over time streamline this a lot.

Once all this is somewhat sorted we can then look into providing some are-we-fast-yet dashboard from performance metrics. (We have something, but it needs a little polishing, …) and again was something someone just built on the side. These things have never been someone’s “real” job. Only ever incidental or scratching an itch of an already spread fairly thin crew.

Finally I’d like to say thank you to @davean, @mpickering, @wz1000, @bgamari and everyone I forgot for working on CI and related tasks, building out this proposal and being usually responsive on short notice on IRC when issues come up. I’m also very grateful to everyone who provides some build resources, especially on odd hardware/configurations.

10 Likes
#14

I like @angerman’s because it contains a number of specific examples of making things more reliable, not just dealing with the unreliability we have today :).

#15

The act of living is constant change. This proposal does not try to chart the trail into the future, merely propose we try a path. It fails to state the evolution and merely propose options because it will depend on how all the parts evolve. From who we hire to what we need.

It is not intended as a permanent position, merely an indefinite one. The period of which is intrinsically tied to the community’s needs, who we might hire, and the Haskell Foundation’s direction. I would expect this job to evolve over a few years at least. Hopefully we get someone who can evolve along with our needs for a decent period of time. If you told me this proposal lasted 20 years though I’d be disappointed, perhaps even upset. Wouldn’t you?

Perhaps this is a place where the proposal missed the audience. We tried to lay out the options and optimization points because this is not a technical solution. The proposal tries to outline what tools we have to build something out of what we find, and propose a plan for constructing something better. Nothing here is contemplated as repeatable or interchangeable though. It is very likely when we look at candidates we will have to pick them via which course they better fit, by our preference of course.

We specifically considered a contractor, but the problem seems bigger and less concrete than would be best served by a contractor. A contractor’s job is to get in an solve a precise problem and be done. We have more of a process and resourcing problem. Further, to some degree we are far from a point where systems do not need management on an ongoing basis. We could bring in a contractor to redo our CI, bring on the systems we’re lacking, and then turn it back over to the same people who are running it now. Even if that continued perfectly it would start another cycle of decay because the world changes. CI must represent the usage of that which is being tested. As such until change ceases, someone will have to keep updating our approach to testing. Even just fixing our CI today would be disappointing today. (As an aside I’m fairly certain some of problems with CI stem from how we test, not just how we run the tests, I’d want someone who could look from both directions on this. Some of my early CI work for GHC had implementations of testing that would have caught several recent issues for example. As importantly though I believe our tests could be more responsive too.)

I would agree with many commenters above though that the proportion of time and focus on GHC’s CI needs specifically should decrease. This was in no way a tacit acceptance of 20 hours a week being taken. This was in fact a specific claim that by focusing on it we could reduce that burden (but no time soon eliminate) and expand the scope. This is why the responsibilities were written as a suggestion list of options leading into the future. Not only would we be trying to select a candidate to best cover them, but the ones the role could best evolve to cover.

The first most logical evolution is expanding what we consider a testing. Its more of the same job and skills. GHC exists as part of an ecosystem. Its the ecosystem we mostly care about. As others have noted and the proposal references though there are a lot of wanted but incomplete projects that for the right person might go together here. Which are appropriate for this person to do would depend entirely on who we get. Its a bidirectional optimization problem and that directly gets into cost.

As for cost, I have surveyed those I know employed in such roles. Most of my data is from the US but not all. Its on the low side of people I surveyed, but around average for the role industry wide. The people in this sort of role I know are generally fairly experienced. Its high for some locales. The amount of per-existing skill and experience will vary with the price point of course. Our main problem is less the skills and more the attitude. We need this person to not be a managerial burden but instead to find and solve problems proactively. My discussion have lead me to believe this is an often lacking skill in less senior candidates.

This price range was selected as a realistic balance of the market and HF’s means. It would be easier to fill the role in a way we’d be excited about if we had more money. I have people I’d directly recommend if we could reasonably afford to add 50k to this. The more people who contribute to HF the easier solving problems like this becomes and the more problems we can solve. Talk to the management at your company today, or contribute yourself.

As for the wider impact I don’t think its gone without notice that the 9.0 and 9.2 releases were, well, problematic. I’d propose that this should make future ones better. For the direct reasons. The indirect reasons though are really out of scope of this proposal. They’re direct benefits of this proposal but how the stake holders in those benefits choose to employee them is not in scope. I do not think it would be appropriate at all to turn around and tell angerman he should take any time freed up and go work on a realtime GC. I might guess than bgamari might (I don’t but I might). This is not an authoritarian system. We don’t even act as oversight for the people who’s problems we’re trying to deal with.

I would hope we’d expand and solve more people’s problems in the community over time. I have a belief that this will reap rewards and I have specific expectations. We should keep proposals to only dictating that which is in scope though I believe. Its unhealthy not to.

TL;DR:
The proposal here is that HF step in to solve a problematic system. The exact method is suggested, not prescribed.

See:

Support additional Haskell server infrastructure (help to the admin team).

I know there’s a lot of overlapping groups taking care of different things so its confusing who does what. I’d hoped that was more clear by not naming ever specific group but perhaps it wasn’t.

4 Likes
#16

Thanks @davean that is all very reassuring and is getting me quite excited about this. Well-running and improved infrastructure is going to unlock a lot of good things!

1 Like
#17

Does this mean we’ll likely open the gitlab instance for a wider audience (e.g. hackage package maintainers) and allow them to re-use the CI? This is interesting for anyone trying to build haskell binaries (cabal, stack, hlint, pandoc, …), because it means you can support all architectures that GHC can support.

At this point, I don’t think it’s possible, because CI is still quite resource limited and GHC needs most of the resources, so PRs etc don’t stall/take forever.

2 Likes
#18

That’s is a personal goal of mine. I think we stand to benefit from sharing the system a lot.

I don’t think we’re that far from it being a reality actually (See “Note on General State of CI” below for). We mostly don’t have the ability to over see it or make use of the fact we’re sharing it to get benefits above and beyond each project being CIed across the entire variety. I give a nod in potential tasks both in “Support related Haskell project CI.” for potentially bring on and improving project’s CIs, along with “Expand CI coverage to the base ecosystem.” for creating integration level CI across important projects. These are written generally though since I’m sure where people wanted to go will be a large discussion between a lot of parties.

If people start piling on though, we need someone manning the ship to head off problems - even the ones easily avoided. I’d very much like to put out the welcome mat.

Note on General State of CI:

From my perspective its not closed to others ATM (Not an official statement). The more interesting varieties on the CI though are constrained so no one is putting signs out or inviting people in. Capacity is only really good ATM in x86/ARM Linuxes. Windows could be easily solved from a capacity standpoint (mostly need a licenses or someone to take a sufficiently backed legal stance we don’t - the licensing situation for us is confusing for reasons far too long to get into here) but its administration is harder so I don’t really want to propose fixing that unless we have more admin resources. In general *nixes and similar would be fairly easy to solve with demand other than the administration side. The only current platform that is somewhat challenging to get sufficient capacity on is Mac due to its specialized (And fairly weak) HW that isn’t really scalable. We have a couple small clusters - mostly M1s. This is the main pain point I see, but I believe it is addressable to a point.

#19

Based on feedback we’ve tried to make the wording more clear and submitted a HFTP.

Thank you for your input and hopeful the proposal is now clearer.

2 Likes
#20

Hi. I’d like to offer to help on a volunteer basis. I’m currently semi-retired, but I still work part time as a Linux system administrator at a small software company. I have plenty of free time and lots of experience with automation and build infrastructure, and I’d be happy to assist in any way. I’ve been a big fan of Haskell for many years and I’m quite comfortable with regular Haskell with an interest in advanced research topics. Is there some way I can get involved?

3 Likes
#21

I can’t speak to unboarding volunteers for operations work—but @pdr if you want an async task to get your feet wet, I can recommend

from @angerman’s list. It is fairly self-contained and we would be very grateful to see have it done!

In any event, might be good to head to #ghc if you weren’t hanging around there already.

1 Like