Haskell Foundation DevOps Monthly Log and Refresher, August 2026

Haskell Foundation
1

Yes, it’s another monthly log.

But it’s also time for a refresher on the DevOps role.

DevOps role refresher

The HF DevOps role was born in 2022 as a full-time position. It became an 80% contract after the first year or so. For the last two years, it has been a 20% contract (one day per week).

In the beginning was GHC

The initial focus in 2022 was on the GHC contribution experience. This meant improving the patch-submission workflow on gitlab.haskell.org. I made limited, modest gains in the face of fractal technical debt; these gains have since been obliterated by the rise of aggressive bot-net scrapers. They hammer the service and create frequent outages. @MangoIV, the current lead on gitlab management, recently wrote a post with some background. See Creepy crawlies — Konstantin Ryabitsev for a write-up of the same problem affecting the Linux kernel project.[1]

And then there were servers

Besides GHC, the DevOps role also has a hand to play in Haskell infrastructure maintenance. There were infra-related side projects right from the start, but in the last three years it has been the full-time focus.

Currently, the role is individually responsible for:

You could also add wiki.haskell.org to that list. Although technically it is a shared responsibility[3], I have been the sole active administrator for some time now.

I also occasionally make small contributions to haskell.org DNS zone maintenance.

And now, August 2026

So with this role context, what have I actually been up to in the last month?

I had about 40 hours of billable work, which is within the limit (20% is not a ceiling) but quite a bit more than my average over the last year. As you may recall, last month I mentioned I have more time for HF now.

In that time:

  • I continued stabilizing the Haskell Wiki.[4]

    • I updated the server configuration to give the system some swap space, which allows more graceful recovery from memory starvation. This is now in effect.
    • I began refactoring the server configuration and removed redundant containerization. This is paying back technical debt and making it easier to work on.

  • I managed the failure of both Hackage mirrors.

    • I “managed” in a project management sense: I figured out the problems and root causes, figured out who the key stakeholders are, and asked dumb questions until the problem was mostly resolved.[5]
    • I also reimplemented some automation to use the “modern” (ca. 2016) Hackage security model and remove reliance on a single, dead mirror.

  • I stood by helplessly as gitlab had a total server failure.

    …Well, okay, I also managed one of the several communication channels we used to keep people updated, and asked dumb questions until I felt like I understood the aftermath. Also, I took notes and scheduled a followup meeting. That last one is surprisingly difficult given timezones and volunteer time constraints…

  • I did basic admin stuff: I wrote the last monthly log, attended one GHC devs’ meeting, and wrote a few emails chasing down some problems before they become problems.

Coming up in September

Every month, new catastrophes strike, and my queue of half-finished work grows longer. The GHC contributor experience is still there on the queue! Four years on!

So, I have to prioritize. My prioritization algorithm is vibes-based, but I take into account these factors:

Priority considerations

  1. Is there an actual emergency? Servers on fire?
  2. Is a system in such bad shape that a major problem is likely?
  3. Does the work impact infrastructure volunteers, present or future?
  4. What is the bus factor in this area?
  5. How hard is it to onboard new people or generally just make changes in this area?

Currently the queue looks like this:

My work queue

  1. Finish stabilizing the Haskell Wiki
  2. Ensure the necessary followups from the gitlab system failure are performed. Uh, by which I mean, swap out the emergency hardware and put the normal hardware back in place. (Not me personally—it’s on a different continent. More project management.)
  3. Implement changes required to upgrade the Stackage server (and do the upgrade)
  4. Finish propagating the refactoring started by a newcomer in GHC perf import module by dyniec · Pull Request #47 · haskellfoundation/hf-infrastructure · GitHub
  5. Clean the bit-rot out of my small piece of the GHC contributor workflow and integrate the GSoC work from 2024 (!!!)
  6. Finish building a monitoring dashboard for all the Haskell servers to which I have access
  7. Continue writing automatic dependency updates for $everything

As it was, so it shall be

Obviously, I won’t get all of these things done. In fact, chances are > 0% that I won’t get any of them done. The next batch of emergencies is undoubtedly already crashing down upon us. :upside_down_face:

But! BUT!! I honestly think things are in much better shape than they would have been if the Haskell Foundation had not created this role. Across the board, open source contributions are going down, and infrastructure has always been a sore point for community projects. And today, we have the universal problem of the rise of angry bot net scrapers. I’m confident (though I haven’t checked) it has literally killed some community projects. [Edit: found one :wilted_flower:]

So, I am still extremely honored to be in this role. Many people assume it is a constant suffer-fest, but because of the flexibility and impact, the pros heavily outweigh the cons.

So… see ya next time!!!

  1. I highly suspect GitHub itself is suffering the same circumstances. I wouldn’t be surprised if this explained part of its unreliability of late. ↩︎

  2. Last month I wrote Hackage provides its own hashes these days · Issue #24 · commercialhaskell/all-cabal-hashes · GitHub, describing my thoughts on the future of this repo. ↩︎

  3. Technically technically, everything is a “shared responsibility”. But I haven’t pushed for it because I know everybody else is a volunteer working extra-super-part-time. ↩︎

  4. The wiki is also suffering from the “creepy crawlies” problem. Bots are hammering its expensive routes, causing high memory and CPU load. ↩︎

  5. · Ancient versions of Cabal are still in the wild and unable to use the mirrors. Even this minor situation demonstrates bugs in Cabal, Hackage, and 3rd-party CI workflows. I don’t even have time to write everything up, much less fix it.
    · This is also one corner of Haskell infra where I don’t have the necessary system access—even if I had the mandate, which is unclear.
    · When does it become appropriate for footnotes to have footnotes? ↩︎

24 Likes