Haskell Foundation DevOps Weekly.. er, Monthly.. no.. YEARLY log.
It’s me again!
It might seem like I’ve been gone for a while, and that’s kind of true. I got a little burned out, and I got busy with other contracts. But that didn’t stop me from making progress here and there, and I’ve taken a renewed interest in some upcoming projects. So like a phoenix at the disco, I’m back and ready to shake my tail feathers.
First off, what’s changed? Well..
Stackage stability
I dealt with a number of issues downstream of moving to OpenCape, our new rackspace and network provider. The new servers are fine, but not as powerful as the old ones.
Stackage was misbehaving because of the resource constraints. To fix this, I put a CDN in front of Stackage, added some process watchdogs, switched some infrastructure to better tooling, and made improvements to disk management. Things seem stable now, although the disks still seem unusually slow. There are no emergencies, at least.
… or are there? I just checked, and disk space is getting tight. I have to look into that right away. [Edit, 20 minutes later: fixed again, for now.]
Infra Monitoring
One thing still desperately missing from Haskell Infra is cohesive monitoring and alerting. We have some pieces here and there, but not very accessible or centralized. (Shout out to https://haskell-status.org/, which has given me early warning of some issues and prevented serious outages!)
To begin tackling this problem, I’ve set up Grafana and Prometheus to record and monitor system-level statistics for the Stackage server. I’ve taken care to make charts that actually measure things I understand and care about, rather than using some off-the-shelf defaults. (These charts, incidentally, are how I just noticed the disk space issue.) This is just the tip of the iceberg, but at least it’s a start.
GitLab hugops
I’ve had very little time to contribute to the GHC GitLab instance. Unfortunately, things are now worse than ever for GitLab and GHC contributors.
The primary maintainer of the GHC GitLab instance is relatively new to the job. Like the last one, he’s supposed to be a GHC developer, but the situation is too unstable for that. GitLab takes a lot of his time.
- The same service migration that gave Stackage a hard time has also affected GitLab.
- Huge waves of aggressive web crawlers defy Hanlon’s razor and makes me wonder if someone is attacking GHC in particular. But logically, I know this is an Internet-wide problem related to the rise of LLMs. This sucks. I don’t even hate LLMs! They’re just really easy to misuse and cause suffering.
- Infrastructural technical debt is still really high.
- The result: a GitLab that is impossible to use for two vocal contributors in particular (hi guys), and hard enough for everybody else. At this point, there are no easy fixes. We’re stuck here until we make a systematic change.
Although I haven’t been able to contribute code-wise, I’ve tried to stay available to offer guidance, system access, and emotional support through this saga. Plus, I handled some basic maintenance tasks and responded to a couple acute issues (but not all).
One systematic change to truly resolve the situation could be me: I would love to wade back into this particular fight. It’s what my role was originally designed to do.
Haskell.org DNS maintenance
Speaking of technical debt, there was a huge mess in the zone file that controls haskell.org, www.haskell.org, downloads.haskell.org, and so on. That file is, after all, the fruit of decades of clickops. (If you don’t know what clickops is, it’s bad.) The carcasses of many partial migrations lay scattered here and there throughout the file.
The upshot of this was recurring website outages.
Our TLS certs are rotated automatically, of course. Except when the automation is broken. Because of all the tech debt, that was hard to avoid. We ended up playing whack-a-mole for half a year; each new fix revealed yet another problem that would surface 2-3 months later.
This is where https://haskell-status.org/ really saved our bacon. The most recent round of errors was identified before certs expired. 
During Zurihac last June, I was able to take a calm look at everything. Combined with the knowledge gained over the last couple rounds, I feel confident that this time, everything is actually fixed. 
Haskell wiki saga
Background: Over the last few years, the number of admins for the Haskell wiki had dwindled to something between zero and one. Then, last year, the situation collapsed with a bang. A long-overdue server upgrade went completely sideways, causing a long outage. Then there was the forced migration to OpenCape. By the end of the chaos, the number of effective Wiki admins was zero, and the wiki was missing styling and functionality.
On the plus side, the configuration had become public code. Together with some drive-by volunteers, I restored missing functionality and (finally, at Zurihac) restored the missing styling. We even enabled a responsive design, meaning the wiki is readable on mobile now!
Besides fixing the wiki itself, I improved the developer process by creating tooling for running the wiki on the developer’s own machine.
There are still open Wiki issues, so feel free to try out the dev experience and report any difficulties. Plus, the wiki is running on an overloaded server. Although the wiki itself is stable, the server is not. That causes occasional outages.
Strategic focus: Haskell infrastructure bus factor
A common concern for all the projects I work on is bus factor. That’s an industry term for how many people would need to get hit by a bus win the lottery and retire permanently for a project to lose all of its contributors. A bigger number is better. It is very small for Haskell infrastructure projects.
I don’t think this problem will go away. (It’s not just Haskell – under-supported Libre/Open Source infrastructure has been a topic since forever.) Instead, I think we need to do more with what we have. That’s how I’m framing the projects I want to work on now.
For instance, our systems are chronically out of date because updates only happen during a crisis. Never mind the bad security, that also makes dealing with crises take more time. That’s why I have been enabling automated updates for the systems I work on.
Automatic updates aren’t a revolutionary idea. It’s just one of the many low-hanging fruit left laying around when the bus factor is chronically low. Making progress is a question of strategically choosing what (not!) to work on rather than thinking hard about complex solutions.
Another not-rocket-science solution near to my heart is making it easy for volunteers to contribute. Volunteer effort is limitless and tightly constrained at the same time. I’ve got a whole philosophy about this I want to write down some time. The takeaway is that every project should have a clear path to project leadership – not just contributing. But obviously it starts with enabling contribution. I’ve worked on this by adding tests, CI, and easy-to-use developer environments for the wiki and many projects within the Stackage infrastructure universe.
Current work and coming up
- Improving server backups to democratize everything from backup management to server recovery
- Assisting with GitLab stabilization to improve GHC contributor workflow
- Continuing to automate system updates for better security and reduced workload
- Continuing to expand system monitoring and alerting
New problems show up every week. I can’t know for certain what I’ll be working on. This is just a list of the things that seem most important to me right now.
That’s all for now! My next monthly update will be coming up soon, so see you then.