DevOps Log, 2023-12-13

Hello, welcome to the n+1’th weekly log. This week has been about the Stackage migration and the standard mix of new operational issues to investigate and fix.

I now have all seven Stackage processes running in the new environment! Unfortunately, one of them gets tripped up on some unexpected Hackage behavior during its attempt at a full data refresh, so I need to fix that before moving to the next phase. The next phase, incidentally, will be updating DNS records so that people actually start using the new services!

Unrelated to this migration, but suspicious enough that people brought it to my attention, Stackage’s Hoogle search had stopped returning results from the base package. I was able to diagnose the problem relatively quickly, after which the GHC team and the Stackage curators were able to implement the fix.

Also unrelated to my work but suspicious enough to bring to my attention, head.hackage pipelines started failing not long after I significantly refactored them. In this case, it’s likely that an access token expired. Although it’s an imperfect solution, I have simply created a calendar reminder for myself to refresh the new one before it expires, as well.

Finally, I have been looking into a fresh set of issues affecting Darwin machines that power Haskell CI. In fact, I just checked, and once again over half of them are offline. There appear to be two distinct issues, and one of them isn’t even with the Darwin machines, themselves. Looks like DevOps Christmas came early this year!

See you next time!

5 Likes

Almost forgot my summary of newly triaged issues:

5 Likes

Great work @chreekat. Your list of things is ever-expanding, soon you’ll need to hire minions…

1 Like

Thanks @simonmic ! I do try to keep my scope limited, but there are so many juicy possibilities…