Stackage outage April 28/29 2025

Hello again,

As promised, I migrated Stackage.org to a new server yesterday. Unfortunately, some time afterward, the server process froze and remained unresponsive. I didn’t learn about the problem until the next morning (obviously a problem by itself), at which point I was able to restart the process.

It froze again 15 minutes later.

Acting on a hunch, I improved the caching (Cloudflare CDN) to reduce server load. The site has been running without issue for three hours now, which hopefully means the acute incident is resolved.

I still don’t know what went wrong. The server process was still running, but was returning some kind of error response with no log output. I will work to understand the root cause and also find ways to mitigate the issue faster should it happen again.

If you’re interested in helping out with this community resource, please bring your ideas to one of these issues:

Thank you!

7 Likes

Is this related to Hoogle’s downgrade? Hoogle.haskell.org no longer searches Stackage, but instead searches Haskell Platform.

Hoogle is searching Stackage for me:

seems to be intermittent, or a problem that was already fixed

The first nightly snapshots built on the new server are out.
Though there is some I/O performance issue via curator that needs to be ironed out.

Also the CDN might be delaying the snapshot diff data,
but Compare nightly-2025-04-24 with nightly-2025-04-30 :: Stackage Server looks okay now.

1 Like

Ok, glad to hear it. All I know about hoogle is that it was also recently migrated.

Hallelujah! I know we’ve got some performance issues to iron out, but this gives me confidence that we can fix them.

1 Like

Bryan has resolved the server io issues we believe, so that is great news!! :raising_hands:

3 Likes