Seeing 503 errors when opening pages from Hoogle or looking up packages on hackage:
Yeah, it happens to me pretty much every midnight UTC. Goes away on its own in 20 minutes or so.
Hmm, although not quite. I’ve seen 503 errors from Varnish cache server before, but not this one.
Seems to be good now ![]()
We started to get these small outages when we switched hackage boxes and haven’t been able to trace down the cause – it appears to be an error with too many file handles opened periodically, but haven’t sourced down where that comes from – although I suspect, if anyone wants to roll up their sleeves, that it could have to do with inefficient storage/calculation of download counts.
Could it be Out of file descriptors! · Issue #100 · haskell/tar · GitHub / cabal haddock failure: out of FDs due to tar · Issue #11131 · haskell/cabal · GitHub? ?
hackage-server still uses tar-0.6: hackage-server/hackage-server.cabal at 350fd8aac61c76481fcccfeee499ab6675171320 · haskell/hackage-server · GitHub
that’s a great suggestion and could well be the culprit. thanks for raising that!
@sclv even after the recent upgrade to tar-0.7, Hackage still gets extremely slow right after midnight UTC, for 10-15 minutes or so. Is there a background backup process or anything similar?..
Ah drat. Yes, I think that’s when our mirroring runs. Maybe we should modify our mirroring to throttle it, as another possible improvement.
I’ve been noticing these 503s a lot recently too; I haven’t paid attention to the time of day, but it’s happening right now, which lines up with midnight UTC. I’ll watch out for it happening other times; I feel like I’ve been seeing it later at night but I’m not sure.
Indeed the problem is getting worse, and it appears to be a result of a change in traffic patterns. Further discussion here hackage-server unresponsive · Issue #1467 · haskell/hackage-server
Just got some now:
(Getting the following two interchangeably, seemingly at random)
==========================
Error 503 error:0A000126:SSL routines::unexpected eof while reading
error:0A000126:SSL routines::unexpected eof while reading
Error 54113
Details: cache-rtm-ehrd2290051-RTM 1772583556 3872556882
Varnish cache server
===================================
Error 503 backend read error
backend read error
Error 54113
etc…
I also got an EOF a few minutes ago, but it works now.