503 errors on Hackage

1

Seeing 503 errors when opening pages from Hoogle or looking up packages on hackage:

Error 503 error:0A000126:SSL routines::unexpected eof while reading

4 Likes
2

Yeah, it happens to me pretty much every midnight UTC. Goes away on its own in 20 minutes or so.

2 Likes
3

Hmm, although not quite. I’ve seen 503 errors from Varnish cache server before, but not this one.

4

Seems to be good now :person_shrugging:

5

We started to get these small outages when we switched hackage boxes and haven’t been able to trace down the cause – it appears to be an error with too many file handles opened periodically, but haven’t sourced down where that comes from – although I suspect, if anyone wants to roll up their sleeves, that it could have to do with inefficient storage/calculation of download counts.

1 Like
6

Could it be Out of file descriptors! · Issue #100 · haskell/tar · GitHub / cabal haddock failure: out of FDs due to tar · Issue #11131 · haskell/cabal · GitHub? ?

hackage-server still uses tar-0.6: hackage-server/hackage-server.cabal at 350fd8aac61c76481fcccfeee499ab6675171320 · haskell/hackage-server · GitHub

2 Likes
New Hackage Server Features
7

that’s a great suggestion and could well be the culprit. thanks for raising that!

1 Like
8

@sclv even after the recent upgrade to tar-0.7, Hackage still gets extremely slow right after midnight UTC, for 10-15 minutes or so. Is there a background backup process or anything similar?..

1 Like
9

Ah drat. Yes, I think that’s when our mirroring runs. Maybe we should modify our mirroring to throttle it, as another possible improvement.

2 Likes
10

I’ve been noticing these 503s a lot recently too; I haven’t paid attention to the time of day, but it’s happening right now, which lines up with midnight UTC. I’ll watch out for it happening other times; I feel like I’ve been seeing it later at night but I’m not sure.

3 Likes
11

Indeed the problem is getting worse, and it appears to be a result of a change in traffic patterns. Further discussion here hackage-server unresponsive · Issue #1467 · haskell/hackage-server

2 Likes
12

Just got some now:
(Getting the following two interchangeably, seemingly at random)

==========================

Error 503 error:0A000126:SSL routines::unexpected eof while reading

error:0A000126:SSL routines::unexpected eof while reading

Error 54113

Details: cache-rtm-ehrd2290051-RTM 1772583556 3872556882

Varnish cache server

===================================

Error 503 backend read error

backend read error

Error 54113

etc…

13

I also got an EOF a few minutes ago, but it works now.