Hackage timeouts (stack error S-5170 again)

Hi.

Since a few days, we often see errors on CI where builds are failing because hackage does not respond in a timely manner:

Tue, 10 Mar 2026 10:59:20 GMT Downloading timestamp
Tue, 10 Mar 2026 10:59:20 GMT Downloading snapshot
Tue, 10 Mar 2026 10:59:50 GMT SomeRemoteError (HttpExceptionRequest Request {
Tue, 10 Mar 2026 10:59:50 GMT   host                 = "hackage.haskell.org"
Tue, 10 Mar 2026 10:59:50 GMT   port                 = 443
Tue, 10 Mar 2026 10:59:50 GMT   secure               = True
Tue, 10 Mar 2026 10:59:50 GMT   requestHeaders       = [("Accept-Encoding",""),("User-Agent","Haskell pantry package")]
Tue, 10 Mar 2026 10:59:50 GMT   path                 = "/snapshot.json"
Tue, 10 Mar 2026 10:59:50 GMT   queryString          = ""
Tue, 10 Mar 2026 10:59:50 GMT   method               = "GET"
Tue, 10 Mar 2026 10:59:50 GMT   proxy                = Nothing
Tue, 10 Mar 2026 10:59:50 GMT   rawBody              = False
Tue, 10 Mar 2026 10:59:50 GMT   redirectCount        = 10
Tue, 10 Mar 2026 10:59:50 GMT   responseTimeout      = ResponseTimeoutDefault
Tue, 10 Mar 2026 10:59:50 GMT   requestVersion       = HTTP/1.1
Tue, 10 Mar 2026 10:59:50 GMT   proxySecureMode      = ProxySecureWithConnect
Tue, 10 Mar 2026 10:59:50 GMT }
Tue, 10 Mar 2026 10:59:50 GMT  ResponseTimeout)
Tue, 10 Mar 2026 10:59:50 GMT 
Tue, 10 Mar 2026 10:59:50 GMT Error: [S-5170]
Tue, 10 Mar 2026 10:59:50 GMT        While trying to unpack HUnit-1.6.2.0, Stack encountered an error.

As you can see we are using stack. Also note the request failed after 30 seconds, which is exactly the ResponseTimeoutDefault. (this was run on Github provided runners see Add test for unpacking · avdv/stack@ebb41d5 · GitHub)

I tried to reproduce with curl, and one time I indeed saw some weird hangup:

curl -v -m 30 https://hackage.haskell.org/package/logict-0.7.0.3.tar.gz -o /dev/null
  % Total    % Received % Xferd  Average Speed  Time    Time    Time   Current
                                 Dload  Upload  Total   Spent   Left   Speed
  0      0   0      0   0      0      0      0                              0* Host hackage.haskell.org:443 was resolved.
* IPv6: (none)
* IPv4: 151.101.0.68, 151.101.64.68, 151.101.128.68, 151.101.192.68
*   Trying 151.101.0.68:443...
* ALPN: curl offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
...
* Server certificate:
*   subject: CN=*.haskell.org
*   start date: Feb  9 16:18:57 2026 GMT
*   expire date: Mar 13 16:18:56 2027 GMT
*   issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign Atlas R3 DV TLS CA 2026 Q1
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   subjectAltName: "hackage.haskell.org" matches cert's "*.haskell.org"
* SSL certificate verified via OpenSSL.
* Established connection to hackage.haskell.org (151.101.0.68 port 443) from 192.168.1.117 port 53384
* using HTTP/1.x
} [5 bytes data]
> GET /package/logict-0.7.0.3.tar.gz HTTP/1.1
> Host: hackage.haskell.org
> User-Agent: curl/8.18.0
> Accept: */*
>
* Request completely sent off
  0      0   0      0   0      0      0      0           00:07              0{ [5 bytes data]
< HTTP/1.1 301 Moved Permanently
< Connection: keep-alive
< Content-Length: 0
< Server: nginx/1.24.0 (Ubuntu)
< Content-Type: text/plain
< Location: /package/logict-0.7.0.3/logict-0.7.0.3.tar.gz
< Accept-Ranges: bytes
< Via: 1.1 varnish, 1.1 varnish
< Date: Tue, 10 Mar 2026 08:20:38 GMT
< X-Served-By: cache-dfw-kdfw8210050-DFW, cache-fra-etou8220058-FRA
< X-Cache: MISS, MISS
< X-Cache-Hits: 0, 0
< X-Timer: S1773130831.082936,VS0,VE7547
<
  0      0   0      0   0      0      0      0           00:07              0
* Connection #0 to host hackage.haskell.org:443 left intact

Note that it took 7 seconds after the request was sent off until the headers arrived.

According to APDEX on https://hackage.haskell.org/ status page – updown.io I should be happy, yet CI is failing all over again :worried:

Anyone experiencing the same issue?

3 Likes

See discussion here 503 errors on Hackage

and ticket here hackage-server unresponsive · Issue #1467 · haskell/hackage-server · GitHub

2 Likes

Oh, I didn’t make the connection to that topic since 503 is not what I’m seeing, but yes the hackage web pages are also quite slow…

2 Likes

Almost everything is configurable with Stack these days … but that default response timeout of 30 seconds is not. Should it be?

I’m pretty sure that most HTTP servers, and importantly proxies, have a 30 second timeout by default. So even if stack has a longer timeout, the connection might be reset by someone else in the chain

1 Like

i haven’t been able to browse hackage without huge delay for days now :disappointed_face:

1 Like

I’ve had to resort to stackage docs because of this issue.

1 Like

Thanks to the work of the infrastructure team, we’ve now moved Hackage to a larger (more RAM) system. The current working hypothesis is that this will mean fewer latency issues.

There is more work to be done, but the immediate frustrations should be eased.

14 Likes

Thanks to everyone involved in this effort! It really flies now

5 Likes