Issue with Empty Responses for `snapshots.json` and `lts-snapshots.json` from Stackage.org

ukari · May 13, 2025, 9:37am

I’ve encountered an issue where requests to https://www.stackage.org/download/snapshots.json or https://www.stackage.org/download/lts-snapshots.json sometimes return empty responses (HTTP 200 with zero content). This seems to occur when cloudflare bypasses its cache and fetches directly from the origin server, while cached responses are consistently valid. Below are detailed curl logs demonstrating the issue. I suspect this may be related to the update mechanism for these files. Has anyone else experienced this, or can you suggest possible causes or fixes?

logs group 1: Empty Response with `cf-cache-status: HIT` and No `age`

$ curl -v https://www.stackage.org/download/snapshots.json
* Host www.stackage.org:443 was resolved.
* IPv6: 2606:4700:3033::6815:2653, 2606:4700:3037::ac43:dcc5
* IPv4: 104.21.38.83, 172.67.220.197
*   Trying [2606:4700:3033::6815:2653]:443...
* Immediate connect fail for 2606:4700:3033::6815:2653: Network is unreachable
*   Trying [2606:4700:3037::ac43:dcc5]:443...
* Immediate connect fail for 2606:4700:3037::ac43:dcc5: Network is unreachable
*   Trying 104.21.38.83:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=stackage.org
*  start date: Apr  6 19:44:16 2025 GMT
*  expire date: Jul  5 20:42:57 2025 GMT
*  subjectAltName: host "www.stackage.org" matched cert's "*.stackage.org"
*  issuer: C=US; O=Google Trust Services; CN=WE1
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA256
*   Certificate level 1: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 2: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384
* Connected to www.stackage.org (104.21.38.83) port 443
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://www.stackage.org/download/snapshots.json
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: www.stackage.org]
* [HTTP/2] [1] [:path: /download/snapshots.json]
* [HTTP/2] [1] [user-agent: curl/8.12.1]
* [HTTP/2] [1] [accept: */*]
> GET /download/snapshots.json HTTP/2
> Host: www.stackage.org
> User-Agent: curl/8.12.1
> Accept: */*
> 
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/2 200 
< date: Tue, 13 May 2025 06:30:50 GMT
< content-type: application/json; charset=utf-8
< server: cloudflare
< vary: Accept-Encoding
< vary: Accept-Encoding, Accept, Accept-Language
< x-xss-protection: 1; mode=block
< last-modified: Tue, 13 May 2025 06:29:12 GMT
< nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800}
< cache-control: max-age=14400
< cf-cache-status: HIT
< report-to: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=CSJF77OZvQOU1A%2FMj8r1LFN9fI8d603Qo19ZR5wus3rR%2BoLYL1CiVM9WTLjZ2iPwgDKnm2c0uX7SAPj7ppQtZxucC6ZFlgpH3zxFfFuNtS9uo2mooL%2BLE4rMvl60ZSTpYOSBbXXVNtH5KdpfJ4pJ"}]}
< cf-ray: 93f01c0188d86845-NRT
< alt-svc: h3=":443"; ma=86400
< 
* Connection #0 to host www.stackage.org left intact

$ curl -v https://www.stackage.org/download/snapshots.json
* Host www.stackage.org:443 was resolved.
* IPv6: 2606:4700:3033::6815:2653, 2606:4700:3037::ac43:dcc5
* IPv4: 104.21.38.83, 172.67.220.197
*   Trying [2606:4700:3033::6815:2653]:443...
* Immediate connect fail for 2606:4700:3033::6815:2653: Network is unreachable
*   Trying [2606:4700:3037::ac43:dcc5]:443...
* Immediate connect fail for 2606:4700:3037::ac43:dcc5: Network is unreachable
*   Trying 104.21.38.83:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=stackage.org
*  start date: Apr  6 19:44:16 2025 GMT
*  expire date: Jul  5 20:42:57 2025 GMT
*  subjectAltName: host "www.stackage.org" matched cert's "*.stackage.org"
*  issuer: C=US; O=Google Trust Services; CN=WE1
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA256
*   Certificate level 1: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 2: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384
* Connected to www.stackage.org (104.21.38.83) port 443
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://www.stackage.org/download/snapshots.json
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: www.stackage.org]
* [HTTP/2] [1] [:path: /download/snapshots.json]
* [HTTP/2] [1] [user-agent: curl/8.12.1]
* [HTTP/2] [1] [accept: */*]
> GET /download/snapshots.json HTTP/2
> Host: www.stackage.org
> User-Agent: curl/8.12.1
> Accept: */*
> 
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/2 200 
< date: Tue, 13 May 2025 06:30:53 GMT
< content-type: application/json; charset=utf-8
< server: cloudflare
< vary: Accept-Encoding
< vary: Accept-Encoding, Accept, Accept-Language
< x-xss-protection: 1; mode=block
< report-to: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=ZpiZ%2Bbai%2BZDLBVUYYhFbe5U%2Fp414IcQ9dB%2Fkr9G3b9Sjh9KDw8fCkSLjjYGTXnpHZ%2FRqq7JFR%2FiMNX0AFyNqnLDAQIdw%2BCAZ2VV%2ByxW0ZEjZCjpxJr51HtE68RLK9gwvDx05f%2FFl%2BxBrJUKKkhK9"}]}
< last-modified: Tue, 13 May 2025 06:29:12 GMT
< nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800}
< age: 3
< cache-control: max-age=14400
< cf-cache-status: HIT
< cf-ray: 93f01c1a4c436845-NRT
< alt-svc: h3=":443"; ma=86400
< 
* Connection #0 to host www.stackage.org left intact
{"lts":"lts-23.21","lts-0":"lts-0.7","lts-1":"lts-1.15","lts-10":"lts-10.10","lts-11":"lts-11.22","lts-12":"lts-12.26","lts-13":"lts-13.30","lts-14":"lts-14.27","lts-15":"lts-15.16","lts-16":"lts-16.31","lts-17":"lts-17.15","lts-18":"lts-18.28","lts-19":"lts-19.33","lts-2":"lts-2.22","lts-20":"lts-20.26","lts-21":"lts-21.25","lts-22":"lts-22.44","lts-23":"lts-23.21","lts-3":"lts-3.22","lts-4":"lts-4.2","lts-5":"lts-5.18","lts-6":"lts-6.35","lts-7":"lts-7.24","lts-8":"lts-8.24","lts-9":"lts-9.21","nightly":"nightly-2025-05-11"}

In the response of the first log, it contains “cf-cache-status: HIT” and not contains “age”. This seems to be cloudflare cache hitted but file is zero size so cloudflare replys origin.

In the second, it contains “cf-cache-status: HIT” and contains “age: 3”. This means the reply is a cloudflare cache.

logs group 2: Empty Response from Another User

$ curl -v https://www.stackage.org/download/snapshots.json
*   Trying [2606:4700:3033::6815:2653]:443...
* Connected to www.stackage.org (2606:4700:3033::6815:2653) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=stackage.org
*  start date: Apr  6 19:44:16 2025 GMT
*  expire date: Jul  5 20:42:57 2025 GMT
*  subjectAltName: host "www.stackage.org" matched cert's "*.stackage.org"
*  issuer: C=US; O=Google Trust Services; CN=WE1
*  SSL certificate verify ok.
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /download/snapshots.json]
* h2h3 [:scheme: https]
* h2h3 [:authority: www.stackage.org]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x557dd68b0290)
> GET /download/snapshots.json HTTP/2
> Host: www.stackage.org
> user-agent: curl/7.88.1
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 200
< date: Tue, 13 May 2025 02:37:54 GMT
< content-type: application/json; charset=utf-8
< server: cloudflare
< vary: Accept-Encoding
< vary: Accept-Encoding, Accept, Accept-Language
< x-xss-protection: 1; mode=block
< last-modified: Tue, 13 May 2025 02:34:33 GMT
< nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800}
< cache-control: max-age=14400
< cf-cache-status: HIT
< report-to: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=zAZuqPIh%2FoVlhbGTK0wP88GvvURRM%2Bf5WPYJTeeu10PBkK8n%2FqE%2FzZgXqW0j6N3rqKUxssgRz2qvaq4SG4HG6cQIY%2F9XiS2CwW6q0JAaWImi7FiYyAn%2Bxac1%2BEnAyWVCDvvBxNWFRhFQuZXIJxLJ"}]}
< cf-ray: 93eec6cd2e93fcec-SIN
< alt-svc: h3=":443"; ma=86400
<
* Connection #0 to host www.stackage.org left intact

This single log shows an another record that snapshots.json is zero size in an another time. Also there is no “age” in http header of response while contains “cf-cache-status: HIT”.

logs group 3: Empty Response with no-cache while Cloudflare ignores the no-cache

$ curl --header 'Cache-Control: no-cache' -v https://www.stackage.org/download/snapshots.json
* Host www.stackage.org:443 was resolved.
* IPv6: 2606:4700:3033::6815:2653, 2606:4700:3037::ac43:dcc5
* IPv4: 104.21.38.83, 172.67.220.197
*   Trying [2606:4700:3033::6815:2653]:443...
* Immediate connect fail for 2606:4700:3033::6815:2653: Network is unreachable
*   Trying [2606:4700:3037::ac43:dcc5]:443...
* Immediate connect fail for 2606:4700:3037::ac43:dcc5: Network is unreachable
*   Trying 104.21.38.83:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=stackage.org
*  start date: Apr  6 19:44:16 2025 GMT
*  expire date: Jul  5 20:42:57 2025 GMT
*  subjectAltName: host "www.stackage.org" matched cert's "*.stackage.org"
*  issuer: C=US; O=Google Trust Services; CN=WE1
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA256
*   Certificate level 1: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 2: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384
* Connected to www.stackage.org (104.21.38.83) port 443
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://www.stackage.org/download/snapshots.json
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: www.stackage.org]
* [HTTP/2] [1] [:path: /download/snapshots.json]
* [HTTP/2] [1] [user-agent: curl/8.12.1]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [cache-control: no-cache]
> GET /download/snapshots.json HTTP/2
> Host: www.stackage.org
> User-Agent: curl/8.12.1
> Accept: */*
> Cache-Control: no-cache
> 
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/2 200 
< date: Tue, 13 May 2025 06:34:04 GMT
< content-type: application/json; charset=utf-8
< server: cloudflare
< vary: Accept-Encoding
< vary: Accept-Encoding, Accept, Accept-Language
< x-xss-protection: 1; mode=block
< report-to: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=FwxqNC5yy7NR4pEn2YuJCnXy%2FdhTfxRnKg0m%2Fwgz2u9cW19N4BVi%2BKQsbVK6arlUjHDxG1W6na%2Ftpa4WpqEIV0p%2BbkBLpReVNI5vsQUKRM5YxqfR3i3UisSTfHTV%2BtNLVUUP3cMyWc0SDw%2FeL%2BLt"}]}
< last-modified: Tue, 13 May 2025 06:29:12 GMT
< nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800}
< age: 194
< cache-control: max-age=14400
< cf-cache-status: HIT
< cf-ray: 93f020bf9ca56845-NRT
< alt-svc: h3=":443"; ma=86400
< 
* Connection #0 to host www.stackage.org left intact
{"lts":"lts-23.21","lts-0":"lts-0.7","lts-1":"lts-1.15","lts-10":"lts-10.10","lts-11":"lts-11.22","lts-12":"lts-12.26","lts-13":"lts-13.30","lts-14":"lts-14.27","lts-15":"lts-15.16","lts-16":"lts-16.31","lts-17":"lts-17.15","lts-18":"lts-18.28","lts-19":"lts-19.33","lts-2":"lts-2.22","lts-20":"lts-20.26","lts-21":"lts-21.25","lts-22":"lts-22.44","lts-23":"lts-23.21","lts-3":"lts-3.22","lts-4":"lts-4.2","lts-5":"lts-5.18","lts-6":"lts-6.35","lts-7":"lts-7.24","lts-8":"lts-8.24","lts-9":"lts-9.21","nightly":"nightly-2025-05-11"}
$ curl --header 'Cache-Control: no-cache' -v https://www.stackage.org/download/snapshots.json
* Host www.stackage.org:443 was resolved.
* IPv6: 2606:4700:3037::ac43:dcc5, 2606:4700:3033::6815:2653
* IPv4: 104.21.38.83, 172.67.220.197
*   Trying [2606:4700:3037::ac43:dcc5]:443...
* Immediate connect fail for 2606:4700:3037::ac43:dcc5: Network is unreachable
*   Trying [2606:4700:3033::6815:2653]:443...
* Immediate connect fail for 2606:4700:3033::6815:2653: Network is unreachable
*   Trying 104.21.38.83:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=stackage.org
*  start date: Apr  6 19:44:16 2025 GMT
*  expire date: Jul  5 20:42:57 2025 GMT
*  subjectAltName: host "www.stackage.org" matched cert's "*.stackage.org"
*  issuer: C=US; O=Google Trust Services; CN=WE1
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA256
*   Certificate level 1: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 2: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384
* Connected to www.stackage.org (104.21.38.83) port 443
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://www.stackage.org/download/snapshots.json
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: www.stackage.org]
* [HTTP/2] [1] [:path: /download/snapshots.json]
* [HTTP/2] [1] [user-agent: curl/8.12.1]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [cache-control: no-cache]
> GET /download/snapshots.json HTTP/2
> Host: www.stackage.org
> User-Agent: curl/8.12.1
> Accept: */*
> Cache-Control: no-cache
> 
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/2 200 
< date: Tue, 13 May 2025 06:39:12 GMT
< content-type: application/json; charset=utf-8
< server: cloudflare
< vary: Accept-Encoding
< vary: Accept-Encoding, Accept, Accept-Language
< x-xss-protection: 1; mode=block
< last-modified: Tue, 13 May 2025 06:37:30 GMT
< nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800}
< cache-control: max-age=14400
< cf-cache-status: HIT
< report-to: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=y5dfBKFy3AJ63zKIE%2FdcqS93ZUn4ssgNbknKbkxY0uKwd7lThZ3dlXG0Br5pIeclVO0xhA%2FUiRWXZv%2FHAzyddq2ZA4RA%2B2IEa%2FFaFFPYi5zdDRkWevpqeFBPD0IewLVDzRmngp8FRGMkM8jgsYd2"}]}
< cf-ray: 93f02845fd3b6845-NRT
< alt-svc: h3=":443"; ma=86400
< 
* Connection #0 to host www.stackage.org left intact

$ curl --header 'Cache-Control: no-cache' -v https://www.stackage.org/download/snapshots.json
* Host www.stackage.org:443 was resolved.
* IPv6: 2606:4700:3037::ac43:dcc5, 2606:4700:3033::6815:2653
* IPv4: 104.21.38.83, 172.67.220.197
*   Trying [2606:4700:3037::ac43:dcc5]:443...
* Immediate connect fail for 2606:4700:3037::ac43:dcc5: Network is unreachable
*   Trying [2606:4700:3033::6815:2653]:443...
* Immediate connect fail for 2606:4700:3033::6815:2653: Network is unreachable
*   Trying 104.21.38.83:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=stackage.org
*  start date: Apr  6 19:44:16 2025 GMT
*  expire date: Jul  5 20:42:57 2025 GMT
*  subjectAltName: host "www.stackage.org" matched cert's "*.stackage.org"
*  issuer: C=US; O=Google Trust Services; CN=WE1
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA256
*   Certificate level 1: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 2: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384
* Connected to www.stackage.org (104.21.38.83) port 443
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://www.stackage.org/download/snapshots.json
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: www.stackage.org]
* [HTTP/2] [1] [:path: /download/snapshots.json]
* [HTTP/2] [1] [user-agent: curl/8.12.1]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [cache-control: no-cache]
> GET /download/snapshots.json HTTP/2
> Host: www.stackage.org
> User-Agent: curl/8.12.1
> Accept: */*
> Cache-Control: no-cache
> 
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/2 200 
< date: Tue, 13 May 2025 06:39:22 GMT
< content-type: application/json; charset=utf-8
< server: cloudflare
< vary: Accept-Encoding
< vary: Accept-Encoding, Accept, Accept-Language
< x-xss-protection: 1; mode=block
< report-to: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=TcRirn1ViqHgpxXqFl%2FXRyXTtE%2FZbVx3RTZRsf%2BTDFLDchbmvyue%2FqG6SL3o4KFJJsxK%2BC7aR33lvPI8jbheUAoq3cnRiw9HjW2DXkFmL%2BIg1%2BhywivLMLO%2B69c0LQzBo8tnFGIX7SRchOT5Iy06"}]}
< last-modified: Tue, 13 May 2025 06:37:30 GMT
< nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800}
< age: 10
< cache-control: max-age=14400
< cf-cache-status: HIT
< cf-ray: 93f028879a2e6845-NRT
< alt-svc: h3=":443"; ma=86400
< 
* Connection #0 to host www.stackage.org left intact
{"lts":"lts-23.21","lts-0":"lts-0.7","lts-1":"lts-1.15","lts-10":"lts-10.10","lts-11":"lts-11.22","lts-12":"lts-12.26","lts-13":"lts-13.30","lts-14":"lts-14.27","lts-15":"lts-15.16","lts-16":"lts-16.31","lts-17":"lts-17.15","lts-18":"lts-18.28","lts-19":"lts-19.33","lts-2":"lts-2.22","lts-20":"lts-20.26","lts-21":"lts-21.25","lts-22":"lts-22.44","lts-23":"lts-23.21","lts-3":"lts-3.22","lts-4":"lts-4.2","lts-5":"lts-5.18","lts-6":"lts-6.35","lts-7":"lts-7.24","lts-8":"lts-8.24","lts-9":"lts-9.21","nightly":"nightly-2025-05-11"}

The first log is cached by cloudflare.

The second from origin site is zero size.

The third is cached.

logs group 4: Empty Response with cf-cache-status: EXPIRED

$ curl --header 'Cache-Control: max-age=0' -v https://www.stackage.org/download/lts-snapshots.json
* Host www.stackage.org:443 was resolved.
* IPv6: 2606:4700:3033::6815:2653, 2606:4700:3037::ac43:dcc5
* IPv4: 104.21.38.83, 172.67.220.197
*   Trying [2606:4700:3033::6815:2653]:443...
* Immediate connect fail for 2606:4700:3033::6815:2653: Network is unreachable
*   Trying [2606:4700:3037::ac43:dcc5]:443...
* Immediate connect fail for 2606:4700:3037::ac43:dcc5: Network is unreachable
*   Trying 104.21.38.83:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=stackage.org
*  start date: Apr  6 19:44:16 2025 GMT
*  expire date: Jul  5 20:42:57 2025 GMT
*  subjectAltName: host "www.stackage.org" matched cert's "*.stackage.org"
*  issuer: C=US; O=Google Trust Services; CN=WE1
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA256
*   Certificate level 1: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 2: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384
* Connected to www.stackage.org (104.21.38.83) port 443
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://www.stackage.org/download/lts-snapshots.json
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: www.stackage.org]
* [HTTP/2] [1] [:path: /download/lts-snapshots.json]
* [HTTP/2] [1] [user-agent: curl/8.12.1]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [cache-control: max-age=0]
> GET /download/lts-snapshots.json HTTP/2
> Host: www.stackage.org
> User-Agent: curl/8.12.1
> Accept: */*
> Cache-Control: max-age=0
> 
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/2 200 
< date: Tue, 13 May 2025 08:25:00 GMT
< content-type: application/json; charset=utf-8
< server: cloudflare
< vary: Accept-Encoding
< vary: Accept-Encoding, Accept, Accept-Language
< x-xss-protection: 1; mode=block
< last-modified: Tue, 13 May 2025 08:25:00 GMT
< nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800}
< cache-control: max-age=14400
< cf-cache-status: EXPIRED
< report-to: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=OtwIfrHtNJe4fExeTpGetykGiDTkQph5W%2Bexpt6dWVCSEtH2N9KkViWT99nCQINoMXlOPh4aWyGxe0xdwdozi7NQIAZe9CL2OOlFITEz7pN6FHbbTZfYXBTLysSfD4Y5wAWQRr95oQ%2BcY3pNUbSC"}]}
< cf-ray: 93f0c3406aee6845-NRT
< alt-svc: h3=":443"; ma=86400
< 
* Connection #0 to host www.stackage.org left intact

$ curl --header 'Cache-Control: max-age=0' -v https://www.stackage.org/download/lts-snapshots.json
* Host www.stackage.org:443 was resolved.
* IPv6: 2606:4700:3033::6815:2653, 2606:4700:3037::ac43:dcc5
* IPv4: 104.21.38.83, 172.67.220.197
*   Trying [2606:4700:3033::6815:2653]:443...
* Immediate connect fail for 2606:4700:3033::6815:2653: Network is unreachable
*   Trying [2606:4700:3037::ac43:dcc5]:443...
* Immediate connect fail for 2606:4700:3037::ac43:dcc5: Network is unreachable
*   Trying 104.21.38.83:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=stackage.org
*  start date: Apr  6 19:44:16 2025 GMT
*  expire date: Jul  5 20:42:57 2025 GMT
*  subjectAltName: host "www.stackage.org" matched cert's "*.stackage.org"
*  issuer: C=US; O=Google Trust Services; CN=WE1
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA256
*   Certificate level 1: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 2: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384
* Connected to www.stackage.org (104.21.38.83) port 443
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://www.stackage.org/download/lts-snapshots.json
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: www.stackage.org]
* [HTTP/2] [1] [:path: /download/lts-snapshots.json]
* [HTTP/2] [1] [user-agent: curl/8.12.1]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [cache-control: max-age=0]
> GET /download/lts-snapshots.json HTTP/2
> Host: www.stackage.org
> User-Agent: curl/8.12.1
> Accept: */*
> Cache-Control: max-age=0
> 
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/2 200 
< date: Tue, 13 May 2025 08:25:07 GMT
< content-type: application/json; charset=utf-8
< server: cloudflare
< vary: Accept-Encoding
< vary: Accept-Encoding, Accept, Accept-Language
< x-xss-protection: 1; mode=block
< report-to: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=y9tsH%2B2GTgt834qC4oTgYo2edsYeCYWtiyCW3WYdSnQk83wBhbL%2Ba%2FVLGgXnvDD4E6YnOihJdZ2XaZgy5tZTRC%2FKWvSzkmmUffyyAas3zrBI3D7TlKjB5TAPxPjiKBu1S8NrFdOxIpwk%2Bc5vIePD"}]}
< last-modified: Tue, 13 May 2025 08:25:00 GMT
< nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800}
< age: 6
< cache-control: max-age=14400
< cf-cache-status: HIT
< cf-ray: 93f0c36d3ba76845-NRT
< alt-svc: h3=":443"; ma=86400
< 
* Connection #0 to host www.stackage.org left intact
{"lts":"lts-23.21","lts-0":"lts-0.7","lts-1":"lts-1.15","lts-10":"lts-10.10","lts-11":"lts-11.22","lts-12":"lts-12.26","lts-13":"lts-13.30","lts-14":"lts-14.27","lts-15":"lts-15.16","lts-16":"lts-16.31","lts-17":"lts-17.15","lts-18":"lts-18.28","lts-19":"lts-19.33","lts-2":"lts-2.22","lts-20":"lts-20.26","lts-21":"lts-21.25","lts-22":"lts-22.44","lts-23":"lts-23.21","lts-3":"lts-3.22","lts-4":"lts-4.2","lts-5":"lts-5.18","lts-6":"lts-6.35","lts-7":"lts-7.24","lts-8":"lts-8.24","lts-9":"lts-9.21","nightly":"nightly-2025-05-11"}

These two logs request the lts-snapshots.json file with the same problem with a little difference. That in the first log, the reply response contains a “cf-cache-status: EXPIRED” http header.

Observations and Questions

I think that when cloudflare replys response without the “age” http header, it means that the file is fetched by cloudflare from the origin site(cached just now or directly fetch through). And the fetched from origin replys response are always zero size.

Here is the questions:

Has anyone else encountered empty responses from these endpoints?
Could this be due to an issue in the file update mechanism on the Stackage server?
Are there recommended workarounds, such as retrying requests or adjusting cache headers?
Should we report this to the Stackage team for further investigation?

Any insights or suggestions would be greatly appreciated!

jmct · May 13, 2025, 2:59pm

Just letting you know that @chreekat is aware of the issue and will be investigating it.

chreekat · May 13, 2025, 6:43pm

Hey @ukari , thanks for the detailed report!

In group 1, a hit with no age header is still a hit. I.e., the 0-byte response was a cached response. The explanation for the missing age header is right at the top here: Cloudflare cache responses · Cloudflare Cache (CDN) docs. I’m using tiered caching, so in our case, I suspect this is a situation where the lower-tier cache is getting populated from a higher tier. But that doesn’t explain the 0-byte response! If it had cached the 0-byte response, I would expect you to see it in the second curl as well. It’s got age: 3, so I would assume it would be the same cached value and thus the same response.

In group 2, it seems like we see the same thing, with one difference. You’ll note there is a cf-ray header that indicates which Cloudflare datacenter is serving the response. Each datacenter has its own cache. So this is another lower-tier cache getting populated from the upper tier, with the same weird 0-byte behavior.

In group 3, Cloudflare does not use the request’s cache-control header to update its own cache, so that may be misleading. If we ignore that, we see something pretty similar to the first two cases. There is one good hit, one “bad” hit with no age, and one followup good hit.

In group 4, we have a variation on the theme. Once again, CF ignores the request cache-control. In this case, we see a full EXPIRED, which must mean CF has gone all the way to the origin to refresh the cache. I think we can assume this entails populating the lower-tier cache, since with tiered-caching there is only one “upper tier” cache that ever contacts the origin. So perhaps the common thread is, “When the lower-tier cache gets populated, there is a 0-byte response”.

There’s one last piece of info to tie this together. On the origin, I don’t see any zero-byte responses. I see plenty of healthy 532-byte responses.

My current working hypothesis is that this has something to do with tiered caching. I’ll turn it off. Let’s see if that improves things.

[Some minutes later]

Ok, that didn’t help. So now the hypothesis is that when any Cloudflare server has its cache populated, it returns 0 bytes for certain routes. Still investigating…

chreekat · May 13, 2025, 7:16pm

For now, I have simply turned off the Cloudflare proxy. (This should be fine.) I can do some more debugging tomorrow.

Please report any difference in behavior you may see!

EDIT (collapsing replies because Discourse thinks I shouldn’t be allowed to post 3 times in a row):

Sorry, I made it worse just now by “disabling the proxy” in the wrong way.

Now we have to wait for DNS to re-propagate so the HTTPS error goes away.

chreekat · May 16, 2025, 6:34am

Hi everyone,

I think I have fixed this now. Please let me know if you see any more problems!

Topic		Replies	Views
[Resolved] Stackage outage 2024-11-04 Announcements	3	362	November 5, 2024
Stackage outage April 28/29 2025 Announcements	7	227	May 2, 2025
Stack build failing Learn	4	3718	March 23, 2020
[ANN] Stack 2.15.5 Announcements	0	558	March 29, 2024
[ANN] Stack 2.15.3 and earlier's default source for list of Stackage snapshots not up to date Announcements	24	2009	July 8, 2024