I have been writing a small CLI tool using http-conduit and aeson to fetch and process public updates from bloxfruit-scripts and similar community resources. While standard GET requests work for basic JSON payloads, the parser frequently fails with invalid UTF-8 byte sequence errors whenever the endpoint returns compressed chunked responses.
I initially tried converting the response stream directly from lazy ByteString into Text before decoding, but the pipeline seems to terminate prematurely on certain network interruptions. Has anyone encountered similar decoding failures when streaming through conduit, and what is the idiomatic way to handle fallback character encodings gracefully in modern Haskell?
Message bodies in HTTP are not (necessarily) UTF-8.
Also see section 2.2:
A recipient MUST parse an HTTP message as a sequence of octets in an encoding that is a superset of US-ASCII [USASCII]. Parsing an HTTP message as a stream of Unicode characters, without regard for the specific encoding, creates security vulnerabilities due to the varying ways that string processing libraries handle invalid multibyte character sequences that contain the octet LF (%x0A). String-based parsers can only be safely used within protocol elements after the element has been extracted from the message, such as within a header field line value after message parsing has delineated the individual field lines. A recipient MUST parse an HTTP message as a sequence of octets in an encoding that is a superset of US-ASCII [USASCII]. Parsing an HTTP message as a stream of Unicode characters, without regard for the specific encoding, creates security vulnerabilities due to the varying ways that string processing libraries handle invalid multibyte character sequences that contain the octet LF (%x0A). String-based parsers can only be safely used within protocol elements after the element has been extracted from the message, such as within a header field line value after message parsing has delineated the individual field lines.