Explicit query-string escaping is coming to Servant! 🆕

Announcements
1

We have just merged servant#1901, which makes query-string encoding explicit throughout Servant’s client stack.

This will be available in the next release of Servant. It is merged into master, but it is not part of a released version yet.

Why is this nice?

Until now, client requests represented a query value as a ByteString, but the type did not tell us what those bytes meant:

  • Was this decoded data that still needed URL encoding?
  • Or was it already encoded and ready to put on the wire?

That ambiguity made it easy to encode a value twice, fail to encode it at all, or handle it differently in different client backends. Reserved characters make the problem visible very quickly: a space, literal +, &, =, %, or Unicode text can change meaning if it is encoded at the wrong stage.

PR #1901 removes that ambiguity by representing client query strings with PartialEscapeQuery. Every section of a value now says explicitly whether it:

  • needs encoding, with QE; or
  • must be left unchanged, with QN.

For example:

import Network.HTTP.Types
  ( EscapeItem (QE, QN)
  , PartialEscapeQuery
  )

query :: PartialEscapeQuery
query =
  [ ("q", [QE "+", QN "+language:haskell"])
  , ("flag", [])
  , ("empty", [QE ""])
  ]

This renders as:

?q=%2B+language:haskell&flag&empty=

The decoded + is safely encoded as %2B, while the trusted query-language syntax is preserved exactly. A value-less parameter such as flag also remains distinct from an empty value such as empty=.

What does this allow us to do?

For ordinary APIs using QueryParam, QueryParams, QueryFlag, or DeepQuery, generated clients continue to look the same at the call site. The improvement is in their behaviour: query data containing spaces, literal plus signs, ampersands, equals signs, Unicode, and other reserved characters is now handled consistently by the in-tree client backends.

The change also means that ToHttpApiData.toEncodedQueryParam is now honoured by QueryParam and QueryParams. This is useful when a type deliberately has a query-specific wire representation—for example, a validated search expression whose + and : characters are meaningful syntax. Previously, putting that encoding logic in toEncodedQueryParam did not have the intended effect in Servant clients; now it does.

DeepQuery values are encoded consistently as well, so decoded values such as:

bob + alice & carol=friend

can round-trip as data rather than accidentally being interpreted as query-string structure.

Finally, custom client combinators, middleware, and backends can now make encoding decisions locally and visibly. There is no longer a convention hidden behind an unannotated ByteString: use QE for decoded bytes that still need encoding, and QN for bytes that are intentionally wire-ready.

Server-side clarity

This work also adds the DecodedQuery alias for server-side QueryString handlers.

handler :: DecodedQuery -> Handler Result

This does not introduce another decoding step. It documents the state handlers already receive: WAI has URL-decoded parameter names and values before the handler runs, including interpreting a wire-level + as a space. The same principle applies to FromDeepQuery.

If your handler or FromDeepQuery instance currently calls urlDecode itself as a workaround, it should be revisited when upgrading; decoding a second time can corrupt literal percent sequences or plus signs.

Who needs to migrate?

Most users of generated Servant clients will not need to change application call sites.

You will need to make a small migration if you maintain:

  • a custom client combinator that edits Request;
  • client middleware that edits requestQueryString;
  • a custom RunClient backend;
  • a client using the QueryString combinator directly.

Conceptually, the main API changes are:

-- Before
requestQueryString :: Seq QueryItem
appendToQueryString :: Text -> Maybe ByteString -> Request -> Request
setQueryString :: Query -> Request -> Request

-- After
requestQueryString :: Seq PartialEscapeQueryItem
appendToQueryString :: Text -> [EscapeItem] -> Request -> Request
setQueryString :: PartialEscapeQuery -> Request -> Request

Use [QE value] for decoded input, [QN value] for a value that is already encoded or intentionally literal, and [] for a value-less parameter. Custom backends should render the result with renderQueryPartialEscape.

One important caution: QN bypasses escaping completely. It should only be used for trusted, validated syntax or correctly percent-encoded bytes. Decoded user input belongs in QE; otherwise characters such as & and = could alter the query structure.

The repository now includes a detailed query-string encoding migration guide covering custom combinators, middleware, backends, direct QueryString clients, and server-side decoding workarounds.

Thanks to everyone involved in designing, reviewing, and landing this improvement. It resolves a long-standing ambiguity in query handling and gives Servant clients a much clearer foundation for both ordinary URL-encoded data and APIs with deliberate query-language syntax.

Again, this has landed on master and will be included in the next Servant release.

7 Likes