# HF Tech Proposal #1: UTF-8 Encoded Text

**URL:** <https://discourse.haskell.org/t/hf-tech-proposal-1-utf-8-encoded-text/2499>\
**Category:** Haskell Foundation\
**Created:** [May 13, 2021, 4:40pm UTC](https://discourse.haskell.org/t/hf-tech-proposal-1-utf-8-encoded-text/2499 "2021-05-13T16:40:24Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![jaror](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/jaror/32/3271_2.png) [@jaror](https://discourse.haskell.org/u/jaror)\
**Post date:** [May 19, 2021, 8:04pm UTC](https://discourse.haskell.org/t/hf-tech-proposal-1-utf-8-encoded-text/2499/21 "2021-05-19T20:04:19Z")

</div>

> [@ketzacoatl](#):
>
> backpack is one of the “alternative preludes”

That is not what I meant. `backpack` is a language feature similar to type classes, but then at the module level. So, I definitely wouldn’t call it an alternative prelude.

`backpack-str` is a Haskell package which provides a specification (similar to a type class) of which functions are required for a string-like type and it provides implementations (similar to instances) of that specification for several types.

An alternative prelude could (and probably should) be developed to show the advantages and disadvantages of the backpack approach before a definite choice is made to switch the actual base package to the backpack approach. I think such an alternative prelude could be developed right now, it does not need anyone’s approval.

---

<div class="post-metadata">

**Author:** ![ezyang](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/ezyang/32/1541_2.png) [@ezyang](https://discourse.haskell.org/u/ezyang)\
**Post date:** [May 20, 2021, 1:39am UTC](https://discourse.haskell.org/t/hf-tech-proposal-1-utf-8-encoded-text/2499/22 "2021-05-20T01:39:54Z")

</div>

Hi, one of the authors of Backpack here.

Why has Backpack not taken off? The biggest reason is no one is working full time on it. Since finishing my PhD, I’m being paid to work on a different open source project (PyTorch), and between that and raising a new baby, I don’t really have time beyond helping answer people’s questions about Backpack in threads like this 🙂

But let’s say that we did have someone pushing Backpack (something I’d love to see! I still believe in the project). What are the things they’d have to do first? Here’s my ordered list:

1. Implement Stack support for Backpack (this is the number one reason why regular libraries can’t easily go off and start using Backpack, unless it’s entirely internal use it locks you out of the Stackage ecosystem)
2. Introduce Backpack support to GHC’s build system (as the most important libraries you might want to parametrize are base which are built as part of GHC’s bootstrap process)
3. Fix some of the major outstanding Backpack bugs; specifically [https://github.com/haskell/cabal/issues/6835](https://github.com/haskell/cabal/issues/6835) is the most important one, [https://github.com/haskell/cabal/issues/5434](https://github.com/haskell/cabal/issues/5434) is not too bad but pretty annoying

These are pretty hefty projects. But if you want to replace String in base, I don’t really see any other route!

---

<div class="post-metadata">

**Author:** ![ketzacoatl](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/ketzacoatl/32/1203_2.png) [@ketzacoatl](https://discourse.haskell.org/u/ketzacoatl)\
**Post date:** [May 20, 2021, 2:04am UTC](https://discourse.haskell.org/t/hf-tech-proposal-1-utf-8-encoded-text/2499/23 "2021-05-20T02:04:43Z")

</div>

Thank you for the response @ezyang!

> [@ezyang](#):
>
> But if you want to replace String in base, I don’t really see any other route!

Could you explain why this is the _only_ route worth considering? Or what about the alternatives falls short and is unacceptable?

---

<div class="post-metadata">

**Author:** ![ezyang](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/ezyang/32/1541_2.png) [@ezyang](https://discourse.haskell.org/u/ezyang)\
**Post date:** [May 20, 2021, 2:01pm UTC](https://discourse.haskell.org/t/hf-tech-proposal-1-utf-8-encoded-text/2499/24 "2021-05-20T14:01:51Z")

</div>

To me, changing String in base is subject to the following constraints:

1. Existing Haskell code must not break; at the very least, you have to go through a deprecation cycle where both the old and new forms are available (but personally, I think that String-based base will always have a place in the language)
2. There must be exactly one canonical implementation of base; it’s not acceptable to copy paste base into base-string and base-text because now all further development on base has to be replicated in two locations  
This puts you squarely in the space of Backpack-like solutions.

---

<div class="post-metadata">

**Author:** ![blamario](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/blamario/32/860_2.png) [@blamario](https://discourse.haskell.org/u/blamario)\
**Post date:** [May 21, 2021, 5:42pm UTC](https://discourse.haskell.org/t/hf-tech-proposal-1-utf-8-encoded-text/2499/25 "2021-05-21T17:42:02Z")

</div>

> This puts you squarely in the space of Backpack-like solutions.

That’s true only if you define “Backpack-like” in a tautological way. Here’s an alternative route to dropping String, which to me doesn’t look much like Backpack:

1. Add more type classes to `base` in the vein of `Semigroup`, `Monoid`, and `IsString`. It could be something in the vein of `StringLike`, my own `TextualMonoid`, or `MonoTraversable`. Whatever the classes are, `String` and `Text` would be their instances.
2. Extend the `default` declaration to default the new classes to `String`.
3. Abstract the types of all `base` functions that currently operate on `String` to accept any `StringLike` instance. Make sure the performance doesn’t suffer in concrete applications.
4. Extend the `default` declaration so that `Text` can override `String` on user demand.
5. Wait for the abstraction to propagate outward. Fix performance regressions as they happen.
6. Switch to `Text` by default in `base`.

The benefit of this approach is that it remains squarely in the language and libraries. It doesn’t depend on an extra-linguistic package-level mechanism (i.e., Backpack), so the end result is less complex. The process to get that end result may be more painful, and performance may never be quite as good.

---

<div class="post-metadata">

**Author:** ![jaror](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/jaror/32/3271_2.png) [@jaror](https://discourse.haskell.org/u/jaror)\
**Post date:** [May 21, 2021, 8:39pm UTC](https://discourse.haskell.org/t/hf-tech-proposal-1-utf-8-encoded-text/2499/26 "2021-05-21T20:39:36Z")

</div>

> [@blamario](#):
>
> It doesn’t depend on an extra-linguistic package-level mechanism (i.e., Backpack), so the end result is less complex.

Typeclasses also introduce complexity in their own way. They obscure error messages and are yet another hurdle for beginners pass when they start learning Haskell.

The complexity that Backpack adds will be for library writers, not for end-users. Libraries can expose their default APIs, e.g. String and Text, as normal fully instantiated Haskell packages.

---

<div class="post-metadata">

**Author:** ![danidiaz](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/danidiaz/32/92_2.png) [@danidiaz](https://discourse.haskell.org/u/danidiaz)\
**Post date:** [May 22, 2021, 6:14am UTC](https://discourse.haskell.org/t/hf-tech-proposal-1-utf-8-encoded-text/2499/27 "2021-05-22T06:14:16Z")

</div>

Having both module signatures and typeclasses does add complexity.

And yet, my feeling is that sometimes module signatures can be the better solution:

- For types that are not the “core” types of an API, but nevertheless used by most functions. Things like the strings used in error messages (possibly nested inside error wrapper types or exceptions). A typeclass solution would complicate the signatures of most functions for something that is a secondary detail.
- For types that are central to your API, but that aren’t likely to vary through a single client’s use of it. For example, parsers: in a given program, you aren’t likely to use more than one stream type with a parser library. Rather than typeclass-heavy signatures, clients would prefer to have functions already specialized so some stream type.

**base** fits those two cases in relation to String.

---

<div class="post-metadata">

**Author:** ![tomjaguarpaw](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/tomjaguarpaw/32/1230_2.png) [@tomjaguarpaw](https://discourse.haskell.org/u/tomjaguarpaw)\
**Post date:** [May 22, 2021, 6:41am UTC](https://discourse.haskell.org/t/hf-tech-proposal-1-utf-8-encoded-text/2499/28 "2021-05-22T06:41:04Z")

</div>

I don’t think you are contradicting @ezyang’s characterisation. Note in particuar point 1: existing Haskell code must break. A type class based solution will break existing code (consider `print (fromString "Hello")`). That’s not to say a type class based solution wouldn’t be the better solution, just that it wouldn’t match @ezyang’s requirements.

---

<div class="post-metadata">

**Author:** ![blamario](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/blamario/32/860_2.png) [@blamario](https://discourse.haskell.org/u/blamario)\
**Post date:** [May 22, 2021, 1:53pm UTC](https://discourse.haskell.org/t/hf-tech-proposal-1-utf-8-encoded-text/2499/29 "2021-05-22T13:53:03Z")

</div>

> [@jaror](#):
>
> Typeclasses also introduce complexity in their own way. They obscure error messages and are yet another hurdle for beginners pass when they start learning Haskell.

> [@danidiaz](#):
>
> - For types that are not the “core” types of an API, but nevertheless used by most functions. Things like the strings used in error messages (possibly nested inside error wrapper types or exceptions). A typeclass solution would complicate the signatures of most functions for something that is a secondary detail.

I agree, and this especially hurts because even beginners would run into it. The reason type signatures would be simpler with Backpack is exactly because Backpack is outside the type system, which is why I call it extra-linguistic. I suspect this means that _advanced_ users will find it constraining, but that may be a good trade-off versus scaring off beginners with complicated type signatures.

> [@danidiaz](#):
>
> - For types that are central to your API, but that aren’t likely to vary through a single client’s use of it. For example, parsers: in a given program, you aren’t likely to use more than one stream type with a parser library. Rather than typeclass-heavy signatures, clients would prefer to have functions already specialized so some stream type.

Majority of parser libraries in use have an abstract input type, I have written some myself. Even the venerable `parsec` has an input type parameter. I think the historical evidence is pretty clear in this case.

> [@tomjaguarpaw](#):
>
> I don’t think you are contradicting @ezyang’s characterisation. Note in particuar point 1: existing Haskell code must break. A type class based solution will break existing code (consider `print (fromString "Hello")`).

That’s exactly the reason I added the `default` extensions to my list, so the existing code doesn’t break.

As a general clarification to no one in particular, the reason I pointed out the typeclass-based solution roadmap was simply to push back on the claim that Backpack (or “Backpack-like”) is the _only_ way forward. It’s not. It may very well be the _best_ way forward, but I’m not qualified to judge that.

---

<div class="post-metadata">

**Author:** ![tomjaguarpaw](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/tomjaguarpaw/32/1230_2.png) [@tomjaguarpaw](https://discourse.haskell.org/u/tomjaguarpaw)\
**Post date:** [May 22, 2021, 7:59pm UTC](https://discourse.haskell.org/t/hf-tech-proposal-1-utf-8-encoded-text/2499/30 "2021-05-22T19:59:42Z")

</div>

> That’s exactly the reason I added the default extensions to my list, so the existing code doesn’t break.

Ah interesting! I’d be interested in seeing a worked example of how this can be made backwards compatible.

---

<div class="post-metadata">

**Author:** ![jhenahan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/jhenahan/32/3578_2.png) [@jhenahan](https://discourse.haskell.org/u/jhenahan)\
**Post date:** [May 22, 2021, 10:14pm UTC](https://discourse.haskell.org/t/hf-tech-proposal-1-utf-8-encoded-text/2499/31 "2021-05-22T22:14:38Z")

</div>

> [@ezyang](#):
>
> Introduce Backpack support to GHC’s build system (as the most important libraries you might want to parametrize are base which are built as part of GHC’s bootstrap process)

Is there anywhere I can read about what’s required for this point? I’m interested in putting some work into this, but I don’t know enough about the intersection of Backpack and building GHC to know what’s actually missing here. I don’t use Stack, so I’d probably do more harm than good on that point, but I’d love to help make Backpack applicable to `base`.

---

<div class="post-metadata">

**Author:** ![blamario](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/blamario/32/860_2.png) [@blamario](https://discourse.haskell.org/u/blamario)\
**Post date:** [May 22, 2021, 11:38pm UTC](https://discourse.haskell.org/t/hf-tech-proposal-1-utf-8-encoded-text/2499/32 "2021-05-22T23:38:43Z")

</div>

> [@tomjaguarpaw](#):
>
> Ah interesting! I’d be interested in seeing a worked example of how this can be made backwards compatible.

Actually your example already works even with `OverloadedStrings`, try it:

```
{-# Language OverloadedStrings #-}
import Data.String
main = print (fromString "Hello, World!")

```

The reason it works is because the `OverloadedStrings` doesn’t merely generalize the type of string literals. It also adds a built-in `default` declaration that makes all ambiguous types default to `String` when possible.

The simplest imaginable way this transition from `String` ends up happening would be by enabling `OverloadedStrings` extension and flipping the switch in that built-in `default` rule from `String` to `Text`. Of course this would be horribly irresponsible, and I’m not advocating for it. What I’m advocating for is exposing this built-in `default` rule and putting it in users’ hands, so they can switch away from `String` at their own pace. I put together a [GHC proposal](https://github.com/ghc-proposals/ghc-proposals/pull/409) to that effect.

---

<div class="post-metadata">

**Author:** ![jaror](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/jaror/32/3271_2.png) [@jaror](https://discourse.haskell.org/u/jaror)\
**Post date:** [May 23, 2021, 8:26am UTC](https://discourse.haskell.org/t/hf-tech-proposal-1-utf-8-encoded-text/2499/33 "2021-05-23T08:26:23Z")

</div>

I think a straightforward first step is to make a `base-indef` that uses the signature from `backpack-str` instead of `String` everywhere.

I would also add proper integration with Hackage to the list of things that need to be done. There was [a post recently on reddit](https://old.reddit.com/r/haskell/comments/ngxyz5/raaz_030_released/) where the author of the Raaz cryptography package ran into issues with backpack on Hackage.

---

<div class="post-metadata">

**Author:** ![tomjaguarpaw](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/tomjaguarpaw/32/1230_2.png) [@tomjaguarpaw](https://discourse.haskell.org/u/tomjaguarpaw)\
**Post date:** [May 23, 2021, 8:39am UTC](https://discourse.haskell.org/t/hf-tech-proposal-1-utf-8-encoded-text/2499/34 "2021-05-23T08:39:40Z")

</div>

I guess I should have tested my example 😃 How about the below?

Basically, I never use `OverloadedStrings` because every time I do my code becomes full of ambiguous type variable errors. If your GHC proposal can resolve that problem and base can be polymorphic in string type without breaking existing code then that’s great!

```haskell
{-# Language OverloadedStrings #-}
{-# Language FlexibleInstances #-}
{-# Options_Ghc -Wall #-}

class Foo a where foo :: a -> ()
instance Foo [Char] where foo = const ()

main :: IO ()
main = print (foo "Hello, World!")

```

```haskell
    • Ambiguous type variable ‘a0’ arising from a use of ‘foo’
      prevents the constraint ‘(Foo a0)’ from being solved.
...

```

---

<div class="post-metadata">

**Author:** ![blamario](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/blamario/32/860_2.png) [@blamario](https://discourse.haskell.org/u/blamario)\
**Post date:** [May 23, 2021, 2:50pm UTC](https://discourse.haskell.org/t/hf-tech-proposal-1-utf-8-encoded-text/2499/35 "2021-05-23T14:50:06Z")

</div>

> [@tomjaguarpaw](#):
>
> I guess I should have tested my example 😃 How about the below?

You found one of the holes I’m trying to patch. The trouble with the built-in `default` that comes with `OverloadedStrings` is that it resolves only the ambiguities that involve the `IsString` class. It doesn’t do anything for your `Foo`. To resolve that, I proposed the extension `NamedDefaults` so you could declare

```
default Foo ([Char])

```

to resolve the ambiguities involving `Foo`. That’s not the end of it though, because if `Foo` is declared in a library module we need a mechanism to export the `default` declarations alongside the class.

---

<div class="post-metadata">

**Author:** ![jaror](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/jaror/32/3271_2.png) [@jaror](https://discourse.haskell.org/u/jaror)\
**Post date:** [May 23, 2021, 3:23pm UTC](https://discourse.haskell.org/t/hf-tech-proposal-1-utf-8-encoded-text/2499/36 "2021-05-23T15:23:52Z")

</div>

Doesn’t that then require adding `default` declarations to every class?

You will also need `default` declarations that feel wrong to me, like `default Foldable ([])` for this code:

```haskell
{-# LANGUAGE OverloadedStrings #-}
main = putStrLn (foldr (:) [] "Hello, World!")

```

Edit: I was a bit confused, this problem above is not about breaking existing code but rather about writing new code with `OverloadedStrings`. I did find an example that does break if `base` gets overloaded, namely:

```haskell
{-# LANGUAGE FlexibleInstances #-}

class Foo a where
  foo :: a
instance Foo [Char] where
  foo = "Hello, World"

main = putStrLn foo

```

This code will be broken if it is not changed and `putStrLn` gets an overloaded type like `putStrLn :: StringLike a => a -> IO ()`. Even if there is a `default StringLike (String)` declaration exported by the prelude.

---

<div class="post-metadata">

**Author:** ![blamario](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/blamario/32/860_2.png) [@blamario](https://discourse.haskell.org/u/blamario)\
**Post date:** [May 23, 2021, 5:59pm UTC](https://discourse.haskell.org/t/hf-tech-proposal-1-utf-8-encoded-text/2499/37 "2021-05-23T17:59:05Z")

</div>

[quote="jaror, post:36, topic:2499]  
Doesn’t that then require adding `default` declarations to every class?  
[/quote]

Most of them I’m afraid. Not quite every class, `Functor` and `Semigroup` for example would be fine.

Your first example already demonstrates a compatibility break, but I don’t think it’s not as bad as what we already went through with FTP. Come to think of it, a `default Foldable ([])` declaration would have come useful back then.

EDIT: On the bright side, the FTP transition is a proof that ambiguous type errors are not the end of the world. So we don’t really need to find and `default` every possible class, only the commonly used ones.

---

<div class="post-metadata">

**Author:** ![danidiaz](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/danidiaz/32/92_2.png) [@danidiaz](https://discourse.haskell.org/u/danidiaz)\
**Post date:** [May 24, 2021, 11:04am UTC](https://discourse.haskell.org/t/hf-tech-proposal-1-utf-8-encoded-text/2499/38 "2021-05-24T11:04:16Z")

</div>

How would the typeclass defaulting mechanism handle datatypes in in `base` that have `String` fields? For example many of the exceptions from [`Control.Exception`](http://hackage.haskell.org/package/base-4.15.0.0/docs/Control-Exception.html), like [`ErrorCall`](http://hackage.haskell.org/package/base-4.15.0.0/docs/Control-Exception.html#t:ErrorCall).

---

<div class="post-metadata">

**Author:** ![blamario](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/blamario/32/860_2.png) [@blamario](https://discourse.haskell.org/u/blamario)\
**Post date:** [May 25, 2021, 12:00am UTC](https://discourse.haskell.org/t/hf-tech-proposal-1-utf-8-encoded-text/2499/39 "2021-05-25T00:00:08Z")

</div>

Not too well, to be honest. If one were determined to completely eradicate `String` from `base`, I suppose one could replace every `String` field with an existential type. But that would probably bring more problems than it would solve. Personally I’d rather leave `ErrorCall` and `error :: String -> a` alone, they’re not suffering from any performance issues.

---

<div class="post-metadata">

**Author:** ![ketzacoatl](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/ketzacoatl/32/1203_2.png) [@ketzacoatl](https://discourse.haskell.org/u/ketzacoatl)\
**Post date:** [May 25, 2021, 2:00am UTC](https://discourse.haskell.org/t/hf-tech-proposal-1-utf-8-encoded-text/2499/40 "2021-05-25T02:00:13Z")

</div>

This is interesting, but also a bit out of my ability/experience (apologies if my question / suggestion is naive or ridiculous), however I am wondering why the “simplest” case I can see doesn’t seem to fit: while not fancy, couldn’t we add what we need to base for the migration, provide the types/functions to make the conversion graceful, and update as we go? Some packages will have versions where a change is breaking and requires an update to downstream code. We have deprecation cycles for this reason.

AFAICT, it’s ok to break base in this way, as it’s part of addressing the wart, and GHC seems to do this. Is it unreasonable to skip the typeclass and polymophism and follow a more simple and direct approach?

[Previous page](https://discourse.haskell.org/t/hf-tech-proposal-1-utf-8-encoded-text/2499.md?page=1)

[Next page](https://discourse.haskell.org/t/hf-tech-proposal-1-utf-8-encoded-text/2499.md?page=3)