# What could replace ByteString when reading from the network/filesystem?

**URL:** <https://discourse.haskell.org/t/what-could-replace-bytestring-when-reading-from-the-network-filesystem/12661>\
**Category:** Uncategorized\
**Created:** [August 6, 2025, 11:20am UTC](https://discourse.haskell.org/t/what-could-replace-bytestring-when-reading-from-the-network-filesystem/12661 "2025-08-06T11:20:48Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kleidukos](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/kleidukos/32/1213_2.png) [@Kleidukos](https://discourse.haskell.org/u/Kleidukos)\
**Post date:** [August 6, 2025, 11:20am UTC](https://discourse.haskell.org/t/what-could-replace-bytestring-when-reading-from-the-network-filesystem/12661/1 "2025-08-06T11:20:48Z")

</div>

In this 2022 [comment](https://discourse.haskell.org/t/base-proposal-around-vector-like-types/2112/63) I highlighted the various responsibilities of `ByteString` today:

It is currently doing three things at once:

- Container for data that crosses the FFI boundaries (needs to be pinned and kept alive so that the GC doesn’t bugger off with your data in the middle of a safe FFI call.)

- String literal (where [`fromString`](https://hackage-content.haskell.org/package/base-4.19.2.0/docs/Data-String.html#v:fromString "Data.String") truncates multi-byte characters to octets. That job is best left to `Text` or `OsString`)

- Binary blob read from the file system or the network, which can be handled freely by the GC.

Since `ByteString` is backed by a `ForeingPtr` with finalisers, I imagine this makes it more suited for holding FFI data.

So, can an unpinned [ByteArray](https://hackage.haskell.org/package/base-4.21.0.0/docs/Data-Array-Byte.html#t:ByteArray) be used a replacement “destination” type when reading data from the outside and letting the GC handle things? After all, it does back `Text`, so it does seem suitable for Haskell-side data manipulation. Not being pinned avoid memory fragmentation as well, which is pretty nice.

---

<div class="post-metadata">

**Author:** ![BurningWitness](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/burningwitness/32/3665_2.png) [@BurningWitness](https://discourse.haskell.org/u/BurningWitness)\
**Post date:** [August 6, 2025, 12:24pm UTC](https://discourse.haskell.org/t/what-could-replace-bytestring-when-reading-from-the-network-filesystem/12661/2 "2025-08-06T12:24:12Z")

</div>

> [@Kleidukos](#):
>
> Container for data that crosses the FFI boundaries

If you’re writing code so low-level that pinnedness matters, you should just use a `ByteArray`.[`primitive`](https://hackage-content.haskell.org/package/primitive) is a must-have for this, I will accept no substitutes (although I do feel like it’s a bit too typeclass-heavy).

> [@Kleidukos](#):
>
> Binary blob read from the file system or the network

As I see it that’s what a strict `ByteString` originally represents, a single chunk of data with an undefined encoding. (and this is why `IsString ByteString` truncates characters to bytes)

> [@Kleidukos](#):
>
> String literal (… That job is best left to `Text`)

I don’t like the `text` package. Examples:

- I’m writing bindings for a C API and I know that some endpoint returns a UTF-8 (or ASCII) string. I should be able to pack that string into a plain UTF-8 byte array in a single copy operation. Currently I have to either decode a `ByteString` to `Text` (linear waste) or repackage using `Text` internals (which have no obligation to uphold the PVP). Also in this particular case it’d be nice if there was no slice information stored, à la `ShortByteString`;
- I have a library that returns arbitrary errors, always in UTF-8. The fastest way to build a string is with `Data.ByteString.Builder`, but it doesn’t have an encoding (except `IsString Builder` does, it’s UTF-8, funny how that works). I’m forced to either use `Data.Text.Lazy.Builder` (not interchangeable, far worse API), or `String` (has its own pitfalls);
- I want to be extremely pedantic with my file name handling. User, on Linux, gives me a raw file name (`OsString`), I encode it using the current locale with errors encoded into UTF-8 surrogates (`decodeFS`), I… manually check if there are surrogates present in the resulting `String` and then if I wish to store it in `Text` it will implicitly wipe said surrogates. UTF-8 with surrogates is a special encoding that would make this both type-safe and fast, but there’s no way to represent it without writing a whole separate library.

I don’t think encodings should be superglued to types. The ecosystem should have generic string types (`Short`/`Slice`/`Stream Short`/`Builder`/`Rope`), and then have separate libraries for each encoding. Ideally you’d also have some way to hide encoding types so that you can talk about `Short {UTF8}` without having to mention `UTF8` constantly (same with byte array pinnedness).

---

<div class="post-metadata">

**Author:** ![bmobn00b](https://avatars.discourse-cdn.com/v4/letter/b/ecccb3/32.png) [@bmobn00b](https://discourse.haskell.org/u/bmobn00b)\
**Post date:** [August 6, 2025, 1:59pm UTC](https://discourse.haskell.org/t/what-could-replace-bytestring-when-reading-from-the-network-filesystem/12661/3 "2025-08-06T13:59:05Z")

</div>

I think [Andrew Martin](https://github.com/andrewthad) who associated with [Layer 3 Communications](https://github.com/layer-3-communications) built a whole ecosystem around ByteArray

---

<div class="post-metadata">

**Author:** ![Swordlash](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/swordlash/32/4604_2.png) [@Swordlash](https://discourse.haskell.org/u/Swordlash)\
**Post date:** [August 7, 2025, 3:46pm UTC](https://discourse.haskell.org/t/what-could-replace-bytestring-when-reading-from-the-network-filesystem/12661/4 "2025-08-07T15:46:39Z")

</div>

I think the answer is yes. There is an unpinned version in `bytestring` - called [ShortByteString](https://hackage-content.haskell.org/package/bytestring-0.12.2.0/docs/Data-ByteString-Short.html#t:ShortByteString) - which is actually a newtype over `ByteArray`.

---

<div class="post-metadata">

**Author:** ![viktor](https://avatars.discourse-cdn.com/v4/letter/v/d6d6ee/32.png) [@viktor](https://discourse.haskell.org/u/viktor)\
**Post date:** [August 11, 2025, 12:18pm UTC](https://discourse.haskell.org/t/what-could-replace-bytestring-when-reading-from-the-network-filesystem/12661/5 "2025-08-11T12:18:49Z")

</div>

The `ShortByteString` objects from the `bytestring` package are precisely the (not necessarily or by default pinned) blobs of binary data that support a very similar API.

The main limitations with `ShortByteString` are:

- There is no chunked variant, but various uses of Lazy bytestrings are in any case better served by monadic byte streams.
- There is no support for `Builders`, precisely because builders leverage pinned memory to support efficient incremental construction (to chunked lazy bytestrings).

Otherwise `ShortByteString` is internally a `ByteArray`, but have a familiar richer API. It is easy to convert between `ByteArray` and `ShortByteString` via the `SBS` pattern synonym.

As for the `IsString` instance, though criticised by some, it does support bona-fide octet string constants, such as SMTP verbts (“EHLO”, “MAIL FROM:”, “RCPT TO:”, “DATA”) or even sample binary data. For truly stripped-down compile-time read-only binary data one can also get away with just Addr# literals, and be duly warned about the 8-bit byte restriction:

```haskell
λ> :set -XMagicHash
λ> import qualified Data.ByteString as BS
λ> import qualified Data.ByteString.Internal as BS
λ> mylit = BS.unsafePackLiteral "foobar"#
λ> BS.length mylit
6
λ> show mylit
"\"foobar\""
λ> mylit2 = BS.unsafePackLiteral "Виктор"#
<interactive>:7:31: error: [GHC-43080]
    primitive string literal must contain only characters <= '\xFF'

```

But see also `unsafePackLenLiteral` if the octet string needs to hold NUL bytes (0-valued octets):

```haskell
λ> mylit1 = BS.unsafePackLiteral "foo\0bar"#
λ> mylit2 = BS.unsafePackLenLiteral 7 "foo\0bar"#
λ> map BS.length $ [mylit1, mylit2]
[3,7]
λ> map (flip BS.indexMaybe 3) $ [mylit1, mylit2]
[Nothing,Just 0]

```

---

<div class="post-metadata">

**Author:** ![Jappie](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/jappie/32/5394_2.png) [@Jappie](https://discourse.haskell.org/u/Jappie)\
**Post date:** [August 12, 2025, 11:27am UTC](https://discourse.haskell.org/t/what-could-replace-bytestring-when-reading-from-the-network-filesystem/12661/6 "2025-08-12T11:27:35Z")

</div>

> There is no support for `Builders`, precisely because builders leverage pinned memory to support efficient incremental construction (to chunked lazy bytestrings).

maybe coulde make one of those linear builders for ShortByteString then  
for example: [text-builder-linear: Builder for Text and ByteString based on linear types](https://hackage.haskell.org/package/text-builder-linear)

---

<div class="post-metadata">

**Author:** ![viktor](https://avatars.discourse-cdn.com/v4/letter/v/d6d6ee/32.png) [@viktor](https://discourse.haskell.org/u/viktor)\
**Post date:** [August 13, 2025, 7:32am UTC](https://discourse.haskell.org/t/what-could-replace-bytestring-when-reading-from-the-network-filesystem/12661/7 "2025-08-13T07:32:20Z")

</div>

I should perhaps have said that, more precisely, what a `Builder` outputs (directly) is a lazy `ByteString`, rather than `ShortByteString`. One can of course take that lazy `ByteString` (with its pinned chunks) and copy it to a `ShortByteString`, releasing the intermediate storage if the lazy `ByteString` is then no longer needed.

On the input side, a `Builder` can consume data from all sorts of serialisable objects, including `ShortByteString`s for which a [combinator](https://hackage-content.haskell.org/package/bytestring-0.12.2.0/docs/Data-ByteString-Builder.html#v:shortByteString) already exists.

Returning to your point, sure if @bodigrim (a maintainer of both `bytestring` and the package you highlighted) is suitably motivated, the new linear builders based on `Buffer` could then be used to construct a `ShortByteString` “directly”, if there advantages in doing so over use of lazy `ByteStrings` as pinned chunked memory during construction.

---

<div class="post-metadata">

**Author:** ![viktor](https://avatars.discourse-cdn.com/v4/letter/v/d6d6ee/32.png) [@viktor](https://discourse.haskell.org/u/viktor)\
**Post date:** [August 14, 2025, 5:21pm UTC](https://discourse.haskell.org/t/what-could-replace-bytestring-when-reading-from-the-network-filesystem/12661/8 "2025-08-14T17:21:18Z")

</div>

Meanwhile, a more disciplined (than directly working with `Addr#` literals) construct for compile-time ByteString literals can be something like the below (for demo purposes stashed away in `.../Data/ByteString/Literal.hs`):

```haskell
module Data.ByteString.Literal (byteStringLiteral) where
import qualified Data.ByteString.Char8 as B
import qualified Language.Haskell.TH.Syntax as TH

byteStringLiteral :: (MonadFail m, TH.Quote m) => String -> TH.Code m B.ByteString
byteStringLiteral s = TH.liftCode $ fmap TH.TExp $
    if all (<= '\xff') s
    then TH.lift (B.pack s)
    else fail $ "Input code points exceed 0xFF in: " ++ show s

```

which then makes possible:

```haskell
$ cat test.hs
module Foo where
import Data.ByteString.Literal

$ ghci -v0 -XTemplateHaskell -ddump-splices -i... test.hs
λ> import qualified Data.ByteString as B
λ> 
λ> bs = $$(byteStringLiteral "foo\0bar\xA0\&baz")
<interactive>:3:8-46: Splicing expression
    byteStringLiteral "foo\0bar\xA0\&baz"
  ======>
    bytestring-0.12.2.0-inplace:Data.ByteString.Internal.Type.unsafePackLenLiteral
      11 "foo\NULbar\\160baz"#
λ> 
λ> B.unpack bs
[102,111,111,0,98,97,114,160,98,97,122]
λ> 
λ> bs = $$(byteStringLiteral "foo\0βαρ")
<interactive>:5:8: error: [GHC-39584]
    • Input code points exceed 0xFF in: "foo\NUL\946\945\961"
    • In the Template Haskell splice $$(byteStringLiteral "foo\0βαρ")
      In the expression: $$(byteStringLiteral "foo\0βαρ")
      In an equation for ‘bs’: bs = $$(byteStringLiteral "foo\0βαρ")

```

---

<div class="post-metadata">

**Author:** ![vshabanov](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/vshabanov/32/2080_2.png) [@vshabanov](https://discourse.haskell.org/u/vshabanov)\
**Post date:** [August 15, 2025, 1:33am UTC](https://discourse.haskell.org/t/what-could-replace-bytestring-when-reading-from-the-network-filesystem/12661/9 "2025-08-15T01:33:14Z")

</div>

> [@Kleidukos](#):
>
> So, can an unpinned [ByteArray](https://hackage.haskell.org/package/base-4.21.0.0/docs/Data-Array-Byte.html#t:ByteArray) be used a replacement “destination” type when reading data from the outside and letting the GC handle things? After all, it does back `Text`, so it does seem suitable for Haskell-side data manipulation. Not being pinned avoid memory fragmentation as well, which is pretty nice.

Large objects are [pinned](https://well-typed.com/blog/2018/05/ghc-special-gc-objects/) too (as constantly copying them on every major collection is not cheap). So if you create a large enough (about 4k or more) `ByteArray` from a `ByteString`, you will get another pinned object and an extra memory copy.

The fragmentation can easily be solved by using chunks of the same size. Lazy `ByteString` uses ~32kB chunks. Builders do the same (they accumulate data into chunks of fixed size). If the goal is to reduce fragmentation then it’s already achieved by lazy `ByteString`s and builders.

Since OS I/O functions require valid memory buffers there’s no way not to use pinned memory. If there’s a need to store a subset of the data in a long-running program then I’m afraid there’s no other way as to create a `Text` or `ShortByteString` from a `ByteString` and make sure it’s forced, so it only contains a copy and doesn’t keep a pointer to the whole original `ByteString`.

---

<div class="post-metadata">

**Author:** ![hasufell](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/hasufell/32/1250_2.png) [@hasufell](https://discourse.haskell.org/u/hasufell)\
**Post date:** [August 15, 2025, 2:09am UTC](https://discourse.haskell.org/t/what-could-replace-bytestring-when-reading-from-the-network-filesystem/12661/10 "2025-08-15T02:09:55Z")

</div>

> [@viktor](#):
>
> There is no chunked variant

There is, but not in bytestring: [Data.Bytes.Chunks](https://hackage.haskell.org/package/byteslice-0.2.13.2/docs/Data-Bytes-Chunks.html)

---

<div class="post-metadata">

**Author:** ![TeofilC](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/teofilc/32/3987_2.png) [@TeofilC](https://discourse.haskell.org/u/TeofilC)\
**Post date:** [August 15, 2025, 8:33am UTC](https://discourse.haskell.org/t/what-could-replace-bytestring-when-reading-from-the-network-filesystem/12661/11 "2025-08-15T08:33:51Z")

</div>

> [@vshabanov](#):
>
> The fragmentation can easily be solved by using chunks of the same size.

Indeed this can help with fragmentation, but it’s not always possible or desirable to do this. Often it’s also very difficult to enforce. The 32KB chunk size is just a default in bytestring and it’s very easy to, for instance, append two chunks without thinking.

These large chunks are also often too big, so you still need to copy it into something smaller, and if that’s pinned then you have the issue again.

---

<div class="post-metadata">

**Author:** ![BurningWitness](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/burningwitness/32/3665_2.png) [@BurningWitness](https://discourse.haskell.org/u/BurningWitness)\
**Post date:** [August 15, 2025, 9:17am UTC](https://discourse.haskell.org/t/what-could-replace-bytestring-when-reading-from-the-network-filesystem/12661/12 "2025-08-15T09:17:34Z")

</div>

> [@TeofilC](#):
>
> Indeed [using chunks of the same size] can help with fragmentation, but it’s not always possible or desirable to do this. Often it’s also very difficult to enforce. The 32KB chunk size is just a default in bytestring and it’s very easy to, for instance, append two chunks without thinking.

Yeah, that’s because `LazyByteString` is just `[StrictByteString]`, it’s a terrible format for anything except streaming from a socket or flattening a `Builder` to (stream to a socket). Best you can do to dissuade people from doing complex text manipulation with it is to make it clear that it’s a list and to offer a better suited type in a nearby module (e.g. a rope).

---

<div class="post-metadata">

**Author:** ![vshabanov](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/vshabanov/32/2080_2.png) [@vshabanov](https://discourse.haskell.org/u/vshabanov)\
**Post date:** [August 15, 2025, 12:07pm UTC](https://discourse.haskell.org/t/what-could-replace-bytestring-when-reading-from-the-network-filesystem/12661/13 "2025-08-15T12:07:03Z")

</div>

> [@TeofilC](#):
>
> The 32KB chunk size is just a default in bytestring and it’s very easy to, for instance, append two chunks without thinking.

Appending combines two lists, not chunks.

But yes, it’s very easy to create small bytestrings and get fragmentation issues.

Unfortunately, I/O is a complex task that cannot be solved simply and efficiently in a general way. One needs to know several things to design an efficient system for each use case:

- I/O buffers are pinned.
- Large `Text` or `ShortByteString` are also pinned.
- Long-lived pinned data allocated at random points in time lead to fragmentation, but long-lived movable data slows down every major GC.
- Chunks of the same size help a lot with fragmentation.
- `Builder`s try to keep chunks of the same size inside, but big `ByteString`s could be added as is.
- If `Text`/`ShortByteString` is not forced it will still contain a closure with a reference to the original large pinned buffer, but closures (or other slice data structures) can use less space than the copied string and be more performant.
- A memory copy is not cheap comparing to modern I/O speeds (easily 1-2GB/s per device). It’s ~20GB/s if not in the cache; much slower if not sequential; much faster if in the cache.

Different tasks have different trade-offs and the solution depends on the [data](https://www.youtube.com/watch?v=rX0ItVEVjHc) you’re processing.

The simple solution of always copying pinned inputs to unpinned memory can lead to:

- extra memory copying
- extra load on every major GC to move the data
- potentially the same pinned array
- potentially greater memory usage if only a small slice of the input is used

It might be fine in many use cases but I wouldn’t add it as the default approach to basic libraries.

---

<div class="post-metadata">

**Author:** ![BurningWitness](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/burningwitness/32/3665_2.png) [@BurningWitness](https://discourse.haskell.org/u/BurningWitness)\
**Post date:** [August 15, 2025, 2:54pm UTC](https://discourse.haskell.org/t/what-could-replace-bytestring-when-reading-from-the-network-filesystem/12661/14 "2025-08-15T14:54:31Z")

</div>

> [@vshabanov](#):
>
> - I/O buffers are pinned.
> - Large `Text` or `ShortByteString` are also pinned

These two aren’t the same kind of pinned though.

A `StrictByteString` can be constructed with externally allocated memory, so it can be used over any pointer you want as long as you manage memory lifetimes properly. I personally haven’t found any use for it, but there may be some unique cases where this is desirable.

A `ByteArray` on the other hand is always safe, and if you need to copy from it you can [check if it’s pinned](https://hackage-content.haskell.org/package/primitive-0.9.1.0/docs/Data-Primitive-ByteArray.html#v:isByteArrayPinned) upfront.

> [@Kleidukos](#):
>
> Since `ByteString` is backed by a `ForeignPtr` with finalisers, I imagine this makes it more suited for holding FFI data.

Not really, C libraries I’ve used always juggle the following three approaches:

- C string size is known upfront, so allocation is user’s responsibility ([fread](https://en.cppreference.com/w/c/io/fread), [ov\_read](https://xiph.org/vorbis/doc/vorbisfile/ov_read.html), [spng\_decoded\_image\_size](https://libspng.org/docs/decode/#spng_decoded_image_size)). This seems to always be when reading or parsing something, so the resulting arrays are large and uniform in size, but you generally feed these somewhere else after instead of keeping them, so I don’t think `malloc` would do anything here;

- C string lifetime is tied to some object ([glfwGetMonitorName](https://www.glfw.org/docs/3.3/group__monitor.html#ga7af83e13489d90379588fb331b9e4b68), strings in [FT\_FaceRec](https://freetype.org/freetype2/docs/reference/ft2-face_creation.html#ft_facerec), [VkPhysicalDeviceProperties.deviceName](https://registry.khronos.org/vulkan/specs/latest/man/html/VkPhysicalDeviceProperties.html)). You may be safe to consume the string on the spot if your application is singlethreaded, but in general using `ForeignPtr`s here is shooting yourself in the foot;

- C string is baked in at compilation time (pretty much just version strings and error messages). These are safe to refer to by the pointer, but they’re also tiny, so it doesn’t matter either way.

As I see it `ForeignPtr` representation is really just there to allow me to use `malloc` instead of a pinned `ByteArray`, and if the difference between the two ever starts to matter I probably have far bigger issues than long garbage collection times.