# Base proposal around vector-like types

**URL:** <https://discourse.haskell.org/t/base-proposal-around-vector-like-types/2112>\
**Category:** Haskell Foundation\
**Created:** [March 16, 2021, 7:09pm UTC](https://discourse.haskell.org/t/base-proposal-around-vector-like-types/2112 "2021-03-16T19:09:24Z")\
**Posts on this page:** 20\
**Page:** 3

<div class="post-metadata">

**Author:** ![adamse](https://avatars.discourse-cdn.com/v4/letter/a/f0a364/32.png) [@adamse](https://discourse.haskell.org/u/adamse)\
**Post date:** [April 1, 2021, 11:16am UTC](https://discourse.haskell.org/t/base-proposal-around-vector-like-types/2112/41 "2021-04-01T11:16:55Z")

</div>

With current ByteString it is possible to have it backed by a `mmap`ed file, this seems like a good feature. Would we lose this if `ByteString` was moved to use something that is not `ForeignPtr`?

---

<div class="post-metadata">

**Author:** ![snoyberg](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/snoyberg/32/190_2.png) [@snoyberg](https://discourse.haskell.org/u/snoyberg)\
**Post date:** [April 1, 2021, 1:35pm UTC](https://discourse.haskell.org/t/base-proposal-around-vector-like-types/2112/42 "2021-04-01T13:35:25Z")

</div>

`mmap`ing came up in the discussions and @bgamari seemed to think it would be supported. But I don’t know enough of the details of how it will work.

---

<div class="post-metadata">

**Author:** ![bgamari](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/bgamari/32/3599_2.png) [@bgamari](https://discourse.haskell.org/u/bgamari)\
**Post date:** [April 1, 2021, 6:20pm UTC](https://discourse.haskell.org/t/base-proposal-around-vector-like-types/2112/43 "2021-04-01T18:20:23Z")

</div>

Sigh, I actually realized that I may have missed something in my discussion with @snoyberg. In particular, I was thinking that we would take advantage of @dcoutts’s [off-heap `ByteArray#` support](https://gitlab.haskell.org/ghc/ghc/-/merge_requests/2561) to facilitate `mmap`'d `ByteArray#`s. However, this could be a bit tricky to accomplish since we would need to ensure that the `ByteArray#` header would need to precede the mapping. It’s not immediately clear to me how we can guarantee this in general. It’s possible that @dcoutts had something in mind here.

---

<div class="post-metadata">

**Author:** ![adamse](https://avatars.discourse-cdn.com/v4/letter/a/f0a364/32.png) [@adamse](https://discourse.haskell.org/u/adamse)\
**Post date:** [April 3, 2021, 6:38am UTC](https://discourse.haskell.org/t/base-proposal-around-vector-like-types/2112/44 "2021-04-03T06:38:48Z")

</div>

The recent updates to the [ticket](https://gitlab.haskell.org/ghc/ghc/-/issues/17747) seem to address this. It looks like it would work to me :)!

---

<div class="post-metadata">

**Author:** ![fede](https://avatars.discourse-cdn.com/v4/letter/f/9fc348/32.png) [@fede](https://discourse.haskell.org/u/fede)\
**Post date:** [April 3, 2021, 9:58am UTC](https://discourse.haskell.org/t/base-proposal-around-vector-like-types/2112/45 "2021-04-03T09:58:19Z")

</div>

Improving the situation with vectors is great. I would like to mention that in scientific applications peoples like to work with multidimensional arrays. It would be great if the new vector would be suitable to perform fast operations on multidimensional arrays.

---

<div class="post-metadata">

**Author:** ![daylily](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/daylily/32/1915_2.png) [@daylily](https://discourse.haskell.org/u/daylily)\
**Post date:** [April 2, 2022, 3:22am UTC](https://discourse.haskell.org/t/base-proposal-around-vector-like-types/2112/46 "2022-04-02T03:22:45Z")

</div>

It’s been a year since this got posted (and Text is already UTF-8), I’m curious if there is any proposal as of now? Really interested to see this get implemented.

From what I read from Michael’s posts, what we hope to do is basically add a vector type to `primitive`, and move ByteString to be implemented on that (then maybe move Text to be implemented on ByteString). That looks like a very clear goal. Maybe I can try to help with that.

One question I have is: how will `vector` fit in? I do not use it frequently (it takes a decade to compile and most of the times either `primitive` or `containers` work better for me), but I have a vague idea that it tries to do data storage and streaming at the same time. The vector type being added to `primitive` could completely replace data storage function of `vector`, and there are other (probably better) streaming libraries too. Is it possible that `vector` will be eventually phased out?

---

<div class="post-metadata">

**Author:** ![daylily](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/daylily/32/1915_2.png) [@daylily](https://discourse.haskell.org/u/daylily)\
**Post date:** [April 2, 2022, 6:16am UTC](https://discourse.haskell.org/t/base-proposal-around-vector-like-types/2112/47 "2022-04-02T06:16:00Z")

</div>

I just found out there is a library `contiguous` that abstracts the array types from `primitive` quite nicely: [contiguous: Unified interface for primitive arrays](https://hackage.haskell.org/package/contiguous)

- It has a class `Contiguous` That unifies most array operations
- It has a type `Slice` that provides basic vector functionalities for all `Contiguous` Types.

---

<div class="post-metadata">

**Author:** ![Bodigrim](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/bodigrim/32/1457_2.png) [@Bodigrim](https://discourse.haskell.org/u/Bodigrim)\
**Post date:** [April 2, 2022, 5:12pm UTC](https://discourse.haskell.org/t/base-proposal-around-vector-like-types/2112/48 "2022-04-02T17:12:50Z")

</div>

Not gonna happen. ByteString is to remain backed by ForeignPtr, and Text does not benefit from unification with vectors, because there are few common operations.

---

<div class="post-metadata">

**Author:** ![jaror](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/jaror/32/3271_2.png) [@jaror](https://discourse.haskell.org/u/jaror)\
**Post date:** [April 3, 2022, 1:52pm UTC](https://discourse.haskell.org/t/base-proposal-around-vector-like-types/2112/49 "2022-04-03T13:52:21Z")

</div>

Most of the API seems to overlap, but only if you can have vectors with variable size elements. Is that really impossible to overcome? I think the [`Contiguous`](https://hackage.haskell.org/package/contiguous-0.6.1.1/docs/Data-Primitive-Contiguous.html#t:Contiguous) class is general enough for that.

Also, text (and `String`/`Char`) uses unicode code points, but I would like to see a library that uses (extended) grapheme clusters. I think code points are a useless middle ground.

---

<div class="post-metadata">

**Author:** ![Ericson2314](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/ericson2314/32/1026_2.png) [@Ericson2314](https://discourse.haskell.org/u/Ericson2314)\
**Post date:** [April 3, 2022, 4:55pm UTC](https://discourse.haskell.org/t/base-proposal-around-vector-like-types/2112/50 "2022-04-03T16:55:40Z")

</div>

Has that been elaborated upon in a blog post? It would be great to have a fully laid out "challenges unifying `vector` and `bytestring`" because I have no idea what they are.

Conversely, I do miss all the container interfaces of Java, and I wonder if we should lean on something like [mono-traversable](https://hackage.haskell.org/package/mono-traversable) (or just optics?!?) more in the Prelude so that people feel less compelled to to make choices like `Map` vs `HashMap` or `Bytestring` vs `Vector Word8` when it doesn’t in fact matter to what they are doing.

---

<div class="post-metadata">

**Author:** ![Bodigrim](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/bodigrim/32/1457_2.png) [@Bodigrim](https://discourse.haskell.org/u/Bodigrim)\
**Post date:** [April 3, 2022, 7:54pm UTC](https://discourse.haskell.org/t/base-proposal-around-vector-like-types/2112/51 "2022-04-03T19:54:48Z")

</div>

> [@jaror](#):
>
> Most of the API seems to overlap, but only if you can have vectors with variable size elements. Is that really impossible to overcome? I think the [`Contiguous`](https://hackage.haskell.org/package/contiguous-0.6.1.1/docs/Data-Primitive-Contiguous.html#t:Contiguous) class is general enough for that.

“Vector with variable size elements” is an oxymoron. If you read closer, the only thing Michael proposes to unify is `Data.Text.Array` and `Data.Primitive.ByteArray`, which from my perspective is of limited utility and not what people usually assume by unification of `Text` and `Vector`.

> [@Ericson2314](#):
>
> Has that been elaborated upon in a blog post? It would be great to have a fully laid out “challenges unifying `vector` and `bytestring`” because I have no idea what they are.

The challenges are the same as for unifying lists and trees, apples and oranges: `ByteString` and `Vector` are vastly different data types for different purposes. If `ByteString` does not serve well your use case, take `Vector` instead, problem solved.

---

<div class="post-metadata">

**Author:** ![Ericson2314](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/ericson2314/32/1026_2.png) [@Ericson2314](https://discourse.haskell.org/u/Ericson2314)\
**Post date:** [April 3, 2022, 8:12pm UTC](https://discourse.haskell.org/t/base-proposal-around-vector-like-types/2112/52 "2022-04-03T20:12:33Z")

</div>

> The challenges are the same as for unifying lists and trees, apples and oranges: `ByteString` and `Vector` are vastly different data types for different purposes. If `ByteString` does not serve well your use case, take `Vector` instead, problem solved.

Well in that case I think the “apple and orange”-ness is far less well known then!

---

<div class="post-metadata">

**Author:** ![Bodigrim](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/bodigrim/32/1457_2.png) [@Bodigrim](https://discourse.haskell.org/u/Bodigrim)\
**Post date:** [April 3, 2022, 8:27pm UTC](https://discourse.haskell.org/t/base-proposal-around-vector-like-types/2112/53 "2022-04-03T20:27:50Z")

</div>

> [@Ericson2314](#):
>
> Well in that case I think the “apple and orange”-ness is far less well known then!

You do not ask to explain why lists and trees are different, right? The default is that since `ByteString` and `Vector` do not share a single line of code, they are different, and GHC is happy to check this automatically. If you wish to argue otherwise, the burden of proof is on you, not on me.

---

<div class="post-metadata">

**Author:** ![jaror](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/jaror/32/3271_2.png) [@jaror](https://discourse.haskell.org/u/jaror)\
**Post date:** [April 3, 2022, 8:50pm UTC](https://discourse.haskell.org/t/base-proposal-around-vector-like-types/2112/54 "2022-04-03T20:50:11Z")

</div>

> [@Ericson2314](#):
>
> It would be great to have a fully laid out “challenges unifying `vector` and `bytestring`” because I have no idea what they are.

As mentioned in the blog post that started this thread, the main difference between the representation of `bytestring` and the representation of unboxed vectors of `Word8`s is that bytestring is pinned to make it easy to pass them to foreign C functions (such as printing functions) while unboxed vectors are unpinned which means they play better with the garbage collector.

I remember reading/hearing recently that large unboxed vectors are already pinned, so that isn’t a problem and small vector can easily be copied without too much overhead. So, I think with that strategy bytestring could switch over to a `ByteArray#` backed (vector-like) data type.

---

<div class="post-metadata">

**Author:** ![Ericson2314](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/ericson2314/32/1026_2.png) [@Ericson2314](https://discourse.haskell.org/u/Ericson2314)\
**Post date:** [April 3, 2022, 9:11pm UTC](https://discourse.haskell.org/t/base-proposal-around-vector-like-types/2112/55 "2022-04-03T21:11:41Z")

</div>

@Bodigrim thats an odd argument to me. People write code which overlaps in purpose all the time. @jaror mentioned the original blog post, which comes to the opposite conclusion that the differences are superficial, and by being able to just mutate whether memory is pinned, we could erase the final non-trivial distinction.

All this just makes me want a blog post retort more!

---

<div class="post-metadata">

**Author:** ![Bodigrim](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/bodigrim/32/1457_2.png) [@Bodigrim](https://discourse.haskell.org/u/Bodigrim)\
**Post date:** [April 3, 2022, 9:20pm UTC](https://discourse.haskell.org/t/base-proposal-around-vector-like-types/2112/56 "2022-04-03T21:20:22Z")

</div>

> [@jaror](#):
>
> the main difference between the representation of `bytestring` and the representation of unboxed vectors of `Word8`s is that bytestring is pinned to make it easy to pass them to foreign C functions (such as printing functions) while unboxed vectors are unpinned which means they play better with the garbage collector.

The main difference is that `ByteString` is a foreign pointer (and pinnedness is just a consequence), while `Vector Word8` is a `ByteArray#` heap object (which could be pinned or unpinned). Yet again, I must emphasize that these are just different data types serving different use cases, same as lists and trees. Why people feel an insistent urge to force their unification at cost of both maintainers and users is beyond my comprehension.

---

<div class="post-metadata">

**Author:** ![jaror](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/jaror/32/3271_2.png) [@jaror](https://discourse.haskell.org/u/jaror)\
**Post date:** [April 3, 2022, 9:48pm UTC](https://discourse.haskell.org/t/base-proposal-around-vector-like-types/2112/57 "2022-04-03T21:48:24Z")

</div>

> [@Bodigrim](#):
>
> Why people feel an insistent urge to force their unification at cost of both maintainers and users is beyond my comprehension.

As I see it, maintainers’ jobs would get easier if these types are unified because they would have to maintain less code. And the main reason for unification is to reduce the amount of choice for users and present one solution that fits all their needs.

What is the benefit of using a foreign pointer if it is not pinnedness? Which use-cases require foreign pointers that cannot also use “normal” `ByteArray#`s? I don’t see why `ByteString` has to be inherently “foreign”.

---

<div class="post-metadata">

**Author:** ![Ericson2314](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/ericson2314/32/1026_2.png) [@Ericson2314](https://discourse.haskell.org/u/Ericson2314)\
**Post date:** [April 3, 2022, 9:58pm UTC](https://discourse.haskell.org/t/base-proposal-around-vector-like-types/2112/58 "2022-04-03T21:58:36Z")

</div>

Well let’s really drill down here.

Lists vs trees is about an intrinsic property of the data structure. Here, however, the intrinsic properties are roughly the same. What is varying instead how how the memory is _managed_. In a language like C++ or rust this is closer to the allocator parameter on `std::vector`/`Vec`.

I did some spelunking because I do not know the details:

1. [bytestring/Data/ByteString/Internal.hs at 707e50bac1f1ef7c1729a9246e05f95a4909fd27 · haskell/bytestring · GitHub](https://github.com/haskell/bytestring/blob/707e50bac1f1ef7c1729a9246e05f95a4909fd27/Data/ByteString/Internal.hs#L644) allocates bytes on master today

2. [`GHC.ForeignPtr.mallocPlainForeignPtrBytes `](https://hackage.haskell.org/package/base-4.15.0.0/docs/GHC-ForeignPtr.html#v:mallocPlainForeignPtrBytes) is what it’s a wrapper around.

3. [`newPinnedByteArray#`](https://hackage.haskell.org/package/ghc-prim-0.8.0/docs/GHC-Prim.html#v:newPinnedByteArray-35-) is the underlying primop its the docs say

🤔

Meanwhile…

1. [vector/vector/src/Data/Vector/Primitive/Mutable.hs at dae9d1739f760027100458928ab96515ecb57cdf · haskell/vector · GitHub](https://github.com/haskell/vector/blob/dae9d1739f760027100458928ab96515ecb57cdf/vector/src/Data/Vector/Primitive/Mutable.hs#L138) Internal module `Data.Vector.Primitive.Mutable` implements `basicUnsafeNew` with `Data.Primitive.ByteArray.newByteArray`.

2. [`Data.Primitive.ByteArray.newByteArray`](https://hackage.haskell.org/package/primitive-0.7.3.0/docs/Data-Primitive-ByteArray.html#v:newByteArray) uses `newByteArray#`

So yes, while using `ForeignPtr` _could_ be intersteresting allowing one to _transfer ownership_ to a foreign library that would deallocate the bytestring, in practice we are actually just getting two marginally different GHC-managed heap objects after all!

@Bodigrim I am sorry but it feels like you boldly asserted something, which sent me on this wild goose chase chasing variable bindings, just to figure out the thing the original blog post from @snoyman casually asserted without much detail is in fact exactly true.

---

<div class="post-metadata">

**Author:** ![Bodigrim](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/bodigrim/32/1457_2.png) [@Bodigrim](https://discourse.haskell.org/u/Bodigrim)\
**Post date:** [April 3, 2022, 10:05pm UTC](https://discourse.haskell.org/t/base-proposal-around-vector-like-types/2112/59 "2022-04-03T22:05:09Z")

</div>

@Ericson2314 it seems you assume that all `ByteString`s are created via `mallocByteString`. That’s not so, you can use `Data.ByteString.Internal.fromForeignPtr` with **any** `ForeignPtr` instead.

---

<div class="post-metadata">

**Author:** ![Ericson2314](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/ericson2314/32/1026_2.png) [@Ericson2314](https://discourse.haskell.org/u/Ericson2314)\
**Post date:** [April 3, 2022, 10:21pm UTC](https://discourse.haskell.org/t/base-proposal-around-vector-like-types/2112/60 "2022-04-03T22:21:10Z")

</div>

OK! Then the last thing I wrote is wrong, and there are yet more subtleties. But then I switch back to arguing merely that all this is quite subtle, far from obvious, and deserving a nice blog post!

It is also interesting to me that `ForeignPtr` is such a highly flexible data type in many respects, but only supports pinned memory. That makes sense for something called `ForeignPtr` — and mere pinning is enough to safely make an `Addr#`. But the use-cases of extendable flat memory include the “pin it sometimes not others” usecase @snoyberg mentions, which still feels natural and obvious, and is a dynamism `ForeignPtr` doesn’t support.

I still feel what we have here is a tower of somewhat awkward abstractions, totally justified for some things, but used in other ways that are far less justified. The accumulation of stuff here, and end result of `Vector Word8` vs `ByteString` _does_ feel like an accident of history, and not something I feel like justifying to new users!

I really do want to clean all this stuff up, from the lowest level RTS assumption of pinness being something that doesn’t vary over the lifetime of a heap object, to the highest level bits of making a Vector/ByteString replacement that is parametric over these details.

For me, this sort of “pull things apart, shake them until they make sense” is why I like breaking up base, though I understand why other people agree with these problems, see the coordination failures, and draw the 180° opposite conclusion that we should throw it all in one monorepo to sort it out.

[Previous page](https://discourse.haskell.org/t/base-proposal-around-vector-like-types/2112.md?page=2)

[Next page](https://discourse.haskell.org/t/base-proposal-around-vector-like-types/2112.md?page=4)