Botan bindings devlog

181

The first issue to solve that the low-level libraries are dependent on bytestring which isn’t the worst, but requires awkward / inefficient copying / marshalling between and adhering to the restrictions of garbage-collected bytestrings

Please don’t invent a new type to do this instead.

If you do, the end result will be way worse than the current situation, because in order to do anything with the bytes that pass though botan people will have to first convert from a bytestring, then convert back to a bytestring, because everything else uses bytestrings.

3 Likes
182

I don’t want to introduce or force a new type, I’m rather actually hoping to avoid it (and instead merely enable alternatives), hence the need for some low-level memory work eg to typeclassify allocation and byte-addressable things and make instances for ByteString so that they just work.

However the awkward and inconvenient truth is that ByteString still is extremely unsuitable for sensitive cryptographic operations because its lifetime and secure erasure cannot be controlled, and as an added malus, when binding to C libraries you have a high chance of having to do your own allocation which Haskell mostly avoids / does with a ByteString by secretly / unsafely yoinking out its Ptr to operate on it, relying on GHC to clean it up - wholly unsuitable.

Note: This is why the ScrubbedBytes data exists in memory

So to cast this in a different light, I am merely trying to codify how we allocate and use the underlying pointers, of which garbage-collected bytestrings are certainly the most common (and thus highly desirable) interface.

Additional clarification:

any […] bytes that pass though […] will have to first convert from a bytestring, then convert back to a bytestring, because everything else uses bytestrings

I want to specifically recognize this, and point out that only the Haskell side of ‘everything else’ uses ByteStrings - anything interacting with foreign C libraries is already not a bytestring, and specifically one of my concerns is avoiding unnecessary conversions from and to bytestrings which are an additional operation from the perspective of the foreign C library or primitive code.

That we can rather-unsafely expose a garbage-collected, pinned ByteString’s buffer to a C API is a statement that a ByteString can act as a memory buffer, not that all memory buffers should be garbage-collected pinned ByteStrings.

In a more succinct manner, ByteStrings are defined by an already-existing low-level memory management / lifetime, and should not be used to define low-level memory management. They just make a nice wrapper, that gets in the way at this lower level.

1 Like