Botan bindings devlog

[ApothecaLabs]

Update: Unit Tests and Pubkey fixes

Once again, the repo has been updated, this time with a good number of things.

• RNG unit tests
• Improvements to SRP6
  • Added synonyms for all the bytestring arguments
  • Fixed srp6ClientAgree
• SRP6 unit tests
  • Documented function srp6_group_identifier missing from Botan FFI
• TOTP unit tests
• Improvements to Utility
  • Replaced size calculations with pointer querying to fix InsufficientBufferSpace exception
• Utility unit tests
• Improvements to PubKey
  • Found authoritative source of algorithm names and parameters
  • Fixed GOST-34.10
  • Removed X25519 (redundant synonym for Curve25519)
  • Renamed awkward flag data types
• PubKey unit tests
• PubKey.Encrypt unit tests
• PubKey.Decrypt unit tests
• PubKey.Sign unit tests
• PubKey.Verify unit tests
• Improvements to PubKey.Sign
  • Fixed signFinish to use allocBytesQuerying
  • Fixed signFinish to use upper bound and then trim
• Found Botan’s test vectors in the C++ source

Almost all of the basic unit tests are done now, and though there are algorithm combos that fail, things are more or less passing in general. There are still some PubKey functions that need testing:

• Key agreement
• Key encapsulation
• Algorithm-specific key loading functions

X509 certificates also don’t have unit tests yet, but I already know I’m going to need to take a deeper look at them so I’m holding off on that and considering it to be a larger, but important issue.

The big thing today, aside from all of the unit tests, is that pubkey encryption, decryption, signing, verification are now all working for all algorithms, though a bit fiddly because of parameters.

Signing was actually slightly broken, and was in some cases producing signatures that would fail verification. A closer look and some experimentation showed that sizes weren’t correct, which was slightly unexpected behavior given that many other functions allow you to query the size pointer by providing a null pointer for input.

In retrospect I realized that the size was non-deterministic due to the random generator. Querying once and running it again with the queried size would actually result in a different random number being generated internally, and thus the occasional InsufficientBufferSpace exception - which is why I couldn’t use querying for these functions (or any other function with non-deterministic buffers, really).

Armed with this knowledge, I figured out that I needed to both poke a size upper bound into a particular pointer, and read its actual size afterwards, and that pretty much got it working properly. Viola!

The Proposal Has been Approved for Recommendation

In some good news that actually happened last Friday, the TWG committee has voted to approve the proposal for recommendation to the Haskell Foundation.

This is a recommendation and not a commitment, and the Foundation still has to accept (or reject) it, but they are meeting Thursday, and this item is on the agenda! I obviously have high hopes, but I’ve made it this far with everyone’s help and feedback, and either way I’m happy to have had your support.

'Til next time!

[ApothecaLabs]

Unit tests for botan-low minimally complete (minus X509)

It has been a busy few days, and I am happy to state today that unit tests for botan-low are minimally complete! I divvied up up the remaining unit tests into similar groups, and managed to get through them all, including a few fixes for key agreement and key encapsulation, which were the final pubkey operations.

• Added unit tests for privKeyGetField and pubKeyGetField in Botan.Low.PubKeySpec
• Added unit tests for PubKey.DH, DSA, ElGamal, and RSA
• Added unit tests for PubKey.ECDH, ECDSA, and SM2
• Added unit tests for PubKey.Ed25519, X25519
• Added fixes and unit tests for PubKey.KeyAgreement
• Added fixes and unit tests for PubKey.KeyEncapsulation

Admittedly, there’s still lots of testing improvements to be made, but every primitive should be passing tests for at least one algorithm, and most of them for several or all. Its awkward to use, but it all seems to be working, aside from a few specific things that need a more dedicated focus (which we are getting closer to now that these tests are done).

After writing these tests, I am also thinking about making a change in responsibilities regarding botan-low vs botan - making botan-low be more responsible for all of the algorithm data types. It’s hard to get 100% algorithm combination testing coverage using only string data types, because not everything is compatible, and the tests help us tell what is - so algorithm ADTs are almost necessary to do better testing.

This would be a slight departure from the proposal already, but not a great one, and I believe the proposal process acknowledges this sort of change anyhow. I’m sure it will all shake loose as things develop.

The repo has been updated.

---

Also I just noticed I goofed on a commit message as I pasted the previous message for format but forgot to edit it - the second commit entitled Added unit tests for PubKey DH, DSA, ElGamal, and RSA should be entitled Added unit tests for PubKey ECDH, ECDSA, and SM2 - pay this no mind

[arybczak]

I am also thinking about making a change in responsibilities regarding botan-low vs botan - making botan-low be more responsible for all of the algorithm data types.

For the record, I like that botan-low API currently simply mimics botan’s FFI API by accepting strings and I think it should stay this way for mirroring and flexibility purposes.

Higher level API with ADTs should be a layer over it.

[Ambrose]

100%. C embeds in Haskell pretty much perfectly (except CBV structs ), so verbatim C bindings are crucial.

I even just call the module C. Like SDL.GPU.C.

Than the layer on top that abstracts over memory allocation and bracketing and other idioms, I call Simple.

[ApothecaLabs]

In that case, rather than give more responsibility to botan-low, I need to take some away and give it to botan instead.

There is a power-responsibility mismatch as over the last few months, botan-low has accrued a few more responsibilities than I’d like (online vs offline processing, unit testing, size queries) that make it more than just managing buffers. I need the strong typing to be able to make some necessary distinctions between algorithms, but moving those items up into botan would suffice instead of moving the types down.

[ApothecaLabs]

An update: Starting work on X509 support

This is exciting! My first update after this project’s proposal was accepted for funding!

So what have I been up to? X509 certificates! Support for this functionality in the Botan C FFI is a little sparse, so I’ve been going over everything to see what’s what in particular. There’s actually quite a bit of functionality missing compared to the full Botan C++ library, and its a section of this project that I know I’ll be writing a little bit of C++ which may get contributed upstream. I’ve started adding lots of notes and data types, and X509 is probably going to be my primary focus for a bit while I regroup over some things and plan out some of the next month’s work.

• X509 basic unit tests
  • Testing against a single simple cert for now
  • CRL unit tests pending
• Stub data types for X509 Distinguished Names, CRL, Status codes, CA, CRL codes, CS

All in all though, the basic certificate loading and queries seem to be functional, but there is no support for Certificate Revocation Lists aside from checking against pre-existing CRLs that can’t be created or modified, and no support for Certificate Authorities or Stores - not in the FFI, which is why we’ll be getting our hands dirty with C++, to add to the C FFI.

The good thing is that what’s there does work, and that’s nice to check off the list. I’ve been digging through the Botan C++ source to see what else is needed - identifying gaps to fill. TLS is probably also going to need similar work, but I’ve been preparing for this

That’s it for the moment - 'til next time!

[ApothecaLabs]

Update: Lots of C++

For the last few days, I’ve been working on extending Botan’s FFI to include X509 Certificate Authorities and Certificate Signing Requests. I’m a bit rusty with the C++, but I’m shaking off that rust quite nicely. Modern C++ has some nice things going for it, if you aren’t burdened with a legacy codebase.

I’ve hit a (temporary) wall with what I can achieve in my attempts to extend Botan’s FFI by adding C++ to the Haskell project directly. Now, this isn’t really bad, as it is something I very much expected, but the practical result is that I just need to spend a bit of time setting up build flags and things earlier than expected.

The problem is that Botan does not expose the proper headers to access the Botan_FFI C++ namespace, and as a result, I cannot obtain a reference to the encapsulated C++ objects which I need in order to pass it to the C++ functions that I am newly wrapping / calling.

For example, I have a Haskell PrivKeyPtr / botan_privkey_t and need to access the Botan::Private_Key underneath. I can see botan_privkey_t exposed in the FFI headers, and I have access to Botan::Private_Key via the public C++ headers, but I can only translate between the two if I have access to Botan’s internal Botan_FFI headers, which I don’t have.

-- Haskell
foreign import ccall unsafe hs_botan_x509_ca_create
    :: Ptr X509CAPtr
    -> X509CertPtr
    -> PrivKeyPtr
    -> Ptr CChar
    -> RNGPtr
    -> IO BotanErrorCode

// C FFI
int hs_botan_x509_ca_create(
    hs_botan_x509_ca_t* ca,
    botan_x509_cert_t cert,
    botan_privkey_t key,
    const char* hash_fn,
    botan_rng_t rng);

// C++
X509_CA X509_CA(
    const X509_Certificate &cert,
    const Private_Key &key,
    const std::string &hash_fn,
    RandomNumberGenerator &rng);

Now, this doesn’t stop me from implementing new C structs for wrapping C++ objects, nor does it stop me from using the old ones, but the issue is I can’t mix the two - the opaque botan_struct pointers remain opaque, despite the functions that need that access.

I could just re-implement botan_privkey_t as hs_botan_privkey_t and then be access its Botan::Private_Key for use with hs_botan_x509_ca_create / X509_CA(...), but then I can’t use that new privkey struct in the old functions. I’d have to rewrite the entire FFI from scratch, or at the very least clone the entire botan/ffi directory into cbits which I do not want to do.

The only real option is to fork Botan and extend the FFI from inside Botan C++. However, now you see why this is not really a problem - we were planning on contributing these improvements back upstream anyway, its just that we have to get set up to do that now. Oh bother

As a result of this, I am gating the extended FFI behind the XFFI flag. Anyone who wants to test the experimental FFI can instead clone the experimental botan-upstream fork fork, build and install it from source, and use the XFFI flag to enable the experimental FFI modules. There are some instructions in the README, and it looks mostly like this:

# Clone
git clone https://github.com/apotheca/botan-upstream $BOTAN_CPP

# Build and install C++
cd $BOTAN_CPP
./configure.py --prefix=$BOTAN_OUT
make
make install

# Play around with it
cd $BOTAN_HASKELL
cabal repl botan-low -fXFFI --extra-lib-dirs=$BOTAN_OUT/lib --extra-include-dirs=$BOTAN_OUT/include

I’m still setting up everything, but I will be moving the C++ FFI code from cbits to the botan-upstream fork. This leaves regular users unaffected, and you can continue to use a regular Botan install via your favorite package manager / distribution if you wish.

For the moment the botan-upstream fork is unmodified, so there’s not much point to it yet, but that will be changing quite soon.

That’s all for now!

[ApothecaLabs]

Great forking success!

So I’ve been doing a lot more tinkering with C++ for the last few days, and have gotten everything set up with the new botan-upstream fork The extended FFI is working, the new C structs now really wrap their C++ counterparts, and the new bindings can all find their symbols which is great. Plus, contributing back upstream to C++ Botan is just a pull request away now, too.

Botan’s FFI tools are reasonably nice to work with, even though they aren’t exposed externally. They have macros and defines for simple tasks, and for safely ‘visiting’ C++ -land from C, and so a lot of it boils down to safely casting between C and C++ as long as the arguments line up. I still have some questions, but for the most part Botan’s FFI follows a consistent formula, which I’ve been able to apply to the new structs and functions.

This has made for some pretty good progress, despite my not having touched C++ for the better part of a decade - I’ve even gotten as far as implementing botan_x509_ca_create, botan_x509_ca_create_padding, and botan_x509_ca_destroy, though it is untested for the moment until I build Botan.Low.X509.CA

Now, it may feel a bit odd to be gallivanting off to C++ -land when there’s still a lot to do in Haskell, but the C++ work to implement better X509 support was both a high-value target, as well as one of the larger “I know this is doable, but I don’t know how hard” tasks. Now I know roughly how hard it will be (and its better to know this early), and I must say I was pleasantly surprised to find that it was less effort than expected so far - I have worked with far worse codebases.

All in all, I’m pretty satisfied with how its going

[ApothecaLabs]

Update: X509 data types, stub functions, and bindings

This update has been building for a few days now, and I’m glad to get it out. You see, you pull on one thread, and things start to unravel - and I can see why the original FFI authors didn’t want to implement more extensive X509 support.

Basic X509 functionality does exist, but is almost entirely limited to loading read-only objects, and I want to be able to use this library to create, sign, revoke certificates, encode / decode PEM and BER/DER formatted objects - so that means getting my hands dirty and doing it myself

It is a bit tedious: to enable one functionality you must implement this other function which requires these data types in order to create these other ones - and and so on until you’re basically forced to implement everything, together. Luckily, that’s our explicit goal, so its no skin off our back!

I’m mostly mirroring the existing C++ API, and after soaking my brain in the X509 C++ source for a few days, I’ve written C FFI data types and function stubs for almost all of the X509 data structures. When I started, the existing X509 data types were just basic read-only Certificates and CRLs - now, we have data types for:

• Distinguished Names
• Extensions
• Certificates
• Certificate Authority
• Certificate Signing Requests
• Certificate Options
• Certificate Stores
• Path validation
• Supplemental data types

All of the things that you need to generate and sign new certificates! There still are some open questions (namely, returning arrays of things, and some questions of ownership), and I still need to do FFI types and function stubs OCSP support, and there’s some missing functions, but we’re getting there!

One large caveat, is that this is just the stub functions, and now I still have to go through and actually implement them all to call the C++ from C. There’s still a lot of work to do, but I’ve basically started in the middle, between the Haskell and the C++ - and with the bindings set, I can get to work on implementing the Haskell and the C++ together.

On the other hand, I’m basically defining the FFI types, and that’s one of the most important parts of programming. Plus, all of this X509 work is effectively doubling the size of the Botan C FFI (it now accounts for half of the FFI header file!) so I shouldn’t be surprised it’s taking a bit of time - I’ll bet the original FFI wasn’t built in a few weeks either!

These changes have been pushed to the repo and upstream fork.

---

If you want to check out this recent work you’ll have to clone the experimental botan-upstream fork, build and install it from source, and use the XFFI flag to enable the experimental FFI modules. There are some instructions in the README, or you can follow along here:

# Clone
git clone https://github.com/apotheca/botan-upstream $BOTAN_CPP

# Build and install C++
cd $BOTAN_CPP
./configure.py --prefix=$BOTAN_OUT
make
make install

# Play around with it
cd $BOTAN_HASKELL
cabal repl botan-low -fXFFI --extra-lib-dirs=$BOTAN_OUT/lib --extra-include-dirs=$BOTAN_OUT/include

That’s all for the moment! Next up, I’ll be fleshing out these function stubs to be vending actual objects that we can interact with!

[jackdk]

I might have missed it, so I’m sorry if you’ve already written this somewhere else, but: what’s the eventual plan for this fork? Are you going to be able to upstream these FFI changes?

[ApothecaLabs]

@jackdk

Are you going to be able to upstream these FFI changes?

Yes, I will (eventually) be submitting these FFI changes as a patch / pull request to the original Botan C++ library repo. It is an open issue that I am happy to contribute to.

[ApothecaLabs]

Update: X509 Cert Options and Cert Store implementations

Now that I’ve got the X509 FFI functions all declared, it’s time to write implementations, and that’s what today’s update is about. Certificate options and certificate stores are more or less first-draft implemented on the C++ side, with certificate signing requests and certificate authorities on the way next. This means that I am very close to being able to generate new X509 certificates through the FFI, which would be a great chunk of new functionality. If I can achieve this, it will be a meaningful contribution not just for the Haskell library but to the upstream C++ library as well.

So close!

Since this is C++, I’ve run into a few questions of string / object ownership and lifetime - it compiles, and the types are correct, but I’ve not really tested it yet. There’s also a few functions that return a shared_ptr, but the Botan FFI is designed for unique_ptr. In general, I’m not up-to-date on the move / copy / ownership semantics yet, and the heavy use of auto and implicit casting makes it difficult to track what is going on. Another issue is a question string encodings, and whether Botan supports null bytes in X509 Distinguished Names, part of the X509 spec which inherits a lot of data types from ASN1 (which does allow for null bytes). I’m just using cstrings for now, but may have to go back and change some things to be uint8_t arrays instead - no biggie if I do.

I plan on posting an update to Botan’s open issue for the X509 FFI asking for some guidance, and in the mean time I’m forging ahead and I’m hoping to have the rest of the X509 interface at least roughly-implemented by the end of the week, barring the open questions about ownership and lifetimes, and for those I will at least make ready to apply the answers for when I do get them.

The repo and upstream fork have been updated. Let’s keep this train rolling!

---

While I’m working on that, check out the impact that this work has had on my github contribution activity - it’s neat to look back, and see the history of the project:

I started this project back in July, and have managed to keep a consistent level of activity and momentum going ever since, making sure to get something meaningful done just about every day. Notably, the contributions to the botan-upstream fork do not appear here (yet) because it is a fork, but they will show if the fork’s changes are accepted in a PR - when that happens, this last month will be even greener

[ApothecaLabs]

Update: More X509 C++ and Haskell over the Holiday

It’s been a bit of a quiet week; I meant to give an update days ago, but the holiday left me worn out, it has been difficult for me to write, so this update is a bit unpolished. I think I am in need of a small break, so I will be taking it a bit easy as I prepare the coming first monthly status update, and I will be trying to maintain a shorter, more frequent update schedule.

X509 Cert Store Bindings and Functions, and More

First off, what have I accomplished since the last update?

• C++ Distinguished Name implementations
• C++ Certificate Extension function declarations and stubs
  • Still have some questions over how I’m going to implement this
• C++ Certificate Options seem to be working
• C++ Created Certificate_Store_In_SQL::find_key_unique, a unique_ptr version of find_key which produces a shared_ptr that we can’t use.
• C++ / Haskell Fixed int botan_x509_cert_store_find_all_certs return type
• C++ / Haskell X509 Path Validation names and types fixed
• Reorganized Haskell X509 types (now spread across multiple modules)
• Haskell Certificate Store bindings and implementations
• Haskell Certificate Signing Request implementations
• Haskell CRL Entry bindings and implementions
• Haskell Certificate Authority implementions
• Haskell Path validation implementions

I’ve spent a fair bit of time wrapping my head around STL pointer lifetimes, to which a friend has helpfully supplied the following ‘guide’ (I can’t tell if he is serious, or if it is in jest, or both):

What a monstrosity! C++ is taking a bit of a toll on me and I can feel my sanity slipping away…

Regardless, I’m having to do a bit more than just wrap functions at this point, I’m having to update or define new C++ functions to get the requisite access that I need (I can’t use shared_ptr objects due to the structure of the FFI, but there are still some functions using them so I have to write unique_ptr variants of them, as I haven’t yet determined if it is “safe” to relax the type constraint even though I suspect that it should auto-cast via move-constructors).

Aside from that, there’s a few functions that need to return jagged 2d arrays, which means an array of array sizes plus the actual size of the array, meaning you’re dealing with 3 pointer variables. Then there’s a few functions that return key-value pairs, which require 5 pointer variables (keys, key lengths, vals, val lengths, count).

Ignoring those two particular issues, I’m still missing Haskell bindings on some things (DN, Extensions), and about 30 or so missing C++ function implementations (Cert Authority, Cert Signing Request, CRL Entry, Cert Store) - otherwise things are coming along decently. Everything is at least declared and either bound or implemented in Haskell or C++, and having one side done makes it easier to fill in the other.

Some thoughts I have as I prepare the monthly update.

• X509 explodes off into a fractal of complicated dependencies: for example, it is reliant on ASN1, and implementing bindings to ASN1 that is far out of scope and I’d rather do that in Haskell anyway.
• Issues / questions over what encoding some things are in - in C++, string just means array of chars, are not cstrings, are not necessarily UTF8
• May ignore DN objects for now in favor of passing string-encoded subject and issuer distinguished names
• Extensions: singular object vs individual extension objects (which do have getters)
• I worry about the ‘query with a null array pointer and we’ll populate the size pointer’ method
  • Doesn’t scale to higher arrays (see issues with 2d jagged arrays)
  • Potential efficiency / non-determinism issues with generated results
  • Eg, the generate_crls method for sql cert stores
  • I would prefer an actual size query function.
  • May revisit this issue, but it applies to several
• Some things are unavailable to me (Sqlite3 store implementation) at this time pending investigation
• Learning some lessons on what I’d do differently if I were to start from scratch (but now is not the time for refactoring)

My biggest concern right now is that I know that the X509 stuff is still going to take continued effort, but I can’t let it take up the entire project’s focus. So, in order to make sure that I hit all of my deliverables in the next two months, I have to figure out a limit to what I’m going to focus on, and cut it off at a certain reasonable point.

Hence my thoughts are that the X509 interface I’ve so far declared is sufficient for now, and definitely an improvement over what was there already. I’ve done enough work on the X509 stuff to know that it should make it in as a deliverable, and that’s good. But it also illustrates why some high-value goals (like X509) are optional deliverables - at this point, it is also safe to say that the other optional deliverables may not make it this stage as I still have much to do.

That’s it for now - I’m preparing a more thorough update for next time to mark the end of the first month, see you then!

The repo and upstream fork have been updated.

[ApothecaLabs]

Yikes! It’s been a little while since I update the devlog; after posting the Monthly Status Report, I had some personally impactful events occur that necessitated my taking a few days off to process. Your patience is appreciated.

Tightening the X509 ship

The big thing of note this update is I’ve sort of worked out the boundaries of what is necessary to implement to get the features that I want. It is a question of functionality vs faithfulness to the full and complete X509 spec, and it can be hard to tell what is X509 spec, and what is opinionated Botan C++ - and Botan certainly has some opinions that don’t really cross the C boundary very well.

At some point it is more sensible to encode things into a known format and then parse it rather than trying to juggle an FFI; if we cross that point, time would then be better served spent on creating / modifying a native X509 implementation while using Botan to supply cryptographic primitives instead - this would mean integrating botan more tightly with / mutually reliant on things like asn1-encoding and crypton-x509 and crypton-x509-store and tls.

We aren’t at that point yet; the convenience functions in Botan serve to both limit our choices, but also make it easier to implement since there is a limit to what they expose. What we need is to be able to:

• Read and verify certificates
• Create self-signed certificates (certificate authority)
• Create and sign requests and new certificates using a certificate authority
• Create and manage certificate revocation lists
• Revoke or affirm certificates using revocation lists
• Read and write certificates and private keys to and from a certificate store

It turns out that we can get (or already have) quite a lot of this without directly implementing a few things:

• Distinguished names are fancy multimaps / key-value lists, and can be represented in an encoded string format rather than constructing a map object and passing it around. However, it is worth noting that there is logic attached to what keys and values are ‘valid’. Ultimately, it may eventually become useful to fully implement, but right now the cost/opportunity balance is against it.
• creating extensions is obviated by the ‘options’ struct, and by implication of supplied arguments in the wrapping functions - in other words, the extensions objects are not used directly but are instead created on-the-fly and immediately consumed. However, this makes querying the finished objects for extensions difficult - they are represented via an abstract class, and having a pointer to that is slightly complicated especially when we don’t know what subclass it is (I am looking into how private keys manage it). If I can get the abstract vs subclass stuff working, it may be worth implementing anyway, due to functions that allow us to query certificates and other objects for their related extensions
• Path validation has already been ‘handled’ by the botan_x509_cert_verify_with_crl which serves to encapsulate the necessary logic to call Botan::x509_path_validate and return the validity. Like distinguished names, it may be worth implementing fully eventually, but at the moment time is better spent elsewhere.

As a result, I may end up temporarily or permanently axing the X509 DN, Extensions, and Path modules. This shrinks our implementation surface a little, and allows us to keep focused on implementing what is necessary to get the desired functionality. So we don’t get everything we wanted, but its still a vast improvement on the basic X509 FFI that I started with, and we can focus on completeness later if that direction still favorable. It is okay to write code that you don’t end up using, it is a problem if you keep that code to your detriment.

This means we can focus on the remaining items for X509:

• PEM and DER encoding for X509 more objects
• Finishing the Cert Authority and new CRL function C++ implementations
• A bunch of small encoding things

---

Codewise, this update features new functions for creating, querying, and updating certificate revocation lists, which before could only be read from a file. They’re still in progress, but huzzah!

I also got the sqlite3 certificate store working; this necessitates configuring botan using the --with-sqlite3 flag, but it should be working now.

The repo and upstream fork have been updated.

[ApothecaLabs]

Update - Fresh New README

Today’s update is, I hope, rather refreshing. After staring a bit too long into the abyss that is C++, I decided to do some long-awaited spring cleaning of the repo, and perform some of the non-code-related tasks.

The big thing for this update is a complete rewrite of the project README. It is a vast improvement, so you should definitely check it out!

I took a few cues from this curated awesome-readme list, it was very helpful. I even managed to get navigation and collapsible sections working, so that it doesn’t get in its own way when scrolling through to where you want to read. There’s some minor formatting to be done on the collapsibles, but I’ll take care of it soon enough.

It is important to remember that non-coding tasks may be just as meaningful to a project as its coding tasks, and 10+ years of industry experience have taught me that a good README is essential - and I have allowed the README to lay fallow far too long. Good documentation doesn’t just happen, you have to make it happen.

Some of the (platform-specific) build instructions haven’t been tested yet, but now it is looking prim and proper, as befitting a well-organized project such as this. I have also split off crypto-schemes into its own repo, due to the narrowing of project scope. Don’t worry, it’s not going away - it’ll be back! I also cleaned up and removed a bunch of non-botan stuff from the repo.

Next up, I’m going to spend the next few days going over building on various platforms, and try to get the ball rolling on the CI stuff.

---

I finally got around and set up a Ko-Fi account for anyone who wishes to donate to help keep this project going, or to just say thank you. It really makes a difference to me and the longevity of this project.

[ApothecaLabs]

Yee-haw! Major botan-bindings CAPI update!

I am absolutely stoked to bring this update, as I’ve been working on it for the last week while checking over everything repeatedly. The major focus of this update is that we have upgraded botan-bindings from using ccall to using capi and the newer CApiFFI.

It is our first significant update to botan-bindings in quite a serious while (XFFI notwithstanding), and is made with the goal of stability, to freeze botan-bindings for its first release. In fact, pending a few things, in the next update (when I have fixed botan-low and merged CAPI-experiments back into main) will involve a version change from 0.0.1 to 0.0.2, and hopefully it will be stable enough to consider as a candidate for versioned release on hackage.

Effectively, this accepts @BurningWitness 's PR that I’ve been referencing for the last several days, though there are some significant tweaks and differences such that I merged by hand, so can’t actually merge the PR, but it saved me a lot of time and effort. Please thank them for their serious contribution to this project.

This means every module in botan-bindings has been somewhat re-written, albeit with the intended goal of requiring minimal changes downstream because the bound interfaces haven’t actually changed, just our way of wrapping them.

So, specifics:

I’ve upgraded from ccall to capi using the newer CApiFFI.

• This gives us a much tighter bindings to the ctypes using CTYPE
• We now get a warning for non-matching Haskell and C types
• The bindings are now safe by default, though I may need to revisit specific functions with callbacks in the future to make sure this is appropriate.
• I’ve renamed things to be more 1:1 consistent with Botan C FFI for predictability, and because we’ll start standardizing with idiomatic naming at higher levels.
• I’ve created a ConstPtr shim for base < 4.18
  • This does cause an issue for GHC 9.0.2

There is now a new method of defining Haskell C data types:

-- Old method
data FooStruct
type FooPtr = Ptr FooStruct

-- New method
data {-# CTYPE "botan/ffi.h" "struct botan_foo_struct" #-} BotanFooStruct
newtype {-# CTYPE "botan/ffi.h" "botan_foo_t" #-} BotanFoo
    = MkBotanFoo { runBotanFoo :: Ptr BotanFooStruct }
        deriving newtype (Eq, Ord, Storable)

The new method is great because I can generate bindings in botan-low now a la:

mkBindings
    ::  (Storable botan)
    =>  (Ptr struct -> botan)
    ->  (botan -> Ptr struct)
    ->  (ForeignPtr struct -> object)
    ->  (object -> ForeignPtr struct)
    ->  FinalizerPtr struct
    ->  (   botan -> IO object
        ,   object -> (botan -> IO a) -> IO a
        ,   object -> IO ()
        ,   (Ptr botan -> IO CInt) -> IO object
        )
mkBindings mkBotan runBotan mkForeign runForeign destroy = bindings where
    bindings = (newObject, withObject, objectDestroy, createObject)
    newObject botan = do
        foreignPtr <- newForeignPtr destroy (runBotan botan)
        return $ mkForeign foreignPtr
    withObject object f = withForeignPtr (runForeign object) (f . mkBotan)
    objectDestroy object = finalizeForeignPtr (runForeign object)
    createObject init = mask_ $ alloca $ \ outPtr -> do
        throwErrorIfNegative_ $ init outPtr
        out <- peek outPtr
        newObject out

I’ll be using this in botan-low as I update it in response to more consistently generate the foreign pointer wrappers and functions. Here’s an example implementation of RNG using it:

newtype RNG = MkRNG { getRNGForeignPtr :: ForeignPtr BotanRNGStruct }

newRNG      :: BotanRNG -> IO RNG
withRNG     :: RNG -> (BotanRNG -> IO a) -> IO a
rngDestroy  :: RNG -> IO ()
createRNG   :: (Ptr BotanRNG -> IO CInt) -> IO RNG
(newRNG, withRNG, rngDestroy, createRNG)
    = mkBindings MkBotanRNG runBotanRNG MkRNG getRNGForeignPtr botan_rng_destroy

It’s a big step towards achieving enough stability for a proper release. Unfortunately, all of this breaks botan-low for the moment, and so it exists in its own CAPI-experiments branch for now.

This is because there are now some issues with the older mkInit functions that I made and used back when FooPtr was a type instead of a newtype, and ConstPtr causes issues with existing functions that predate it but now require it due to updated bindings.

I’m working on fixing all of that next, while performing a similar ‘refactor-and-freeze’ on botan-low.

---

Update on CI work

Aside from all of that, there’s also @ocramz 's PR, which I think I am going to accept and then probably heavily edit. I don’t like including things into projects that I don’t understand or have a thorough grasp on, so I’ve been reading up heavily on Github CI to better understand their markup, because I’m almost certainly going to go with github actions / workflows for CI - the bar for entry is very low.

To that end, I’ve gathered the following options:

• get-tested (which ocramz PR used, and embeds into a workflow)
• haskell-ci (which generated a hard-to-read workflow)
• Writing workflows by hand

It turns out that I need some flexibility that I don’t know if haskell-ci affords (I need to be able to edit the workflow, and its a generator), so I ended up not considering it as an option.

On the other hand,get-tested does one thing simply and does it well - it generates the ghc-os test matrix for me, and the PR includes useful steps like caching that I knew I’d need to be using. However, I have other matrix needs - specifically, the library target also needs to be a part of the matrix, so I need more power than get-tested gives me.

In the end I’ll probably write the CI workflow by hand, as its only a few lines of code to write our own matrix, but it is quite fine for now, as I’ll need a few more days going over github CI workflows to produce something better.

---

Well, I’m a bit bushed after all of this. The next few days are going to be a bit quiet as I go visiting family, and I hope everyone enjoys the holidays too

[romes]

Great work Leo and collaborators! It’s exciting to see this work being done.

[Vlix]

Props to @BurningWitness and @ocramz for the PRs, and of course to @ApothecaLabs for the continued effort

[jackdk]

That’s a pretty great update. I maintain a small (nearly-irrelevant) package which does FFI bindings. I’m going to have to play with CApiFFI and those CTYPE annotations, and try and modernise it.

[ApothecaLabs]

Holiday Update: Fixing botan-low for the new CAPI changes

The last update was a good one, but it broke a few things in botan-low - hence why I’ve been working off of the CAPI-experiments branch. This update is about as big, and once again touches every module in a library - this time botan-low. It’s a bit late, so I’ll keep this short and sweet.

Change log / greatest hits

• botan-low now uses the newer better data types and their newtype constructors
• The mkBindings generator function has been moved to botan-low and expanded
• Initializers now use generated createObject functions
• Destructors are now generated
• Functions now use ConstPtr where appropriate
• FooCtx-style objects have been renamed Foo
• withFooPtr-style functions have been renamed withFoo
• mkFoo functions in Botan.Low.Make are being re-worked
• Suite of more consistent marshalling functions are being developed
• “Fixed” unit tests to the point of compiling if the XFFI test modules are deleted and removed from the cabal file (havent actually scrutinized the results yet)
• botan is still broken

Whew! That covered a lot and I’m still testing everything (in fact, the unit tests are still broken, but that’s because hspec-discover doesn’t respect our if flag(XFFI) and tries to include modules that won’t exist), so this update is still living on the CAPI-experiments branch for the time being.

There’s more work to do on botan-low but I’m aiming to get it release-stable like how botan-bindings effectively is - that’s all for now!

---

This update has been pushed to the CAPI-experiments branch branch.