Botan bindings devlog

Perhaps ‘axing’ is a tad strong - lets say, relegated to the status of ‘prefer the other interface’ along side the mutable interface, in favor of this (hopefully) better new approach.

a group of pattern synonyms that contain all the valid ones

One of the difficulty of creating pattern synonyms for valid algorithm combinations is that there’s literally thousands of them - I’d have to rely on TemplateHaskell in order to make that feasible. I first ran into the issue with unit testing when the sheer number of tests (one per combination) caused the unit tests to terminate without outputting the results - this is why there are so many individual unit test targets now :upside_down_face:

Note that although I can’t provide convenient aliases for every algorithm combination, I certainly can make sure to do that for all priority / best-in-class algorithms as I did in Botan.Hash.SHA3, which exports not just the family sha3 but also all of the individual SHA3 variants (sha3_224, sha3_256, sha3_384, sha3_512).

make newtypes for every section that only accepts parts of certain algos/etc

Things have already been improved beyond yesterday’s update. With the new data families, we are accomplishing exactly this :smiley:

For example, AES128SecretKey (formerly AES128CipherKey, but ) is now actually a newtype wrapper around a GSecretkey type from which I can automatically derive everything.

First, GSecretKey is itself a newtype wrapper around Bytestring:

newtype GSecretKey = MkGSecretKey { unGSecretKey :: ByteString }
    deriving newtype (Eq, Ord, Encodable)

Then I then wrap the algorithm-specific newtype around that, plus a pattern and function to hide the GSecretKey:

newtype instance SecretKey AES128 = MkAES128SecretKey GSecretKey
    deriving newtype (Eq, Ord, Show, Encodable)

pattern AES128SecretKey :: ByteString -> SecretKey AES128
pattern AES128SecretKey bytes = MkAES128SecretKey (MkGSecretKey bytes)

getAES128SecretKey :: SecretKey AES128 -> ByteString
getAES128SecretKey (AES128SecretKey bs) = bs

And now I can just say:

newSecretKey @AES128
-- 591b3de67f882893a11af874fbf40bdd

Of course, this won’t be just for SecretKey and BlockCipher - I’m doing the same with other algorithms and supporting data types such as nonces, ciphertext, mac codes, salts, wherever appropriate.

It’ll all be out in the next update - I’ve already made some mad big strides since yesterday’s update (those dominoes keep tumbling down, and I’ve been in a happy little coding groove for the last few days.

1 Like

Update: Families!

After a serious shakedown with the recent classy update, things have really settled out.

  • Botan.Types.Class now provides reusable data families SecretKey, Nonce, Digest, Ciphertext, LazyCiphertext for common cryptography components
  • Also provides Has[Foo] / Is[Foo] typeclasses and G[Foo] generic types for data families
  • KeySpec has been generalized to SizeSpecifier in Botan.Types.Class
  • Botan.Types.Class is now considered the gold standard (nominally ready for release / documentation / tutorials)
  • Modules must conform to Botan.Types.Class to be gold standard

These modules have been updated to the current gold standard:

  • Botan.BlockCipher.Class
  • Botan.BlockCipher.AES
  • Botan.Cipher.Class
  • Botan.Cipher.ChaCha20Poly1305

These typeclasses are still being worked on to integrate with Botan.Types.Class, but work at this point is mostly mechanical.

  • Botan.Hash.Class
    • Needs to be updated to use the Digest / HasDigest / GDigest in Botan.Types.Class
  • Botan.MAC.Class
    • Ditto SecretKey / HasSecretKey / GSecretKey
    • Ditto Digest / HasDigest / GDigest
  • Botan.OneTimeAuth.Class
    • Ditto SecretKey / HasSecretKey / GSecretKey
    • Ditto Nonce / HasNonce / GNonce
    • Ditto Digest / HasDigest / GDigest

These typeclasses still need to be developed and are considered critical for first release:

  • Botan.PubKey.Class
  • Botan.PubKey.Encrypt.Class
  • Botan.PubKey.Sign.Class

Other typeclasses still need to be developed, but are not considered critical for first release as their concrete interfaces currently suffice.

  • Password hashing (for Bcrypt, etc)
  • FPE
  • HOTP / TOTP
  • KDF
  • Key Wrapping
  • KeyAgreement / KeyExchange / PAKE (for SRP6, DHKE)
  • ZFEC

We’re delaying our hackage release candidate by a week, maybe two, but its worth it, I promise. Slow is smooth, and smooth is fast. I want this library to be smooth.

As always, this has been pushed to the repo .

2 Likes

It really feels like the kind of abstraction problem you’re trying to solve between having an abstract interface of crypto algorithms with different backends such as botan is a perfect fit for backpack… :school_satchel:

2 Likes

Someone’s already done that: raaz: Fast and type safe cryptography.

I’m actually surprised this didn’t make the prior art of the tech proposal.

2 Likes

@romes

It certainly helps to have every gear polished and lubricated, ready to fit into place…

@jaror

Oooh, I don’t know how I missed raaz, but I’mma have to give it the ol’ look-see-and-digest for a bit - I’m sure there’s things to learn from it. :smiley:


Small aside, providing different backends for implementing the same algorithm, and providing typeclasses for implementing similar but separate algorithms, are two distinct goals. They do work together quite nicely, however, as they are related.

Update: More gold-standard modules, imminent Hackage release

Hello, all! It is time for another update, but first:

I must apologize, for I have fallen prey to my tendency to keep my eyes on the horizon instead of on the finish line, and have ended up moving my own goalposts. Truth be told, botan-bindings and botan-low passed the finish line some time ago, and I really should have pulled the trigger on publishing them already. :grimacing:

This is where I really appreciate the community framework; weekly meetings with José have been helping me to keep my updates more regular, and also keep me on track and appraised of any concerns. Frankly, I don’t want to be a silo’d developer, and communication is an essential part of that.

As such, a minor course correction: I’m switching focus, and the goal of the next update is going to be the publishing of botan-bindings and botan-low to hackage as a candidate package (alongside the third monthly status report, of course, which is also now due! Egads!).

I am a bit anxious, never having published a package to hackage before. I look at the project and its very easy for me to forget what I’ve done, and only see what still needs to be done, or could be done - perfectionist’s disease. Its fine, it doesn’t have to be perfect, it just has to be sufficient for initial release, and it is. We can still keep working on it after we publish.

There’s a few things that must happen before publishing:

First, according to hackage requirements, I need someone to endorse me, so if a fine citizen or two could assist me in this, I would appreciate it.

Secondly, there’s the minor question of pkgconfig-depends vs extra-libraries. Right now, the stanzas in botan-bindings.cabal look like this:

library
    ...
    includes:
        botan/ffi.h
    if os(windows)
        extra-libraries: botan-3
    else
        pkgconfig-depends: botan-3 >= 3.0.0
    ...

This has been working so far, but I want to ensure is configured properly, since it is how we link to the Botan C++ library. There are more than a few different ways of handling this, spread over several discussions:

I’d like to handle it properly - if I’ve read the discussions correctly, we can crank the cabal version dependency up and improve it. I’d appreciate any suggestions here.

Thirdly, I need to go over botan-bindings and botan-low with a fine-toothed comb and do a few things like establish dependency version constraints to make the package acceptable to hackage. I’ll be on this for the next few days, and then hopefully upload it as a candidate on Monday.

So, time to get it done!


Now, back to this update.

  • The following modules have been completed to gold-standard, and conform to Botan.Types.Class:
    • Botan.BlockCipher
      • AES
      • ARIA
      • Blowfish
      • Camellia
      • CAST
      • DES
      • GOST
      • IDEA
      • Noekeon
      • SEED
      • Serpent
      • SHALCAL
      • SM4
      • Threefish
      • Twofish
  • I’ve added a temporary BlockCipher128 typeclass while a proper blocksize constraint is developed.
  • There have been improvements to error reporting, and the last exception message is now attached to any thrown Haskell exceptions.
  • I’ve laid out a Botan.Easy module that exposes a saltine-like interface of recommended algorithms

That’s all for now, it’s been pushed to the repo. We’ll be back on this stuff after we publish to hackage.

5 Likes

pkg-config

I strongly recommend pkgconfig-depends, if botan upstream publishes .pc files. The main reason is that it means botan developers are deciding what the link flags are, rather than all their library consumers making a best guess. It also means you’re declaring a dependency on a native package instead of linker flags (which are more of an implementation detail), which seems more like the correct level of abstraction.

I would hope that you’d be able to use pkg-config on Windows but I haven’t played in that space for many years. Doesn’t GHC come with some sort of MSYS setup for this sort of thing?

4 Likes

Yes, pkg-config works on windows (or at least it is not any worse than other options). See my experience report: Installing a library with C dependencies on Windows

1 Like

It’s worth to provide a fallback to extra-libraries (for the case when library is installed, but pkg-config isn’t) and a fallback with bundled C-sources (very helpful on Windows / CentOS and inevitable for cross-compilation). This requires two cabal flags: a manual one to force bundled sources and an automatic one to switch between pkg-config and extra-libraries.

2 Likes

@jackdk @jaror @Bodigrim This is very helpful. I do intend to keep using pkgconfig-depends wherever possible, and wish to ensure that I am using it properly.


Alright so this is what I have so far, though my inference is incomplete and so I have a few questions.

First, we have our pkgconfig flag, and the first question is whether the preferred convention is to express it in the positive or the negative:

-- We can express it in the positive
flag use-pkgconfig
    description: Enable pkgconfig
    default: True
    manual: False

-- Or we can express it in the negative
flag no-pkgconfig
    description: Disable pkgconfig
    default: False
    manual: False

This should be purely nomenclative, as the two flags should behave the same (with an appropriate negation): it should attempt to use pkgconfig-depends first, with the dependency solver automatically falling back on negating the flag if it fails.

Secondly, can you be more explicit about what you mean by bundled sources? I think I understand, but wish for clarification to avoid assumptions - I’d rather know exactly.

My inference is that we have our force-bundled flag:

flag force-bundled
    description: Force bundled sources
    default: False
    manual: True

Unlike use-pkgconfig (or no-pkgconfig), force-bundled is manual: True (and default: False) in order to require the user to positively activate it.

Then we use the flags as such:

library
    ...
    -- Or !flag(no-pkgconfig) && !flag(force-bundled) / !(flag(no-pkgconfig) || flag(force-bundled))
    if flag(use-pkgconfig) && !flag(force-bundled)
        pkgconfig-depends:
            botan-3 >= 3.0.0
    else
        includes:
            botan/ffi.h
        extra-libraries:
            botan-3

Combined with the use-pkgconfig flag, the dependency solver should 1) try to use pkgconfig, falling back if it fails 2) unless the force-bundled is set, in which case it skips to the falls back immediately.

I may also need to bump cabal-version: 3.8 because of this issue.


Is this understanding correct? Is this what you meant by forcing bundled sources? And which method is preferred / standard: use-pkgconfig or no-pkgconfig?

NOTE: I have some other flags (eg, for mtl and random support, plus the XFFI flag) that will also try to follow the established convention, but I believe that they should be flag no-foo and default: False / manual: True (like force-bundled)

I never like negation if it is not necessary.

The default is the thing that assumes if it’s on or not and you won’t get situations where you disable the no-foo flag… because then you get a double negation, which in programming you almost never want to encounter.

E.g. I’d rather --flag botan:-pkg-config than --flag botan:no-pkg-config. We did the same in the password library, where the default is that all 4 algorithms are enabled, and you have to disable them to not include them, instead of enabling the negative flag.

All my opinion, obviously. I wonder how other people feel about this.

4 Likes

@ApothecaLabs yes, correct. The full setup could look like this:

2 Likes

Minor update: Readying for 0.0.1 release

Awrighty, I’ve got a minor update today, and its that I’ve gotten botan-bindings and botan-low ready for initial release.

This includes futzing with package flags, adding dependency version bounds, and getting low-level tutorials completed for critical path use cases. These tutorials are available in the README, and will be making their way into the haddock module documentation in the near future.

It all looks good, passes cabal check, and I can cabal sdist botan-bindings/botan-bindings.cabal and cabal sdist botan-low/botan-low.cabal to get my package’s *.tar.gz archive. I’ve uploaded the package candidates:

I can also cabal new-haddock --haddock-for-hackage botan-bindings and cabal new-haddock --haddock-for-hackage botan-low to generate haddock documentation. However, I don’t know how to preview the documentation on hackage (though I can look at it on my machine). Documentation has been generated and uploaded as well. It isn’t the greatest, but I’ll be working on it the next few days

I’ve pushed this to the repo, and I think we’re set for 0.0.1 aside from this question of documentation - what does everyone think? Do I pull the trigger and publish?

2 Likes

You can generate documentation locally and upload with cabal upload -d.

2 Likes

I clicked on a random module and ran into this:

image

Surely that extra slash shouldn’t be there?

1 Like

It should not! I am cleaning up Botan.Low’s documentation today :slight_smile:

1 Like

I think a convenient way to install everything through cabal, especially on Windows, would be crucial for wide adoption. It’s hard to imagine packages migrating from, say, cryptonite to botan otherwise; installing botan yourself is quite a chore.

Prior art is packages like lzma-clib and especially lzma-static. The idea is that you can autoconf on several, most-popular OS / arch and package headers to switch between depending on conditionals in Cabal file.

2 Likes

Agreed - I’ve been putting some time into looking at a custom Setup.hs and all that entails. This is a thornier issue for botan, however, because the source code is extremely large and complex compared to something like lzma: it is C++, has a ton of configuration and customization, and takes between 5 minutes to half-an-hour to build depending.

This is not to dissuade, of course. It will not be as simple as including the source files and mentioning them in the appropriate stanzas, but it would be incredibly valuable, especially for getting it working for other backends such as WASM. I’ll have to devote some time to investigating how readily this can be achieved. I do know that the previous z-botan managed some form of bundling.

Major update: 0.0.1 release imminent

My laptop was in the shop over the weekend, and I am extremely happy to have it back. I have otherwise spent my time recently focusing on 2 things: Squishing a few critical bugs, and producing tutorials for everthing botan-low.

In that regard:

  • Fixed memory leak in Botan.Low.PubKey.KeyAgreement
  • Renamed pattern BOTAN_PUBKEY_PEM_FORMAT_SIGNATURE to BOTAN_PUBKEY_STD_FORMAT_SIGNATURE because it is not guaranteed to be PEM format.
  • Created tutorials for:
    • Botan.Low.HOTP
    • Botan.Low.PubKey.KeyAgreement
    • Botan.Low.PubKey.KeyEncapsulation
    • Botan.Low.SRP6
    • Botan.Low.TOTP
    • Botan.Low.ZFEC
  • Tutorials are now present in the README as well as their respective module’s haddock documentation

There are a few modules remaining left to document / produce tutorials for (Botan.Low.KDF, Botan.Low.PwdHash) but they are both non-critical modules, and will not hold up the release, which is planned for tomorrow! I’ll sneak them in if I can, but I don’t want to hold things up just for them!

The package candidates have been updated:

So has the repo. :partying_face:

2 Likes

Yes, Z-Botan includes entire Botan source code and builds it during Setup.hs. Not bad, even while it requires Python and other tools installed, but I do not see what blocks us from distributing precompiled amalgamated Botan source at least for most popular platforms (foremost Windows, because this is where getting build tools is most cumbersome).

1 Like