Botan bindings devlog

#120

It really feels like the kind of abstraction problem you’re trying to solve between having an abstract interface of crypto algorithms with different backends such as botan is a perfect fit for backpack… :school_satchel:

2 Likes
#121

Someone’s already done that: raaz: Fast and type safe cryptography.

I’m actually surprised this didn’t make the prior art of the tech proposal.

2 Likes
#122

@romes

It certainly helps to have every gear polished and lubricated, ready to fit into place…

@jaror

Oooh, I don’t know how I missed raaz, but I’mma have to give it the ol’ look-see-and-digest for a bit - I’m sure there’s things to learn from it. :smiley:

Small aside, providing different backends for implementing the same algorithm, and providing typeclasses for implementing similar but separate algorithms, are two distinct goals. They do work together quite nicely, however, as they are related.

#123

Update: More gold-standard modules, imminent Hackage release

Hello, all! It is time for another update, but first:

I must apologize, for I have fallen prey to my tendency to keep my eyes on the horizon instead of on the finish line, and have ended up moving my own goalposts. Truth be told, botan-bindings and botan-low passed the finish line some time ago, and I really should have pulled the trigger on publishing them already. :grimacing:

This is where I really appreciate the community framework; weekly meetings with José have been helping me to keep my updates more regular, and also keep me on track and appraised of any concerns. Frankly, I don’t want to be a silo’d developer, and communication is an essential part of that.

As such, a minor course correction: I’m switching focus, and the goal of the next update is going to be the publishing of botan-bindings and botan-low to hackage as a candidate package (alongside the third monthly status report, of course, which is also now due! Egads!).

I am a bit anxious, never having published a package to hackage before. I look at the project and its very easy for me to forget what I’ve done, and only see what still needs to be done, or could be done - perfectionist’s disease. Its fine, it doesn’t have to be perfect, it just has to be sufficient for initial release, and it is. We can still keep working on it after we publish.

There’s a few things that must happen before publishing:

First, according to hackage requirements, I need someone to endorse me, so if a fine citizen or two could assist me in this, I would appreciate it.

Secondly, there’s the minor question of pkgconfig-depends vs extra-libraries. Right now, the stanzas in botan-bindings.cabal look like this:

library
    ...
    includes:
        botan/ffi.h
    if os(windows)
        extra-libraries: botan-3
    else
        pkgconfig-depends: botan-3 >= 3.0.0
    ...

This has been working so far, but I want to ensure is configured properly, since it is how we link to the Botan C++ library. There are more than a few different ways of handling this, spread over several discussions:

I’d like to handle it properly - if I’ve read the discussions correctly, we can crank the cabal version dependency up and improve it. I’d appreciate any suggestions here.

Thirdly, I need to go over botan-bindings and botan-low with a fine-toothed comb and do a few things like establish dependency version constraints to make the package acceptable to hackage. I’ll be on this for the next few days, and then hopefully upload it as a candidate on Monday.

So, time to get it done!

Now, back to this update.

  • The following modules have been completed to gold-standard, and conform to Botan.Types.Class:
    • Botan.BlockCipher
      • AES
      • ARIA
      • Blowfish
      • Camellia
      • CAST
      • DES
      • GOST
      • IDEA
      • Noekeon
      • SEED
      • Serpent
      • SHALCAL
      • SM4
      • Threefish
      • Twofish
  • I’ve added a temporary BlockCipher128 typeclass while a proper blocksize constraint is developed.
  • There have been improvements to error reporting, and the last exception message is now attached to any thrown Haskell exceptions.
  • I’ve laid out a Botan.Easy module that exposes a saltine-like interface of recommended algorithms

That’s all for now, it’s been pushed to the repo. We’ll be back on this stuff after we publish to hackage.

5 Likes
#124

pkg-config

I strongly recommend pkgconfig-depends, if botan upstream publishes .pc files. The main reason is that it means botan developers are deciding what the link flags are, rather than all their library consumers making a best guess. It also means you’re declaring a dependency on a native package instead of linker flags (which are more of an implementation detail), which seems more like the correct level of abstraction.

I would hope that you’d be able to use pkg-config on Windows but I haven’t played in that space for many years. Doesn’t GHC come with some sort of MSYS setup for this sort of thing?

4 Likes
#125

Yes, pkg-config works on windows (or at least it is not any worse than other options). See my experience report: Installing a library with C dependencies on Windows

1 Like
#126

It’s worth to provide a fallback to extra-libraries (for the case when library is installed, but pkg-config isn’t) and a fallback with bundled C-sources (very helpful on Windows / CentOS and inevitable for cross-compilation). This requires two cabal flags: a manual one to force bundled sources and an automatic one to switch between pkg-config and extra-libraries.

2 Likes
#127

@jackdk @jaror @Bodigrim This is very helpful. I do intend to keep using pkgconfig-depends wherever possible, and wish to ensure that I am using it properly.

Alright so this is what I have so far, though my inference is incomplete and so I have a few questions.

First, we have our pkgconfig flag, and the first question is whether the preferred convention is to express it in the positive or the negative:

-- We can express it in the positive
flag use-pkgconfig
    description: Enable pkgconfig
    default: True
    manual: False

-- Or we can express it in the negative
flag no-pkgconfig
    description: Disable pkgconfig
    default: False
    manual: False

This should be purely nomenclative, as the two flags should behave the same (with an appropriate negation): it should attempt to use pkgconfig-depends first, with the dependency solver automatically falling back on negating the flag if it fails.

Secondly, can you be more explicit about what you mean by bundled sources? I think I understand, but wish for clarification to avoid assumptions - I’d rather know exactly.

My inference is that we have our force-bundled flag:

flag force-bundled
    description: Force bundled sources
    default: False
    manual: True

Unlike use-pkgconfig (or no-pkgconfig), force-bundled is manual: True (and default: False) in order to require the user to positively activate it.

Then we use the flags as such:

library
    ...
    -- Or !flag(no-pkgconfig) && !flag(force-bundled) / !(flag(no-pkgconfig) || flag(force-bundled))
    if flag(use-pkgconfig) && !flag(force-bundled)
        pkgconfig-depends:
            botan-3 >= 3.0.0
    else
        includes:
            botan/ffi.h
        extra-libraries:
            botan-3

Combined with the use-pkgconfig flag, the dependency solver should 1) try to use pkgconfig, falling back if it fails 2) unless the force-bundled is set, in which case it skips to the falls back immediately.

I may also need to bump cabal-version: 3.8 because of this issue.

Is this understanding correct? Is this what you meant by forcing bundled sources? And which method is preferred / standard: use-pkgconfig or no-pkgconfig?

NOTE: I have some other flags (eg, for mtl and random support, plus the XFFI flag) that will also try to follow the established convention, but I believe that they should be flag no-foo and default: False / manual: True (like force-bundled)

#128

I never like negation if it is not necessary.

The default is the thing that assumes if it’s on or not and you won’t get situations where you disable the no-foo flag… because then you get a double negation, which in programming you almost never want to encounter.

E.g. I’d rather --flag botan:-pkg-config than --flag botan:no-pkg-config. We did the same in the password library, where the default is that all 4 algorithms are enabled, and you have to disable them to not include them, instead of enabling the negative flag.

All my opinion, obviously. I wonder how other people feel about this.

4 Likes
#129

@ApothecaLabs yes, correct. The full setup could look like this:

2 Likes
#130

Minor update: Readying for 0.0.1 release

Awrighty, I’ve got a minor update today, and its that I’ve gotten botan-bindings and botan-low ready for initial release.

This includes futzing with package flags, adding dependency version bounds, and getting low-level tutorials completed for critical path use cases. These tutorials are available in the README, and will be making their way into the haddock module documentation in the near future.

It all looks good, passes cabal check, and I can cabal sdist botan-bindings/botan-bindings.cabal and cabal sdist botan-low/botan-low.cabal to get my package’s *.tar.gz archive. I’ve uploaded the package candidates:

I can also cabal new-haddock --haddock-for-hackage botan-bindings and cabal new-haddock --haddock-for-hackage botan-low to generate haddock documentation. However, I don’t know how to preview the documentation on hackage (though I can look at it on my machine). Documentation has been generated and uploaded as well. It isn’t the greatest, but I’ll be working on it the next few days

I’ve pushed this to the repo, and I think we’re set for 0.0.1 aside from this question of documentation - what does everyone think? Do I pull the trigger and publish?

2 Likes
#131

You can generate documentation locally and upload with cabal upload -d.

2 Likes
#132

I clicked on a random module and ran into this:

image

Surely that extra slash shouldn’t be there?

1 Like
#133

It should not! I am cleaning up Botan.Low’s documentation today :slight_smile:

1 Like
#134

I think a convenient way to install everything through cabal, especially on Windows, would be crucial for wide adoption. It’s hard to imagine packages migrating from, say, cryptonite to botan otherwise; installing botan yourself is quite a chore.

Prior art is packages like lzma-clib and especially lzma-static. The idea is that you can autoconf on several, most-popular OS / arch and package headers to switch between depending on conditionals in Cabal file.

2 Likes
#135

Agreed - I’ve been putting some time into looking at a custom Setup.hs and all that entails. This is a thornier issue for botan, however, because the source code is extremely large and complex compared to something like lzma: it is C++, has a ton of configuration and customization, and takes between 5 minutes to half-an-hour to build depending.

This is not to dissuade, of course. It will not be as simple as including the source files and mentioning them in the appropriate stanzas, but it would be incredibly valuable, especially for getting it working for other backends such as WASM. I’ll have to devote some time to investigating how readily this can be achieved. I do know that the previous z-botan managed some form of bundling.

#136

Major update: 0.0.1 release imminent

My laptop was in the shop over the weekend, and I am extremely happy to have it back. I have otherwise spent my time recently focusing on 2 things: Squishing a few critical bugs, and producing tutorials for everthing botan-low.

In that regard:

  • Fixed memory leak in Botan.Low.PubKey.KeyAgreement
  • Renamed pattern BOTAN_PUBKEY_PEM_FORMAT_SIGNATURE to BOTAN_PUBKEY_STD_FORMAT_SIGNATURE because it is not guaranteed to be PEM format.
  • Created tutorials for:
    • Botan.Low.HOTP
    • Botan.Low.PubKey.KeyAgreement
    • Botan.Low.PubKey.KeyEncapsulation
    • Botan.Low.SRP6
    • Botan.Low.TOTP
    • Botan.Low.ZFEC
  • Tutorials are now present in the README as well as their respective module’s haddock documentation

There are a few modules remaining left to document / produce tutorials for (Botan.Low.KDF, Botan.Low.PwdHash) but they are both non-critical modules, and will not hold up the release, which is planned for tomorrow! I’ll sneak them in if I can, but I don’t want to hold things up just for them!

The package candidates have been updated:

So has the repo. :partying_face:

2 Likes
#137

Yes, Z-Botan includes entire Botan source code and builds it during Setup.hs. Not bad, even while it requires Python and other tools installed, but I do not see what blocks us from distributing precompiled amalgamated Botan source at least for most popular platforms (foremost Windows, because this is where getting build tools is most cumbersome).

1 Like
#138

I don’t see anything blocking us per se, but from my efforts to extend it, I am aware that Botan C++'s internal build structure is highly unorthodox and does a bunch of include-path swizzling.

Even if it does give us Python and other tools as environmental dependencies, I’d rather we wrap Botan C++ entirely as-is (eg, as a submodule / subrepo pointed towards a specific botan commit), than create any sort of custom Botan C++ distribution unless it requires very little effort.

This subject (packaging C++ into a Haskell repo) is outside my area of experience, and I am not necessarily aware of all of the potential solutions to this problem, so I do appreciate any help or information.

Edit: I missed: distributing precompiled amalgamated Botan source can you explain in a little more detail what you have in mind?

#139

Quick question, @ApothecaLabs

Does Botan have PBKDF2? I see it has bcrypt (even though you can’t supply your own salt), but other password hash functions like scrypt and Argon2 also seem to be missing?
Is that something that’s explicitly not in the botan library, or something they just haven’t gotten to yet?