Oh boy do I empathize with you here. The FFI layer can definitely get in the way and make it difficult to determine whether a flaw is present in the FFI or the bound library, and I have experienced this with Botan.
However, in this case, your current frustration (at least, the non-determinism) isn’t maybe isn’t FFI-related, because (EC)DSA is not (necessarily) a deterministic algorithm
You see, many cryptographic operations are non-deterministic, and require a random nonce that must never be reused (because it is a number only used once).Ideally, this value should be exposed as an input, and the resulting function would be pure and deterministic. However, nonce mismanagement has caused many security lapses, and so some libraries remove user nonce handling in one of two ways:
-
they make a variant that generates the nonce non-deterministically and attaches it to the cryptext output, thus obviating user handling but also resulting in a different output each time
-
they make a variant that derives the nonce deterministically based on the key and plaintext (eg, using HMAC) and attaches it to the cryptext output, resulting in the same output each time
Note that attaching the nonce makes the cryptext longer, which may also be one of the issues you are experiencing.
This is all and well, except that sometimes the library doesn’t bother calling it something different, and so it can be hard to tell exactly which variant you are using - and this is on top of the fact that many algorithms already have families that must be distinguished. This is one of the reason why test vectors are so important - it helps you know exactly which variant you are using.
This stackexchange post may be relevant to your issues.