Based on my understanding of the subject, key commitment is only necessary for those AE (Authenticated Encryption), or AEAD (Authenticated Encryption with Additional Data), algorithms that use Poly1305 or similar (like GCM).
However, if one uses for the authentication tag an HMAC (or something like Blake3), then this commitment is unnecessary. (Or so I understand at the moment…)
The reason – and this is something the article missed to introduce – is that for some AE / AEAD algorithms (like mentioned above for example based on Poly1305), there is an easy way to compute decryption keys that would generate a correct authentication tag, thus providing broken data from the decryption.
Thus the whole issue boils down to this: given a message (with encrypted data and authentication tag) and given a decryption key, we can’t be “sure” (in the cryptographic sense), that this particular key was actually used to encrypt this message.
You are right, that using something like HMAC (which is also a PRF suitable for the constructions mentioned in the post) would alleviate this problem. However, then you either have 1) a non-affected AE(AD) and you don’t worry about this or 2) you make a new Encrypt-then-MAC construction (or other AE) which doesn’t suffer from this.
Thanks for the feedback though! I probably should’ve added a small intro in terms of the problem. I sort of assumed that if someone were to use information from this post, they knew the problem and the fact that they needed to solve it.
So then, I’m asking in general out of curiosity as I’m not a cryptographer, why do we bother in general with constructs that have so many pitfalls?
If we already know that we can construct something like XChaCha-Blake3, why don’t we just create that, validate it, implement it in a few languages and push people towards it?
I understand that Blake3 might be overkill in terms of performance compared with Poly1305, however in 99% of the cases performance isn’t an issue, because 99% of people using these are non-cryptographers and most likely will just fall in those pitfalls…
One example is, if you can’t choose which AE/AEAD scheme you can use. In some contexts, such as .NET, you’re relatively restricted in what AE/AEADs you can choose (I think only AES-GCM and ChaCha20-Poly1305). With these, you’re not able to simply go ahead and modify the underlying implementation directly and here something that can be put on top is easier to deal with, instead of reimplementing.
Yes, you can make an AE work for Encrypt-then-MAC XChaCha20-BLAKE3, but for example, if you want to use this as an AEAD, then you also need a canonical way to add the ciphertext and AD to the MAC (as one of the comments at the bottom of the article you linked also mentions).
Very cool! IMO the README sells it short — the formally-verified arithmetic and X25519 support are worth calling out.
Very cool that you use strong typing in the API, as this is a pet peeve of mine with C crypto APIs. They give you a box of delicate machinery whose misuse can leave your code vulnerable to attack, but then declare all the parameters as void*. That’s like trying to hook up a time bomb whose wires are all the same color.
IMO the README sells it short — the formally-verified arithmetic and X25519 support are worth calling out.
Thanks, we’ll get this added!
Very cool that you use strong typing in the API, as this is a pet peeve of mine with C crypto APIs.
Something else that’s pretty cool, is the use of the ownership model, in the high-level key-exchange API that gives compile-time guarantees that a private key can only be used once, for ephemeral key-exchange (unless intentionally misused).
I’ve already commented on the /r/crypto submission, but just wanted to make sure I also mention here: these are absolutely great points that any crypto library should, at least, take into consideration.
It’s an introduction to cryptography concepts that are used in applications (encryption, authentication, etc.) with more in-depth explanations on how some of the actual algorithms work (for example, AES) and how they can be used in different types of applications (SSL/TLS, secure messaging like the signal protocol, etc.) The second part of the book also covers extra topics like hardware cryptography, post-quantum cryptography, cryptocurrencies, and even some advanced topics like zero-knowledge proofs.
Wait, is the full book available online? I’ve been waiting for this book, perhaps I’ve missed an announcement? Regardless, thanks for your hard work. I really enjoy your posts on cryptologie.net, and I’ve been looking forward to “Real-World Cryptography”.
I think it is slated for print release in a few months, and it seems like 15 / 16 chapters are available online. I grabbed a copy because this was almost exactly what I was looking for (judging by the preview).
I think one big resource that’s pretty popular is cryptopals. I’ve only done the first few sets but the exercises are a lot of fun and teach you real attacks
In terms of hands-on things, I’ve seen https://cryptohack.org/ be mentioned once or twice. Haven’t tried it myself, so I can’t really say anything about.
Isabelle, a powerful prover with support for code generation in (mostly functional) languages. The sel4 project is a notable user of Isabelle. There is also a new framework called Igloo, which aims to provide language agnostic end-to-end proofs from protocol to implementation.
Then there is Event-B, of which I know less about, but which powers the autonomous Paris metro line, an impressive feat.
When I read the full two paragraphs around that sentence, I get the feeling it’s for educational purposes, not for real security use, which is why they encourage a security disclaimer.
I might be projecting my own views though because I think “rolling your own crypto” is a fantastic way of learning how the system works and how things can fall apart. I’ve learned how easy it is to make mistakes and I’ve never read papers and documentation more thoroughly than with cryptography and compression algorithms since both are so incredibly hard to debug. I’ve also learned a lot about tooling to mitigate these issues.
I’m a big fan of “rolling your own crypto” and here I’m talking about implementing known algorithms. I do it myself. I even think making it available on GitHub or similar, to ask for feedback, is good (if users are warned that no security can be expected).
However, a problem arises when projects that don’t even uphold the bare minimum of testing test vectors, are published with no warnings at all. Had there been used test vectors in this case, it wouldn’t have left IdentityModel completely broken.
When I read the full two paragraphs around that sentence, I get the feeling it’s for educational purposes, not for real security use, which is why they encourage a security disclaimer.
Yes, this is exactly what I mean. And I totally agree with your benefits, that come from exploring it for educational purposes.
One reason it’s not a good idea: After showing a snippet of code that calls some XChaCha20-Poly1305 crypto functions, the author notes:
This is not a XChaCha20-Poly1305 construction.
In other words, the APIs exposed by low-level libraries are like bags of loose components. They have to be wired up correctly in order to work right, and it’s not always obvious how to do so. Even if you know about padding and nonces, a specific cipher can have its own limitations you also need to be aware of.
That’s why I’m a fan of the higher-level libraries stemming from and inspired by Daniel Bernstein’s NaCl (libSodium, Monocypher, CryptoKit, etc) which give you bigger building blocks like “cryptoBox” that do a specific thing and do it right, and choose appropriate algorithms under the hood. That makes it a lot easier to successfully implement a crypto construction, and in a way that’s compatible with other programs that use the same construction.
That’s why I’m a fan of the higher-level libraries stemming from and inspired by Daniel Bernstein’s NaCl (libSodium, Monocypher, CryptoKit, etc) which give you bigger building blocks like “cryptoBox” that do a specific thing and do it right, and choose appropriate algorithms under the hood.
Thanks for citing my work, appreciated. :-)
Working on Monocypher had me realise that the NaCl family ((Tweet)NaCl, Libsodium, Monocypher) is actually fairly low level. Yes, providing authenticated encryption and clean key exchange out of the box was a huge improvement. But my work on authenticated key exchange told me that high-level protocols are often fairly delicate, and require significant effort for untrained people to get them right. (Implementing an existing protocol like Noise isn’t too bad, though.) That’s in my opinion a big reason why Latacora’s Cryptographic Right Answers still recommends freaking TLS for client/server security.
I’d say the NaCl family of APIs is a good start. More work is needed to provide even higher-level facilities: full authenticated key exchange, PAKE, encrypted file formats (I’m leering at PURB), certificates… On top of those, we should then provide fully fledged network libraries, that actually handle the I/O (so far I’ve limited myself to manipulating buffers, to minimise dependencies and maximise portability). My, I’m afraid I still have a couple years of work ahead of me…
This is why it’s important to ‘salt‘ your hash with a secret key
It’s important to use a password hash function (~= key derivation function) like argon2, scrypt or bcrypt. It handles salt and iteration (it’s important too, regular hash functions are way too fast and don’t require much memory!) and everything. You should not think about salts yourself.
That article also mentions the use of password hashing algorithms, as @myfreeweb pointed out. The only thing to note is that PBKDF2 is quite outdated and better alternatives, such as Argon2, exist and should be preferred.
Based on my understanding of the subject, key commitment is only necessary for those AE (Authenticated Encryption), or AEAD (Authenticated Encryption with Additional Data), algorithms that use Poly1305 or similar (like GCM).
However, if one uses for the authentication tag an HMAC (or something like Blake3), then this commitment is unnecessary. (Or so I understand at the moment…)
The reason – and this is something the article missed to introduce – is that for some AE / AEAD algorithms (like mentioned above for example based on Poly1305), there is an easy way to compute decryption keys that would generate a correct authentication tag, thus providing broken data from the decryption.
Thus the whole issue boils down to this: given a message (with encrypted data and authentication tag) and given a decryption key, we can’t be “sure” (in the cryptographic sense), that this particular key was actually used to encrypt this message.
You are right, that using something like HMAC (which is also a PRF suitable for the constructions mentioned in the post) would alleviate this problem. However, then you either have 1) a non-affected AE(AD) and you don’t worry about this or 2) you make a new Encrypt-then-MAC construction (or other AE) which doesn’t suffer from this.
Thanks for the feedback though! I probably should’ve added a small intro in terms of the problem. I sort of assumed that if someone were to use information from this post, they knew the problem and the fact that they needed to solve it.
So then, I’m asking in general out of curiosity as I’m not a cryptographer, why do we bother in general with constructs that have so many pitfalls?
If we already know that we can construct something like XChaCha-Blake3, why don’t we just create that, validate it, implement it in a few languages and push people towards it?
I understand that Blake3 might be overkill in terms of performance compared with Poly1305, however in 99% of the cases performance isn’t an issue, because 99% of people using these are non-cryptographers and most likely will just fall in those pitfalls…
One example is, if you can’t choose which AE/AEAD scheme you can use. In some contexts, such as .NET, you’re relatively restricted in what AE/AEADs you can choose (I think only AES-GCM and ChaCha20-Poly1305). With these, you’re not able to simply go ahead and modify the underlying implementation directly and here something that can be put on top is easier to deal with, instead of reimplementing.
Yes, you can make an AE work for Encrypt-then-MAC XChaCha20-BLAKE3, but for example, if you want to use this as an AEAD, then you also need a canonical way to add the ciphertext and AD to the MAC (as one of the comments at the bottom of the article you linked also mentions).
Very cool! IMO the README sells it short — the formally-verified arithmetic and X25519 support are worth calling out.
Very cool that you use strong typing in the API, as this is a pet peeve of mine with C crypto APIs. They give you a box of delicate machinery whose misuse can leave your code vulnerable to attack, but then declare all the parameters as
void*. That’s like trying to hook up a time bomb whose wires are all the same color.Thanks, we’ll get this added!
Something else that’s pretty cool, is the use of the ownership model, in the high-level key-exchange API that gives compile-time guarantees that a private key can only be used once, for ephemeral key-exchange (unless intentionally misused).
I’ve already commented on the /r/crypto submission, but just wanted to make sure I also mention here: these are absolutely great points that any crypto library should, at least, take into consideration.
Most real-world books are pretty outdated at this point, I’ve been writing the book I wish I had when I got into applied cryptography. I spent more than 2 years writing it and it’s going to print soon, but you can read it online here: https://www.manning.com/books/real-world-cryptography?a_aid=Realworldcrypto&a_bid=ad500e09
It’s an introduction to cryptography concepts that are used in applications (encryption, authentication, etc.) with more in-depth explanations on how some of the actual algorithms work (for example, AES) and how they can be used in different types of applications (SSL/TLS, secure messaging like the signal protocol, etc.) The second part of the book also covers extra topics like hardware cryptography, post-quantum cryptography, cryptocurrencies, and even some advanced topics like zero-knowledge proofs.
I also wrote a post here on why I’m writing another book on cryptography: https://cryptologie.net/article/504/why-im-writing-a-book-on-cryptography/ which should give you more background about what kind of book it is.
Wait, is the full book available online? I’ve been waiting for this book, perhaps I’ve missed an announcement? Regardless, thanks for your hard work. I really enjoy your posts on cryptologie.net, and I’ve been looking forward to “Real-World Cryptography”.
I think it is slated for print release in a few months, and it seems like 15 / 16 chapters are available online. I grabbed a copy because this was almost exactly what I was looking for (judging by the preview).
Serious Cryptography by Jean-Philippe Aumasson is one of my favorites.
I would definitely also recommend this one.
I think one big resource that’s pretty popular is cryptopals. I’ve only done the first few sets but the exercises are a lot of fun and teach you real attacks
In terms of hands-on things, I’ve seen https://cryptohack.org/ be mentioned once or twice. Haven’t tried it myself, so I can’t really say anything about.
I stumbled upon it this morning as I wrote my post :). I’m going to sprinkle these in, seems like a great way to get my feet wet.
Some unmentioned theorem provers/frameworks:
Isabelle, a powerful prover with support for code generation in (mostly functional) languages. The sel4 project is a notable user of Isabelle. There is also a new framework called Igloo, which aims to provide language agnostic end-to-end proofs from protocol to implementation.
Then there is Event-B, of which I know less about, but which powers the autonomous Paris metro line, an impressive feat.
There’s also Verifpal.
Isn’t this article a great example of why NOT to roll your own crypto?
When I read the full two paragraphs around that sentence, I get the feeling it’s for educational purposes, not for real security use, which is why they encourage a security disclaimer.
I might be projecting my own views though because I think “rolling your own crypto” is a fantastic way of learning how the system works and how things can fall apart. I’ve learned how easy it is to make mistakes and I’ve never read papers and documentation more thoroughly than with cryptography and compression algorithms since both are so incredibly hard to debug. I’ve also learned a lot about tooling to mitigate these issues.
Yes, this is exactly what I mean. And I totally agree with your benefits, that come from exploring it for educational purposes.
Not why. How. Avoiding most mistakes only requires following a few rules. Those rules aren’t easy to follow, but they are pretty easy to know about.
Exactly, I was going to post the same thing.
One reason it’s not a good idea: After showing a snippet of code that calls some XChaCha20-Poly1305 crypto functions, the author notes:
In other words, the APIs exposed by low-level libraries are like bags of loose components. They have to be wired up correctly in order to work right, and it’s not always obvious how to do so. Even if you know about padding and nonces, a specific cipher can have its own limitations you also need to be aware of.
That’s why I’m a fan of the higher-level libraries stemming from and inspired by Daniel Bernstein’s NaCl (libSodium, Monocypher, CryptoKit, etc) which give you bigger building blocks like “cryptoBox” that do a specific thing and do it right, and choose appropriate algorithms under the hood. That makes it a lot easier to successfully implement a crypto construction, and in a way that’s compatible with other programs that use the same construction.
Thanks for citing my work, appreciated. :-)
Working on Monocypher had me realise that the NaCl family ((Tweet)NaCl, Libsodium, Monocypher) is actually fairly low level. Yes, providing authenticated encryption and clean key exchange out of the box was a huge improvement. But my work on authenticated key exchange told me that high-level protocols are often fairly delicate, and require significant effort for untrained people to get them right. (Implementing an existing protocol like Noise isn’t too bad, though.) That’s in my opinion a big reason why Latacora’s Cryptographic Right Answers still recommends freaking TLS for client/server security.
I’d say the NaCl family of APIs is a good start. More work is needed to provide even higher-level facilities: full authenticated key exchange, PAKE, encrypted file formats (I’m leering at PURB), certificates… On top of those, we should then provide fully fledged network libraries, that actually handle the I/O (so far I’ve limited myself to manipulating buffers, to minimise dependencies and maximise portability). My, I’m afraid I still have a couple years of work ahead of me…
It’s important to use a password hash function (~= key derivation function) like argon2, scrypt or bcrypt. It handles salt and iteration (it’s important too, regular hash functions are way too fast and don’t require much memory!) and everything. You should not think about salts yourself.
I follow this guide as well:
https://crackstation.net/hashing-security.htm
(It seems legit)
That article also mentions the use of password hashing algorithms, as @myfreeweb pointed out. The only thing to note is that PBKDF2 is quite outdated and better alternatives, such as Argon2, exist and should be preferred.
Another good resource for recommendations on this is Cryptographic Right Answers.
Argon2 is mentioned as well. Thanks for the link!