This is an interesting article, but I feel that it leans too heavily on a few jargon terms that are not as widely understood as the author perhaps suspects, and doesn’t look hard enough at the existing systems out there.
It assumes more knowledge of how existing packaging tools work than I’ve ever needed to acquire. I have never built a package. What’s a hook? Why does it exist? What does it do? Is it needed? What for?
It also assumes more foreknowledge of container systems than I’ve ever needed. I predicted the rise of containers two years before Docker, but that doesn’t mean I’ve ever built or deployed one. That’s not my job.
What precisely does “deterministic” mean in this context? I interviewed the CEO of Rocky Linux and he talked a lot about deterministic compilation, and when I asked him what that was and what it meant, he floundered. He expected me to know and to care and was unable to clearly explicate what it was when I did not.
For comparison, I have written about packaging systems too; here is part 1 and part 2.
You are right that I’m assuming from the reader quite a bit of familiarity with OS package management. (The container stuff was tangent to the subject, thus not essential.)
Regarding the hooks, I’ve tried to briefly describe what they are (“scripts that …”), but I’ve focused more on the functionality they provide.
Regarding “determinism” I’ve succinctly described it as “in the cryptographic sense of identical checksums”, and I’ve stated that, starting from the same “base-image” (say an existing installation), installing (in one operation, or the same sequence of multiple installations) the same set of packages (given in any order), regardless of how many times (or the phase of the moon), should yield an identical “image”. (Here by “image” I actually mean the same directories, the same files, with the same contents, the same timestamps, the same owners, etc. If one creates a reproducible tar after each install, the md5sum of those tar archives would be identical.)
I don’t think this actually pertains to the problem I describe. I don’t think we need a new OS (as in kernel), I think we need to clean up our distributions packaging. (Do you have a link where they describe the way packaging and installation is done?)
Genode supports several kernels, including Linux, but generally not preferable, as third generation microkernels such as seL4 are simply that much better.
By default, it works with perfectly reproduce-able scenarios similar to nix. They have a decent webbrowser as well as 3d-acceleration so Linux is seldom needed. When it is, they have a working port of Virtualbox.
The Genode Foundations handbook (linked in their frontpage) is the go-to for describing the architecture; Chapter 4.6 and 5.5 and 6 is where I’d look at for this, but the intro chapter is worth reading.
The Yocto / Bitbake system is often used to create a Linux filesystem (and kernels, bootloaders, etc.) from a set of recipes. These recipes usually specify a released tarball or git commit for the software being built. Primarily for embedded systems.
If you have all your recipes and such in version control, then theoretically you can exactly reproduce a system image.
And of course they do care quite a bit about reproducibility, since it’s so core to bitbake’s functionality. The output of each build step is cached and hashed. If the hash hasn’t changed, then its outputs are reused. This is applied recursively, so you can make a change and be assured that the output is the same as if you had built everything from scratch, even though only a few build steps were ran. It’s common to share state (sstate) within an organization, which can greatly speed up builds.
My suggestion is of course a bit tongue-in-cheek, since yocto (or rather openembedded) is sort of like a cross-compile-only version of gentoo. In their quest for reproducibility, they don’t even use the host toolchain (or any utilities). While you certainly could run yocto in yocto, it would be pretty masochistic. Plus, many of the defaults are oriented towards producing small, efficient binaries. While you can certainly enable all the bells and whistles, it’s not really the intention of the distro.
While I generally agree with what you are saying, I was hedging a little bit. Mostly with regards to creating two bit-exact build images when you have two different machines building from scratch from the same set of recipes. It is easy, for example, to have timestamps squirreled away inside the build image. Though of course those two build images should be functionally identical. I’m not so worried about bit-for-bit identical builds, but I am under the impression that some people are.
Well, that’s why we have SOURCE_DATE_EPOCH and the like.
It actually is quite important to have bit-for-bit identical images in bitbake. For example, say that you modify a dependency of some package and do_build has to be re-run. If that dependency change doesn’t affect the build output, then the sstate hash for that build step will be the same. Then, since the hash of do_build didn’t change, all the dependent build steps can be reused. In this way, changes to variables which might otherwise require large rebuilds can result in much smaller effort. But if you don’t have a bit-for-bit reproducibility, then everything else in the dependency cone will need to be rebuilt. I’m not sure if this is as feasible on multi-machine setups, but it certainly is useful for me.
Indeed, after reading the landing page, it seems Michael has reached (way before me) to some of the same conclusions (he has more articles on the topic that I’ll have to read).
However, distri is clearly an experimental distribution (say so on the landing page), thus unfortunately still not a proper production-ready solution.
Nix is only as deterministic as it’s builders and compilers that’s true. It gives you the scaffolding you need but you still have to ensure you follow the rules. If you use a builder that creates random data somewhere then you still won’t have a deterministic output. That’s true of pretty much anything that tries to solve this problem though. They are working with tools that make it a little too easy to violate the rules unexpectedly.
Yeah I was actually surprised by this, from some brief experience with a shell.nix
It seems like it is extremely easy to write Nix files “wrong” and break the properties of Nix
Whereas with Bazel, the build breaks if you do something wrong. It helps you write a correct configuration
Similar issue with Make – Make provides you zero help determining dependencies, so basically all Makefiles have subtle bugs. To be fair Ninja also doesn’t strictly enforce things, but it’s easy to read and debug, and has a few tools like a dep graph exporter.
nix-shell is a bit of a lukewarm compromise between Nix’s ability to supply arbitrary dependencies and the guarantees of the build sandbox. It’s not really trying to be hermetic, just ~humane.
I would like to hear more about your shell.nix experience. It has some issues, for sure, but I have found the platform to be extremely reliable overall.
Is that really relevant? A determined user could flash their BIOS with garbage and break anything they want. Does it matter that you can go out of your way to break your own NixOS system? The point is that as a user you’re not supposed to mess with your system manually, but you’re supposed to make changes by changing your configuration.nix. This is not the case for most other distros where you make changes to files under /etc side by side with files maintained by the distro packages.
The point is that I don’t think anyone is checking. IDK how many options in nixpkgs break this. I don’t think we need to stop the user from purposely breaking it but we should make “official” flags not break it and make it easy for users to not break this.
However, I do think there is a place for classical distributions, as there is a place for distributions like Nix, Guix, and even something like GoboLinux.
Pretty much every concern you share in that post is solved by NixOS. Some points
You don’t modify files in /etc, you use the NixOS configuration to do so
Changes are idempotent
Changes are reversible, there’s a history of config revisions called generations, or you can just remove the config and apply to roll forward
System configuration is reproducible and deterministic
Services are systemd units declared through the configuration
Packages reference their direct dependencies allowing for multiple versions of the same lib/package/etc
NixOS isn’t perfect by any means, but it is a solution you seem to be looking for. Guix probably as well, but it’s a smaller and more limited distro.
I took the multiple comments that “NixOS may do this but I’ll have to research” to mean you haven’t tried it. And this entire blog post screams to me that NixOS is a solution for you.
You don’t modify files in /etc, you use the NixOS configuration to do so
In a way, that’s second system syndrome. Suse Linux was bashed for doing something like that with YaST for a long time…
In case of NixOS, I found that new language (not the syntax, the semantics) changed too fast for my liking…
System configuration is reproducible and deterministic
With a few caveats: Software might still get updated (I suppose you can lock that down to specific revisions, but who does?). From a bugfix/security perspective this may be desirable, but it’s not entirely reproducible, and updates can always introduce new bugs or incompatibilities, too.
You’ll get the same version with the same Nixpkgs, every time. buildInputs = [ bash ] will always point at the same bash package for some given Nixpkgs. Package versioning is a different issue.
I think you’re misunderstanding the way nixpkgs works with versions and what the versions actually mean there. Check it out in practice. Unless you update the channels/flakes or whatever you use between runs, nothing will change - there’s no explicit pinning required.
The way I manage this is using several different versions of nixpkgs; a “known good” version for each package I want to lock, so for agda I would have nixpkgsAgda, the regular nixpkgs which is pinned to a stable release, and nixpkgsLatest which is pinned to master.
Most of my packages are on stable nixpkgs. Every now and then when I run nix flake update, it pulls in new versions of software. Things pinned to stable have never broken so far. Things pinned to latest are updated to their latest versions, and things pinned to specific packages never change.
While it does involve pulling in several versions of nixpkgs, I build a lot of software from source anyway so this doesn’t matter to me very much. I do hope that nixpkgs somehow fixes the growing tarball size in the future…
buildInputs = [ bash ] may or may not point at the same version
That’s a snippet of code written in the Nix expression language. bash is not a keyword in that language, it’s just some arbitrary variable name (indeed, so is buildInputs). Just like any other language, if that bash variable is defined/computed to be some pure, constant value, then it will always evaluate to that same thing. If it’s instead defined/computed to be some impure, under-determined value then it may vary between evaluations. Here’s an example of the former:
This evaluates to { buildInputs = [ «derivation /nix/store/6z1cb92fmxq2svrq3i68wxjmd6vvf904-bash-5.2-p15.drv» ]; } and (as far as I’m aware) always will do; even if github.com disappears, it may still live on if that sha256 appears in a cache!
Here’s an example of an impure value, which varies depending on the OS and CPU architecture, on the presence/contents of a NIX_PATH environment variable, the presence/contents of a ~/.config/nixpkgs/overlays.nix file, etc.
PS: I’ve avoided the word “version”, since that concept doesn’t really exist in Nix. It’s more precise to focus on the .drv file’s path, since that includes a hash (e.g. 6z1cb92fmxq2svrq3i68wxjmd6vvf904) which is the root of a Merkle tree that precisely defines that derivation and all of its transitive dependencies; whereas “version” usually refers to an optional, semi-numerical suffix on the file name (e.g. 5.2-p15) which (a) is easy to spoof, and (b) gives no indication of the chosen dependencies, compiler flags, etc. which may have a large effect on the resulting behaviour (which is ultimately what we care about).
In case of NixOS, I found that new language (not the syntax, the semantics) changed too fast for my liking…
Do you happen to have an example of such an semantic change at hand? Curious as nix is often regarded as rather conservative and trying to keep backwards-compatibility as good as possible.
Part of the reason Nix/Guix take their particular approach is specifically because the global mutable state of a traditional FHS style distribution is nigh-impossible to manipulate
in a deterministic way; not just because of the manipulation by the package manager, but because there is no delineation between mutations by the package manager and those by the user. A great example is /etc/passwd, which contains both locally created users as well as “service accounts” the distro maintainers make to separate out some unprivileged services.
it needs a lot of work to enforce 1) determinism/reproducibility, 2) non-hysterisis (when you stretch an object and it doesn’t return /exactly/ back… like hidden side effects in ansible), 3) ease of use, & 4) acceptance testing.
Interesting take on authentication, however perhaps a bit too convoluted?
Let aside the fact it relies on JWT that has a bad reputation of being impossible to get right in implementation, but what caught my eye was how the verification (by PyPI for example) is being done: by applying certain policies over the verified claims in the received JWT.
Because the article doesn’t say, and doesn’t give any screenshots in case of PyPI, I can only wonder how this is actually achieved:
either it supports a predefined set of providers (like GitHub, GitLab, etc.) and provides out-of-the-box policies for them; (but then the openness argument dissipates;)
or the user now has to write some policies themselves (in some DSL perhaps) to check the various claims (user, repository, branch, etc.);
So, I’m left wondering if they haven’t just moved the problem from one place to another by introducing yet another attack surface (which besides JWT is now the policies)?
They have even identified one themselves – that of resurrecting dead or renamed accounts – and they’ve solved it with the intervention of GitHub, which means that this issue has to be solved by each individual provider.
I can’t believe we can’t find simpler approaches than this…
This was meant to be just a technical summary of the architecture. The link at the very top of the post goes to the the PyPI blog, which contains an announcement that’s more specific to the features available at launch-time; it also contains a link to the technical documentation1, which should address some of your questions around policies and security model.
To summarize:
Yes, JWTs at bad. They’re also, practically speaking, the only game in town. OAuth2 and OIDC use them, so we have to use them. There would be no trusted publishing at all without these underlying technologies.
At the moment, PyPI supports just one provider: GitHub. That provider was selected because it accounts for an overwhelming majority of publishing activity, meaning it’s the one that would be most immediately useful to potential users. I’m currently working on support for two more providers: Google’s IdP and GitLab.
It would be fantastic if we could do vendor neutral policy verification, but neither OAuth2 nor OIDC is rigid enough for us to assume a ‘baseline’ of shared relevant claims. Each IdP needs to be added manually, since each has its own claim set and quirks; eliminating that problem is a problem for standards bodies, not relying parties like PyPI.
The “policies” in question are just HTML form inputs for (in GitHub’s case) the username, repository name, workflow name, and an optional actions environment. You don’t have to know a DSL; that’s something we considered early on in design, and rejected for complexity reasons. There are examples of this in the docs (linked above).
Trusted publishing is indeed architecturally complex. But it’s using OIDC as it’s intended, and we’ve tried to abstract nearly all of this complexity from the end user (who only needs to know which user/repo and workflow they trust). The surface that it exposes is roughly equivalent to granting an attacker access to your CI (including all secrets in it), so I don’t think that part is a significant new risk.
I’m not trying to dismiss the work you have done with this integration. It is a first step towards a better solution for a quite hard to solve problem.
However, PyPI is now in the same league as Rust’s crates.io in that it requires a GitHub account to be able to “play” in Rust’s (and now Python’s) ecosystem. I don’t have anything against GitHub either, I use them for all my code, however there are developers (and certainly also amongst Python ones) that don’t want to have a GitHub account. Thus for this category of users the problem is still unsolved.
However, assuming GitHub is only the first such identity provider, what are the next steps to support multiple independent providers? How would the UI change to support something that doesn’t have GitHub’s particularities? would a separate HTML UI (i.e. form with inputs) be developed for each?
Because each time such a proposal appears, that in theory supports multiple identity providers, it falls short and supports only the largest ones like Google / Facebook / Apple.
To summarize, I understand why PyPI (and crates.io for that matter) have chosen to start (and stop) with GitHub: at the moment it has a large user-base and it’s pretty secure (or at least with deeper pockets to throw money at the problem if needed); in a few words it’s a practical approach (that I would have taken myself). But I doubt there would be a time when the “trusted publishing” would expand to other providers…
So, if GitHub is the only one, why bother with all the complexity?
Google’s IdP and GitLab.
GitLab is perhaps similar to GitHub, thus it might have the concepts of repositories, branches, etc. But would you support only the “official hosted” GitLab instance, or will you support independent GitLab deployments?
But what are the equivalents for Google? They’ve killed their code.google.com long ago. Are you targeting another of their products?
However, PyPI is now in the same league as Rust’s crates.io in that it requires a GitHub account to be able to “play” in Rust’s (and now Python’s) ecosystem.
This new feature, which offers an ability to publish from CI with a short-lived token in order to make CI compromises less severe, chose GitHub as the first supported IdP, not the only one that will ever be supported, and you’re literally replying to someone who says they’re working on adding support for others.
And if you don’t have or want a GitHub account, or object to using a GitHub account, or don’t want to publish from CI at all, you can still use plenty of other PyPI publishing workflows to put your packages onto PyPI, all the way down to just running python -m build && python -m twine upload dist/* manually from your own personal machine. So claiming that PyPI now “requires a GitHub account” is false.
I’ll state it unambiguously: trusted publishing is not intended to deprecate ordinary PyPI API tokens. Users will continue to be able to create API tokens, which can then be used however they like. Trusted publishing is only intended to provide a faster and more convenient alternative, where the platform (i.e. GitHub, and soon others) permit.
Support for independent providers is something that PyPI’s admins will probably consider on a case-by-case basis, as a balancing act between implementation complexity (each IdP needs special handling), the security margins of each new IdP (there’s relatively little value in a random self-hosted IdP versus a normal API token), and the impact to the community (e.g., major CI platforms).
But what are the equivalents for Google? They’ve killed their code.google.com long ago. Are you targeting another of their products?
Yes, sorry if this was unclear: in this context, the IdP being supported is the one that provides proofs for machine identities on Google Cloud Build. In other words, similar to GitHub Actions’ machine identities.
age support[s] the encryption mechanism [of a] shared password – i.e. type some stuff on the keyboard, and the tool will base the encryption keys on that
Does it? AFAIK age-keygen does not let you create keypairs from a user-provided shared password? Or do you mean something else?
at decryption time you need either the shared password either one of the private keys, but not necessarily both
AFAIK the only way to decrypt an age message is with a private key, not a shared password? Am I mistaken?
age [--encrypt] --passphrase [--armor] [-o OUTPUT] [INPUT]
[...]
-p, --passphrase Encrypt with a passphrase.
Files can be encrypted with a passphrase by using -p/–passphrase.
By default age will automatically generate a secure passphrase.
Passphrase protected files are automatically detected at decrypt time.
Granted, the age tool doesn’t provide at this moment support for encrypting both with an identity and a passphrase, but the format (as specified here) does technically support it.
The textual file header wraps the file key for one or more recipients, so that it can be unwrapped by one of the corresponding identities. It starts with a version line, followed by one or more recipient stanzas, and ends with a MAC.
The first is the string scrypt, the second is a base64-encoded salt computed by the recipient implementation as 16 bytes from a CSPRNG, and the third is the base-two logarithm of the scrypt work factor in decimal.
Granted, the age tool doesn’t provide at this moment support for encrypting both with an identity and a passphrase,
I’m not sure what “and” would mean in this context. Is it first one and then the other? If so, in what order? And I guess this is possible without explicit support in the tool: encrypt with a passphrase to get ciphertext 1, and then encrypt that ciphertext with an identity to get ciphertext 2; reverse the process to decrypt. Also, as most identity files can themselves be protected with a passphrase, what would be the benefit of using a passphrase as a separate step?
I’m not sure what “and” would mean in this context. Is it first one and then the other?
This is exactly related to my article, and why I’ve wanted to design “multi-factor encryption” (which is a very poor name…)
Getting back to age, as said, although the current CLI (and documentation) doesn’t support providing at the same time both a password and a recipient identity, the format does permit it. So indeed saying “a password and a recipient identity” isn’t very clear; it could mean either of the following at decryption time (which is the important part):
“and” – at decryption, the user must have both the shared secret password and the private key pertaining to the identity used at encryption; (this is what I call “multi-factor encryption”, and neither age nor gnupg supports this;)
“or” – at decryption, the user must have either the password or the private key; this is what both age or gnupg implement;
Is it first one and then the other? If so, in what order?
If it were supported, and this is how I’ve implemented it in my tool, the user specifies everything at once, in any order, and the tool knows how to handle everything.
encrypt with a passphrase to get ciphertext 1, and then encrypt that ciphertext with an identity to get ciphertext 2; reverse the process to decrypt.
Indeed, this is one possible solution, but it implies two encryption phases.
A second solution might be to just encrypt the “session key” (or “packet key”) with the passphrase and then that output with the identity.
A thind approach, which I’ve implemented, is combining both the passphrase, the identity, (other factors), and the “session nonce” to obtain the “session key”.
Also, as most identity files can themselves be protected with a passphrase, what would be the benefit of using a passphrase as a separate step?
If you are thinking only about decryption, then indeed the user has to give two passwords, which has zero added security as opposed to just one password (provided both have enough entropy, because their entropy is compounded). That is because both can be easily intercepted by a key-logger, leaked from memory, etc.
However if you think of the whole workflow you can see why they could be useful. Imagine this: you and a colleague are working remotely on a project; you want to share a very secret information over chat; your colleague has already created an identity file and shared that with you (over another secure mean); you could just use that identity to encrypt your very secret information, but then anyone that gets access to his laptop (which has both the private key for the identity and the chat history) can decrypt it (especially since age doesn’t necessarily encourage encrypted identity files); however, now if you add a shared password (that you communicate over phone for example), the attacker now must also wait (or trick) your colleague into revealing that password.
If you need secret key X and passphrase Y in order to decrypt some ciphertext, then however you compose X+Y will produce some composite secret Z, which is the actual secret you need for decryption. As far as I can tell this puts you back to square one, the secret is really Z, you’ve just Horcruxed it up into a few separate pieces. You can do that with any secret. Chop up your ED25519 key into 4 byte chunks, each on its own USB stick, and ship them to the far corners of the earth — this will probably annoy your attackers, but it won’t improve the strength of the encryption. As far as I understand.
As far as I can tell this puts you back to square one, the secret is really Z, you’ve just Horcruxed it up into a few separate pieces.
Exactly. It’s not “magic crypto”. It’s just “how can I make it easy for the user to provide two passwords instead of one”. (Although, in my case, because I offer multiple “kinds” of passwords, PINs and secrets, by choosing one or another, the user also chooses the resources required to brute-force them, via passing them through Argon2 with different parameters.)
To put it another way: Age doesn’t do anything magical; it just combines various cryptographic primitives to provide a tool to the user; I could just as well take a piece of paper and compute everything myself (perhaps in a millennium). That doesn’t make Age useless; it makes it a dependable tool.
As we all know: never implement your own cryptography!
Cryptography is not magic, and sometimes you have to implement it even if you’re not an expert (those expert’s aren’t always available). It will take more than a weekend though.
BTW, I’ve read your article initially when it appeared. I’m glad you’ve written it, because to my knowledge it’s the only one encouraging other non-cryptography-experts to give it a try. :)
Agree. I’ve been ruminating, experimenting, and tweaking this since since January, and slowly I want to publish some of my ideas to find some feedback (as I’ve already received so far here).
Moreover, I’ve not actually “invented” or “implemented” my own cryptographic primitives, but instead relied on (what currently is considered) state of the art as implemented by others (in Rust); for example I’ve used only the following primitives:
Blake3 for anything hashing related, from MAC to key derivations; (and I’ve used religiously key derivation with context, thus reusing the same key for different purposes is eliminated;)
Argon2 (id) for deriving anything that might be human generated (mainly PIN’s, secrets) or for reducing the possibility of brute-force;
ChaCha20 for symmetric encryption; (I know of XChaCha20, however I limit the encryption to at most 128 MiB; any larger than that and it doesn’t fit the purpose of this tool;)
X25519 (and DHE) for asymmetric keys;
In the end I didn’t do anything that libsodium wouldn’t do, except I’ve put everything together for a specific purpose.
ChaCha20 for symmetric encryption; (I know of XChaCha20, however I limit the encryption to at most 128 MiB; any larger than that and it doesn’t fit the purpose of this tool;)
Implying ChaCha20 has a practical limit on message lengths is a bit of a shortcut: it is not widely understood, but there are three ChaCha20 variants:
The original, and in my opinion “one true” variant, as was designed by DJB: 64-bit nonce, 64-bit counter. The big counter means message length is effectively unlimited (we’ll never reach 2^73 bytes in practice).
The IETF variant: 96-bit nonce, 32-bit counter. I understand the modification was done for TLS compatibility. The nonce is still too small to be random though, and now messages are limited to a realistically attainable length (256GiB).
XChaCha20: 192-bit nonce, 64-bit counter. It would be the best, if it wasn’t for that pesky HChacha20 overhead. Still, it’s my preferred default when I make an AEAD API for end users.
When I design a protocol myself however, I design for a 32-bit counter and a 64-bit nonce, thus ensuring good efficiency and maximum compatibility.
Indeed, choosing the “right” ChaCha20 was quite problematic for me (especially since I’m not a cryptographer). However I took the practical approach: choose the “standard” one (as in the one ratified by IETF), because that is the one most likely to be implemented in other libraries.
I don’t get what you want to do. Do you want to open a encrypted channel, encrypt a message and send it to someone else, or encrypt a message to yourself (i.e. for backups or full disk encryption)? Only in the last case you can control the security of the used secrets.
Also your text is full of misconceptions. I don’t even know where to start. Some examples:
Oracles are mostly a theoretical construct used in security proofs. Sometimes oracles are used in attacks to break the security of a protocol (i.e. a RSA padding oracle can help to decrypt RSA encrypted messages).
PINs are for authentication not for encryption. Yes you could use them for encryption, but it would be easy to break.
DHE is a Key exchange algorithms. It can be used to exchange a key in a online protocol (like TLS). This key then can be used for encryption.
I don’t get what you want to do. Do you want to open a encrypted channel, encrypt a message and send it to someone else, or encrypt a message to yourself (i.e. for backups or full disk encryption)? Only in the last case you can control the security of the used secrets.
In all the main use-cases (but this doesn’t have to be so) the same “owner” is at the both ends (encryption and decryption) although it might be at different times or with different levels of involvment. For example, here are a few use-cases:
encrypting small files (scripts or configurations) to be used in semi-unattended setups; (for example scripts to be used at the startup of a VM: create an X25519 key for the VM, create another key for the administrator, put the script on S3; etc.)
encrypting other sensitives pieces of information (like service credentials) and have them decrypted while running over SSH (with SSH-agent forwarding);
Oracles are mostly a theoretical construct used in security proofs. […]
Indeed. However, think at the functionality that an oracle exposes: give it as input some data and it will deterministically output back some other data (hopefully random), however nobody can guess without access to the oracle what the output might be. (In real life, any HMAC-like construct, any RSA / Ed25519 signature, could be considered instantiation of such oracles.)
So, by using such oracles one could achieve the following (which can’t be achieved solely based on secrets):
acting as a two-factor by protecting access to the oracle; (the USB token isn’t plugged, you aren’t in the same network, etc.)
rate-limiting guesses; provided the attacker doesn’t also compromise the machine where the code runs, that code could rate-limit (either in software or by actual hardware means) the rate at which its services are called;
acting as an audit log; each interaction could be logged, so that in case of a breach, one could check who might have tried to access it;
So, in the end, what is (in terms of functionality) the difference between a KMS, an USB RSA Token, your ssh-agent or a simple service that does a HMAC with a secret key, when all you want from them is to “mix” some data you gave them as inputs?
PINs are for authentication not for encryption. Yes you could use them for encryption, but it would be easy to break.
And that’s exactly why I’ve called them “PIN’s”; nobody would be taken by surprise if someone brute-forces a 4 digit pin.
DHE is a Key exchange algorithms. It can be used to exchange a key in a online protocol (like TLS). This key then can be used for encryption.
So what am I doing wrong then? DHE is not only for online protocols. In fact it’s one of the main constructs used all over libsodium for various kinds of “boxes”.
give [an oracle] as input some data and it will deterministically output back some other data (hopefully random), however nobody can guess without access to the oracle what the output might be. (In real life, any HMAC-like construct, any RSA / Ed25519 signature, could be considered instantiation of such oracles.)
Are you speaking about (general) oracles, or random oracles?
Based on my understanding of the Matthew Green articles and the Wikipedia entry a random oracle is not satisfied by the properties you list here, which AFAICT describe a cryptographic signature. Random oracles rely on state derived from an input history, whereas signatures are essentially stateless. And in fact many papers like this one explicitly state that “random oracles do not exist in the real world!”
So, by using such oracles one could achieve the following (which can’t be achieved solely based on secrets):
A cryptographic signature is based on a private key, right? Is a private key not just a secret?
Are you speaking about (general) oracles, or random oracles?
Obviously we can’t actually use “true random oracles” because these are not implementable in software; just as we can’t speak of “true random generators”.
However, colloquially we call “pseudo random generators” just plain “random generators” (or “secure random generators”) and everybody understands that they are imperfect instances of the theoretical term. The same here: when I say “random oracle” I mean a “pseudo random oracle”, as obtained for example from a HMAC.
Based on my understanding of the Matthew Green articles and the Wikipedia entry a random oracle is not satisfied by the properties you list here, which AFAICT describe a cryptographic signature.
In cryptography, a random oracle is an oracle (a theoretical black box) that responds to every unique query with a (truly) random response chosen uniformly from its output domain. If a query is repeated, it responds the same way every time that query is submitted.
Now, getting back to Matthew’s series on the subject of random oracles, although they are very insightful, I think he spent 5 long articles talking about them, but failed to give a concrete definition in one phrase as Wikipedia did. :)
However, if you read between the lines, you’ll see that most often than not random oracles are embodied by hash functions or pseudo random functions.
Random oracles rely on state derived from an input history, whereas signatures are essentially stateless.
<<Random oracles rely on state derived from an input history,>> is completely wrong. Matthew Green gave an example on how a random oracle might be implemented (by starting with an empty history, and for each item, coming up with a random output, and remembering that). However, random oracles, respond to inputs the same way regardless of the order the inputs are given.
And in fact many papers like this one explicitly state that “random oracles do not exist in the real world!”
As said, I’m interested in practical, imperfect, real-world instantiations of this concept.
So, by using such oracles one could achieve the following (which can’t be achieved solely based on secrets):
A cryptographic signature is based on a private key, right? Is a private key not just a secret?
Indeed, a private key (or anything that can be used to implement a “pseudo random oracle”), is nothing else than “cryptographic key material” that has to be kept secret. And from a functional point of view, it’s just like our algorithm contains inline what the oracle would do.
However, and this is what I’ve found enlightening, we forget that the concept of an “oracle” implies that it’s a third party (in real terms a different process, on different hardware, or a different machine, perhaps even on a different network)!
Thus the attacker now has to compromise two systems:
the user’s system to get the encrypted data and any decryption “secrets” that might be needed;
the oracle’s system, by tricking it to reply to our calls; (if the oracle requires additional authentication, like physical presence, it makes the life of an attacker even harder;)
So, just like USB RSA tokens are nothing else than circuitry that implement RSA (perhaps even in software on a micro-controller) and a place to keep safe the RSA private key, there is nothing else “magical” about them. So with the oracles I speak about: they use plain algorithms, but “at a distance”.
<<Random oracles rely on state derived from an input history,>> is completely wrong.
Can you ELI5 why?
However, and this is what I’ve found enlightening, we forget that the concept of an “oracle” implies that it’s a third party (in real terms a different process, on different hardware, or a different machine, perhaps even on a different network)!
If you allow a cryptographic signature, derived from a private key, to satisfy a random oracle, then it’s no longer guaranteed to be a third party, right? Anyone with access to the private key becomes the oracle, indistinguishable from any other. Furthermore, does this not mean that the output is not actually random, but rather a pure function of the input and the private key?
I was almost about to ask you what ELI5 is; however I think it stands for explain like I'm 5 (years old), according to Google. :)
<<Random oracles rely on state derived from an input history,>> is completely wrong.
As stated (in the wikipedia article and in many places in Matthew’s articles) an oracle is a “black box” that receives inputs and deterministically answers “random looking” outputs, plus it’s formally defined as f (input) -> output.
Thus, because the inputs history (which can be formally defined as an ordered sequence of inputs) is not part of the actual input, the oracle is deterministic only based on the individual (current) input, regardless of other inputs that it might have received in the past.
Therefore, the only requirement of the oracle is to respond to the same input with the same output regardless of who, when, in what order, (or even if) one asks it.
Stating <<random oracles rely on state derived from an input history>> implies that an oracle’s reply would depend on the order in which the inputs are given, and not only on a particular individual input. (i.e. for i in 1..5 do oracle(i); o_6_a = oracle(6); for i in 5..1 do oracle (i); o_6_b = oracle(6); assert (o_6_a == o_6_b);.)
Granted, one could envisage an oracle that does the following (basing this algorithm on Matthew’s example with the history):
the oracle starts with an empty “history” (i.e. a hash-map);
the oracle starts with a 128 bit register initialized to a random state;
on each input, the oracle looks in the history hash-map, and if an entry exists, it replies with the cached value;
else (the output is not cached), the oracle computes register = md5 (concatenate (register, input)), then use the current register as the output (which is then cached);
Such an oracle would technically “rely on state derived from an input history”, however once an input is presented, the oracle makes up its mind on the chosen output and will forever give that as output (for the same input). Also, granted, if the oracle (or the virtual world it lives in) would be “reset”, and the inputs would be presented in another order, the outputs would be different (provided the register is initialized to a constant value like all zeroes).
But all this doesn’t break the following two properties of the oracle:
once an oracle is instantiated (which happens when the virtual world is reset), it is deterministic in its outputs (for each individual given input);
the oracle is a black-box, and we don’t know its implementation, thus we can’t guess its outputs without first asking; (it might, for all we know, return the hash of a counter, with the caching to keep the determinism;)
I think I understand your position to be that (1) a random oracle is reasonably approximated by a cryptographic hash function, in practical terms; and maybe also that (2) the good stuff we get from random oracles we can also get from cryptographic hash functions. Is that correct?
I certainly buy (1), but I’m stuck on (2). As far as I can tell, the whole point of random oracles is to define something explicitly with stronger guarantees than what can be provided by hash functions. Quoting Wikipedia,
Random oracles are typically used as an idealised replacement for cryptographic hash functions in schemes where strong randomness assumptions are needed of the hash function’s output.
So when you say
the only requirement of the oracle is to respond to the same input with the same output regardless of who, when, in what order, (or even if) one asks
I don’t really get it. This property is satisfied by a function f(x) = 1, right? And that can’t possibly be a valid random oracle, right?
(1) a random oracle is reasonably approximated by a cryptographic hash function, in practical terms;
Exactly. (In fact I think our whole cryptographic ecosystem is based on this assumption. From what I gather most of the security constructs we have are proved in the “random oracle model”, and then the “random oracle” is replaced with hashes or pseudo-random-functions. At least based on what I’ve read, and especially Matthew’s articles.)
(2) the good stuff we get from random oracles we can also get from cryptographic hash functions.
This is, from my point of view, the same as point (1).
However, and I want to stress this, you are completely missing one characteristic of the random oracle (both in theoretical terms, and how it can be used in practice): random oracles are not just another algorithm; they are independent agents running outside the constructs we implement, test or try to prove; in practical terms, a random oracle (as I view it in the real world) might be implemented with a HMAC, but it’s running in a completely separate process than the software trying to encrypt / decrypt. That’s the whole thing with them; the attacker can’t (at all theoretically, but realistically easily) peek into them.
To reiterate an example: 2FA (two factor authentication), either implemented as a USB token, or via TOTP in a software “authenticator” running on mobile (where the input is the time), that is a random oracle. It hasn’t anything magic in it; it follows a standard HMAC-based algorithm; but it strength lies mainly in the fact that the attacker which might have compromised my workstation / laptop, can’t easily also compromise my mobile.
So when you say:
the only requirement of the oracle is to respond to the same input with the same output regardless of who, when, in what order, (or even if) one asks
I don’t really get it. This property is satisfied by a function f(x) = 1, right? And that can’t possibly be a valid random oracle, right?
I have considered as asserted the fact that a “random oracle” should return “random outputs” (for whatever the definition of “random output” we accept). Thus the two properties I’ve listed there were mainly focused to prove the point that a random oracle’s output doesn’t (in general) depend on the order of its inputs.
Thus, the function f(x) = 1 might be an “oracle”, but it’s not a “random oracle”.
To reiterate an example: 2FA (two factor authentication), either implemented as a USB token, or via TOTP in a software “authenticator” running on mobile (where the input is the time), that is a random oracle. It hasn’t anything magic in it; it follows a standard HMAC-based algorithm; but it strength lies mainly in the fact that the attacker which might have compromised my workstation / laptop, can’t easily also compromise my mobile.
The term “random oracle” means something stronger than what you are describing here. If when you write “random oracle” you actually mean “pseudorandom oracle” then I guess I’d ask you to please make that explicit.
random oracles are not just another algorithm; they are independent agents running outside the constructs we implement, test or try to prove; in practical terms, a random oracle (as I view it in the real world) might be implemented with a HMAC, but it’s running in a completely separate process than the software trying to encrypt / decrypt. That’s the whole thing with them; the attacker can’t (at all theoretically, but realistically easily) peek into them.
If a cryptographic-hash-based oracle runs in the same process as my application, and its secret material exists in the same memory space as my code, then it is relatively easy for me to exfiltrate it. If that same oracle runs as a separate process on the same host as my application, then it is harder for me to access that secret material. If the oracle and my code run as separate VMs under the same hypervisor on a single host, then it is that much more difficult. If the oracle runs on a separate host than my code, then it is harder still. If it runs in a secure enclave on a separate host behind a tightly controlled firewall, even harder. But as long as the oracle, running in any of those environments, is performing a cryptographic hash/signature based on a private key, then these are all differences of degree, not of kind. As far as I understand.
The term “random oracle” means something stronger than what you are describing here. If when you write “random oracle” you actually mean “pseudorandom oracle” then I guess I’d ask you to please make that explicit.
OK, please imagine that everywhere I’ve said “oracle” or “random oracle”, I’ve actually meant “pseudo random oracle”. :)
If a cryptographic-hash-based oracle runs in the same process as my application, and its secret material exists in the same memory space as my code, then it is relatively easy for me to exfiltrate it.
But that is not an “pseudo random oracle”; the whole purpose of these is not to run in the same process space as our software. That’s the whole trick I was trying to stress.
But as long as the oracle, running in any of those environments, is performing a cryptographic hash/signature based on a private key, then these are all differences of degree, not of kind. As far as I understand.
Theoretically you are right. It’s a difference of degree, not of kind.
But then, a 4 digit password is the same as a 100 random printable ASCII character password; they have the same “kind”; but in practical terms the “difference of degree” is what matters.
Obviously we can’t actually use “true random oracles” because these are not implementable in software; just as we can’t speak of “true random generators” … when I say “random oracle” I mean a “pseudo random oracle”, as obtained for example from a HMAC.
The term “random oracle” is a theoretical concept with certain properties that are, as you note, not implementable in practice, but nevertheless is well-defined. There is a categorical difference between a random oracle and a pseudorandom oracle. Conflating the former with the latter is, I think, one of the points of confusion in this conversation.
However, colloquially we call “pseudo random generators” just plain “random generators” (or “secure random generators”) and everybody understands that they are imperfect instances of the theoretical term.
There is definitely also a clear difference between a pseudorandom number generator vs. a cryptographically secure random number generator. One common manifestation of that difference is the behavior when entropy is exhausted: the weaker generator will make shit up, and the stronger one will block or error out.
There is definitely also a clear difference between a pseudorandom number generator vs. a cryptographically secure random number generator.
What is a “pseudorandom number generator”?
If you ask Google or Wikipedia it either shows you “weak random number generators” (like the Mersene Twister) or “cryptographic random number generator”; both are “pseudo”. Thus, even with this long term “pseudorandom number generator”, we know what we mean from the context… The same with my “oracle” (or “random oracle”) name: in this context, it means some software or hardware agent that tries to have the properties of a “theoretical random oracle”.
One common manifestation of that difference is the behavior when entropy is exhausted: the weaker generator will make shit up, and the stronger one will block or error out.
Can you point to even a single cryptographic secure random number generator that just panics or errors (or even blocks) when it has exhausted its “entropy”? (Which I assume you mean “when it has reached the risk of wrapping around and repeating”.)
The Linux one (/dev/random which recently I think is the same as /dev/urandom) just “makes shit up” continuously. The rand Rust crate doesn’t seem to return error conditions (unless there is an I/O error).
Can you point to even a single cryptographic secure random number generator that just panics or errors (or even blocks) when it has exhausted its “entropy”? (Which I assume you mean “when it has reached the risk of wrapping around and repeating”.)
Entropy is a resource on a system, comparable to CPU or memory, which can be exhausted in similar ways. There isn’t really any concept of “wrapping around” as far as I know. Typically, /dev/random will block when entropy is exhausted, and /dev/urandom will not. Similarly, Go’s math/rand is pseudorandom, and crypto/rand is cryptographically secure. I’m not super familiar with Rust, but I’d certainly expect there to be a way to read both non-blocking pseudorandom bytes, and blocking/erroring cryptographically secure bytes, based on user need.
Entropy is a resource on a system, comparable to CPU or memory, which can be exhausted in similar ways.
I would say “it depends”. For a physical / hardware system, indeed entropy is something that is accumulated over time, thus it can’t be exhausted.
There isn’t really any concept of “wrapping around” as far as I know.
However, for an algorithm based solution, like for example many use ChaCha for random number generation, there is a limit on the amount of data that can be generated before the counter wraps around, and the generator starts outputting the same values. Thus, we could say that in this case the “pseudo entropy” is exhausted.
Typically, /dev/random will block when entropy is exhausted, and /dev/urandom will not.
It used to be so… Not anymore… I can’t find now the reference, but I remember in some recent Linux kernel they’ve made both /dev/random and /dev/urandom the same.
For example pv < /dev/random > /dev/null just yielded ~15 GiB of data in less than a minute… Clearly that entropy isn’t real for my laptop.
Similarly, Go’s math/rand is pseudorandom, and crypto/rand is cryptographically secure.
Yes, but neither blocks or errors (or panics).
I’m not super familiar with Rust, but I’d certainly expect there to be a way to read both non-blocking pseudorandom bytes, and blocking/erroring cryptographically secure bytes, based on user need.
Apparently there isn’t. Because apparently nobody wants his software to stall until the user jerks the mouse. :)
Some applications … need random data. … To supply this data, a system keeps a pool of random data, called entropy, that it collects from various sources of randomness on the system … These sources of randomness can only supply data at certain rates. If a system is used to do a lot of work that needs random data, it can use up more random data than is available. Then software that wants random data has to wait for more to be generated or it has to accept lower-quality data. This is called entropy starvation or entropy depletion.
—
Yes, but neither blocks or errors (or panics).
crypto/rand will block when entropy is exhausted.
edit: “Exhausted” here means “unavailable at the moment” and not “unavailable forever”.
Because apparently nobody wants his software to stall until the user jerks the mouse. :)
If my application requires cryptographically strong random information, and it isn’t available, then I absolutely want my application to stall or error, rather than carry on with data that doesn’t meet my requirements.
Your use of PIN is slightly confusing because normally it’s not that these are for low security situations. It’s that they are used for places that have some intrinsic immunity to offline attacks. For example, the PIN in credit card transactions is validated by the chip on the card. The PIN for Windows or iOS / Android device unlock is used to access a key in a TPM or other hardware root of trust. The PIN in Signal is used to recover your key from some code running in an SGX enclave in the cloud. All of these things do rate limiting, so it doesn’t matter that you can generate all of the possible values in a few ms, you get to try them once per second with something like exponential back off so that, practically, you get at most tens to hundreds of guesses.
The key phrase that you want to search for is ‘key derivation function’. It’s common to have a single secret stored in some hardware token (TPM, U2F device, and so on) that can never be extracted but can be used as one of the inputs to a KDF. If you have such a thing, then there’s a simple flow of using a password hashing algorithm (Argon2 unless you have a really good reason to do something else) to generate the other input. This flow can be chained: the result of the first KDF can be used as an input to the next one, so if you have a chain of passwords or other secrets then you can hash them and then apply them.
If you want to use n out of m passwords then you have a much harder problem. Something like Shamir’s Secret Sharing algorithm lets you derive a set of parts of a key and there is some work to allow each holder to do part of an encryption independently, but I’m not aware of a way of defining the shares and then a function that combines a subset of them to consistently generate the same output and that sounds like a very hard problem. You’re effectively running error correction the wrong way around and I don’t see a way of doing that unless the system can generate one of the secrets that you must provide (then it’s easy, but it’s n user-provided secrets plus one fixed one and owing that one and some of the secrets probably lets an attacker guess some of the others with high probability, so that one would have to be stored in tamper-proof hardware).
Your use of PIN is slightly confusing because normally it’s not that these are for low security situations. […]
Indeed what you are describing is correct. However, and the reason I called them “PIN”, is because these have very low entropy. How else could I call them so that users understand their intended meaning?
The key phrase that you want to search for is ‘key derivation function’. […] This flow can be chained: the result of the first KDF can be used as an input to the next one, so if you have a chain of passwords or other secrets then you can hash them and then apply them.
And this is exactly how it’s implemented. Each kind of token (PIN, secret, “ballast”) is passed through Argon2id with various CPU and memory requirements.
It’s common to have a single secret stored in some hardware token (TPM, U2F device, and so on) that can never be extracted but can be used as one of the inputs to a KDF.
And this is exactly how it’s implemented. As part of the derivation chain, intermediary values are passed through what I call “oracles” (which at the moment only ssh-agent-based is implemented, but anything that has an RSA or Curve25519 key would work).
If you want to use n out of m passwords then you have a much harder problem.
At the moment this is not implemented. The main reason being that I don’t think there are any mature enough implementations out there for me to use.
BTW, besides Shamir Secret Sharing I’ve seen a few other papers that describe alternatives (I don’t have the reference now) but I don’t think they have been yet properly implemented (or reviewed).
I may be misunderstanding what you’re trying to do, but can’t you just concatenate the secrets (with appropriate delimiters), run that through a cryptographic digest, and use that as the key? Or for added security, use a key-derivation function instead of a digest.
Technically (i.e. from a cryptographic point of view) mashing (through canonical concatenation with or without hashing or intermediary derivation) two (or more) “passwords” together and using the result as the new “password” is equivalent (to using separate “passwords”).
However, there are a few issues that I try to solve:
first of all the user experience – one “password” could come from an environment variable and another from a file; with proper arguments you don’t need a wrapper script to mash them together;
as mentioned earlier, the key of doing all this correctly is canonicalization; just joining multiple tokens with a space is not a very good idea;
then, at least in my proposed case, there are multiple types of “passwords”, mainly differentiated by the intended use-case and by their derivation requirements (in terms of CPU and memory); for example PIN’s (short and weak), “secrets” (high entropy) and “ballasts” (high entropy with high demand on the derivation), plus “associated context” (public information binding the encrypted file to a particular context or purpose); you can’t just mash these together;
I don’t think using JSON is a good idea, mainly for the following reasons:
there is no standard canonical representation; in essence, each algorithm has to define its own; (and string representation, especially in the presence of Unicode, I think it’s the hardest part;)
there aren’t many implementations that actually support canonical representation (for any arbitrary definition of “canonical”);
In recent months I think CBOR is preferred for such purposes.
However, at least in my implementation, for practical purposes (because performance isn’t a major concern for me), I don’t need canonicalization, because internally I always use 32 octets (256 bits) “keys” or “hashes” (either from Blake3 or Argon2id or DHE) with either a fixed number, or prefixed by their count.
Oops, you’re right about JSON; in my main project we do use it for that purpose, but that’s only because we had already implemented a canonical encoder for other reasons.
I wonder if it’s the UX of mail clients vs other things that’s a contributor. I can’t think of why this is a problem since mailing lists also have topics within them. Maybe the time is ripe for people to gather and come up with best practices for this type of email use case in this day and age?
The UX of mail clients is definitively a major issue for high-volume mailing lists…
I have extensive “experience” into this with GMail, and if it weren’t for the “mute” option I would have renounced using mailing lists much sooner. Also, sticking with GMail here, creating and maintaining filtering rules is almost a complete madness, so much so that at one point I’ve written my own Racket-based DSL to generate (and import) GMail rules via their XML format. However, besides basic labeling, GMail is useless in this regard, which is the main reason I’ve ditched it a few years ago.
I since use Thunderbird for “normal” emails, but not for mailing lists… Because Thunderbird also seems to be useless in this regard…
I had some success with notmuch, which allows quite advanced labeling workflows, but that seriously lacks a proper UI.
Thus, at the moment I do my mailing list consumption via GMail, and any serious searching via notmuch. But all in all, I would say I’ve completely abandoned mailing lists…
Unfortunately, Chris describes exactly what I’ve used to do also, namely be subscribed to a firehose of technical mailing lists, at it’s peak up to ~350 mailing lists at once; now I’m down to about a handful, especially for niche projects. These days I’ve switched to following Lobsters links and directly following various authors blogs RSS.
However, I still think that email is (technically) superior to “forums” (which these days means Discourse), because anyone is free to choose his user-agent and implement whatever filtering or archival workflow one wants. With Discourse you can’t… (Although it has support for emails, that is broken, or at least was broken a few years ago.)
Also forums, at least for me, didn’t solve the high-volume issue, as for example I’ve tried following the Rust users Discourse forum, and it was as “bad” (in terms of follow-ability) just like other users mailing list. Both had high-volume (sometimes low-quality) traffic.
Perhaps what we are missing are low-volume and/or high-quality mailing lists focused on particular topics.
The flip side of being able to choose your own client is that other people can too. The FreeBSD lists are increasingly difficult to follow because people read them on the web and reply with things that then don’t have the thread ID metadata, so it’s impossible to see where their reply fits in a long thread and the threads end up being displayed as a set of disjoint threads. In contrast, LLVM’s Discourse doesn’t have this problem and also doesn’t have the problem that someone replying to a five-year-idle thread (e.g. a new person asked a question, it was answered, and the answer is now wrong after API changes and the next person wants help) is referencing context that a load of subscribers don’t have.
The biggest win for LLVM was that the project had a large set of lists with overlapping audiences. C++ things might need to go to the libc++ and clang lists, but then some people would hit reply all and have their replies bounced from the lists they were not on. It’s much easier to follow a set of tags and have threads gain tags once people realise a particular audience would care about something.
Indeed, a “forum software” (mainly Discourse) might be much better equipped to handle long-form threaded communications between multiple people; while also supporting things like proper Markdown and code snippets. However, most of the problems you’ve described stem mostly from not respecting best practices (use proper quoting, use proper email clients, etc.), which forums seem to properly enforce.
In fact, I agree that perhaps emails aren’t the best solution as “forums”, however they have something that no forum software has: flexibility.
Picking on Discourse (because it’s the dominant software in this area):
can I have one account and use that for all my communities? (not with Discourse, but yes with emails;)
can I have the same settings (user preferences) applied to all my Discourse accounts? no; I have to manually tweak each user-preference (time-zone, notifications, theme, etc.) across all my communities;
can I have an archive of all my interactions in a certain community (or even across all)? no; to my knowledge I can’t easily export all the threads I’ve been involved in a certain community; and even if it would be possible I would have to manually do it;
can I use an alternative user-agent as opposed to the standard one (i.e. the web-based UI)? no; although as said I could use email-based workflow but that was broken, specifically due to the fact that Discourse didn’t actually understand the List-* header purposes;
So, perhaps in terms of direct usability Discourse is a better choice, but it completely fails in other respects.
Perhaps we need an email-like protocol specifically for this purpose? Perhaps the Discourse team could propose one? (Although Discourse is a business, thus it’s perhaps against their incentives to do so…)
can I have one account and use that for all my communities? (not with Discourse, but yes with emails;)
I use the log in with GitHub option, so effectively yes.
can I have the same settings (user preferences) applied to all my Discourse accounts? no; I have to manually tweak each user-preference (time-zone, notifications, theme, etc.) across all my communities;
That’s definitely missing from the web site, though I believe the app does better.
can I have an archive of all my interactions in a certain community (or even across all)? no; to my knowledge I can’t easily export all the threads I’ve been involved in a certain community; and even if it would be possible I would have to manually do it;
It’s possible to export everything from an instance, I don’t know if you can export all of your threads. Sounds like something that would be useful though. Have you tried raising a feature request?
can I use an alternative user-agent as opposed to the standard one (i.e. the web-based UI)? no; although as said I could use email-based workflow but that was broken, specifically due to the fact that Discourse didn’t actually understand the List-* header purposes;
From what I can tell, everything that the Discourse app and web app use is exported over their REST API, so you could in theory. I don’t know if there are other clients that do this. Note that your other requests could also be done by writing some code that calls their API.
Regarding feature requests, since I don’t use Discourse, I don’t think asking for any features would be fair (or helpful for me)…
I did at one time try to convince the developers to properly handle List-* headers and completely failed, thus I gave up… (The issue was, as I remember, that they did add some List-* headers, but also added the To: me@example.com header, which basically broke the mailing lists conventions, and thus labeling efforts.)
Regarding the “app”: is there a “desktop app”? I’m not so much into “mobile”, and anyway I kind of hate that everything nowadays has an “official app”… Is there some choice in “apps”, or is there only a single “blessed one”?
Hmm, randomization is a bit different, yes, but it is open-loop in the sense that it doesn’t care about how well does the server respond to the requests, it just sends as many requests as specified. Connection count in this case entirely depends on how well the server is responding, if it’s time to send a new request and all of the existing connections are waiting for a response, it opens up a new connection and sends a request there.
Adding some noise generation options to it would be nice though, and possibly quite easy.
Frequently the problem is exposing a concept that seems useful but isn’t. For example, the author has a couple of examples about Python datetimes, but the core problem with Python’s date libraries (besides the awkward API) is that naive datetimes are an extremely niche usecase. Basically, they should only be used for calendar apps to have repeating events happen at the same local time. But it seems like a naive datetime is a convenient idea, so there it is and everything else follows from that core mistake.
It’s also probably worth splitting these things into intentional and unintentional approaches. Things like OpenSSL are well intentioned but have a load of API misuse footguns. Similarly, GPG tries to do the right thing but provides users with a bewildering interface that can cause them to not have security by accident. In contrast, things like cookie banners and Facebook intentionally try to pull users into these paths.
Speaking strictly of either OpenSSL and GnuPG, I agree that many of their complexities and API foot-guns are because they need to implement complex (and sometimes) broken protocols.
However, I think they are missing a wrapping layer, something like libsodium: give the user two libraries:
the current low-level API with a big warning that normal users shouldn’t touch it;
the libsodium alternative, made only of safe functions that are (if not impossible then at least) hard to misuse;
Speaking strictly of either OpenSSL and GnuPG, I agree that many of their complexities and API foot-guns are because they need to implement complex (and sometimes) broken protocols
That’s not quite what I was saying. They could do a lot better with their APIs, but if you use the APIs as they intend then you have a benefit, it’s just that they didn’t do a great job. In contrast, with so-called ‘dark patterns’ the designers are deliberately trying to make you use them in a way that is harmful to the user. Both appear similar (any sufficiently advanced incompetence is indistinguishable from malice) but the fixes are very different. In the case of OpenSSL, it’s learning from things like libsodium (huge fan of that API). In the case of Facebook, it’s jail time for executives.
I think I understand the distinction you are trying to make, between involuntary “complications” and malicious “dark patterns”. However, I think the author is focusing on the unintended ones.
With regard to the dark patterns, I think the only possible solution is user education. There is no incentive for Facebook (or any other site with “adware”) to renounce the dark patterns. And I don’t think we can (or should) force them to by law (as long as they don’t hurt people in the process). However, we could try to give them negative publicity, or even better, come-up with better solutions.
Although I hope I’m wrong, I don’t think the current standards (i.e. WebAuthn, or recently PassKeys) are a proper solution to passwords (and password managers) for a few simple reasons:
they are mostly tied to hardware (either some USB thing, or some TPM / enclave on the user’s laptop / mobile);
they can’t be backed-up or stored or cloned as one would store a password; (which stems mostly from the previous point;)
However, there is no technical reason why this should be so, because both proposed standards could easily be implemented in software. The main issue is that in practical terms both of these are described in terms of API’s that the browsers have to implement, and for the moment the only implementations are either in the OS (iOS, Android) or via USB… (There are some “virtual tokens” that use USB-over-IP, but in practical terms this is ridiculous because it asks me to run something as root to “improve” my security…)
So, for the foreseeable future it appears I’ll have to stick to passwords…
(Although for anyone that doesn’t have decent security practice, they are much better than plain passwords. It’s just that for advanced users they aren’t practical enough.)
FWIW, PassKeys is essentially just a profile of WebAuthn - the requests and responses are essentially the same, you can reuse essentially all of the code for both. PassKeys is just how Apple and Google decided to market their phones being able to act as WebAuthn authenticators.
Also, for what it’s worth, I don’t see the problem with WebAuhtn/PassKeys being tied to hardware and having no key backup or cloning. The strategy propagated by them is to have multiple authenticators, that are independent, and if one is lost, another one can still be used. I feel this is a lot better than to have the one true key, that you need to move around. After all, the least secure location for a key is being in transfer, and WebAuthn/PassKeys aim to eliminate the need for key movement.
I do agree that the API’s available for the methods is currently lacking though.
FWIW, PassKeys is essentially just a profile of WebAuthn […]
I’m not so sure that PassKeys and WebAuthn are exactly the same thing, with different branding name; they might be, they certainly might have common cryptographic underpinnings and details, but when I’ve tried to understand how they actually work I don’t think I’ve stumbled-upon something simple enough (but with technical details) to understand exactly what they are underneath… A lot of marketing, a lot of BS, not enough sensible documentation…
With regard to backups, they are a complete show-stopper for me personally:
what is the cost of a physical FIDO token, compared with a dumb plain USB stick or SD card? for the price of one FIDO token I can buy at least 10+ USB sticks / SD cards and store as many copies of my passwords as I want;
most physical hardware is more brittle than even the cheapest piece of paper; (I can print out my passwords, in text or QRcode, laminate the page, and put it in a safe place;)
Furthermore, what are the security measures for FIDO (and the like) tokens?
some have none, plug them and you’re in!
some have fingerprints; do we actually trust fingerprint readers for authentication?
can I apply my own policy? protect them with an extra passwords?
can I review the “implementation”?
So, for the time being, I would actually prefer software-based (i.e. open source) tokens.
Webauthn spec is indeed not incredibly readable, but AFAIK that’s the case for all W3C specifications that deal with JS interfaces… The overview on MDN is a lot easier to understand since it explains things from a higher level.
Communication between a browser/OS and a physical token is described by CTAP2 specification, which is pretty lengthy, but it deals with a lot more stuff than just authentication.
Passkeys seem to not have a specification, but rather just be a term that’s co-marketed for essentially an embedded platform token. The cases where a phone is used for authenticating in a desktop browser is just CTAP2 over Bluetooth.
Cost for FIDO tokens right now is about ~20-30$ per key, depending on the vendor. Not the lowest cost ever, but IMO fairly bearable. The hardware I have has survived for about 4 years already, while staying with my keys of the time.
Newer hardware available is a lot tougher - I’ve heard stories of tokens going through a washing machine and surviving.
As for security: there’s a userVerification parameter in the API - with the possible values of discouraged, preferred and required - the default one is preferred. What this means depends on the authenticator, but even the simplest ones usually have PIN codes (entered through computer) as an option, and will ask user to enter them if the PIN has been enrolled in preferred and required use cases. Application of any additional policy depends on the authenticator, some do allow this to be customized, some not.
Open source tokens do exist by the way, for example Solo which is my token of choice. It doesn’t have any custom silicon, it’s just a secure micro-controller with an open source firmware. They sell models that have been flashed with an unlocked bootloader, so if you want, you can load up your own custom version of the firmware and lock the bootloader so only versions could be used.
For a syncable embedded platform token. That’s the whole point, that’s why they decided that it’s marketable to regular users already. Apple’s had a soft WebAuthn token tied to one device before and it was hidden in developer options. Now it’s “production ready” exactly because now the keys are synced to all your devices via iCloud Keychain (/ Google whatever). And yeah e.g. Apple does have recovery for when you lost all devices too.
Thanks for the video / interview! It’s a refreshing and apparently non-mainstream take, especially now in 2023 when Emacs almost seems reasonable in comparison with VS-Code and others… :)
It completely and exactly describes my own development workflow:
tiling window-manager (i3 in my case) with a terminal multiplexer (tmux);
simple text editor (no IDE-features, not even auto-completion, syntax-highlighting and other niceties) – my own sce in my case; (complemented with mc to browse between the files;)
another shell for testing; actually multiple shells for all kind of tasks;
script everything for easy use – with my own z-run; (although make or just would do just fine;)
and obviously Firefox for reading the documentation;
In essence: keep close to CLI and TUI tools!
Another interesting (and unpopular take) is reading the actual reference manual / documentation instead of randomly copy-pasting stuff from stack-overflow until something barely works!
Indeed — I really enjoyed hearing how he reads the manual, which is the opposite of what Julia Evans said she does in this other episode. Interesting to see two programmers I respect a lot taking very different approaches to the same problem.
It’s a very insightful write-up on (the perils of) client TLS certificates, especially the part regarding the trust you need to have in your load-balancer (or TLS terminator) doing the right thing (also making sure no one can smuggle in another X-Client-Certificate header and trick the actual backend).
With regard to internal authentication, besides whitelists one could employ something like Wireguard, which is lightweight enough, and could replace TLS or other internal authentication mechanisms. (Assuming one runs only one service per node; although there are also solutions for the opposite case.)
However, and in relation with the previous article about keygen, I would really have liked for client TLS certificates to be more user friendly and more widely supported…
I really wished IPv6 would become “mainstream” and just replace IPv4 so, at least from my point of view of home / small-business administrator, we can all get rid of port forwarding and NAT’s.
However, I religiously disable IPv6 anywhere I can, but especially on (home) routers, due to the same reason described in the article: many of the consumer “devices” passively support IPv6, but who knows if they are ready or not to handle IPv6 traffic “safely” (for whatever “safely” means with these security disasters)…
For example I’ve currently standardized on Mikrotik small routers and access points mainly because their software is “enterprise-grade” but at an affordable price, and they do have proper IPv6 support (and lately Wireguard). However, many of the stock firmware on consumer routers (i.e. TP-Link, Linksys/Cisco, etc.) although support IPv6, they didn’t have proper firewall support for IPv6.
On the other hand, as a freelance developer / administrator, I never bother with IPv6 (unless supported by the CDN I use), because it’s a potential reliability issue.
Thus, when combining all these factors, I think the failure of IPv6 is a self-fulfilling prophecy…
However, I religiously disable IPv6 anywhere I can, but especially on (home) routers, due to the same reason described in the article: many of the consumer “devices” passively support IPv6, but who knows if they are ready or not to handle IPv6 traffic “safely” (for whatever “safely” means with these security disasters)…
I got the opposite of this from the article. If you disable IPv6 then that doesn’t actually disable IPv6. If you booted Windows Vista on a v4-only network, it would establish a v6 tunnel and then advertise itself as a router. Any other device on your network now has a routable, public, v6 address and is not protected by the firewall that you think that you have. It would be fairly easy for malware on a phone to do this, so someone brining the phone onto your network would tunnel all v6 traffic via a network that they control. If you’re not setting up IPv6, then you’re telling attackers ‘[encapsulated] IPv6 is reserved for your use’.
Adding a device in your local network (be it physically connected with a wire, wireless via WPA2, etc.) is always a risk for it to start poking at its neighbors. Especially in home / small office setups that don’t have DHCP snooping / port isolation / etc., a malicious device can even start giving IPv4 DHCP leases and route traffic through himself, thus taking over IPv6 is the least concern.
However, my main concern is with regard to traffic that crosses the network boundary (in or out), for which I don’t think we are that ready to tame as we are with IPv4.
This is an interesting article, but I feel that it leans too heavily on a few jargon terms that are not as widely understood as the author perhaps suspects, and doesn’t look hard enough at the existing systems out there.
It assumes more knowledge of how existing packaging tools work than I’ve ever needed to acquire. I have never built a package. What’s a hook? Why does it exist? What does it do? Is it needed? What for?
It also assumes more foreknowledge of container systems than I’ve ever needed. I predicted the rise of containers two years before Docker, but that doesn’t mean I’ve ever built or deployed one. That’s not my job.
What precisely does “deterministic” mean in this context? I interviewed the CEO of Rocky Linux and he talked a lot about deterministic compilation, and when I asked him what that was and what it meant, he floundered. He expected me to know and to care and was unable to clearly explicate what it was when I did not.
For comparison, I have written about packaging systems too; here is part 1 and part 2.
You are right that I’m assuming from the reader quite a bit of familiarity with OS package management. (The container stuff was tangent to the subject, thus not essential.)
Regarding the hooks, I’ve tried to briefly describe what they are (“scripts that …”), but I’ve focused more on the functionality they provide.
Regarding “determinism” I’ve succinctly described it as “in the cryptographic sense of identical checksums”, and I’ve stated that, starting from the same “base-image” (say an existing installation), installing (in one operation, or the same sequence of multiple installations) the same set of packages (given in any order), regardless of how many times (or the phase of the moon), should yield an identical “image”. (Here by “image” I actually mean the same directories, the same files, with the same contents, the same timestamps, the same owners, etc. If one creates a reproducible
tarafter each install, themd5sumof thosetararchives would be identical.)Conceded, yes. But the things are this:
Have you?
Why is that (there being scripts) a problem?
OK, which switches the requirement from “knowing about packaging systems” to “knowing about cryptography”.
Which effectively doubles the prior knowledge needed. That’s not helping, that’s making it worse.
Ever looked into Genode?
I don’t think this actually pertains to the problem I describe. I don’t think we need a new OS (as in kernel), I think we need to clean up our distributions packaging. (Do you have a link where they describe the way packaging and installation is done?)
Genode supports several kernels, including Linux, but generally not preferable, as third generation microkernels such as seL4 are simply that much better.
By default, it works with perfectly reproduce-able scenarios similar to nix. They have a decent webbrowser as well as 3d-acceleration so Linux is seldom needed. When it is, they have a working port of Virtualbox.
The Genode Foundations handbook (linked in their frontpage) is the go-to for describing the architecture; Chapter 4.6 and 5.5 and 6 is where I’d look at for this, but the intro chapter is worth reading.
the solution is clearly yocto :^)
Could you please elaborate more?
The Yocto / Bitbake system is often used to create a Linux filesystem (and kernels, bootloaders, etc.) from a set of recipes. These recipes usually specify a released tarball or git commit for the software being built. Primarily for embedded systems.
https://www.yoctoproject.org/
If you have all your recipes and such in version control, then theoretically you can exactly reproduce a system image.
And of course they do care quite a bit about reproducibility, since it’s so core to bitbake’s functionality. The output of each build step is cached and hashed. If the hash hasn’t changed, then its outputs are reused. This is applied recursively, so you can make a change and be assured that the output is the same as if you had built everything from scratch, even though only a few build steps were ran. It’s common to share state (sstate) within an organization, which can greatly speed up builds.
My suggestion is of course a bit tongue-in-cheek, since yocto (or rather openembedded) is sort of like a cross-compile-only version of gentoo. In their quest for reproducibility, they don’t even use the host toolchain (or any utilities). While you certainly could run yocto in yocto, it would be pretty masochistic. Plus, many of the defaults are oriented towards producing small, efficient binaries. While you can certainly enable all the bells and whistles, it’s not really the intention of the distro.
While I generally agree with what you are saying, I was hedging a little bit. Mostly with regards to creating two bit-exact build images when you have two different machines building from scratch from the same set of recipes. It is easy, for example, to have timestamps squirreled away inside the build image. Though of course those two build images should be functionally identical. I’m not so worried about bit-for-bit identical builds, but I am under the impression that some people are.
Well, that’s why we have
SOURCE_DATE_EPOCHand the like.It actually is quite important to have bit-for-bit identical images in bitbake. For example, say that you modify a dependency of some package and
do_buildhas to be re-run. If that dependency change doesn’t affect the build output, then the sstate hash for that build step will be the same. Then, since the hash ofdo_builddidn’t change, all the dependent build steps can be reused. In this way, changes to variables which might otherwise require large rebuilds can result in much smaller effort. But if you don’t have a bit-for-bit reproducibility, then everything else in the dependency cone will need to be rebuilt. I’m not sure if this is as feasible on multi-machine setups, but it certainly is useful for me.Neat! I didn’t know that reproducible builds had been taken that far with Yocto. I’ve still got a lot to learn.
How about something like distri?
I’ll post the link to the main site and GitHub of the project: https://distr1.org/ and https://github.com/distr1/distri.
Indeed, after reading the landing page, it seems Michael has reached (way before me) to some of the same conclusions (he has more articles on the topic that I’ll have to read).
However,
distriis clearly an experimental distribution (say so on the landing page), thus unfortunately still not a proper production-ready solution.You already have your solution, but you haven’t tried it. NixOS.
Note that Nix isn’t quite completely deterministic due to the activation script which is a giant pile of shell commands.
(It maybe be mostly deterministic, but as a user you can add basically whatever you want to it and break this.)
Nix is only as deterministic as it’s builders and compilers that’s true. It gives you the scaffolding you need but you still have to ensure you follow the rules. If you use a builder that creates random data somewhere then you still won’t have a deterministic output. That’s true of pretty much anything that tries to solve this problem though. They are working with tools that make it a little too easy to violate the rules unexpectedly.
Yeah I was actually surprised by this, from some brief experience with a shell.nix
It seems like it is extremely easy to write Nix files “wrong” and break the properties of Nix
Whereas with Bazel, the build breaks if you do something wrong. It helps you write a correct configuration
Similar issue with Make – Make provides you zero help determining dependencies, so basically all Makefiles have subtle bugs. To be fair Ninja also doesn’t strictly enforce things, but it’s easy to read and debug, and has a few tools like a dep graph exporter.
nix-shell is a bit of a lukewarm compromise between Nix’s ability to supply arbitrary dependencies and the guarantees of the build sandbox. It’s not really trying to be hermetic, just ~humane.
I would like to hear more about your shell.nix experience. It has some issues, for sure, but I have found the platform to be extremely reliable overall.
I linked some comments from this page - https://github.com/oilshell/oil/wiki/Can-Oil-Use-Nix%3F
Is that really relevant? A determined user could flash their BIOS with garbage and break anything they want. Does it matter that you can go out of your way to break your own NixOS system? The point is that as a user you’re not supposed to mess with your system manually, but you’re supposed to make changes by changing your
configuration.nix. This is not the case for most other distros where you make changes to files under/etcside by side with files maintained by the distro packages.The point is that I don’t think anyone is checking. IDK how many options in nixpkgs break this. I don’t think we need to stop the user from purposely breaking it but we should make “official” flags not break it and make it easy for users to not break this.
I did try Nix (briefly).
However, I do think there is a place for classical distributions, as there is a place for distributions like Nix, Guix, and even something like GoboLinux.
Pretty much every concern you share in that post is solved by NixOS. Some points
NixOS isn’t perfect by any means, but it is a solution you seem to be looking for. Guix probably as well, but it’s a smaller and more limited distro.
I took the multiple comments that “NixOS may do this but I’ll have to research” to mean you haven’t tried it. And this entire blog post screams to me that NixOS is a solution for you.
In a way, that’s second system syndrome. Suse Linux was bashed for doing something like that with YaST for a long time…
In case of NixOS, I found that new language (not the syntax, the semantics) changed too fast for my liking…
With a few caveats: Software might still get updated (I suppose you can lock that down to specific revisions, but who does?). From a bugfix/security perspective this may be desirable, but it’s not entirely reproducible, and updates can always introduce new bugs or incompatibilities, too.
All packages are locked in Nix. So With a few caveats: Software might still get updated is not a problem there.
https://www.software.ac.uk/blog/2017-10-05-reproducible-environments-nix offers “download a specific release tarball of buildInput specs” to improve matters. Other than that,
buildInputs = [ bash ]may or may not point at the same version. https://github.com/NixOS/nixpkgs/issues/93327 discusses this for 3 years and there doesn’t seem to be a resolution.You’ll get the same version with the same Nixpkgs, every time.
buildInputs = [ bash ]will always point at the samebashpackage for some given Nixpkgs. Package versioning is a different issue.I count package versioning as part of the system configuration: Sure, your package names are always the same, but the content might differ wildly.
The content will be the same every time with the same system configuration, assuming you’ve pinned your Nixpkgs version.
Or using flakes.
I think you’re misunderstanding the way nixpkgs works with versions and what the versions actually mean there. Check it out in practice. Unless you update the channels/flakes or whatever you use between runs, nothing will change - there’s no explicit pinning required.
The way I manage this is using several different versions of nixpkgs; a “known good” version for each package I want to lock, so for agda I would have
nixpkgsAgda, the regular nixpkgs which is pinned to a stable release, andnixpkgsLatestwhich is pinned to master.Most of my packages are on stable nixpkgs. Every now and then when I run
nix flake update, it pulls in new versions of software. Things pinned to stable have never broken so far. Things pinned to latest are updated to their latest versions, and things pinned to specific packages never change.While it does involve pulling in several versions of nixpkgs, I build a lot of software from source anyway so this doesn’t matter to me very much. I do hope that nixpkgs somehow fixes the growing tarball size in the future…
That’s a snippet of code written in the Nix expression language.
bashis not a keyword in that language, it’s just some arbitrary variable name (indeed, so isbuildInputs). Just like any other language, if thatbashvariable is defined/computed to be some pure, constant value, then it will always evaluate to that same thing. If it’s instead defined/computed to be some impure, under-determined value then it may vary between evaluations. Here’s an example of the former:This evaluates to
{ buildInputs = [ «derivation /nix/store/6z1cb92fmxq2svrq3i68wxjmd6vvf904-bash-5.2-p15.drv» ]; }and (as far as I’m aware) always will do; even ifgithub.comdisappears, it may still live on if thatsha256appears in a cache!Here’s an example of an impure value, which varies depending on the OS and CPU architecture, on the presence/contents of a
NIX_PATHenvironment variable, the presence/contents of a~/.config/nixpkgs/overlays.nixfile, etc.I recommend sticking to the former ;)
PS: I’ve avoided the word “version”, since that concept doesn’t really exist in Nix. It’s more precise to focus on the
.drvfile’s path, since that includes a hash (e.g.6z1cb92fmxq2svrq3i68wxjmd6vvf904) which is the root of a Merkle tree that precisely defines that derivation and all of its transitive dependencies; whereas “version” usually refers to an optional, semi-numerical suffix on the file name (e.g.5.2-p15) which (a) is easy to spoof, and (b) gives no indication of the chosen dependencies, compiler flags, etc. which may have a large effect on the resulting behaviour (which is ultimately what we care about).Do you happen to have an example of such an semantic change at hand? Curious as nix is often regarded as rather conservative and trying to keep backwards-compatibility as good as possible.
Part of the reason Nix/Guix take their particular approach is specifically because the global mutable state of a traditional FHS style distribution is nigh-impossible to manipulate in a deterministic way; not just because of the manipulation by the package manager, but because there is no delineation between mutations by the package manager and those by the user. A great example is
/etc/passwd, which contains both locally created users as well as “service accounts” the distro maintainers make to separate out some unprivileged services.Relevant writing from Lennart Poettering on state in Linux distributions and how to accomplish a factory reset mechanism.
There is a place for “classical” (read: legacy) distributions, yes, in the same way there is a place for collecting stamps.
I want to switch to NixOS so badly, but I also want SELinux support.
it needs a lot of work to enforce 1) determinism/reproducibility, 2) non-hysterisis (when you stretch an object and it doesn’t return /exactly/ back… like hidden side effects in ansible), 3) ease of use, & 4) acceptance testing.
It resembles a lot with bubblewrap, although I think
minijailis a bit more advanced in the options it presents.In fact I think (by reading the documentation of
minijail) that their main use-case is security isolation, meanwhilebwrapis more about portability.Just the previous day I was toying with
bwrapto start a container without Podman / Docker. I’m curios howminijailworks for this use-case.Interesting take on authentication, however perhaps a bit too convoluted?
Let aside the fact it relies on JWT that has a bad reputation of being impossible to get right in implementation, but what caught my eye was how the verification (by PyPI for example) is being done: by applying certain policies over the verified claims in the received JWT.
Because the article doesn’t say, and doesn’t give any screenshots in case of PyPI, I can only wonder how this is actually achieved:
So, I’m left wondering if they haven’t just moved the problem from one place to another by introducing yet another attack surface (which besides JWT is now the policies)?
They have even identified one themselves – that of resurrecting dead or renamed accounts – and they’ve solved it with the intervention of GitHub, which means that this issue has to be solved by each individual provider.
I can’t believe we can’t find simpler approaches than this…
This was meant to be just a technical summary of the architecture. The link at the very top of the post goes to the the PyPI blog, which contains an announcement that’s more specific to the features available at launch-time; it also contains a link to the technical documentation1, which should address some of your questions around policies and security model.
To summarize:
Yes, JWTs at bad. They’re also, practically speaking, the only game in town. OAuth2 and OIDC use them, so we have to use them. There would be no trusted publishing at all without these underlying technologies.
At the moment, PyPI supports just one provider: GitHub. That provider was selected because it accounts for an overwhelming majority of publishing activity, meaning it’s the one that would be most immediately useful to potential users. I’m currently working on support for two more providers: Google’s IdP and GitLab.
It would be fantastic if we could do vendor neutral policy verification, but neither OAuth2 nor OIDC is rigid enough for us to assume a ‘baseline’ of shared relevant claims. Each IdP needs to be added manually, since each has its own claim set and quirks; eliminating that problem is a problem for standards bodies, not relying parties like PyPI.
The “policies” in question are just HTML form inputs for (in GitHub’s case) the username, repository name, workflow name, and an optional actions environment. You don’t have to know a DSL; that’s something we considered early on in design, and rejected for complexity reasons. There are examples of this in the docs (linked above).
Trusted publishing is indeed architecturally complex. But it’s using OIDC as it’s intended, and we’ve tried to abstract nearly all of this complexity from the end user (who only needs to know which
user/repoand workflow they trust). The surface that it exposes is roughly equivalent to granting an attacker access to your CI (including all secrets in it), so I don’t think that part is a significant new risk.I’m not trying to dismiss the work you have done with this integration. It is a first step towards a better solution for a quite hard to solve problem.
However, PyPI is now in the same league as Rust’s
crates.ioin that it requires a GitHub account to be able to “play” in Rust’s (and now Python’s) ecosystem. I don’t have anything against GitHub either, I use them for all my code, however there are developers (and certainly also amongst Python ones) that don’t want to have a GitHub account. Thus for this category of users the problem is still unsolved.However, assuming GitHub is only the first such identity provider, what are the next steps to support multiple independent providers? How would the UI change to support something that doesn’t have GitHub’s particularities? would a separate HTML UI (i.e. form with inputs) be developed for each?
Because each time such a proposal appears, that in theory supports multiple identity providers, it falls short and supports only the largest ones like Google / Facebook / Apple.
To summarize, I understand why PyPI (and
crates.iofor that matter) have chosen to start (and stop) with GitHub: at the moment it has a large user-base and it’s pretty secure (or at least with deeper pockets to throw money at the problem if needed); in a few words it’s a practical approach (that I would have taken myself). But I doubt there would be a time when the “trusted publishing” would expand to other providers…So, if GitHub is the only one, why bother with all the complexity?
GitLab is perhaps similar to GitHub, thus it might have the concepts of repositories, branches, etc. But would you support only the “official hosted” GitLab instance, or will you support independent GitLab deployments?
But what are the equivalents for Google? They’ve killed their
code.google.comlong ago. Are you targeting another of their products?This new feature, which offers an ability to publish from CI with a short-lived token in order to make CI compromises less severe, chose GitHub as the first supported IdP, not the only one that will ever be supported, and you’re literally replying to someone who says they’re working on adding support for others.
And if you don’t have or want a GitHub account, or object to using a GitHub account, or don’t want to publish from CI at all, you can still use plenty of other PyPI publishing workflows to put your packages onto PyPI, all the way down to just running
python -m build && python -m twine upload dist/*manually from your own personal machine. So claiming that PyPI now “requires a GitHub account” is false.I appreciate this concern!
I’ll state it unambiguously: trusted publishing is not intended to deprecate ordinary PyPI API tokens. Users will continue to be able to create API tokens, which can then be used however they like. Trusted publishing is only intended to provide a faster and more convenient alternative, where the platform (i.e. GitHub, and soon others) permit.
Support for independent providers is something that PyPI’s admins will probably consider on a case-by-case basis, as a balancing act between implementation complexity (each IdP needs special handling), the security margins of each new IdP (there’s relatively little value in a random self-hosted IdP versus a normal API token), and the impact to the community (e.g., major CI platforms).
Yes, sorry if this was unclear: in this context, the IdP being supported is the one that provides proofs for machine identities on Google Cloud Build. In other words, similar to GitHub Actions’ machine identities.
Does it? AFAIK age-keygen does not let you create keypairs from a user-provided shared password? Or do you mean something else?
AFAIK the only way to decrypt an age message is with a private key, not a shared password? Am I mistaken?
According to the readme:
Granted, the
agetool doesn’t provide at this moment support for encrypting both with an identity and a passphrase, but the format (as specified here) does technically support it.Huh, TIL! Thanks!
I’m not sure what “and” would mean in this context. Is it first one and then the other? If so, in what order? And I guess this is possible without explicit support in the tool: encrypt with a passphrase to get ciphertext 1, and then encrypt that ciphertext with an identity to get ciphertext 2; reverse the process to decrypt. Also, as most identity files can themselves be protected with a passphrase, what would be the benefit of using a passphrase as a separate step?
This is exactly related to my article, and why I’ve wanted to design “multi-factor encryption” (which is a very poor name…)
Getting back to
age, as said, although the current CLI (and documentation) doesn’t support providing at the same time both a password and a recipient identity, the format does permit it. So indeed saying “a password and a recipient identity” isn’t very clear; it could mean either of the following at decryption time (which is the important part):agenorgnupgsupports this;)ageorgnupgimplement;If it were supported, and this is how I’ve implemented it in my tool, the user specifies everything at once, in any order, and the tool knows how to handle everything.
Indeed, this is one possible solution, but it implies two encryption phases.
A second solution might be to just encrypt the “session key” (or “packet key”) with the passphrase and then that output with the identity.
A thind approach, which I’ve implemented, is combining both the passphrase, the identity, (other factors), and the “session nonce” to obtain the “session key”.
If you are thinking only about decryption, then indeed the user has to give two passwords, which has zero added security as opposed to just one password (provided both have enough entropy, because their entropy is compounded). That is because both can be easily intercepted by a key-logger, leaked from memory, etc.
However if you think of the whole workflow you can see why they could be useful. Imagine this: you and a colleague are working remotely on a project; you want to share a very secret information over chat; your colleague has already created an identity file and shared that with you (over another secure mean); you could just use that identity to encrypt your very secret information, but then anyone that gets access to his laptop (which has both the private key for the identity and the chat history) can decrypt it (especially since
agedoesn’t necessarily encourage encrypted identity files); however, now if you add a shared password (that you communicate over phone for example), the attacker now must also wait (or trick) your colleague into revealing that password.If you need secret key X and passphrase Y in order to decrypt some ciphertext, then however you compose X+Y will produce some composite secret Z, which is the actual secret you need for decryption. As far as I can tell this puts you back to square one, the secret is really Z, you’ve just Horcruxed it up into a few separate pieces. You can do that with any secret. Chop up your ED25519 key into 4 byte chunks, each on its own USB stick, and ship them to the far corners of the earth — this will probably annoy your attackers, but it won’t improve the strength of the encryption. As far as I understand.
Exactly. It’s not “magic crypto”. It’s just “how can I make it easy for the user to provide two passwords instead of one”. (Although, in my case, because I offer multiple “kinds” of passwords, PINs and secrets, by choosing one or another, the user also chooses the resources required to brute-force them, via passing them through Argon2 with different parameters.)
To put it another way: Age doesn’t do anything magical; it just combines various cryptographic primitives to provide a tool to the user; I could just as well take a piece of paper and compute everything myself (perhaps in a millennium). That doesn’t make Age useless; it makes it a dependable tool.
Why? Is it your belief that splitting a key into N parts, with different security properties, is more secure than simply using the entire key itself?
As I’ve said before, its mainly about usability.
Cryptography is not magic, and sometimes you have to implement it even if you’re not an expert (those expert’s aren’t always available). It will take more than a weekend though.
BTW, I’ve read your article initially when it appeared. I’m glad you’ve written it, because to my knowledge it’s the only one encouraging other non-cryptography-experts to give it a try. :)
Agree. I’ve been ruminating, experimenting, and tweaking this since since January, and slowly I want to publish some of my ideas to find some feedback (as I’ve already received so far here).
Moreover, I’ve not actually “invented” or “implemented” my own cryptographic primitives, but instead relied on (what currently is considered) state of the art as implemented by others (in Rust); for example I’ve used only the following primitives:
In the end I didn’t do anything that
libsodiumwouldn’t do, except I’ve put everything together for a specific purpose.Implying ChaCha20 has a practical limit on message lengths is a bit of a shortcut: it is not widely understood, but there are three ChaCha20 variants:
When I design a protocol myself however, I design for a 32-bit counter and a 64-bit nonce, thus ensuring good efficiency and maximum compatibility.
Indeed, choosing the “right” ChaCha20 was quite problematic for me (especially since I’m not a cryptographer). However I took the practical approach: choose the “standard” one (as in the one ratified by IETF), because that is the one most likely to be implemented in other libraries.
I don’t get what you want to do. Do you want to open a encrypted channel, encrypt a message and send it to someone else, or encrypt a message to yourself (i.e. for backups or full disk encryption)? Only in the last case you can control the security of the used secrets.
Also your text is full of misconceptions. I don’t even know where to start. Some examples:
Oracles are mostly a theoretical construct used in security proofs. Sometimes oracles are used in attacks to break the security of a protocol (i.e. a RSA padding oracle can help to decrypt RSA encrypted messages).
PINs are for authentication not for encryption. Yes you could use them for encryption, but it would be easy to break.
DHE is a Key exchange algorithms. It can be used to exchange a key in a online protocol (like TLS). This key then can be used for encryption.
In all the main use-cases (but this doesn’t have to be so) the same “owner” is at the both ends (encryption and decryption) although it might be at different times or with different levels of involvment. For example, here are a few use-cases:
Indeed. However, think at the functionality that an oracle exposes: give it as input some data and it will deterministically output back some other data (hopefully random), however nobody can guess without access to the oracle what the output might be. (In real life, any HMAC-like construct, any RSA / Ed25519 signature, could be considered instantiation of such oracles.)
So, by using such oracles one could achieve the following (which can’t be achieved solely based on secrets):
So, in the end, what is (in terms of functionality) the difference between a KMS, an USB RSA Token, your
ssh-agentor a simple service that does a HMAC with a secret key, when all you want from them is to “mix” some data you gave them as inputs?And that’s exactly why I’ve called them “PIN’s”; nobody would be taken by surprise if someone brute-forces a 4 digit pin.
So what am I doing wrong then? DHE is not only for online protocols. In fact it’s one of the main constructs used all over
libsodiumfor various kinds of “boxes”.Are you speaking about (general) oracles, or random oracles?
Based on my understanding of the Matthew Green articles and the Wikipedia entry a random oracle is not satisfied by the properties you list here, which AFAICT describe a cryptographic signature. Random oracles rely on state derived from an input history, whereas signatures are essentially stateless. And in fact many papers like this one explicitly state that “random oracles do not exist in the real world!”
A cryptographic signature is based on a private key, right? Is a private key not just a secret?
Yesterday evening I’ve written an article dedicated to this subject, which I think addresses my intentions.
Obviously we can’t actually use “true random oracles” because these are not implementable in software; just as we can’t speak of “true random generators”.
However, colloquially we call “pseudo random generators” just plain “random generators” (or “secure random generators”) and everybody understands that they are imperfect instances of the theoretical term. The same here: when I say “random oracle” I mean a “pseudo random oracle”, as obtained for example from a HMAC.
Quoting the Wikipedia article on the subject:
Now, getting back to Matthew’s series on the subject of random oracles, although they are very insightful, I think he spent 5 long articles talking about them, but failed to give a concrete definition in one phrase as Wikipedia did. :)
However, if you read between the lines, you’ll see that most often than not random oracles are embodied by hash functions or pseudo random functions.
<<Random oracles rely on state derived from an input history,>> is completely wrong. Matthew Green gave an example on how a random oracle might be implemented (by starting with an empty history, and for each item, coming up with a random output, and remembering that). However, random oracles, respond to inputs the same way regardless of the order the inputs are given.
As said, I’m interested in practical, imperfect, real-world instantiations of this concept.
Indeed, a private key (or anything that can be used to implement a “pseudo random oracle”), is nothing else than “cryptographic key material” that has to be kept secret. And from a functional point of view, it’s just like our algorithm contains inline what the oracle would do.
However, and this is what I’ve found enlightening, we forget that the concept of an “oracle” implies that it’s a third party (in real terms a different process, on different hardware, or a different machine, perhaps even on a different network)!
Thus the attacker now has to compromise two systems:
So, just like USB RSA tokens are nothing else than circuitry that implement RSA (perhaps even in software on a micro-controller) and a place to keep safe the RSA private key, there is nothing else “magical” about them. So with the oracles I speak about: they use plain algorithms, but “at a distance”.
Can you ELI5 why?
If you allow a cryptographic signature, derived from a private key, to satisfy a random oracle, then it’s no longer guaranteed to be a third party, right? Anyone with access to the private key becomes the oracle, indistinguishable from any other. Furthermore, does this not mean that the output is not actually random, but rather a pure function of the input and the private key?
I was almost about to ask you what
ELI5is; however I think it stands forexplain like I'm 5 (years old), according to Google. :)As stated (in the wikipedia article and in many places in Matthew’s articles) an oracle is a “black box” that receives inputs and deterministically answers “random looking” outputs, plus it’s formally defined as
f (input) -> output.Thus, because the inputs history (which can be formally defined as an ordered sequence of inputs) is not part of the actual input, the oracle is deterministic only based on the individual (current) input, regardless of other inputs that it might have received in the past.
Therefore, the only requirement of the oracle is to respond to the same input with the same output regardless of who, when, in what order, (or even if) one asks it.
Stating <<random oracles rely on state derived from an input history>> implies that an oracle’s reply would depend on the order in which the inputs are given, and not only on a particular individual input. (i.e.
for i in 1..5 do oracle(i); o_6_a = oracle(6); for i in 5..1 do oracle (i); o_6_b = oracle(6); assert (o_6_a == o_6_b);.)Granted, one could envisage an oracle that does the following (basing this algorithm on Matthew’s example with the history):
register = md5 (concatenate (register, input)), then use the currentregisteras the output (which is then cached);Such an oracle would technically “rely on state derived from an input history”, however once an input is presented, the oracle makes up its mind on the chosen output and will forever give that as output (for the same input). Also, granted, if the oracle (or the virtual world it lives in) would be “reset”, and the inputs would be presented in another order, the outputs would be different (provided the register is initialized to a constant value like all zeroes).
But all this doesn’t break the following two properties of the oracle:
Thanks for the reply.
I think I understand your position to be that (1) a random oracle is reasonably approximated by a cryptographic hash function, in practical terms; and maybe also that (2) the good stuff we get from random oracles we can also get from cryptographic hash functions. Is that correct?
I certainly buy (1), but I’m stuck on (2). As far as I can tell, the whole point of random oracles is to define something explicitly with stronger guarantees than what can be provided by hash functions. Quoting Wikipedia,
So when you say
I don’t really get it. This property is satisfied by a function f(x) = 1, right? And that can’t possibly be a valid random oracle, right?
Exactly. (In fact I think our whole cryptographic ecosystem is based on this assumption. From what I gather most of the security constructs we have are proved in the “random oracle model”, and then the “random oracle” is replaced with hashes or pseudo-random-functions. At least based on what I’ve read, and especially Matthew’s articles.)
This is, from my point of view, the same as point (1).
However, and I want to stress this, you are completely missing one characteristic of the random oracle (both in theoretical terms, and how it can be used in practice): random oracles are not just another algorithm; they are independent agents running outside the constructs we implement, test or try to prove; in practical terms, a random oracle (as I view it in the real world) might be implemented with a HMAC, but it’s running in a completely separate process than the software trying to encrypt / decrypt. That’s the whole thing with them; the attacker can’t (at all theoretically, but realistically easily) peek into them.
To reiterate an example: 2FA (two factor authentication), either implemented as a USB token, or via TOTP in a software “authenticator” running on mobile (where the input is the time), that is a random oracle. It hasn’t anything magic in it; it follows a standard HMAC-based algorithm; but it strength lies mainly in the fact that the attacker which might have compromised my workstation / laptop, can’t easily also compromise my mobile.
I have considered as asserted the fact that a “random oracle” should return “random outputs” (for whatever the definition of “random output” we accept). Thus the two properties I’ve listed there were mainly focused to prove the point that a random oracle’s output doesn’t (in general) depend on the order of its inputs.
Thus, the function
f(x) = 1might be an “oracle”, but it’s not a “random oracle”.The term “random oracle” means something stronger than what you are describing here. If when you write “random oracle” you actually mean “pseudorandom oracle” then I guess I’d ask you to please make that explicit.
If a cryptographic-hash-based oracle runs in the same process as my application, and its secret material exists in the same memory space as my code, then it is relatively easy for me to exfiltrate it. If that same oracle runs as a separate process on the same host as my application, then it is harder for me to access that secret material. If the oracle and my code run as separate VMs under the same hypervisor on a single host, then it is that much more difficult. If the oracle runs on a separate host than my code, then it is harder still. If it runs in a secure enclave on a separate host behind a tightly controlled firewall, even harder. But as long as the oracle, running in any of those environments, is performing a cryptographic hash/signature based on a private key, then these are all differences of degree, not of kind. As far as I understand.
OK, please imagine that everywhere I’ve said “oracle” or “random oracle”, I’ve actually meant “pseudo random oracle”. :)
But that is not an “pseudo random oracle”; the whole purpose of these is not to run in the same process space as our software. That’s the whole trick I was trying to stress.
Theoretically you are right. It’s a difference of degree, not of kind.
But then, a 4 digit password is the same as a 100 random printable ASCII character password; they have the same “kind”; but in practical terms the “difference of degree” is what matters.
The term “random oracle” is a theoretical concept with certain properties that are, as you note, not implementable in practice, but nevertheless is well-defined. There is a categorical difference between a random oracle and a pseudorandom oracle. Conflating the former with the latter is, I think, one of the points of confusion in this conversation.
There is definitely also a clear difference between a pseudorandom number generator vs. a cryptographically secure random number generator. One common manifestation of that difference is the behavior when entropy is exhausted: the weaker generator will make shit up, and the stronger one will block or error out.
What is a “pseudorandom number generator”?
If you ask Google or Wikipedia it either shows you “weak random number generators” (like the Mersene Twister) or “cryptographic random number generator”; both are “pseudo”. Thus, even with this long term “pseudorandom number generator”, we know what we mean from the context… The same with my “oracle” (or “random oracle”) name: in this context, it means some software or hardware agent that tries to have the properties of a “theoretical random oracle”.
Can you point to even a single cryptographic secure random number generator that just panics or errors (or even blocks) when it has exhausted its “entropy”? (Which I assume you mean “when it has reached the risk of wrapping around and repeating”.)
The Linux one (
/dev/randomwhich recently I think is the same as/dev/urandom) just “makes shit up” continuously. TherandRust crate doesn’t seem to return error conditions (unless there is an I/O error).Entropy is a resource on a system, comparable to CPU or memory, which can be exhausted in similar ways. There isn’t really any concept of “wrapping around” as far as I know. Typically, /dev/random will block when entropy is exhausted, and /dev/urandom will not. Similarly, Go’s math/rand is pseudorandom, and crypto/rand is cryptographically secure. I’m not super familiar with Rust, but I’d certainly expect there to be a way to read both non-blocking pseudorandom bytes, and blocking/erroring cryptographically secure bytes, based on user need.
I would say “it depends”. For a physical / hardware system, indeed entropy is something that is accumulated over time, thus it can’t be exhausted.
However, for an algorithm based solution, like for example many use ChaCha for random number generation, there is a limit on the amount of data that can be generated before the counter wraps around, and the generator starts outputting the same values. Thus, we could say that in this case the “pseudo entropy” is exhausted.
It used to be so… Not anymore… I can’t find now the reference, but I remember in some recent Linux kernel they’ve made both
/dev/randomand/dev/urandomthe same.For example
pv < /dev/random > /dev/nulljust yielded ~15 GiB of data in less than a minute… Clearly that entropy isn’t real for my laptop.Yes, but neither blocks or errors (or panics).
Apparently there isn’t. Because apparently nobody wants his software to stall until the user jerks the mouse. :)
https://stackoverflow.com/questions/20959924/what-is-entropy-starvation
—
crypto/rand will block when entropy is exhausted.
edit: “Exhausted” here means “unavailable at the moment” and not “unavailable forever”.
If my application requires cryptographically strong random information, and it isn’t available, then I absolutely want my application to stall or error, rather than carry on with data that doesn’t meet my requirements.
Your use of PIN is slightly confusing because normally it’s not that these are for low security situations. It’s that they are used for places that have some intrinsic immunity to offline attacks. For example, the PIN in credit card transactions is validated by the chip on the card. The PIN for Windows or iOS / Android device unlock is used to access a key in a TPM or other hardware root of trust. The PIN in Signal is used to recover your key from some code running in an SGX enclave in the cloud. All of these things do rate limiting, so it doesn’t matter that you can generate all of the possible values in a few ms, you get to try them once per second with something like exponential back off so that, practically, you get at most tens to hundreds of guesses.
The key phrase that you want to search for is ‘key derivation function’. It’s common to have a single secret stored in some hardware token (TPM, U2F device, and so on) that can never be extracted but can be used as one of the inputs to a KDF. If you have such a thing, then there’s a simple flow of using a password hashing algorithm (Argon2 unless you have a really good reason to do something else) to generate the other input. This flow can be chained: the result of the first KDF can be used as an input to the next one, so if you have a chain of passwords or other secrets then you can hash them and then apply them.
If you want to use n out of m passwords then you have a much harder problem. Something like Shamir’s Secret Sharing algorithm lets you derive a set of parts of a key and there is some work to allow each holder to do part of an encryption independently, but I’m not aware of a way of defining the shares and then a function that combines a subset of them to consistently generate the same output and that sounds like a very hard problem. You’re effectively running error correction the wrong way around and I don’t see a way of doing that unless the system can generate one of the secrets that you must provide (then it’s easy, but it’s n user-provided secrets plus one fixed one and owing that one and some of the secrets probably lets an attacker guess some of the others with high probability, so that one would have to be stored in tamper-proof hardware).
Indeed what you are describing is correct. However, and the reason I called them “PIN”, is because these have very low entropy. How else could I call them so that users understand their intended meaning?
And this is exactly how it’s implemented. Each kind of token (PIN, secret, “ballast”) is passed through Argon2id with various CPU and memory requirements.
And this is exactly how it’s implemented. As part of the derivation chain, intermediary values are passed through what I call “oracles” (which at the moment only
ssh-agent-based is implemented, but anything that has an RSA or Curve25519 key would work).At the moment this is not implemented. The main reason being that I don’t think there are any mature enough implementations out there for me to use.
BTW, besides Shamir Secret Sharing I’ve seen a few other papers that describe alternatives (I don’t have the reference now) but I don’t think they have been yet properly implemented (or reviewed).
I may be misunderstanding what you’re trying to do, but can’t you just concatenate the secrets (with appropriate delimiters), run that through a cryptographic digest, and use that as the key? Or for added security, use a key-derivation function instead of a digest.
Technically (i.e. from a cryptographic point of view) mashing (through canonical concatenation with or without hashing or intermediary derivation) two (or more) “passwords” together and using the result as the new “password” is equivalent (to using separate “passwords”).
However, there are a few issues that I try to solve:
Right, you use a proper encoding like length/value or even JSON to eliminate ambiguity.
I don’t think using JSON is a good idea, mainly for the following reasons:
In recent months I think CBOR is preferred for such purposes.
However, at least in my implementation, for practical purposes (because performance isn’t a major concern for me), I don’t need canonicalization, because internally I always use 32 octets (256 bits) “keys” or “hashes” (either from Blake3 or Argon2id or DHE) with either a fixed number, or prefixed by their count.
Oops, you’re right about JSON; in my main project we do use it for that purpose, but that’s only because we had already implemented a canonical encoder for other reasons.
I wonder if it’s the UX of mail clients vs other things that’s a contributor. I can’t think of why this is a problem since mailing lists also have topics within them. Maybe the time is ripe for people to gather and come up with best practices for this type of email use case in this day and age?
The UX of mail clients is definitively a major issue for high-volume mailing lists…
I have extensive “experience” into this with GMail, and if it weren’t for the “mute” option I would have renounced using mailing lists much sooner. Also, sticking with GMail here, creating and maintaining filtering rules is almost a complete madness, so much so that at one point I’ve written my own Racket-based DSL to generate (and import) GMail rules via their XML format. However, besides basic labeling, GMail is useless in this regard, which is the main reason I’ve ditched it a few years ago.
I since use Thunderbird for “normal” emails, but not for mailing lists… Because Thunderbird also seems to be useless in this regard…
I had some success with
notmuch, which allows quite advanced labeling workflows, but that seriously lacks a proper UI.Thus, at the moment I do my mailing list consumption via GMail, and any serious searching via
notmuch. But all in all, I would say I’ve completely abandoned mailing lists…Unfortunately, Chris describes exactly what I’ve used to do also, namely be subscribed to a firehose of technical mailing lists, at it’s peak up to ~350 mailing lists at once; now I’m down to about a handful, especially for niche projects. These days I’ve switched to following Lobsters links and directly following various authors blogs RSS.
However, I still think that email is (technically) superior to “forums” (which these days means Discourse), because anyone is free to choose his user-agent and implement whatever filtering or archival workflow one wants. With Discourse you can’t… (Although it has support for emails, that is broken, or at least was broken a few years ago.)
Also forums, at least for me, didn’t solve the high-volume issue, as for example I’ve tried following the Rust users Discourse forum, and it was as “bad” (in terms of follow-ability) just like other users mailing list. Both had high-volume (sometimes low-quality) traffic.
Perhaps what we are missing are low-volume and/or high-quality mailing lists focused on particular topics.
The flip side of being able to choose your own client is that other people can too. The FreeBSD lists are increasingly difficult to follow because people read them on the web and reply with things that then don’t have the thread ID metadata, so it’s impossible to see where their reply fits in a long thread and the threads end up being displayed as a set of disjoint threads. In contrast, LLVM’s Discourse doesn’t have this problem and also doesn’t have the problem that someone replying to a five-year-idle thread (e.g. a new person asked a question, it was answered, and the answer is now wrong after API changes and the next person wants help) is referencing context that a load of subscribers don’t have.
The biggest win for LLVM was that the project had a large set of lists with overlapping audiences. C++ things might need to go to the libc++ and clang lists, but then some people would hit reply all and have their replies bounced from the lists they were not on. It’s much easier to follow a set of tags and have threads gain tags once people realise a particular audience would care about something.
Indeed, a “forum software” (mainly Discourse) might be much better equipped to handle long-form threaded communications between multiple people; while also supporting things like proper Markdown and code snippets. However, most of the problems you’ve described stem mostly from not respecting best practices (use proper quoting, use proper email clients, etc.), which forums seem to properly enforce.
In fact, I agree that perhaps emails aren’t the best solution as “forums”, however they have something that no forum software has: flexibility.
Picking on Discourse (because it’s the dominant software in this area):
List-*header purposes;So, perhaps in terms of direct usability Discourse is a better choice, but it completely fails in other respects.
Perhaps we need an email-like protocol specifically for this purpose? Perhaps the Discourse team could propose one? (Although Discourse is a business, thus it’s perhaps against their incentives to do so…)
I use the log in with GitHub option, so effectively yes.
That’s definitely missing from the web site, though I believe the app does better.
It’s possible to export everything from an instance, I don’t know if you can export all of your threads. Sounds like something that would be useful though. Have you tried raising a feature request?
From what I can tell, everything that the Discourse app and web app use is exported over their REST API, so you could in theory. I don’t know if there are other clients that do this. Note that your other requests could also be done by writing some code that calls their API.
Regarding feature requests, since I don’t use Discourse, I don’t think asking for any features would be fair (or helpful for me)…
I did at one time try to convince the developers to properly handle
List-*headers and completely failed, thus I gave up… (The issue was, as I remember, that they did add someList-*headers, but also added theTo: me@example.comheader, which basically broke the mailing lists conventions, and thus labeling efforts.)Regarding the “app”: is there a “desktop app”? I’m not so much into “mobile”, and anyway I kind of hate that everything nowadays has an “official app”… Is there some choice in “apps”, or is there only a single “blessed one”?
Regarding the REST API: that’s not a standard.
If anyone knows a HTTP benchmark / stress-test tool that is an open loop system let me know. (
wrkand all the others like it are closed loop systems.)I’ve been using vegeta for HTTP benchmarking and stress testing, it is a very nice and simple open loop system.
How does this tool achieve the “open loop system” requirements?
Because, it should at least support the following features (which I don’t see mentioned in the readme, at least at a quick glance):
Based on the arguments supported by the tool, it follows exactly the same model as
wrk.Hmm, randomization is a bit different, yes, but it is open-loop in the sense that it doesn’t care about how well does the server respond to the requests, it just sends as many requests as specified. Connection count in this case entirely depends on how well the server is responding, if it’s time to send a new request and all of the existing connections are waiting for a response, it opens up a new connection and sends a request there.
Adding some noise generation options to it would be nice though, and possibly quite easy.
Interesting article. I would say it applies to more than programming languages and runtimes. For example:
Frequently the problem is exposing a concept that seems useful but isn’t. For example, the author has a couple of examples about Python datetimes, but the core problem with Python’s date libraries (besides the awkward API) is that naive datetimes are an extremely niche usecase. Basically, they should only be used for calendar apps to have repeating events happen at the same local time. But it seems like a naive datetime is a convenient idea, so there it is and everything else follows from that core mistake.
It’s also probably worth splitting these things into intentional and unintentional approaches. Things like OpenSSL are well intentioned but have a load of API misuse footguns. Similarly, GPG tries to do the right thing but provides users with a bewildering interface that can cause them to not have security by accident. In contrast, things like cookie banners and Facebook intentionally try to pull users into these paths.
Speaking strictly of either OpenSSL and GnuPG, I agree that many of their complexities and API foot-guns are because they need to implement complex (and sometimes) broken protocols.
However, I think they are missing a wrapping layer, something like
libsodium: give the user two libraries:libsodiumalternative, made only of safe functions that are (if not impossible then at least) hard to misuse;That’s not quite what I was saying. They could do a lot better with their APIs, but if you use the APIs as they intend then you have a benefit, it’s just that they didn’t do a great job. In contrast, with so-called ‘dark patterns’ the designers are deliberately trying to make you use them in a way that is harmful to the user. Both appear similar (any sufficiently advanced incompetence is indistinguishable from malice) but the fixes are very different. In the case of OpenSSL, it’s learning from things like libsodium (huge fan of that API). In the case of Facebook, it’s jail time for executives.
I think I understand the distinction you are trying to make, between involuntary “complications” and malicious “dark patterns”. However, I think the author is focusing on the unintended ones.
With regard to the dark patterns, I think the only possible solution is user education. There is no incentive for Facebook (or any other site with “adware”) to renounce the dark patterns. And I don’t think we can (or should) force them to by law (as long as they don’t hurt people in the process). However, we could try to give them negative publicity, or even better, come-up with better solutions.
Although I hope I’m wrong, I don’t think the current standards (i.e. WebAuthn, or recently PassKeys) are a proper solution to passwords (and password managers) for a few simple reasons:
However, there is no technical reason why this should be so, because both proposed standards could easily be implemented in software. The main issue is that in practical terms both of these are described in terms of API’s that the browsers have to implement, and for the moment the only implementations are either in the OS (iOS, Android) or via USB… (There are some “virtual tokens” that use USB-over-IP, but in practical terms this is ridiculous because it asks me to run something as
rootto “improve” my security…)So, for the foreseeable future it appears I’ll have to stick to passwords…
(Although for anyone that doesn’t have decent security practice, they are much better than plain passwords. It’s just that for advanced users they aren’t practical enough.)
FWIW, PassKeys is essentially just a profile of WebAuthn - the requests and responses are essentially the same, you can reuse essentially all of the code for both. PassKeys is just how Apple and Google decided to market their phones being able to act as WebAuthn authenticators.
Also, for what it’s worth, I don’t see the problem with WebAuhtn/PassKeys being tied to hardware and having no key backup or cloning. The strategy propagated by them is to have multiple authenticators, that are independent, and if one is lost, another one can still be used. I feel this is a lot better than to have the one true key, that you need to move around. After all, the least secure location for a key is being in transfer, and WebAuthn/PassKeys aim to eliminate the need for key movement.
I do agree that the API’s available for the methods is currently lacking though.
I’m not so sure that PassKeys and WebAuthn are exactly the same thing, with different branding name; they might be, they certainly might have common cryptographic underpinnings and details, but when I’ve tried to understand how they actually work I don’t think I’ve stumbled-upon something simple enough (but with technical details) to understand exactly what they are underneath… A lot of marketing, a lot of BS, not enough sensible documentation…
Where are the RFC’s?
Here is the “thing” for WebAuthn https://www.w3.org/TR/webauthn-3/; doesn’t even come close to an IETF RFC in terms of readability…
With regard to backups, they are a complete show-stopper for me personally:
Furthermore, what are the security measures for FIDO (and the like) tokens?
So, for the time being, I would actually prefer software-based (i.e. open source) tokens.
Webauthn spec is indeed not incredibly readable, but AFAIK that’s the case for all W3C specifications that deal with JS interfaces… The overview on MDN is a lot easier to understand since it explains things from a higher level.
Communication between a browser/OS and a physical token is described by CTAP2 specification, which is pretty lengthy, but it deals with a lot more stuff than just authentication.
Passkeys seem to not have a specification, but rather just be a term that’s co-marketed for essentially an embedded platform token. The cases where a phone is used for authenticating in a desktop browser is just CTAP2 over Bluetooth.
As for security: there’s a
userVerificationparameter in the API - with the possible values ofdiscouraged,preferredandrequired- the default one ispreferred. What this means depends on the authenticator, but even the simplest ones usually have PIN codes (entered through computer) as an option, and will ask user to enter them if the PIN has been enrolled inpreferredandrequireduse cases. Application of any additional policy depends on the authenticator, some do allow this to be customized, some not.Open source tokens do exist by the way, for example Solo which is my token of choice. It doesn’t have any custom silicon, it’s just a secure micro-controller with an open source firmware. They sell models that have been flashed with an unlocked bootloader, so if you want, you can load up your own custom version of the firmware and lock the bootloader so only versions could be used.
Maybe in the US, but in Europe, more specifically Romania, it’s somewhere closer to 50 EUR, which is quite a lot…
Would you bet your online presence on that? :)
For a syncable embedded platform token. That’s the whole point, that’s why they decided that it’s marketable to regular users already. Apple’s had a soft WebAuthn token tied to one device before and it was hidden in developer options. Now it’s “production ready” exactly because now the keys are synced to all your devices via iCloud Keychain (/ Google whatever). And yeah e.g. Apple does have recovery for when you lost all devices too.
Thanks for the video / interview! It’s a refreshing and apparently non-mainstream take, especially now in 2023 when Emacs almost seems reasonable in comparison with VS-Code and others… :)
It completely and exactly describes my own development workflow:
i3in my case) with a terminal multiplexer (tmux);mcto browse between the files;)makeor just would do just fine;)In essence: keep close to CLI and TUI tools!
Another interesting (and unpopular take) is reading the actual reference manual / documentation instead of randomly copy-pasting stuff from stack-overflow until something barely works!
Indeed — I really enjoyed hearing how he reads the manual, which is the opposite of what Julia Evans said she does in this other episode. Interesting to see two programmers I respect a lot taking very different approaches to the same problem.
It’s a very insightful write-up on (the perils of) client TLS certificates, especially the part regarding the trust you need to have in your load-balancer (or TLS terminator) doing the right thing (also making sure no one can smuggle in another
X-Client-Certificateheader and trick the actual backend).With regard to internal authentication, besides whitelists one could employ something like Wireguard, which is lightweight enough, and could replace TLS or other internal authentication mechanisms. (Assuming one runs only one service per node; although there are also solutions for the opposite case.)
However, and in relation with the previous article about
keygen, I would really have liked for client TLS certificates to be more user friendly and more widely supported…I really wished IPv6 would become “mainstream” and just replace IPv4 so, at least from my point of view of home / small-business administrator, we can all get rid of port forwarding and NAT’s.
However, I religiously disable IPv6 anywhere I can, but especially on (home) routers, due to the same reason described in the article: many of the consumer “devices” passively support IPv6, but who knows if they are ready or not to handle IPv6 traffic “safely” (for whatever “safely” means with these security disasters)…
For example I’ve currently standardized on Mikrotik small routers and access points mainly because their software is “enterprise-grade” but at an affordable price, and they do have proper IPv6 support (and lately Wireguard). However, many of the stock firmware on consumer routers (i.e. TP-Link, Linksys/Cisco, etc.) although support IPv6, they didn’t have proper firewall support for IPv6.
On the other hand, as a freelance developer / administrator, I never bother with IPv6 (unless supported by the CDN I use), because it’s a potential reliability issue.
Thus, when combining all these factors, I think the failure of IPv6 is a self-fulfilling prophecy…
I got the opposite of this from the article. If you disable IPv6 then that doesn’t actually disable IPv6. If you booted Windows Vista on a v4-only network, it would establish a v6 tunnel and then advertise itself as a router. Any other device on your network now has a routable, public, v6 address and is not protected by the firewall that you think that you have. It would be fairly easy for malware on a phone to do this, so someone brining the phone onto your network would tunnel all v6 traffic via a network that they control. If you’re not setting up IPv6, then you’re telling attackers ‘[encapsulated] IPv6 is reserved for your use’.
Adding a device in your local network (be it physically connected with a wire, wireless via WPA2, etc.) is always a risk for it to start poking at its neighbors. Especially in home / small office setups that don’t have DHCP snooping / port isolation / etc., a malicious device can even start giving IPv4 DHCP leases and route traffic through himself, thus taking over IPv6 is the least concern.
However, my main concern is with regard to traffic that crosses the network boundary (in or out), for which I don’t think we are that ready to tame as we are with IPv4.