1.  

    I’m considering building some kitchen management software. During the pandemic I did a lot more cooking at home and so far all the recipe/menu/inventory/shopping software has been a very poor fit.

    1. 73

      First, their argument for Rust (and against C) because of memory safety implies that they have not done due diligence in finding and fixing such bugs. […] And with my bc, I did my due diligence with memory safety. I fuzzed my bc and eliminated all of the bugs.

      This seems like such a short-sighted, limited view. Software is not bug-free. And ruling out a class of bugs by choice of technology as a measure of improving overall robustness won’t fix everything, but at the very least it’s a trade-off that deserves more thorough analysis than this empty dismissal.

      1. 19

        I think he is probably right (after all, he wrote it) when he says rewriting his bc in Rust would make it more buggy. I disagree this is an empty dismissal, since it is backed by his personal experience.

        For the same reason, I think cryptography developers are probably right (after all, they wrote it) when they say rewriting their software in Rust would make it less buggy. So the author is wrong about this. His argument is not convincing why he knows better than developers.

        1. 15

          I think there’s a big difference between programs and libraries with stable requirements and those that evolve here. The bc utility is basically doing the same thing that it did 20 years ago. It has a spec defined by POSIX and a few extensions. There is little need to modify it other than to fix bugs. It occasionally gets new features, but they’re small incremental changes.

          Any decision to rewrite a project is a trade off between the benefits from fixing the accumulated technical debt and the cost of doing and validating the rewrite. For something stable with little need of future changes, that trade is easy to see: the cost of the rewrite is high, the benefit is low. In terms of rewriting in a memory-safe language, there’s an additional trade between the cost of a memory safety vulnerability and the cost of the rewrite. The cost of Heartbleed in OpenSSL was phenomenal, significantly higher than the cost of rewriting the crypto library. In the cast of bc, the cost of a memory safety bug is pretty negligible.

          Data from Microsoft’s Security Response Center and Google’s Project Zero agree that around 70-75% of vulnerabilities are caused by memory safety bugs. Choosing a language that avoids those by construction means that you can focus your attention on the remaining 25-30% of security-related bugs. The author talks about fuzzing, address sanitiser, and so on. These are great tools. They’re also completely unnecessary in a memory-safe language because they try to find classes of bugs that you cannot introduce in the first place in a memory-safe language (and they do so probabilistically, never guaranteeing that they’ve found them all).

          If you’re starting a new project, then you need a really good reason to start it in C and pay the cost of all of that fuzzing.

          1. 17

            Data from Microsoft’s Security Response Center and Google’s Project Zero agree that around 70-75% of vulnerabilities are caused by memory safety bugs. Choosing a language that avoids those by construction means that you can focus your attention on the remaining 25-30% of security-related bugs.

            There’s an implied assumption here that if a language is memory safe, those memory safe bugs will simply go away. In my experience, that is not quite true. Sometimes those memory safety bugs will turn into logic bugs.

            Not to pick on Rust here, but in Rust it is very common to put values into an array and use array indices instead of pointers when you have some kind of self-referential data structure that’s impossible to express otherwise using rust’s move semantics. If you simply do such a naive transformation of your C algorithm, your code will be memory safe, but all your bugs, use after free, etc, will still be there. You just lifted them to logical bugs.

            Rust has no good abstractions to deal with this problem, there are some attempts but they all have various practical problems.

            Other languages like ATS and F* have abstractions to help with this problem directly, as well as other problems of logical soundness.

            1. 13

              Right - but in lifting these from memory bugs to logic bugs, you get a runtime panic/abort instead of a jump to a (likely-attacker-controllable) address. That’s a very different kind of impact!

              1. 9

                You don’t get a panic if you access the “wrong” array index. The index is still a valid index for the array. Its meaning (allocated slot, free slot, etc), is lost to the type system, though in a more advanced language it need not be. This later leads to data corruption, etc, just like in C.

                1. 7

                  It leads to a much safer variant of data corruption though. Instead of corrupting arbitrary memory in c or c++ (like a function pointer, vtable, or return address), you are only corrupting a single variable’s value in allocated and valid and aligned memory (like a single int).

                  You would get a panic in rust for every memory corruption bug that could cause arbitrary code execution, which is what matters.

                  1. 1

                    This later leads to data corruption, etc, just like in C.

                    Can you expand on this? I had expected the behaviour in Rust to be significantly safer than C here. In C, the data corruption caused by use-after free often allows an attacker to execute arbitrary code.

                    I totally see your point about logical corruption (including things like exposing critical secrets), but I don’t follow that all the way to “just like C”. How would an array index error be exploited in Rust to execute arbitrary code?

                    1. 11

                      I have once written a bytecode interpreter in C++, for a garbage collected scripting language. I implemented my own two-space garbage collector. For performance reasons, I didn’t use malloc() directly, but instead allocated a big enough byte array to host all my things. If I overflew that array, Valgrind could see it. But if I messed up it’s internal structure, no dice. That heap of mine was full of indices and sizes, and I made many mistakes that caused them to be corrupted, or somehow not quite right. And I had no way to tell.

                      I solved this by writing my own custom heap analyser, that examined the byte array and tell me what’s in there. If I see all my “allocated” objects in order, all was well. Often, I would see something was amiss, and I could go and fix the bug. Had I written it in Rust instead, I would have had to write the exact same custom heap analyser. Because Rust wouldn’t have prevented me from putting the wrong values inside my array. It’s perfectly “safe” after all, to write gibberish in that array as long as I don’t overflow it.

                      Now could this particular bug lead to arbitrary code execution? Well, not quite. It would generate wrong results, but it would only execute what my C++/Rust program would normally execute. In this case however, I was implementing a freaking scripting language. The code an attacker could execute wasn’t quite arbitrary, but it came pretty damn close.

                      1. 6

                        The effects of data corruption depend on what the code does with the data. This often means arbitrary code execution, but not always. It’s not a property of C, it’s a property of the code. This doesn’t change when you change the implementation language.

                        Fundamentally there is no semantic difference between a pointer in a C heap and an array index into a Rust array. In fact some sophisticated blog authors that explain this array technique often point out they compile to the exact same assembly code. It’s what the code does with the data that leads to exploitation (or not).

                        Of course Rust has many additional safety advantages compared to C, buffer overflows don’t smash the stack, etc, and using references in Rust if you can is safe. And when using references, there’s a great deal of correlation between Rust’s notion of memory safety and true logic safety. This is good! But many people don’t realise that this safety is predicated on the lack of aliasing. The borrow checker is only a mechanism to enforce this invariant, it’s not an operative abstraction. It’s the lack of aliasing that gets you the safety, not the borrow checker itself. When you give up aliasing, you lose a lot of what Rust can do for you. Virtually everybody understands that if you introduce unsafe pointers, they give up safety, but less people seem to understand that introducing aliasing via otherwise safe mechanism has the same effect. Of course, the program continues to be memory safe in Rust terms, but you lose the strong correlation between memory safety and logic safety that you used to have.

                        Not that there’s anything wrong with this, mind you, it’s just something people need to be aware of, just as they are already aware of the tradeoffs that they make when using unsafe. It does make a projection for the number of bugs that Rust can prevent in practice more difficult, though.

                        1. 6

                          I think this is incorrect. Arbitrary code execution does not mean “can execute an arbitrary part of my program due to a logic bug”, it means “can execute arbitrary code on the host, beyond the code in my program”. Even a rust alias logic bug dies not open up this kind of arbitrary code execution exposure because you can’t alias an int with a function pointer or a vtable or a return address on the stack, like you can in c or c++. You can only alias an int with an int in safe rust, which is an order of magnitude safer and really does eliminate an entire class of vulnerabilities.

                          1. 6

                            I think this is incorrect. Arbitrary code execution does not mean “can execute an arbitrary part of my program due to a logic bug”, it means “can execute arbitrary code on the host, beyond the code in my program”.

                            In the security research world, we usually treat control of the program counter (the aptly named rip on x86-64) as “arbitrary code execution.” You can do a surprising of programming using code that’s already in a process without sending any byte code of your own with return-oriented programming.

                            1. 3

                              But does Rust let you do that here? What does a snippet of Rust code look like that allows attacker-controlled indexing into an array escalate to controlling the program counter?

                              1. 2

                                Surely you agree that “variables changing underfoot” implies “programs flow becomes different from what I expect”. That’s why we use variables, to hold the Turing machine state which influences the next state. A logical use after free means “variables changing underfoot”. You don’t expect a free array slot’s value (perhaps now reallocated) to change based on some remote code, but it does.

                                1. 3

                                  Right, but “program flow becomes different from what I expect, but it still must flow only to instruction sequences that the original program encoded” is much much safer than “program flow can be pointed at arbitrary memory, which might not even contain instructions, or might contain user-supplied data”.

                                  1. 2

                                    With ROP, the program flow only goes through “instruction sequences that the original program encoded”, and yet ROP is pretty much fatal.

                                    1. 7

                                      ROP is not possible when you index an array wrong in rust, what is your point?

                                      1. 6

                                        And you can’t do rop in safe rust.

                                        1. 2

                                          Maybe not directly within the native code of the program itself, but I think (at least part of) 4ad’s point is that that’s not the only level of abstraction that matters (the memory bug vs. logic bug distinction).

                                          As an example, consider a CPU emulator written entirely in safe Rust that indexes into a u8 array to perform its emulated memory accesses. If you compile an unsafe program to whatever ISA you’re emulating and execute it on your emulator, a bad input could still lead to arbitrary code execution – it’s at the next semantic level up and not at the level of your program itself, but how much does that ultimately matter? (It’s not really terribly different than ROP – attacker-controlled inputs determining what parts of your program get executed.)

                                          That’s admittedly a somewhat “extreme” case, but I don’t think the distinction between programs that do fall into that category and those that don’t is terribly clear. Nearly any program can, if you squint a bit, be viewed essentially as a specialized interpreter for the language of its config file (or command-line flags or whatever else).

                                          1. 2

                                            There’s no distinction here. If your program implements a cpu emulator then your program can execute with no arbitrary code execution at all and still emulate arbitrary code execution on the virtual cpu. If you want the virtual program executing to not possibly execute arbitrary virtual instructions, you need to generate the virtual program’s instructions using a safe language too.

                                            In most cases, though, arbitrary virtual code execution is less dangerous than arbitrary native code execution, though that’s beside the point.

                                            1. 2

                                              So…we agree? My point was basically that attacker-controlled arbitrary code execution can happen at multiple semantic levels – in the emulator or in the emulated program (in my example), and writing the emulator in a safe language only protects against the former, while the latter can really be just as bad.

                                              Though I realize now my example was poorly chosen, so a hopefully better one: even if both the emulator and the emulated program are written in memory-safe languages, if the emulator has a bug due to an array-index use-after-free that causes it to misbehave and incorrectly change the value of some byte of emulated memory, that destroys the safety guarantees of the emulated program and we’re back in arbitrary-badness-land.

                                              1. 1

                                                Sure but this is just as meaningful as talking about a cpu hardware bug that might cause a native safe program to run amok. Technically true but not very useful when evaluating the safe programming language

                                2. 3

                                  Right, I agree, and safe rust aliasing that the GP described is not possible to control the program counter arbitrarily.

                                3. 4

                                  Yeah exactly, this is the part I thought @4ad was arguing was possible. Eg. in C, use-after-free often allows me to make the program start interpreting attacker-provided data as machine code. I thought this is what 4ad was saying was also possible in Rust, but I don’t think that’s what they are claiming now.

                                  To me, that’s a big difference. Restricting the possible actions of a program to only those APIs and activities the original code includes, vs C where any machine code can be injected in this same scenario, is a major reduction in attack surface, to me.

                                  1. 4

                                    One thing to note is that code is data and data is code, in a true, hard-mathematical sense.

                                    The set of

                                    the possible actions of a program to only those APIs and activities the original code includes,

                                    and

                                    C where any machine code can be injected in this same scenario

                                    is exactly the same (unbounded!). Of course it is much easier in practice to effect desired behavior when you can inject shell code into programs, but that’s hardly required. You don’t need to inject code with ROP either (of course ROP itself is not possible in Rust because of other mitigations, this is just an example).

                                    Please note that in no way I am suggesting that Rust is doing anything bad here. Rust is raising the bar, which is great. I want the bar raised even higher, and we know for a fact that this is possible today both in theory and practice. Until we raise the bar, I want people to understand why we need to raise the bar.

                                    At the end of a day you either are type safe or you aren’t. Of course the specifics of what happens when you aren’t type safe depend on the language!

                                    PS: arrays can contain other things than integers, e.g. they can contain function pointers. Of course you can’t confuse an int with a function pointer, but using the wrong function pointer is pretty catastrophic.

                                    1. 3

                                      is exactly the same (unbounded!).

                                      I guess this is what I don’t understand, sorry for being dense. Can you show a concrete code example?

                                      In my mind I see a program like this:

                                      
                                      enum Action {
                                          GENERATE_USER_WEEKLY_REPORT,
                                          GENERATE_USER_DAILY_REPORT,
                                          LAUNCH_NUCLEAR_MISSILES
                                      }
                                      
                                      impl Action {
                                        pub fn run(&self) {
                                          ...
                                        }
                                      }
                                      
                                      // Remember to remove the nuclear missile action before calling!
                                      fn exploitable( my_actions:  &Vec<Box<Action>>, user_controlled: usize ) {
                                        my_actions[user_controlled].run();
                                      }
                                      
                                      

                                      In my mind, there are two differences between this code in Rust and similar code in C:

                                      1. This only allows the user to launch nuclear missiles; it does not allow them to, say, write to the harddrive or make network calls (unless one of the actions contained code that did that ofc); in C, I’d likely be able to make something like this call any system function I wanted to, whether machine code to do that was present in the original binary or not.

                                      2. In Rust, this doesn’t allow arbitrary control flow, I can’t make this jump to any function in the binary, I can only trick it into running the wrong Action; in C, I can call run on any arbitrary object anywhere in the heap.

                                      ie. in C, this would let me execute anything in the binary, while in Rust it still has to abide by the control flow of the original program, I thought was the case, anyway.

                                      I think you’re saying this is wrong, can you explain how/why and maybe show a code example if you can spare the time?

                                      1. 4

                                        This is correct and 4ad is mistaken. I’m not sure why 4ad believes the two are equivalent; they aren’t.

                                      2. 3

                                        “is exactly the same”

                                        It simply isn’t, and I’m not sure why you think it is.

                                  2. 1

                                    In fact some sophisticated blog authors that explain this array technique often point out they compile to the exact same assembly code.

                                    Do you have any links on this that you recommend?

                          2. 2

                            Good analysis. You didn’t use the words, but this is a great description of the distinction between stocks and flows: https://en.wikipedia.org/wiki/Stock_and_flow. I wish more people talking about software paid attention to it.

                            1. 2

                              Author here.

                              I would also argue that crypto should not change often, like bc. You might add ciphers, or deprecate old ones, but once a cipher is written and tested, there should be very little need for it to change. In my opinion.

                            2. 8

                              For the same reason, I think cryptography developers are probably right (after all, they wrote it) when they say rewriting their software in Rust would make it less buggy.

                              Have they actually rewrote anything? Or have they instead selected a different crypto library they trust better than the previous one? On the one hand, Rust has no advantage over C in this particular context. On the other hand, they may have other reasons to trust the Rust library better than the C one. Maybe it’s better tested, or more widely used, or audited by more reputable companies.

                              If I take your word for it however, I have to disagree. Rewriting a cryptographic library in Rust is more likely to introduce new bugs, than it is to fix bugs that haven’t already been found and fixed in the C code. I do think however that the risk is slim, if they take care to also port the entire test suite as well.

                              1. 7

                                In the Cryptography case isn’t the Rust addition some ASN.1 parsing code? This is cryptography adjacent but very much not the kind of different that your point about cryptography code makes. Parsing code unless it is very trivial and maybe not even then tends to be some of the more dangerous code you can write. In this particular case Rust is likely a large improvement in both ergonomics for the parsing as well as safety.

                                1. 1

                                  You’ve got a point. I can weaken it somewhat, but not entirely eliminate it.

                                  I don’t consider ASN.1 “modern”. It’s over complicated for no good reason. Certificates can be much, much simpler than that: at each level, you have a public key, ID & expiration date, a certificate of the CA, and a signature from the CA. Just put them all in binary blobs, and the only thing left to parse are the ID & expiration date, which can be left to the application. And if the ID is an URL, and the expiration date is an 64-bit int representing seconds from epoch, there won’t be much parsing to do… Simply put, parsing certificate can be “very trivial”.

                                  Another angle is that if you need ASN.1 certificates, then you are almost certainly using TLS, so you’re probably in a context where you can afford the reduced portability of a safer language. Do use the safer language in this case.

                                  Yet another angle is that in practice, we can separate the parsing code from the rest of the cryptographic library. In my opinion, parsing of certificate formats do not belong to a low-level cryptographic library. In general, I believe the whole thing should be organised in tiers:

                                  • At the lowest level, you have the implementation of the cryptographic primitives.
                                  • Just above that, you have constructions: authenticated encryption, authenticated key exchange, PAKE…
                                  • Higher up still, you have file format, network packet formats, and certificates. They can (and should) still be trivial enough that even C can be trusted with them. They can still be implemented with zero dependencies, so C’s portability can still be a win. Though at that level, you probably have an idea of the target platforms, making portability less of a problem.
                                  • Higher up still is interfacing with the actual system: getting random numbers, talking to the file system, actually sending & receiving network packets… At that level, you definitely know which set of platforms you are targetting, and memory management & concurrency start becoming real issues. At that point you should seriously consider switching to a non-C, safer language.
                                  • At the highest level (the application), you should have switched away from C in almost all cases.
                              2. 2

                                For the same reason, I think cryptography developers are probably right (after all, they wrote it) when they say rewriting their software in Rust would make it less buggy. So the author is wrong about this. His argument is not convincing why he knows better than developers.

                                This is a fair point. When it comes down to it, whether I am right or wrong about it will only be seen in the consequences of the decision that they made.

                              3. 14

                                Here’s the more thorough analysis you’re asking for: this is cryptographic code we’re talking about. Many assumptions that would be reasonable for application code simply does not apply here:

                                • Cryptographic code is pathologically straight-line, with very few branches.
                                • Cryptographic code has pathologically simple allocation patterns. It often avoids heap allocation altogether.
                                • Cryptographic code is pathogenically easy to test, because it is generally constant time: we can test all code paths by covering all possible input & output lengths. If it passes the sanitizers & valgrind under those conditions, it is almost certainly correct (with very few exceptions).

                                I wrote a crypto library, and the worst bug it ever had wasn’t caused by C, but by a logic error that would have happened even in Haskell. What little undefined behaviour it did have didn’t have any visible effect on the generated code.

                                Assuming you have a proper test suite (that tests all input & output lengths), and run that test suite with sanitisers & Valgrind, the kind of bug Rust fixes won’t occur in your cryptographic C code to begin with. There is therefore no practical advantage, in this particular case to using Rust over C. Especially when the target language is Python: you have to write bindings anyway, so you can’t really take advantage of Rust’s better APIs.

                                1. 2

                                  These bugs still occur in critical software frequently. It is more difficult and time consuming to do all of the things you proposed than it is to use a safer language (in my opinion), and the safer language guarantees more than your suggestions would. And there’s also no risk of someone forgetting to run those things.

                                  1. 6

                                    These bugs still occur in critical software frequently.

                                    Yes they do. I was specifically talking about one particular kind of critical software: cryptographic code. It’s a very narrow niche.

                                    It is more difficult and time consuming to do all of the things you proposed than it is to use a safer language (in my opinion)

                                    In my 4 years of first hand experience writing cryptographic code, it’s really not. Rust needs the same test suite as C does, and turning on the sanitizers (or Valgrind) on this test suite is a command line away. The real advantage of Rust lies in its safer API (where you can give bounded buffers instead of raw pointers). Also, the rest of the application will almost certainly be much safer if it’s written in Rust instead of C.

                                    And there’s also no risk of someone forgetting to run those things.

                                    Someone who might forget those things has no business writing cryptographic code at all yet, be it in C or in Rust. (Note: when I started out, I had no business writing cryptographic code either. It took over 6 months of people findings bugs and me learning to write a better test suite before I could reasonably say my code was “production worthy”.)

                                    1. 6

                                      Rusts advantage goes much further than at the api boundary, but again the discussion should be around how to get safer languages more widely used (ergonomics, platform support) and not around “super careful programmers who have perfect test suites and flawless build pipelines don’t need safer languages”. To me it is like saying “super careful contractors with perfect tools don’t need safety gear”, except if you make a mistake in crypto code, you hurt more than just yourself. Why leave that up to human fallability?

                                      1. 4

                                        Rusts advantage goes much further than at the api boundary

                                        Yes it does. In almost all domains. I’m talking about modern cryptographic code.

                                        again the discussion should be around how to get safer languages more widely used (ergonomics, platform support)

                                        Write a spec. A formal one if possible. Then implement that spec for more platforms. Convincing projects to Rewrite It In Rust may work as a way to coerce people into supporting more platforms, but it also antagonises users who just get non-working software; such a strategy may not be optimal.

                                        not around “super careful programmers who have perfect test suites and flawless build pipelines don’t need safer languages”.

                                        You’re not hearing me. I’m not talking in general, I’m talking about the specific case of cryptographic code (I know, I’m repeating myself.)

                                        • In this specific case, the amount of care required to write correct C code is the same as the amount of care required to write Rust code.
                                        • In this specific case, Rust is not safer.
                                        • In this specific case, you need that perfect test suite. In either language.
                                        • In this specific case, you can write that perfect test suite. In either language.

                                        except if you make a mistake in crypto code, you hurt more than just yourself. Why leave that up to human fallability?

                                        I really don’t. I root out potential mistakes by expanding my test suite as soon as I learn about a new class of bugs. And as it happens, I am painfully aware of the mistakes I made. One of them was even a critical vulnerability. And you know what? Rust wouldn’t have saved me.

                                        Here are the bugs that Rust would have prevented:

                                        • An integer overflow that makes elliptic curves unusable on 16-bit platforms. Inconvenient, but (i) it’s not a vulnerability, and (ii) Monocypher’s elliptic curve code is poorly suited to 16-bit platforms (where I recommend C25519 instead).
                                        • An instance of undefined behaviour the sanitizers didn’t catch, that generated correct code on the compilers I could test. (Note that TweetNaCl itself also have a couple instances of undefined behaviour, which to my knowledge never caused anyone any problem so far. Undefined behaviour is unclean, but it’s not always a death sentence.)
                                        • A failure to compile code that relied on conditional compilation. I expect Rust has better ways than #ifdef, though I don’t actually know.

                                        Here are the bugs that Rust would not have prevented:

                                        • Failure to wipe internal buffers (a “best effort” attempt to erase secrets from the computer’s RAM).
                                        • A critical vulnerability where fake signatures are accepted as if they were genuine.

                                        Lesson learned: in this specific case, Rust would have prevented the unimportant bugs, and would have let the important ones slip through the cracks.

                                        1. 8

                                          I’m talking about modern cryptographic code.

                                          In this discussion, I think it is important to remind that cryptography developers are explicitly and intentionally not writing modern cryptographic code. One thing they want to use Rust on is ASN.1 parsing. Modern cryptographic practice is that you shouldn’t use ASN.1 and they are right. Implementing ASN.1 in Rust is also right.

                                          1. 4

                                            I’m talking about modern cryptographic code.

                                            So am I.

                                            In this specific case, the amount of care required to write correct C code is the same as the amount of care required to write Rust code.

                                            I disagree.

                                            In this specific case, Rust is not safer.

                                            I disagree here too.

                                            In this specific case, you need that perfect test suite. In either language.

                                            I partially agree. There is no such thing as a perfect test suite. A good crypto implementation should have a comprehensive test suite, of course, no matter the language. But that still isn’t as good as preventing these classes of bugs at compile time.

                                            Rust wouldn’t have saved me.

                                            Not really the point. Regardless of how lucky or skilled you are that there are no known critical vulnerabilities in these categories in your code, that disregards both unknown vulnerabilities in your code, and vulnerabilities in other people’s code as well. A safe language catches all three and scales; your method catches only one and doesn’t scale.

                                            1. 1

                                              Note that I did go the extra mile and went a bit further than Valgrind & the sanitisers. I also happen to run Monocypher’s test suite under the TIS interpreter, and more recently TIS-CI (from TrustInSoft). Those things guarantee that they’ll catch any and all undefined behaviour, and they found a couple bugs the sanitisers didn’t.

                                              that disregards both unknown vulnerabilities in your code

                                              After that level of testing and a successful third party audit, I am confident there are none left.

                                              and vulnerabilities in other people’s code as well

                                              There is no such code. I have zero dependencies. Not even the standard library. The only thing I have to fear now is a compiler bug.

                                              your method catches only one and doesn’t scale.

                                              I went out of my way not to scale. Yet another peculiarity of modern cryptographic code, is that I don’t have to scale.

                                              1. 1

                                                There is no such code.

                                                Sure there is. Other people write cryptographic code too. Unless you are here just arguing against safe languages for only this single project? Because it seemed like a broader statement originally.

                                                I went out of my way not to scale.

                                                I mean scale as in other developers also writing cryptographic software, not scale as in your software scaling up.

                                                1. 1

                                                  Sure there is. Other people write cryptographic code too. Unless you are here just arguing against safe languages for only this single project

                                                  I was talking about Monocypher specifically. Other projects do have dependencies, and any project that would use Monocypher almost certainly has dependencies, starting with system calls.

                                                  I mean scale as in other developers also writing cryptographic software, not scale as in your software scaling up.

                                                  Fair enough. I was thinking from the project’s point of view: a given project only need one crypto library. A greenfield project can ditch backward compatibility and use a modern crypto library, which can be very small (or formally verified).

                                                  Yes, other people write cryptographic code. I myself added my own to this ever growing pile because I was unsatisfied with what we had (not even Libsodium was enough for me: too big, not easy to deploy). And the number of bugs in Monocypher + Libsodium is certainly higher than the number of bugs in Libsodium alone. No doubt about that.

                                                  Another reason why crypto libraries written in unsafe languages don’t scale, is the reputation game: it doesn’t matter how rigorously tested or verified my library is, if you don’t know it. And know it you cannot, unless you’re more knowledgeable than I am, and bother to audit my work yourself, which is prohibitively expensive. So in practice, you have to fall back to reputation and external signs: what other people say, the state of documentation, the security track record, issues from the bug tracker…

                                  2. 6

                                    This made me twitch!

                                    Why make a choice which prevents an entire class of bugs when you could simply put in extra time and effort to make sure you catch and fix them all?

                                    Why lock your doors when you can simply stand guard in front of them all night with a baseball bat?

                                    While personally would back the cryptography devs’ decision here, I think there is a legitimate discussion to be had around whether breaking compatibility for some long-standing users is the right thing to do. This post isn’t contributing well to that discussion.

                                  1. 12

                                    We go through this every time some language designer comes up with a way of packaging and distributing: CPAN,pip,gems,npm,crates and it goes on and on. It seems like everybody likes re-inventing the distribution wheel.

                                    Short version is, RPMs and debs have been around for 25 years, and, while they were originally designed with C in mind, they’re flexible enough to incorporate programs written in any language.

                                    Yes, they have restrictions, and some of these restrictions are uncomfortable to people who do all their work in one or two particular languages, but distros like Debian and Redhat target people who just want to have a system that works out of the box, with programs written in many different languages co-existing in one reasonably coherent system. This is the problem distro packages like RPM and deb are trying to solve.

                                    I appreciate that if I’m working in ruby, I usually have something like rvm, and bundler, and other utilities for keeping multiple ruby development environments in a sane way, and I appreciate that others like these tools for programming in their preferred environment.

                                    However. If I just want to be able to install and use a script written in python (take for example, “Terminator” that is written in python), as a distro, I just want to be able to install the script, and ensure that it works cleanly with the rest of the system, I don’t care about setting up a special environment and managing dependencies, and all of the other baggage that these other distribution methods involve.

                                    1. 18

                                      Short version is, RPMs and debs have been around for 25 years, and, while they were originally designed with C in mind, they’re flexible enough to incorporate programs written in any language.

                                      No, they don’t. They don’t even work well for C++, and have been hobbling the development of template libraries for over 10 years now because of how bad they are at handling API-stable-ABI-unstable libraries.

                                      1. 2

                                        What’s the problem? It’s been a while since I looked at these in Linux environments, but in the FreeBSD ports collections if you upgrade a C++ library, it bumps the PORTREVISION of all of the ports that depend on that library. The next time a package set is built, the packages use the new version of the library. The FreeBSD project explicitly moved away from building packages outside of package sets to address this kind of thing - it’s easy for things to get out of sync and computers are so fast now that building everything (30,000+ packages) on a single machine is feasible in about a day.

                                      2. 10
                                        • apt doesn’t handle multiple versions of the same library well. You can do it by renaming packages and an indirection layer of virtual packages, but this is tortured compared to auto deduplicating dependencies according to semver.

                                        • People in charge of Linux package managers generally don’t care about Windows. There is nothing technically preventing RPM/deb from working on Windows, except that it “sucks” (for shits and giggles I’ve made cargo-deb Windows compatible, and it happily builds Windows-only deb packages).

                                          npm has got a huge traction as a method of distributing dev tools, because it supports Windows as a first-class platform.

                                        • RPM/deb as an archive format doesn’t make that much of a difference. Rust/Python/JS could use .deb as their format, after all it’s just a bunch of files together. The real issue is where do you publish to, and how other people find it. Linux distros set themselves as gatekeepers, which has its own value, but it’s very different from npm/cargo, etc. that have npm publish free for all with zero hassle, way easier than maintaining PPAs.

                                        Having written all that, I realize it all comes to usability. The distinction between “it can do it” vs “it just works”.

                                        1. 8

                                          I never want to care about “the rest of the system”. I want to write complete programs, not shells of programs that work if slotted into the right system.

                                          The more programs are like this, the less a system that works out of the box is a problem to think about.

                                          1. 5

                                            I’d argue the packaging system put up by Linux distros is actually not flexible, but rather the human sitting between applications to be packaged and the Linux distribution sinking hours into patching software to fit into the packaging system is. If rpm/deb were actually flexible we would not have these conflicts.

                                            It seems like everybody likes re-inventing the distribution wheel.

                                            I can say the same about the state of Linux distributions. It seems that I as an application developer can only rely on the featureset that is the intersection of all the distro’s package managers if I were to follow your advice.

                                            1. 2

                                              rpm/deb is fairly flexible. The reason you get these arguments is mostly because of distribution policies, not because deb/rpm can’t do it.

                                              https://lwn.net/Articles/843313/rss

                                              1. 2

                                                I did not claim rpm/deb cannot deal with large bundled apps (that’s fairly trivial, curl|sh can do that too). I’m saying rpm/deb cannot deal with dependency management to the granularity npm/cargo can, and then not in an efficient manner. Kornel already replied with other examples where rpm/deb can’t do things.

                                            2. 7

                                              Then the distribution maintainers should solve that problem within the constraints of the language and ecosystem the tool was developed in. The language and ecosystem are not going to change nor should they. If RPMs and debs can handle the change then they should just package them and move on. Complaining that new ways of doing development and deployment make the job harder helps no one. Either the distributions will adapt or they will lose the next generation of software developed in these new paradigms.

                                              1. 9

                                                A CVE gets assigned some widely popular library. For fun we will say they have a monthly release cadence and the bug is published mid release cycle. Upstream is not done with their integration tests and don’t want the release just for a 3 line patch, even if the issue is severe. Lets say it’s used for around 30 binaries.

                                                What do you do?

                                                If the solution here is to do some wonkey patching of Cargo.toml and Cargo.lock across 30 packages to ensure they are pulling the correct patch (is it even possible), how does this scale?

                                                This isn’t the question of distributions adapting to anything, this is “the next generation of software” digging a deep grave and ignoring almost 3 decades worth of experience. This isn’t me claiming we should pretend Rust is C and package all crates like individual packages. Nobody has the time for that. But pretending this isn’t a problem is deeply unsettling.

                                                1. 10

                                                  I don’t know, it’s not my problem, but if it were, I guess I would try solving it rather than trying to wedge everything into this old C paradigm.

                                                  1. 4

                                                    Not having a solution doesn’t mean you can just paint an old solution as terrible.

                                                    Lots of smart people are working on separating language ecosystems from system administration, and we have these problems. So now, what do we do?

                                                  2. 4

                                                    Upstream is not done with their integration tests and don’t want the release just for a 3 line patch, even if the issue is severe.

                                                    In the case of uncooperative and irresponsible upstreams, what Debian does is say “we will package this, but it is not covered by the security support that we provide for the rest of the OS”. They used to do this for webkit and node.

                                                    What else can you do? At some point packaging implies cooperation.

                                                    1. 3

                                                      Cargo has multiple features for replacing dependencies. For individual packages you drop [patch.crates-io] (it works across the whole tree, so no need to patch deps-of-deps recursively). To do it at a distro scale, you can globally configure Cargo to use a local copy of the index instead of crates-io, and replace deps in your copy (it’s just a git repo with JSON, easy to modify).

                                                      Binaries are built from lockfiles, so you can read them to know exactly what to rebuild. There are tools like cargo-audit that already deal with this.

                                                      1. 1

                                                        So there are then 30 patches to modify cargo. You would also need a secondary package repository to provide the patched cargo package? Does cargo build everything from source or would it require the patched packages to be built?

                                                        1. 3

                                                          You can tell Cargo to get code from a local directory. Distros already replace crates-io with their own packages, so they already have it all set up.

                                                          Distros have managed to tame dozens of weird C build systems. Cargo is quite simple in comparison.

                                                          1. 2

                                                            Well, no. Most doesn’t. Just the two largest ones because of distro policies. But if you look at the raw numbers most distribution bundle everything vendored and does not provide this at all. I’m not even sure if Ubuntu follows the Debian guidelines?

                                                            This is why I bring it up to begin with.

                                                      2. 2

                                                        I get it. Distro’s are downstream of everything. They get the sewage and the refuse and whatever else that results from trying to get hundreds if not thousands of different applications and libraries to work together and keep them patched appropriately. But that’s the job. If you don’t want that job then don’t work on a distro.

                                                        In your particular example I would feel free to blacklist a project that doesn’t want to patch and test their code when it has a CVE. If the code is absolutely necessary and blacklisting it isn’t an option then patch it locally and move on. This isn’t substantially different from a terrible maintainers of a C application. Distributions have been carrying patches forward for libraries and applications for as long as I’ve been using distributions and longer.

                                                        1. 7

                                                          Back in the C days, it was considered basic etiquette to make sure your Makefile worked with DESTDIR properly.

                                                          What happens now is simply Parkinson’s law in it’s finest. Flatpak, Snap and Docker included.

                                                          It puzzles me that nobody is worried about our inability to maintain a coherent culture given the influx of programmers, but then again… Eternal September, right?

                                                          Must be weird for old geezers to live through that the second time. I am far too young for that so I can’t tell.

                                                          We need to negotiate a packaging standard that would not suck for most and then push it hard so that it gets adopted far and wide from Rust to ES. Funny times.

                                                          I’m especially curious whether it can be done without effectively merging all distros into one. But hey, project maintainers seem to know the best how is their code supposed to be built and configured. Maybe it’s time.

                                                          1. 3

                                                            I am one of the old geezers and I’m fully on board with the new way of doing things. Let the software maintainers bear the burden of patching and securing their software when CVEs happen. In a way this could reduce the burden on distro packagers. Download or build the latest binaries and distribute that. If upstream won’t patch their stuff then warn the user and refer them to upstream. Yes this means packagers have to build more stuff. But we unlock a whole lot of benefits in the process. Less issues with shared libraries. More reliable software with these new tools.

                                                          2. 4

                                                            This isn’t substantially different from a terrible maintainers of a C application. Distributions have been carrying patches forward for libraries and applications for as long as I’ve been using distributions and longer.

                                                            Previously we just needed to patch the shared library and move on. Now we suddenly need to care about what a number of upstreams vendor with complete disregard for what that implies.

                                                            The comments reads as extremely unsympathetic to distributions. But why? This is a model Rust/Go/npm picked. This wasn’t decided by us, and you still need to deal with the issue regardless if there is a distribution involved or not. We are told “take this and deal with it”. Upstreams are not the one going to deal with user inquiries why XYZ isn’t fixed and what we are going to do about it. We are understaffed and given more problems to deal with.

                                                            If you don’t want us to package the “next generation of software” say so… but users are going to disagree.

                                                            1. 1

                                                              I acknowledge the fact that you have to run cargo build more times than before. But that is the price you pay for packinging in a distro. If your users want the rust application then package it for them. Rust isn’t going to adapt for a whole host of really good reasons. And I as both a developer and someone who deploys some of these in production get a lot of benefit out of those reasons and as the primary user of the language would resist any such change.

                                                              For the security issues if upstream won’t patch then remove them from the security policy and tell the user they need to take CVEs up with maintainer of the software. This isn’t hard and complaining about it gives no value to any end-user.

                                                              1. 2

                                                                Are we going to claim everything Rust touches to be unsupported? Seriously?

                                                                I don’t think Rust is the C replacement the community thinks it is.

                                                                1. 1

                                                                  If the distro needs something that Rust touches then they need to build the tooling to be able to package it. It’s more expensive to build it all but if that’s what you need to do then do that.

                                                            2. 3

                                                              But that’s the job. If you don’t want that job then don’t work on a distro.

                                                              Note that Debian security team found the job so onerous that they decided to remove security support for all Go packages in Debian. See https://www.debian.org/releases/stable/amd64/release-notes/ch-information.en.html#golang-static-linking.

                                                              1. 1

                                                                This is a perfectly valid approach. If you can’t make your tooling do the job with that software then drop support for it.

                                                                1. 2

                                                                  This works for Go, but does not work for Rust, because GNOME C libraries are starting to depend on Rust and distributions don’t want to drop support for GNOME.

                                                                  1. 1

                                                                    Then I guess in the case of GNOME it is by definition not too onerous. It is not the case that you can’t package and ship updates for the software. It’s just harder and more complicated. But that’s why we write tooling. I don’t get to complain when my business wants a feature that requires me to write complicated hard code. I just role up my sleeves and start designing/typing. This is no different.

                                                      1. 5

                                                        What is the actual path forward fixing the problem? Bringing Rust/LLVM support to all of those platforms? I can understand the reasoning by the maintainers that C is inherently insecure, but not being able to use the package for the foreseeable future isn‘t really an option either. Well it might spark some innovation :D

                                                        1. 14

                                                          Speaking in realistic terms and fully acknowledging that it is in some ways a sad state of affairs. Most of those platforms are dying and continuing to keep the alive is effectively volunteering to keep porting advancements from the rest of the world onto the platform. If you want to use third party packages on an AIX box you sort of just have to expect that you’ll have to share the labor of keeping that third party package working on AIX. The maintainers are unlikely to be thinking about you and for good reason.

                                                          For users of Alpine linux the choice to use a distro that is sufficiently outside the mainstream means you are also effectively committing to help port advancements from the rest of the world onto the platform if you want to consistently use them.

                                                          For both categories you can avoid that implicit commitment by moving to more current and mainstream platforms.

                                                          1. 12

                                                            Exactly. And as a long term implication, if Rust is here to stay, the inevitable fate of those platforms is “Gain Rust support or die”. Maintaining a C implementation of everything in the name of backward compatibility is only delaying the inevitable, and is ultimately a waste of time.

                                                          2. 6

                                                            I see it like this: Rust is not going away. This won’t be the last project introducing rust support.

                                                            Either your plattform supports rust or the world will stop supporting your plattform.

                                                            1. 5

                                                              This definitely seems to be true. I helped drive Rust support for a relatively niche platform (illumos), and while it took a while to get it squared away everybody I spoke with was friendly and helpful along the way. We’re a Tier 2 platform now, and lots of Rust-based software just works. We had a similar experience with Go, another critical runtime in 2021, which also required bootstrapping, being no longer written in C and so on.

                                                            2. 8
                                                              1. The package maintainers agree not to break shit, or
                                                              2. Someone from among those affected volunteers to maintain a fork.
                                                              1. 5

                                                                I mean, you can always pin your dependency to the version before this one. No way that could come back and bite you </sarcasm>

                                                                1. 2

                                                                  I think GCC frontend for Rust, which recently got funded, will solve this problem.

                                                                  1. 3

                                                                    Rust crates tend to rely on recently added features and/or library functions. Given that GCC releases are far less frequent, I think there will be a lot of friction when using the GCC frontend.

                                                                    1. 3

                                                                      Maintaining compatibility with a 6/9/12 month old rust compiler is a much smaller ask of the maintainers than maintaining a C library indefinitely.

                                                                1. 1

                                                                  I’m very pleased that you at least discussed what you mean by “functional”. Everyone who even begins to discuss whether something is “functional” or “object oriented” or “procedural” or whatever, should be required to begin with a definition of what those things mean to them.

                                                                  Your post makes me feel mixed things about whether Rust is “functional”.

                                                                  The borrow checker means that you are free to write impure functions without sacrificing correctness/safety. It’s also more performant to do so. So why would you write functional style Rust? You’re sacrificing (a little) performance for no real gain in correctness.

                                                                  Granted, I agree with your point that Rust’s controlled mutation mechanism does encourage you to write less mutation-happy code because passing around too many mutable references gets painful compared to other languages with no mutation control.

                                                                  I’d say that Rust, fundamentally is not a functional language. Taking a copy when you could mutate would be something of an anti-pattern, IMO.

                                                                  It’s just that working with the borrow checker will make you want to write code with less mutation. On the other hand, writing Java code ALSO makes me want to write code with less mutation because of how buggy and incomprehensible my code becomes with prolific mutation. Is Java functional? Probably not.

                                                                  1. 3

                                                                    Funnily enough when you are first starting out with Rust a common way to make the borrow checker happy is liberal use of .clone(). Eventually you grow out of it. But in a way this supports the authors point in that immutable copies is a somewhat natural if less performant style that the borrow checker pushes you towards.

                                                                    1. 1

                                                                      Interesting point!

                                                                  1. 5

                                                                    Betteridge’s law strikes again.

                                                                    One of the key features of a blockchain, which the author tries to handwave away, is that every link in the chain is verifiable, and unalterable. The author tries to claim that because each commit carries a reference to its parent, it’s a “chain of blocks”, but it’s not so much a chain as just an order. You can edit the history of a git repo easily, reparent, delete, squash, and perform many other operations that entirely modify the entire chain. It was kinda made that way.

                                                                    1. 12

                                                                      The technical properties of git’s and common block chain data structures are relatively similar.

                                                                      You can also fork a bitcoin block chain and pretend that your fork is the canonical one. The special bit about block chains is that there’s some mechanism for building agreement about the HEAD pointer. Among other things, there’s no designated mover of that pointer (as in a maintainer in a git-using project), but an algorithm that decides which among competing proposals to take.

                                                                      1. 16

                                                                        They are technically similar because both a blockchain and a git repo are examples of a merkle tree. As you point out though the real difference is in the consensus mechanism. Git’s consensus mechanism is purely social and mostly manual. Bitcoin’s consensus mechanism is proof of work and mostly automated.

                                                                        1. 2

                                                                          Please stop referring to “Proof of _” as a consensus mechanism. It is an anti-sybil mechanism, the consensus mechanism is called “longest chain” or “nakomoto consensus” - you can use a different anti-sybil mechanism with the same consensus mechanism (but you may lose some of the properties of bitcoin).

                                                                          The point is that there are various different combinations available of these two components and conflating them detracts from people’s ability to understand what is going on.

                                                                          1. 2

                                                                            You are right. I was mixing definitions there. Thanks for pointing it out. The main point still stands though. The primary distinction between a blockchain and git is the consensus mechanism and not the underlying merkle tree datastructure that they both share.

                                                                          2. 1

                                                                            Mandatory blockchain != bitcoin. Key industrial efforts listed in https://wiki.hyperledger.org/ are mostly not proof-of-work in any way (the proper term for this is permissioned blockchain, which is where industrial applications are going).

                                                                            1. 2

                                                                              You are correct. I don’t disagree at all. I used bitcoin as an example because it’s well known. There are lots of different blockchains with different types of consensus mechanisms.

                                                                        2. 2

                                                                          You can make a new history but it will always be distinct from the original one.

                                                                          I think what you’re really after is the fact that there is no one to witness that things like the author and the date of a commit are genuine – that is, it’s not just that I can edit the history, I can forge a history.

                                                                          1. 1

                                                                            what was all that hullabaloo about git moving away from SHA-1 due to vulnerabilities? why where they using a cryptographic hash function in the first place?

                                                                            what you said makes sense, but it seems to suggest this SHA-1 thing was a bit of bikeshedding or theater

                                                                            1. 2

                                                                              Git uses a cryptographic hash function because it wants to be able to assume that collisions never occur, and the cost of doing so isn’t too large. A collision was demonstrated in SHA-1 in 2017.

                                                                              1. 3

                                                                                SHA-1 still prevents accidental collisions. Was Git really designed to be robust against bad actors?

                                                                                1. 2

                                                                                  ¯_(ツ)_/¯

                                                                                  1. 1

                                                                                    The problem is that it was never properly defined what properties people expect from Git.

                                                                                    You can find pieces of the official Git documentation and public claims by Linus Torvalds that are seemingly in contradiction to each other. And the whole pgp signing part does not seem to be very well thought through.

                                                                                2. 2

                                                                                  Because you can sign git commits and hash collisions ruins that.

                                                                                  1. 1

                                                                                    ah that makes some sense

                                                                                3. 1

                                                                                  Technically you haven’t really made the others disappear. They are all still there just not easily viewed without using reflog. All you are really doing is creating a new branch point and moving the branch pointer to the head of that new branch when you do those operations. But to the average user it appears that you have edited history.

                                                                                1. 1

                                                                                  I self host a prometheus and grafana setup with a custom written ISP quality monitor tool I wrote in rust for fun. I also host an assortment of small personal projects in various states of completeness.

                                                                                  I’m considering moving my source code hosting in-house as well with Github just being a mirror but haven’t quite tackled that yet.

                                                                                  All on a personal FreeNAS instance

                                                                                  1. 20

                                                                                    Autotools solves problems I do not have. I’m sure there are people out there who do have them but I don’t know how many there are. While I agree that there is something to be said for nearly 30 yrs of hard won knowledge about portability and compilation if that knowledge is slowly growing obsolete with no clear or safe way of identifying the obsolete vs the still necessary then the total value of that knowledge is decreased. It’s like having a book where 30% of it is correct and the rest is not but you don’t know which 30% it is.

                                                                                    I’m honestly not sure it’s worth the effort to keep maintaining it. There are build systems out there with better underlying architecture and better guarantees for your build. It may be time for Autotools to be put on ice.

                                                                                    1. 16

                                                                                      This is a particular reputational problem for unusual language choices because failures of a team or product tend to be blamed on the weirdest things that a team does. When a Go project fails due to design issues, people rarely say “this failed because of Go” because there are so many examples of projects written in Go that did not fail. (On the flip side, design failure is also more likely because there are so many more examples of successful Go projects to follow than there are examples of successful Haskell projects.)

                                                                                      1. 5

                                                                                        On the flip side, design failure is also more likely because there are so many more examples of successful Go projects to follow than there are examples of successful Haskell projects.

                                                                                        I feel like this is a big part of it. Another issue is that often the developers choosing languages like Haskell are less likely to have used it at a large scale than people working with conventional languages. For many people this is because you don’t get the opportunity to work with a language at scale without using it at work.

                                                                                        1. 8

                                                                                          Indeed. And things work much differently at scale than they do in the small, the tradeoffs get different, and one has to develop a plan to compartmentalize ugliness and complexity.

                                                                                          1. 2

                                                                                            For an average project… I seriously doubt the “less likely to have used it at a large scale” part is true. Many teams don’t have a person on board who’d have practical experience using something at an actually large scale, much less designing for a large scale.

                                                                                            Just because there are more people with that experience in the general population, doesn’t mean a particular team has one of those. Also, many projects never reach a truly large scale, and with modern hardware, large is really quite large.

                                                                                            Of course, “using $language isn’t a stand-in for good software design”. But first, it’s true for any language, and second, there are already enough people trying to discourage everyone from moving beyond the current mainstream.

                                                                                            1. 3

                                                                                              I can’t tell you the number of Python shops that said the same thing, only to have their engineers perform expensive rewrites in more performant language runtimes. Twitter here is the poster child of doing a large-scale rewrite from Rails into Scala for performance reasons. It’s true that large scale does not happen to every company, but I also think optimizing solely for developer ergonomics will put you in a bind any time you do need to scale.

                                                                                              Of course, “using $language isn’t a stand-in for good software design”. But first, it’s true for any language, and second, there are already enough people trying to discourage everyone from moving beyond the current mainstream.

                                                                                              Who? So many Silicon Valley companies have seen success embracing new technological stacks (like Rails and Django when they were hot, to Scala when it became cool, Go when it came onto the scene, and now Rust), so I can’t see why a new company would not want to use a new language if their devs were comfortable in it and confident in its ability to scale to the market they wished to address.

                                                                                              the current mainstream

                                                                                              There’s wisdom in accepting tried-and-true solutions. I’ve dug into new language libraries only to find undocumented settings or badly chosen constants more times than I can count. Edge cases in LRU caches, bugs in returning connections back into connection pools, circuit breakers that will oscillate between on-and-off when degenerate conditions are hit, these are all examples of algorithms that were incorrectly implemented, and it’s not always clear if the value proposition of a new language overcomes the complexity of implementing tried-and-true algorithms. Choosing an “unproven” language is not a light decision, especially when several engineers will spend their time and money on making it work.

                                                                                              1. 3

                                                                                                Most places aren’t Silicon Valley. Most companies aren’t startups. Most projects aren’t web services with open registration.

                                                                                                And it’s not about optimizing for developer ergonomics. Advanced type systems are about correctness first of all. It’s just sort of taking off with Scala, Swift, and Rust. Sort of. Those clearly still face more opposition than 70’s designs like Python, Ruby, or Go.

                                                                                                1. 5

                                                                                                  Most places aren’t Silicon Valley. Most companies aren’t startups. Most projects aren’t web services with open registration.

                                                                                                  According to StackOverflow [1], Typescript is a top 10 language, and Rust, Swift, and Typescript are top 20 languages. So while there is a bias for tried-and-true languages like Python and Java, languages with “advanced” type systems are in the top 20.

                                                                                                  Advanced type systems are about correctness first of all

                                                                                                  Is it? I’m not sure if there’s any proof that advanced type systems actually lead to more correct programs. I certainly don’t remember that when I was in grad school for PLT either. I understand that practitioners feel that type systems allow them to make more correct programs with less cognitive load, but I would bin that into developer ergonomics and not correctness until there’s proof positive that ML-based type systems do actually result in more correct programs.

                                                                                                  It’s just sort of taking off with Scala, Swift, and Rust. Sort of. Those clearly still face more opposition than 70’s designs like Python, Ruby, or Go

                                                                                                  Or, perhaps, most programmers genuinely do not enjoy working with heavily restrictive type systems or with monad transformer stacks and free monads. Haskell is a year older than Python and is an order of magnitude less widely used than Python. It’s clear that recent languages like Rust, Swift, Typescript, and Scala do use a lot of ideas from ML-descended type systems but it’s not clear whether they are popular as a whole or not. Many languages have taken pieces of these type systems (such as the Optional/Maybe sum type) and bolted them onto their language (Kotlin), but I think that ML-style languages will always be enjoyed by a minority, though how large that minority is time will tell.

                                                                                                  1: https://insights.stackoverflow.com/survey/2020#technology-programming-scripting-and-markup-languages-all-respondents

                                                                                                  1. 2

                                                                                                    I understand that practitioners feel that type systems allow them to make more correct programs with less cognitive load, but I would bin that into developer ergonomics and not correctness until there’s proof positive that ML-based type systems do actually result in more correct programs.

                                                                                                    This is an accurate statement I think. No type system can guarantee you are more correct unless the “spec” you are implementing is more correct. What it does help with is giving the developer more information on where they are starting to deviate from the spec. But even then it doesn’t guarantee that they are implementing the spec correctly in the type system. ML Type systems give you early information that your code won’t work as implemented when it hits certain edge cases. Those edge cases will sometime but not always manifest as bugs in production. So depending on the ergonomics you favor in your programming language you will either love that early information, be ambivalent, or you will dislike it.

                                                                                        1. 14

                                                                                          This stack will also make supporting the following features difficult:

                                                                                          • Paging
                                                                                          • Site Navigation
                                                                                            • By category
                                                                                            • By tag
                                                                                          • Common Elements
                                                                                          1. 12

                                                                                            It’s so annoying that (from our current perspective) this could have been so easily done in the <head> tag, by adding a few <meta ...> pointers to the next/previous page or something <neighbours>, but instead the only people who use anything like this are search engines.

                                                                                            1. 4

                                                                                              Why do you need paging? You’d need thousands of pages before your site exceeds the size of e.g. NYTimes which amounts to tens of megabytes.

                                                                                              1. 4

                                                                                                Maybe you don’t. But you don’t need to scale to thousands of pages before paging is useful.

                                                                                                • 5 most recent stories? Needs paging.
                                                                                                • List of articles about topic X? Needs paging.
                                                                                                • Blog posts in the month of June? Needs paging.

                                                                                                Not everyone needs those but for the ones that do? Plain HTML/CSS is going to suck and be really error prone.

                                                                                                1. 1

                                                                                                  wait what even is paging in this context

                                                                                                  1. 2

                                                                                                    I suppose a more fitting word for that concept is ‘pagination’.

                                                                                                    1. 1

                                                                                                      well the things in /u/zaphar’s comment don’t seem to be about that… why do you need pagination to list the 5 most recent stories?

                                                                                                      1. 2

                                                                                                        Because that list would otherwise need to be updated manually.

                                                                                                        1. 1

                                                                                                          how would pagination help with that? a list of 5 stories probably fits on a single page anyway.

                                                                                                          1. 1

                                                                                                            It’s the same process. The top-five-most-recent stories is just a page with 5 stories on it.

                                                                                                            1. 1

                                                                                                              pagination is the process of dividing a document into pages.

                                                                                                              maybe /u/zaphar was using “paging” to refer to any dynamic content; I am not familiar with that usage.

                                                                                                              1. 1

                                                                                                                The “document” is the entire list of stories. The page is the first five.

                                                                                                                1. 1

                                                                                                                  this is extremely dumb

                                                                                              2. 1

                                                                                                Yes, but he also doesn’t need to maintain things that he doesn’t support, and that works in his favor.

                                                                                                1. 8

                                                                                                  I really really want 2021 to be the year of Nix.

                                                                                                  1. 8

                                                                                                    See also: “backlash against Kubernetes”

                                                                                                    1. 5

                                                                                                      I struggle to see how building infrastructure on top of a library with howmanythousandslinesofcode of some Standard ML derivative and bash constitutes simplicity in orchestration.

                                                                                                      NixOS is neat, but it’s not simple.

                                                                                                      1. 10

                                                                                                        Hot take: I’d prefer building systemd units that automatically get managed over orchestrating k8s pods any day. Nothing is simple, but Nix provides another approach to managing complexity well.

                                                                                                        1. 4

                                                                                                          I had an idea to rebuild what was once CoreOS’s fleetctl. Orchestration built on top of systemd, without anything more fancy on top.

                                                                                                          1. 2

                                                                                                            I think people generally complain about systemd, among many reasons, because distros started with /etc/init.d, picked up systemd, and started using it in the same way. So “why do we need all this crap” makes sense when all the distro uses is the init system features. But systemd, for better or worse, is really a daemon manager, and the argument for systemd vs. sysvinit with NixOS or a tool like that is much stronger.

                                                                                                        2. 1

                                                                                                          Not sure how can be something too simple. A tool either solves the problem or it doesnt.

                                                                                                        3. 4

                                                                                                          And a blow against Ansible, Terraform, Salt etc. for free!

                                                                                                          1. 1

                                                                                                            I’m a Nix n00b, but I don’t understand the comparison between Nix/NixOS and Kubernetes. To me, k8s = “distributed” vs Nix = “single host”. Did I miss something?

                                                                                                            1. 1

                                                                                                              NixOps and similar let you do a form of distributed Terraform-style configuration. It’s not automatic scheduling like Kubes, but gets you part of the way there at least. Scheduling would be cool, though. If NixOps could do it, it would beat Kubernetes at its own game.

                                                                                                              And that’s why Nix is a technology to watch. I think its current adoption is partially a backlash against Docker being inefficient - you can generate Docker images with docker-tools in nixpkgs now without ever installing Docker. The principles behind the tech are solid - per-application install prefixes are widely used on clusters, Nix takes it to the next level and uses them as part of the dataflow of building new derivations. In 2021, I bet we’ll see the tools and documentation start to mature in a way that will ultimately set the stage for it being able to do more of what Kubernetes does, but without YAML. (Seriously. I’d rather write Nix than YAML.)

                                                                                                              And Nix has done all this without Docker-style growth hacking or Kubernetes-style aggressive advertising, but nixpkgs is now creeping up on FreeBSD Ports in number of maintainers. The ecosystem has a pretty bright future, IMO.

                                                                                                          2. 1

                                                                                                            Flakes and the cli update will finally make it make sense.

                                                                                                          1. 19

                                                                                                            I believe pretty much everything that LibreSSL originally was (consistent codingstyle, cleanup of obsolete/dead code etc.) has happened in OpenSSL these days. It’s more that there’s some myth around LibreSSL from these early days (where the people behind it raised back then valid criticism about OpenSSL) than any real value.

                                                                                                            This prompted me to check if OpenSSL still had support code for big endian x86_64. They do.

                                                                                                            #   elif defined(B_ENDIAN)
                                                                                                                   /*
                                                                                                                    * Most will argue that x86_64 is always little-endian. Well, yes, but
                                                                                                                    * then we have stratus.com who has modified gcc to "emulate"
                                                                                                                    * big-endian on x86. Is there evidence that they [or somebody else]
                                                                                                                    * won't do same for x86_64? Naturally no. And this line is waiting
                                                                                                                    * ready for that brave soul:-)
                                                                                                                    */
                                                                                                            
                                                                                                            1. 3

                                                                                                              That comment is begging for someone to scream YAGNI as a response.

                                                                                                              1. 1

                                                                                                                YMNISD

                                                                                                                (You might need it someday…)

                                                                                                            1. 23

                                                                                                              Part I starts with a faulty premise. This means that our explanations might not fulfill the explainable-AI requirements. Why? Because the discovery of the Higgs boson was made by theories about why particles have mass having various implications, and those implications being followed through with formal logic. In this arena of reasoning, we are not doing statistical analysis on lots of pseudo-continuous data, but instead our inputs are proof statements.

                                                                                                              If I had to explain to a lay reporter why we expected the Higgs boson, I would start my attempt with something like, “Electricity and magnetism are two ways to look at the same single happenstance in physics, where negatively- and positively-charged objects interact with each other. When we look at particle physics, we see a way to combine the behaviors we see, with atoms splitting and joining, with electromagnetism. This helps us build our models. But imagine that we switched ‘positive’ and ‘negative’. Look up the history with Ben Franklin, it’s a great story. The point is that those labels could have been switched, and that gives us a sort of symmetry for particles. The Higgs boson is a particle that should exist according to what we know about particle symmetries, and we call it ‘Higgs’ after one of the people who first noticed that it should exist.”

                                                                                                              Crucially, this explanation uses a real example as an analogy to bootstrap intuition about the unknown. Rather than handwaving, poisoning the well, or appealing to authority; the explanation lays out a context, including specific symbols (keywords) which can link to further details. The explanation does not rely on a vague statistical argument made using many weak indicators, but uses one main concept, symmetry, as its focus.

                                                                                                              Now, having said all this, I strongly suspect that the author might reasonably reply that the question they wanted to ask was more like, “what pattern in experimental data prompted the theoretical work which led to the proposal of the Higgs mechanism?” This does sound like something that could be answered with a data-driven correlation. And, indeed, that is what happened; the models of that time were faulty and predicted that certain massive particles should be massless. But the actual statistical techniques that were used were the standard ones; the explanation could begin and end with a t-test.

                                                                                                              All of this context is necessary to understand what will happen to poor Steve. Historically, Steve’s last name might be the most important input to the algorithm, or possibly their ethnic background if the algorithm can get hold of it more directly. And Steve’s inability to participate in society is explained away by the reassurance that there are millions of parameters which might have influenced the decision. This is exactly the sort of situation that explainable-AI proponents are trying to avoid.

                                                                                                              But in both cases, the reasoning is not simple, there’s no single data point that is crucial, if even a few inputs were to change slightly the outcome might be completely different, but the input space is so fast it’s impossible to reason about all significant changes to it.

                                                                                                              I don’t agree with this. Specifically, I don’t think that those millions of parameters are actually used much. Instead, I think that NNAEPR and there are only a handful of parameters which account for almost all of the variance in loan amounts, and that the error of the remaining parameters is subject to roundoff. Similarly, only one measurement, mass, needed to be wrong to provoke the development of the Higgs mechanism in theory.

                                                                                                              The explanation in part III is not a valid proof, because correlation is not transitive. I do appreciate the exploration here into epistemology and the nature of justification. But I can’t ignore the fact that the maths are incorrect; if an AI can generate English which successfully bullshits people, then is it really explaining or just lying? In a hypothetical world where AIs have civil rights, we would expect AI explanations to be just as cross-examinable as human explanations, and thus to stand up under scrutiny. What difference is there between an opaque AI loan officer and an opaque human loan officer?

                                                                                                              As we have explored here before, we must be extremely skeptical of the argument that it is simply too hard to explain ourselves to others, in the context of the immense social harm which results from being judged by opaque corporations. Specifically, when they claim that they cannot be legible to outsiders, they are trying to find ways to be less responsible for their own actions; be assured that the corporation is legible to itself.

                                                                                                              1. 10

                                                                                                                we must be extremely skeptical of the argument that it is simply too hard to explain ourselves to others, in the context of the immense social harm which results from being judged by opaque corporations

                                                                                                                Just want to say that I think this is a really thoughtful and true thing, beyond the rest of your commentary. Ultimately the worth of these tools, surely, must be measured in how beneficial they are to society.

                                                                                                                If a neural net loan officer saves society a few tens of of thousands human-labor-hours a year, subsequently making loans slightly cheaper and more profitable, that’s good. But if they do that while also making it impossible to answer the question “why was this loan denied”, then well, the net effect is that you made the world worse and more open to exploitation and your approach should be curtailed.

                                                                                                                1. 5

                                                                                                                  Back in 1972 John Kemeny (co-developer of BASIC) was warning about uninterrogable decision-making (in Man and the Computer):

                                                                                                                  I have heard a story about the design of a new freeway in the City of Los Angeles. At an open hearing a number of voters complained bitterly that the freeway would go right through the midst of a part of the city heavily populated by blacks and would destroy the spirit of community that they had slowly and painfully built up. The voters’ arguments were defeated by the simple statement that, according to an excellent computer, the proposed route was the best possible one. Apparently none of them knew enough to ask how the computer had been instructed to evaluate the variety of possible routes. Was it asked only to consider costs of building and acquisition of property (in which case it would have found routing through a ghetto area highly advantageous), or was it asked to take into account the amount of human suffering that a given route would cause? Perhaps the voters would even have agreed that it is not possible to measure human suffering in terms of dollars. But if we omit considering of human suffering, then we are equating its cost to zero, which is certainly the worst of all procedures!

                                                                                                                  (This message brought to you by the Campaign for the Rehabilitation of BASIC.)

                                                                                                                  1. 3

                                                                                                                    You raise an important point about model interpretability. All models that predict the future by training on historical data propagate historical bias. This is an effect, not a side-effect.

                                                                                                                    A simple example can be found in natural language processing, where words become numbers to be usable as model features. With a model trained on a corpus of human-written documents, you’ll be able to “subtract” the word “woman” from the word “king” to get the result of “queen” and think yourself quite clever. Then, you’ll subtract the word “woman” from the word “doctor” and find yourself uncomfortable to discover the result is “nurse”.

                                                                                                                    An additional example drawing from the above comment: if it is illegal and unethical to deny a loan on the basis of race, but you build an opaque model to predict loan outcome that (under the hood) incorporates e.g. census block as a feature, you will still have built a redlining AI that reinforces historical racial segregation.

                                                                                                                  2. 1

                                                                                                                    I don’t agree with this. Specifically, I don’t think that those millions of parameters are actually used much. Instead, I think that NNAEPR and there are only a handful of parameters which account for almost all of the variance in loan amounts, and that the error of the remaining parameters is subject to roundoff. Similarly, only one measurement, mass, needed to be wrong to provoke the development of the Higgs mechanism in theory.

                                                                                                                    Okay, How do you explain why you believe the parameters necessary are smaller? If you want to counter his argument based on the maths being wrong you have to explain why you think the maths are wrong. And that in some sense is playing straight into his argument.

                                                                                                                    1. 4

                                                                                                                      I strongly suggest that you spend some time with the linked paper. From a feature-based POV, polynomial regression directly highlights the relatively few robust features which exist in a dataset. Neural nets don’t do anything desirable on top of it; indeed, they are predicted and shown to have a sort of collinearity which indicates redundancy in their reasoning and can highlight spurious features rather than the robust features which we presumably desire.

                                                                                                                      Even leaving that alone, we can use the idea of entropy and surprise to double-check the argument. It would be extremely surprising if variance in Steve’s last name caused variance in Steve’s loan qualifications, given the expectation that loan officers do not discriminate based on name. Similarly, it would be extremely surprising if variance in Steve’s salary did not cause variance in Steve’s loan qualifications, given the expectation that salaries are correlated with ability to pay back loans. This gives us the ability to compare an AI loan officer with a human loan officer.

                                                                                                                    2. 1

                                                                                                                      This means that our explanations might not fulfill the explainable-AI requirements. Why? Because the discovery of the Higgs boson was made by theories about why particles have mass having various implications, and those implications being followed through with formal logic. In this arena of reasoning, we are not doing statistical analysis on lots of pseudo-continuous data, but instead our inputs are proof statements.

                                                                                                                      If I had to explain to a lay reporter why we expected the Higgs boson, I would start my attempt with something like, “Electricity and magnetism are two ways to look at the same single happenstance in physics, where negatively- and positively-charged objects interact with each other. When we look at particle physics, we see a way to combine the behaviors we see, with atoms splitting and joining, with electromagnetism. This helps us build our models. But imagine that we switched ‘positive’ and ‘negative’. Look up the history with Ben Franklin, it’s a great story. The point is that those labels could have been switched, and that gives us a sort of symmetry for particles. The Higgs boson is a particle that should exist according to what we know about particle symmetries, and we call it ‘Higgs’ after one of the people who first noticed that it should exist.”

                                                                                                                      I think we might have different intuitions for what “explanation” means here.

                                                                                                                      The above is the kind of news-conference explanation that is not at all satisfying. It’s a just-so story, not something you’d want from, e.g., an emergency system controlling the launch of nuclear missiles… or even from an organ transplant model that decides who the most likely to benefit patients are.

                                                                                                                      Maybe, if you actually know physics yourself, try to answer a question like:

                                                                                                                      “Why is the mass of the higgs boson not 125.18 ± 0.15 GeV/c^2 instead of 125.18 ± 0.16 (as per Wikipedia) ?”

                                                                                                                      or

                                                                                                                      “What would it take for the mass to be 125.18 ± 0.15 ? How would the theory or experiments have to differ ?”

                                                                                                                      Those are the kind of explanations that, I presume, a physicist working on the Higgs boson could give (not 100% sure how accessible they are, maybe anyone with a physics PhD could given a bit of digging). But the likelihood of me understand them is small, you can’t make a “just-so story” to explain those kinds of details.

                                                                                                                      Yet ultimately it’s the details that matter, the metaphysics are important but explaining those does not give a full picture…. they are more like intuition pumps to begin learning from.

                                                                                                                      I don’t agree with this. Specifically, I don’t think that those millions of parameters are actually used much. Instead, I think that NNAEPR and there are only a handful of parameters that account for almost all of the variance in loan amounts, and that the error of the remaining parameters is subject to roundoff. Similarly, only one measurement, mass, needed to be wrong to provoke the development of the Higgs mechanism in theory.

                                                                                                                      This I actually agree with, and my last 4 months of leisure time research have basically been spent on reading up on parameter pruning and models that use equations that are quick to forward prop but take long to fit (quick to forward prop is often ~= easy to understand, but I prefer to focus on it since it has industry application).

                                                                                                                      That being said it’s not all clear to me that this is the case, it could be the case in some scenarios but (see III.2 and IV in the article) it’s probably an NP-hard problem to distinguish those scenarios. And maybe my intuition that most models are over-parameterized and using backprop-optimized operations is just wishful thinking.

                                                                                                                      Again, not to say our intuitions differ here, but I tried to go against my intuitions when writing this article, as state in it.

                                                                                                                    1. 5

                                                                                                                      An old piece of mine that I recently rediscovered. This was back before I got deep into history and formal methods and was mostly writing silly Python satires. Apologies for the medium link; it’s also before I realized writing on my own blog was better. I should probably move it over :/

                                                                                                                      Enjoy!

                                                                                                                      1. 4

                                                                                                                        I was literally giggling while reading this. Well done!

                                                                                                                        1. 3

                                                                                                                          It reads like an AI taught to write medium posts. I sense a new technologically generated mode of satire on the horizon.

                                                                                                                        1. 1

                                                                                                                          The static compile could be a pretty massively useful feature. It simplifies so much of the typescript application deployment.

                                                                                                                          1. 1

                                                                                                                            Errors are a part of your API. If you don’t treat them like a first class part of the API however then you will have poor error types and messages as a result. It’s one reason why I kind of liked Java’s checked exceptions, Go’s err as a return type and Rust’s Result type. They allow you to clearly document the errors that are part of your API and give the consumer the tools they need to interoperate with your errors. It’s still on you to design your errors though if you want people to properly consume them.

                                                                                                                            1. 3

                                                                                                                              All the examples were things I literally never had an issue with, so personally going for “less powerful” over “more powerful” isn’t something I got convinced in doing.

                                                                                                                              As long as it’s statically typed and the language is my friend, not my nemesis, I’ll pick that.

                                                                                                                              1. 4

                                                                                                                                A dynamically typed language is more powerful than a statically typed language. You can do things in them that the statically typed language won’t allow you to do. It seems like you have indeed made the choice to restrict the power of your languages in order the gain the benefits of the more restrictive type systems of a statically typed language.

                                                                                                                                1. 5

                                                                                                                                  A dynamically typed language is more powerful than a statically typed language. You can do things in them that the statically typed language won’t allow you to do

                                                                                                                                  Indeed, like failing at runtime due to a type mismatch like a string getting in where you expected a number.

                                                                                                                                  1. 1

                                                                                                                                    I could gain the benefits of an open sawblade but decided that I preferred one full of protections, I lost in freedom but also will save myself most of the pain.

                                                                                                                                1. 3

                                                                                                                                  I use an IDE to browse and Evil-Mode Emacs to edit. Why? Because IDE’s are better at browsing code. But Vim or Evil Mode blow everything else out of the water when it comes to the plain and simple task of editing text which most code is.

                                                                                                                                  1. 1

                                                                                                                                    Do you use code completion?

                                                                                                                                    It seems like there are three main problems.

                                                                                                                                    • Browsing code
                                                                                                                                    • Editing text
                                                                                                                                    • Listing available completions (syntactically and grammatically correct)

                                                                                                                                    You know instead of having autocomplete build the perfect thing in advance, have autocorrect fix the buggy code or prompt for else block or missing match arms. I guess the third one is a service that offers patches against code you haven’t written and code that you have and then another system that applies those patches. God I hate code

                                                                                                                                    1. 3

                                                                                                                                      Honestly, I really don’t use it. I find that I remember the details of the APIs I use better if I don’t. Doing a code review gets a lot easier if you remember the APIs.

                                                                                                                                  1. 7

                                                                                                                                    Without any changes, I’m scoring quite okay : https://www.websitecarbon.com/website/raymii-org/

                                                                                                                                    • Hurrah! This web page is cleaner than 94% of web pages tested
                                                                                                                                    • Only 0.06g of CO2 is produced every time someone visits this web page.
                                                                                                                                    • Over a year, with 10,000 monthly page views, this web page produces
                                                                                                                                    • 7.39kg of CO2 equivalent. The same weight as 0.05 sumo wrestlers and as much CO2 as boiling water for 1,001 cups of tea
                                                                                                                                    • 1 tree This web page emits the amount of carbon that 1 tree absorbs in a year.
                                                                                                                                    • 16kWh of energy That’s enough electricity to drive an electric car 100km.
                                                                                                                                    1. 7

                                                                                                                                      I just redesigned my site to use less resources, including images. Beat you by 4 percentage points :-)

                                                                                                                                      1. 4

                                                                                                                                        I redesigned mine quite a while back and while I wish I could say I beat both of you the best I can do is as tie between @johnaj and I. https://www.websitecarbon.com/website/jeremy-marzhillstudios-com/

                                                                                                                                        1. 2

                                                                                                                                          I also redesigned mine awhile back in the name of speed and simplicity. I guess that’s good for the environment because apparently I beat everyone ever: https://www.websitecarbon.com/website/cosmo-red/

                                                                                                                                          1. 4

                                                                                                                                            Haha that’s cool, I wonder what type of car that is, I want one:

                                                                                                                                            • 0kWh of energy
                                                                                                                                            • That’s enough electricity to drive an electric car 2km.
                                                                                                                                            1. 3

                                                                                                                                              Maybe it’s got a sail? It’s a sailcar!

                                                                                                                                            2. 3

                                                                                                                                              My results tell me my car would move 1km further, but I’m using sustainable energy, so who wins that?

                                                                                                                                              1. 2

                                                                                                                                                Heh, I vote for you. Where do you host your website?

                                                                                                                                                1. 3

                                                                                                                                                  Strato, but mainly because I get 200GB for 5 Euro a month.

                                                                                                                                        2. 4

                                                                                                                                          I have a small personal, statically-generated blog which managed to score 99% by emitting 0.00g.

                                                                                                                                          https://www.websitecarbon.com/website/danso.ca

                                                                                                                                          Which makes me wonder about the methodology, obviously.

                                                                                                                                          A weakness of this method is that it does not account for the cost of building the website from the markdown and Haskell source files. Compiling a program is not free, nor is running it. But maybe that cost is negligible compared to serving the website after it’s made?

                                                                                                                                        1. 18

                                                                                                                                          The cited law doesn’t seem like it’s about combating abuse. It seems like it’s about providing a way to obtain evidence for prosecution. This article seems to fail to address the particular concerns the law is addressing.

                                                                                                                                          I believe backdoors are fundamentally unworkable and wrong. But the arguments against it need to do better than this.

                                                                                                                                          1. 10

                                                                                                                                            The point of the article is to prove that there is no viable solution to the particular concern the proposed law is suggesting (see also https://lobste.rs/s/ntyvtw/combating_abuse_matrix_without#c_52uuvc) and proposes an alternative which could help mitigate abuse across the board if embraced properly.

                                                                                                                                            But the arguments against it need to do better than this.

                                                                                                                                            go on then, we’re all ears :)

                                                                                                                                            1. 9

                                                                                                                                              I think where these arguments seem to go off the rails is in two ways.

                                                                                                                                              • They treat the backdoor proposals as a black and white proposition.
                                                                                                                                              • They propose solutions to a different problem than the proponents are trying to solve.

                                                                                                                                              If I had the audience to effect change here I would be focusing my arguments instead on the tradeoffs and I would be honest about them. We can’t allow shining a light on bad behavior like child exploitation, hate groups, and terrorism without sacrificing privacy/security in the long term. What we are actually talking about are tradeoffs that society has to make.

                                                                                                                                              To be clear the authorities actually don’t care about the 1/1 gpg encrypted communication between bad parties here. For them that’s a straw man argument. It’s such a small skillset that by definition it’s a smaller problem. The danger in there eyes is the mass enablement of more people to hide their bad behavior and the accompanying fear that the ability to hide results in an explosion of bad behavior because it’s safer and easier to hide it.

                                                                                                                                              When we refuse to engage them on that point we lose the argument. One small part of the argument is touched on in your article. The cat is out of the bag for something like Matrix. You only need one competent administrator in a different juristiction to provide a way to hide for other.

                                                                                                                                              But the other part of the argument that seems to be shied away from is that we regard the risk of to society of a backdoor as greater than the risk to children that good privacy and encryption poses. But unless you convince the rest of society that this is the case and get them to own it you will lose the argument.

                                                                                                                                              1. 9

                                                                                                                                                They treat the backdoor proposals as a black and white proposition

                                                                                                                                                That’s because they are a black and white proposition. There’s no such thing as half a back door. And “ability to decrypt anything on command and we promise to have a good reason” is a back door, no matter how many times politicians pretend otherwise.

                                                                                                                                                1. 2

                                                                                                                                                  I think you misunderstand what I mean by black and white. Ubiquitious encryption is not an unambiguous good in many peoples eyes. It’s a tradeoff.

                                                                                                                                                2. 5

                                                                                                                                                  Governments care about the ability to spy on their citizens. They are hiding this agenda behind the more politically potent cause of preventing child abuse.

                                                                                                                                                  1. 6

                                                                                                                                                    But you shouldn’t be trying to convince the government. The people you need to convince are the governed. The government are on one side of the argument and you are the other.