1. 18

    What this rant does not focus on: It’s a good thing that these usecases are broken. Wayland prohibits your desktop applications from capturing keystrokes or recording other apps’ screens by default. X’s security model (and low level graphics APIs) is/are severely outdated, and Wayland promises not only to be more secure, but also expose cleaner APIs at the lower level (rendering, etc.)

    These usecases are/will still be supported though, but this time via standardized interfaces, many of which already exist and are implemented in today’s clients.

    X is based on a 30 year old code base and an outdated model (who runs server-side display servers these days?). Of course, switching from X to Wayland will break applications, and until they are rewritten with proper Wayland support they will stay that way. For most X11 apps there even is Xwayland, which allows you to run X11 apps in Wayland if you must.

    1. 27

      What this rant does not focus on: It’s a good thing that these usecases are broken

      You should have more compassion for users and developers who have applications that have worked for decades, are fully featured, and are being asked to throw all of that away. For replacements that are generally very subpar. With no roadmap when party will be reached. For a system that does not offer any improvements they care about (you may care about this form of security, not everyone does).

      I could care less about whether when I run ps I see Xorg or wayland. And I doubt that most of the people who are complaining really care about x vs wayland. They just don’t want their entire world broken for what looks to them like no reason at all.

      1. 5

        I’m not saying that those apps should be thrown away immediately. Some of these work under XWayland (I sometimes stream using OBS and it records games just fine).

        If your application really does not run under XWayland, then run an X server! X is not going to go away tomorrow, rather it is being gradually replaced.

        I’m simply explaining that there are good reasons some applications don’t work on Wayland. I’m a bit sore of hearing “I switched to Wayland and everything broke” posts: Look behind the curtain and understand why they broke.

      2. 17

        I’m kind of torn on the issue.

        On the one hand, the X security model is clearly broken. Like the UNIX security model, it assumes that every single application the user wants to run is 100% trusted. It’s good that Wayland allows for sandboxing, and “supporting the use cases, but this time via standardized interfaces” which allow for a permission system sounds good.

        On the other hand, there’s clearly no fucking collaboration between GNOME and the rest of the Wayland ecosystem. There’s a very clear rift between the GNOME approach which uses dbus for everything and the everything-else approach which builds wayland protocol extensions for everything. There doesn’t seem to be any collaboration, and as a result, application authors have to choose between supporting only GNOME, supporting everything other than GNOME, or doing twice the work.

        GNOME also has no intention of ever supporting applications which can’t draw their own decorations. I’m not opposed to the idea of client-side decorations, they’re nice enough in GTK applications, but it’s ridiculous to force all the smaller graphics libraries which just exist to get a window on the screen with a GL context - like SDL, GLFW, GLUT, Allegro, SFML, etc - to basically reimplement GTK just to show decorations on GNOME on Wayland. The proposed solution is libdecorations, but that seems to be at least a decade away from providing a good, native-feeling experience.

        This isn’t a hate post. I like Wayland and use Sway every day on my laptop. I like GNOME and use it every day on my desktop (though with X because nvidia). I have written a lot of wayland-specific software for wlroots-based compositors. But there’s a very clear rift in the wayland ecosystem which I’m not sure if we’ll ever solve. Just in my own projects, I use the layer-shell protocol, which is a use-case GNOME probably won’t ever support, and the screencopy protocol, which GNOME doesn’t support but provides an incompatible dbus-based alternative to. I’m also working on a game which uses SDL, which won’t properly support GNOME on Wayland due to the decorations situation.

        1. 13

          the X security model is clearly broken

          To be honest I feel the “brokenness” of the security model is vastly overstated. How many actual exploits have been found with this?

          Keyloggers are a thing, but it’s not like Wayland really prevents that. If I have a malicious application then I can probably override firefox to launch something that you didn’t intend (via shell alias, desktop files) or use some other side-channel like installing an extension in ~/.mozilla/firefox, malicious code in ~/.bashrc to capture ssh passwords, etc. Only if you sandbox the entire application is it useful, and almost no one does that.

          1. 10

            This isn’t a security vulnerability which can be “exploited”, it’s just a weird threat model. Every single time a user runs a program and it does something to their system which they didn’t want, that’s the security model being “exploited”.

            You might argue that users should never run untrusted programs, but I think that’s unfair. I run untrusted programs; I play games, those games exist in the shape of closed-source programs from corporations I have no reason to trust. Ideally, I should be able to know that due to the technical design of the system, those closed source programs can’t listen to me through my microphone, can’t see me through my webcam, can’t read my keyboard inputs to other windows, and can’t see the content in other windows, and can’t rummage through my filesystem, without my expressed permission. That simply requires a different security model than what X and the traditional UNIX model does.

            Obviously Wayland isn’t enough on its own, for the reasons you cite. A complete solution does require sandboxing the entire application, including limiting what parts of the filesystem it can access, which daemons it can talk to, and what hardware it can access. But that’s exactly what Flatpak and Snaps attempts to do, and we can imagine sandboxing programs like Steam as well to sandbox all the closed source games. However, all those efforts are impossible as long as we stick with X11.

            1. 3

              Every single time a user runs a program and it does something to their system which they didn’t want, that’s the security model being “exploited”.

              If you think a permission system is going to solve that, I going to wish you good luck with that.

              Ideally, I should be able to know that due to the technical design of the system, those closed source programs can’t listen to me through my microphone, can’t see me through my webcam, can’t read my keyboard inputs to other windows, and can’t see the content in other windows, and can’t rummage through my filesystem, without my expressed permission.

              Ah yes, and those closed-source companies will care about this … why exactly?

              They will just ask for every permission and won’t run otherwise, leaving you just as insecure as before.

              But hey, at least you made the life of “trustworthy” applications worse. Good job!

              But that’s exactly what Flatpak and Snaps attempts to do […]

              Yes, letting software vendors circumvent whatever little amount of scrutiny software packagers add, that will surely improve security!

              1. 7

                If you think a permission system is going to solve that, I going to wish you good luck with that.

                It… will though. It’s not perfect, but it will prevent software from doing things without the consent of the user. That’s the goal, right?

                You may be right that some proprietary software vendors will just ask for every permission and refuse to launch unless given those permissions. Good. That lets me decide between using a piece of software with the knowledge that it’ll basically be malware, or not using that piece of software.

                In reality though, we don’t see a lot of software which takes this route from other platforms which already have permission systems. I’m not sure I have ever encountered a website, Android app or iOS app which A) asked for permissions to do stuff it obviously didn’t need, B) refused to run unless given those permissions, and C) wasn’t obviously garbage.

                What we do see though is that most apps on the iOS App Store and websites on the web, include analytics packages which will gather as much info on you as possible and send it back home as telemetry data. When Apple, for example, put the contacts database behind a permission wall, the effect wasn’t that every app suddenly started asking to see your contacts. The effect was that apps stopped snooping on users’ contacts.

                I won’t pretend that a capability/permission system is perfect, because it isn’t. But in the cases where it has already been implemented, the result clearly seems to be improved privacy. I would personally love to be asked for permission if a game tried to read through my ~/.ssh, access my webcam or record my screen, even if just to uninstall the game and get a refund.

                Yes, letting software vendors circumvent whatever little amount of scrutiny software packagers add, that will surely improve security!

                I mean, if you wanna complain about distros which use snaps and flatpaks for FOSS software, go right ahead. I’m not a huge fan of that myself. I’m talking about this from the perspective of running closed source software or software otherwise not in the repos, where there’s already no scrutiny from software packagers.

                1. 3

                  There’s probably evidence from existing app stores on whether users prefer to use software that asks for fewer permissions. There certainly seems to be a market for that (witness all the people moving to Signal).

                  1. 3

                    But hey, at least you made the life of “trustworthy” applications worse. Good job!

                    “Trustworthy software” is mostly a lie. Every application is untrustworthy after it gets remotely exploited via a security bug, and they all have security bugs. If we lived in a world without so much memory-unsafe C, then maybe that wouldn’t be true. But we don’t live in that world so it’s moot.

                    Mozilla has its faults, but I trust them enough to trust that Firefox won’t turn on my webcam and start phoning home with the images. I could even look at the source code if I wanted. But I’d still like Firefox sandboxed away from my webcam because Firefox has memory bugs all the time, and they’re probably exploitable. (As does every other browser, of course, but I trust those even less.)

                  2. 1

                    A complete solution does require sandboxing the entire application, including limiting what parts of the filesystem it can access, which daemons it can talk to, and what hardware it can access. But that’s exactly what Flatpak and Snaps attempts to do

                    But that’s quite limited sandboxing, I think? To be honest I’m not fully up-to-speed with what they’re doing exactly, but there’s a big UX conundrum here because write access to $HOME allows side-channels, but you also really want your applications to do $useful_stuff, which almost always means accessing much (or all of) $HOME.

                    Attempts to limit this go back a long way (e.g. SELinux), and while this works fairly well for server applications, for desktop applications it’s a lot harder. I don’t really fancy frobbing with my config just to save/access a file to a non-standard directory, and for non-technical users this is even more of an issue.

                    So essentially I don’t really disagree with:

                    I should be able to know that due to the technical design of the system, those closed source programs can’t listen to me through my microphone, can’t see me through my webcam, can’t read my keyboard inputs to other windows, and can’t see the content in other windows, and can’t rummage through my filesystem, without my expressed permission. That simply requires a different security model than what X and the traditional UNIX model does.

                    and I’m not saying that the Wayland model isn’t better in theory (aside from some pragmatical implementation problems, which should not be so casually dismissed as some do IMHO), but the actual practical security benefit that it gives you right now is quite limited, and I think that will remain the case for the foreseeable future as it really needs quite a paradigm shift in various areas, which I don’t really seeing that happening on Linux any time soon.

                    1. 2

                      I don’t really fancy frobbing with my config just to save/access a file to a non-standard directory

                      If a standard file-picker dialog were used, it could be granted elevated access & automatically grant the calling application access to the selected path(s).

                      1. 1

                        there’s a big UX conundrum here because write access to $HOME allows side-channels, but you also really want your applications to do $useful_stuff, which almost always means accessing much (or all of) $HOME.

                        This is solved on macOS with powerboxes. The Open and Save file dialogs actually run as a separate process and update the application’s security policy dynamically to allow it to access files that the user has selected, but nothing else. Capsicum was designed explicitly to support this kind of use case, it’s a shame that NIH prevented Linux from adopting it.

                        1. 1

                          This sounds like a good idea! I’d love to see that in the X11/Wayland/Unix ecosystem, even just because I hate that awful GTK file dialog for so many reasons and swapping it out with something better would make my life better.

                          Still; the practical security benefit I – and most users – would get from Wayland today would be very little.

                    2. 5

                      I think “broken” is too loaded; “no longer fit for purpose” might be better.

                      1. 2

                        Well, the security model is simply broken.

                        I agree that a lot of focus is put on security improvements compared to Wayland’s other advantages (tear-free rendering being the one most important to me). But it’s still an advantage over X, and I like software which is secure-by-default.

                        1. 1

                          How many actual exploits have been found with this?

                          They were very common in the ‘90s, when folks ran xhost +. Even now, it’s impossible to write a secure password entry box in X11, so remember that any time you type your password into the graphical sudo equivalents that anything that’s currently connected to your X server could capture it. The reason it’s not exploited in the wild is more down to the fact that *NIX distros don’t really do much application sandboxing and so an application that has convinced a user to run it already has pretty much all of the access that it needs for anything malicious that it wants to do. It’s also helped by the fact that most *NIX users only install things from trusted repositories where it’s less likely that you’ll find malware but expect that to change if installing random snap packages from web sites becomes common.

                        2. 4

                          It’s good that Wayland allows for sandboxing

                          If I wanted to sandbox an X application, I’d run it on a separate X server. Maybe even an Xnest kind of thing.

                          I’ve never cared to do this (if I run xnest it is to test network transparency or new window managers or something, not security), so I haven’t tried, but it seems to me it could be done fairly easily if someone really wanted to.

                          1. 2

                            Whoa, I’ve never heard about the GNOME issues (mostly because I’m in a bubble including sway and emersion, and what they do looks sensible to me). That sucks though, I hope they somehow reconcile.

                            Regarding Nvidia I think Simon mentioned something that hinted at them supporting something that has to do with Wayland, but I could just as easily have misunderstood.

                          2. 8

                            Wayland prohibits your desktop applications from capturing keystrokes or recording other apps’ screens by default

                            No, it doesn’t. Theoretically it might enable doing this by modifying the rest of the system too, but in practice (and certainly the default environment) it is still trivial for malware to keylog and record screen on current Wayland desktop *nix installs.

                            1. 3

                              it is still trivial for malware to keylog and record screen on current Wayland desktop *nix installs.

                              I don’t think that’s true. The linked article says recording screens and global hotkeys is “broken” by Wayland. How can it be so trivial for “malware” to do something, and absolutely impossible for anyone else?

                              Or is this malware that requires I run it under sudo?

                              1. 10

                                It’s the difference between doing something properly and just doing it. Malware is happy with the latter while most non malware users are only happy with the former.

                                There are numerous tricks you can use if you are malware, from using LD_PRELOAD to inject code and read events first (since everyone uses libwayland this is really easy), to directing clients to connect to your mitm Wayland server, to just using a debugger, and so on and so forth. None of these are really Wayland’s fault, but the existence of them means there is no meaningful security difference on current desktops.

                                1. 2

                                  I don’t know if I agree that the ability to insert LD_PRELOAD in front of another application is equivalent to sending a bytestring to a socket that is already open, but at least I understand what you meant now.

                              2. 5

                                I’m sick of this keylogger nonsense.

                                X11 has a feature which allows you to use the X11 protocol to snoop on keys being sent to other applications. Wayland does not have an equivalent feature.

                                Using LD_PRELOAD requires being on the other side of an airtight hatch. It straight-up requires having arbitrary code execution, which you can use to compromise literally anything. This is not Wayland’s fault. Wayland is a better lock for your front door. If you leave your window open, it’s not Wayland’ fault when you get robbed.

                                1. 7

                                  Indeed, it’s not waylands fault, and I said as much in response to the only reply above yours, an hour and 20 minutes before you posted this reply. You’re arguing against a straw man.

                                  What is the case is that that “airtight hatch” between things that can interact with wayland and things that can do “giant set of evil activities” has been propped wide open pretty much everywhere on desktop linux, and isn’t reasonably easy to close given the rest of desktop software.

                                  If you were pushing “here’s this new desktop environment that runs everything in secure sandboxes” and it happened to use wayland there would be the possibility of a compelling security argument here. Instead what I see is people making this security argument in a way that could give people the impression it secures things when it doesn’t actually close the barn doors, which is outright dangerous.

                                  In fact, as far as I know the only desktop *nix OS that does sandbox everything thing is QubesOS, and it looks like they currently run a custom protocol on top of an X server…

                                  1. 3

                                    Quoting you:

                                    Wayland prohibits your desktop applications from capturing keystrokes or recording other apps’ screens by default

                                    No, it doesn’t.

                                    Yes, it does. Wayland prohibits Wayland clients from using Wayland to snoop on other Wayland clients. X11 does allow X11 clients to use X11 to snoop on other X11 clients.

                                    Other features of Linux allow you to circumvent this within the typical use-case, but that’s a criticism of those features moreso than of Wayland, and I’m really tired of it being trotted out in Wayland discussions. Wayland has addressed its part of the problem. Now it’s on the rest of the ecosystem to address their parts. Why do you keep dragging it into the Wayland dicsussion when we’ve already addressed it?

                                    1. 7

                                      This

                                      Wayland prohibits your desktop applications from capturing keystrokes or recording other apps’ screens by default

                                      And this

                                      Wayland prohibits Wayland clients from using Wayland to snoop on other Wayland clients.

                                      Are two very different statements. The latter partially specifies the method of snooping, the former does not.

                                      Why do you keep dragging it into the Wayland dicsussion when we’ve already addressed it?

                                      I do not, I merely reply to incorrect claims brought up in support of wayland claiming that it solves a problem that it does not. It might one day become part of a solution to that problem. It might not. It certainly doesn’t solve it by itself, and it isn’t even part of a solution to that problem today.

                              3. 4

                                X’s design has many flaws, but those flaws are well known and documented, and workarounds and extensions exist to cover a wide range of use cases. Wayland may have a better design regarding modern requirements, but has a hard time catching up with all the work that was invested into making X11 work for everyone over the last decades.

                                1. 3

                                  X’s design has many flaws, but those flaws are well known and documented, and workarounds and extensions exist to cover a wide range of use cases.

                                  Once mere flaws become security issues it’s a different matter though.

                                  [Wayland] has a hard time catching up with all the work that was invested into making X11 work for everyone over the last decades.

                                  This may be true now, but Wayland is maturing as we speak. New tools are being developed, and there isn’t much missing in the realm of protocol extensions to cover the existing most-wanted X features. I see Wayland surpassing X in the next two, three years.

                                  1. 2

                                    Yeah, I started to use sway on my private laptop and am really happy with it. Everything works flawlessly, in particular connecting an external HiDPI display and setting different scaling factors (which does not work in X). However, for work I need to be able to share my screen in video calls occasionally and record screencasts with OBS, so I’m still using X there.

                                2. 4

                                  I wonder if X’s security model being “outdated” is partly due to the inexorable slide away from user control. If all your programs are downloaded from a free repo that you trust, you don’t need to isolate every application as if it’s out to get you. Spotify and Zoom on the other hand are out to get you, so a higher level of isolation makes sense, but I would still prefer this to be the exception rather than the rule.

                                  In practice 99.9% of malicious code that is run on our systems is done via the web browser, which has already solved this problem, albeit imperfectly, and only after causing it in the first place.

                                  1. 4

                                    If all your programs are downloaded from a free repo that you trust, you don’t need to isolate every application as if it’s out to get you

                                    I completely agree, as long as all of my programs are completely isolated from the network and any other source of untrusted data, or are formally verified. Otherwise, I have to assume that they contain bugs that an attacker could exploit and I want to limit the damage that they can do. There is no difference between a malicious application and a benign application that is exploited by a malicious actor.

                                    1. 1

                                      all of your programs are completely isolated from the network?

                                      how are you posting here?

                                      1. 2

                                        They’re not, that’s my point and that’s why I’m happy that my browser runs sandboxed. Just because I trust my browser doesn’t mean that I trust everyone who might be able to compromise it.

                                        1. 1

                                          that makes sense for a browser, which is both designed to run malicious code and too complex to have any confidence in its security. but like i said i would prefer cases like this to be the exception. if the rest of your programs are relatively simple and well-tested, isolation may not be worth the complexity and risk of vulnerabilities it introduces. especially if the idea that your programs are securely sandboxed leads you to install less trustworthy programs (as appears to be the trend with desktop linux).

                                          1. 2

                                            Okay, what applications do you run that never consume input from untrusted sources (i.e. do not connect to the network or open files that might come from another application)?

                                            1. 1

                                              I don’t think you are looking at this right. The isolation mechanism can’t be 100% guaranteed free of bugs any more than an application can. Your rhetorical question is pretty far from what I thought we were discussing so maybe you could rephrase your argument.

                                  2. 1

                                    This argument seems similar to what happened with cinnamon-screensaver a few weeks ago:

                                    https://github.com/linuxmint/cinnamon-screensaver/issues/354#issuecomment-762261555 (responding to https://www.jwz.org/blog/2021/01/i-told-you-so-2021-edition/)

                                    It’s a good thing for security (and maybe for users in the long term once they work again) that these usecases are broken, but it is not a good thing for users in the short term that these usecases don’t work on Wayland.

                                  1. 12

                                    It’s an interesting exercise to replace “Wayland” with “systemd” and “Xorg” with “sysv-init” in this article.

                                    1. 10

                                      That is an interesting exercise indeed. There are at least two valid complaints about systemd which don’t apply to Wayland: scope and portability. systemd has its fingers in too many pies, and explicitly uncares about non-Linux platforms. Wayland has a pretty small, mostly closed scope, and runs on BSDs in practice, and more platforms are hypothetically possible if someone put a little bit of work in (no more work than, say, porting X11 requires).

                                      1. 11

                                        Can I agree with both of you on this? I think systemd gets a lot of “NEW==BAD” hate, and I think the proponents of sysvinit are definitely overstating it’s relevance to computers today.

                                        However, it seems like whenever the systemd devs hit any kind of problem, they’re like “I know, we’ll just write our own systemd version and tie it into this giant opaque blob that does everything”

                                        Though the systemd devs will say “It’s all a modular buffet, you can take what you want and use your old tools for stuff you don’t”, unfortunately, nobody’s interested in taking the good bits of systemd, which is basically just the init system, and packaging that up with tried and tested utilities that have worked for the past 20 years, and continue to work today.

                                      2. 13

                                        Disliking Wayland and disliking systemd are orthogonal.

                                        1. 6

                                          I think that some amount of rejection may come from the shared feeling that “newer” hardly means “better” these days. (Gnome, Firefox, Flatpak, Go …)

                                        1. 34

                                          Disclaimer: I represent a GitHub competitor.

                                          The opening characterization of GitHub detractors is disingenuous:

                                          The reasons for being against GitHub hosting tend to be one or more of:

                                          1. it is an evil proprietary platform
                                          2. it is run by Microsoft and they are evil
                                          3. GitHub is American thus evil

                                          GitHub collaborated with US immigration and customs enforcement under the Trump administration, which is a highly controversial organization with severe allegations of “evil”. GitHub also recently fired a Jewish employee for characterising armed insurrectionists wearing Nazi propeganda as Nazis.

                                          It’s not nice to belittle the principles of people who have valid reasons to cite ethical criticisms of GitHub. Even if you like the workflow and convenience, which is Daniel’s main justification, other platforms offer the same conveniences. As project leaders, we have a responsibility to support platforms which align with our values. There are valid ethical and philosophical complaints about GitHub, and dismissing them because of convenience and developer inertia is cowardly.

                                          1. 27

                                            GitHub collaborated with US immigration and customs enforcement under the Trump administration

                                            This makes it sound worse than it actually was, ICE bought a Github Enterprise Server license through a reseller. Github then tried to compensate by donating 500.000$ to “nonprofit organizations working to support immigrant communities”.

                                            … other platforms offer the same conveniences.

                                            Maybe, but they definitely lack the networking effect that was one of main points for curl to use Github.

                                            1. 24

                                              The inconsistency is what kills me here. Allowing ICE to have an account became a heinous crime against neoliberalism, meanwhile how many tech companies openly collaborated with the US military while we killed a million innocent people in Iraq? Or what about Microsoft collaborating with our governments surveillance efforts?

                                              I’m not even engaging in what-about-ism here in the sense that you must be outraged at all the things or none. I’m suggesting that ICE outrage is ridiculous in the face of everything else the US government does.

                                              Pick less ridiculous boogeymen please.

                                              1. 20

                                                I see a lot of the same people (including myself) protesting all of these things…

                                                I feel like I should say something to make this remark longer, and less likely to be taken as hostile, but that’s really all I have to say. Vast numbers of people are consistently opposing all the things you object to. If you’re attempting to suggest that people are picking only one issue to care about and ignoring the other closely related issues, that’s simply wrong - factually, that is not what is happening. If you’re not trying to suggest that, I don’t understand the purpose of your complaint.

                                                1. 13

                                                  The inconsistency is what kills me here.

                                                  Also:

                                                  1. Free Software and Open Source should never discriminate against fields of endeavour!
                                                  2. GitHub should discriminate against this particular organisation!

                                                  and:

                                                  1. We need decentralised systems that are resistant to centralised organisation dictating who can or can’t use the service!
                                                  2. GitHub should use its centralised position to deny this service to this particular organisation!

                                                  Anyway, how exactly will curl moving away from GitHub or GitHub stopping their ICE contract help the people victimized by ICE? I don’t see how it does, and the entire thing seems like a distraction to me. Fix the politics instead.

                                                  1. 14

                                                    Is some ideological notion of consistency supposed to weigh more heavily than harm reduction in one’s ontological calculus? Does “not discriminating against a field of endeavor” even hold inherent virtue? The “who” and “on what grounds” give the practice meaning.

                                                    If I endeavor to teach computer science to under-served groups, and one discriminated against my practice due to bigotry, then that’s bad. If I endeavor to make a ton of money by providing tools and infrastructure to a power structure which seeks to violate the human rights of vulnerable populations, you would be right to “discriminate” against my endeavor.

                                                    Anyway, how exactly will curl moving away from GitHub or GitHub stopping their ICE contract help the people victimized by ICE?

                                                    I don’t think anyone here has suggested that if curl were to move away from github that it would have an appreciable or conclusive impact on ICE and it’s victims. The point of refusing to work for or with with ice or their enablers is mainly to raise awareness of the issue and to build public opposition to them, which is a form of direct action - “fixing the politics” as you put it. It’s easy to laugh at and dismiss people making noise online, or walking out of work, or writing a heated blog post, but as we’ve seen over the last decade, online movements are powerful forces in democratic society.

                                                    1. 8

                                                      Is some ideological notion of consistency supposed to weigh more heavily than harm reduction in one’s ontological calculus?

                                                      If you’re first going to argue that 1) is unethical and should absolutely never be done by anyone and then the next day you argue that 2), which is in direct contradiction to 1), is unethical and should absolutely never be done by anyone then I think there’s a bit of a problem, yes.

                                                      Because at this point you’re no longer having a conversation about what is or isn’t moral, and what the best actions are to combat injustices, or any of these things, instead you’re just trying to badger people in to accepting your viewpoint on a particular narrow issue.

                                                      1. 3

                                                        If you’re first going to argue that 1) is unethical and should absolutely never be done by anyone and then the next day you argue that 2), which is in direct contradiction to 1), is unethical and should absolutely never be done by anyone then I think there’s a bit of a problem, yes.

                                                        does anyone say that though

                                                    2. 12

                                                      Your first two points are a good explanation of the tension between the Open Source and Ethical Source movements. I think everyone close to the issue is in agreement that, yes, discriminating against militant nationalism is a form of discrimination, just one that ought to happen.

                                                      There was some open conflict last year between the Open Source Institute, and the group that became the Organization for Ethical Source. See https://ethicalsource.dev/ for some of the details.

                                                      Your second two points, also, highlight a real and important concern, and you’ve stated it well. I’m personally against centralized infrastructure, including GitHub. I very much want the world to move to decentralized technical platforms in which there would be no single entity that holds the power that corporations presently do. However, while centralized power structures exist, I don’t want those structures to be neutral to injustice. To do that is to side with the oppressor.

                                                      (Edit: I somehow wrote “every” instead of “everyone”. Too many editing passes, I guess. Oops.)

                                                      1. 11

                                                        To clarify: this wasn’t really intended as a defence of either the first or second points in contradictions, I just wanted to point out that people’s views on this are rather inconsistent, to highlight that the issue is rather more complex than some people portray it as. To be fair, most people’s worldviews are inconsistent to some degree, mine certainly are, but then again I also don’t make bold absolute statements about these sort of things and insult people who don’t fit in that.

                                                        I think that both these issues are essentially unsolvable; similar to how we all want every criminal to be convicted but also want zero innocent people to be convicted unjustly. This doesn’t mean we shouldn’t try, but we should keep a level head about what we can and can’t achieve, and what the trade-offs are.

                                                        I don’t want those structures to be neutral to injustice. To do that is to side with the oppressor.

                                                        In Dutch we have a saying I rather like: “being a mayor in wartime”. This refers to the dilemma of mayors (and journalists, police, and so forth) during the German occupation. To stay in your position would be to collaborate with the Nazis; but to resign would mean being replaced with a Nazi sympathizer. By staying you could at least sort of try to influence things. This is a really narrow line to walk though, and discussions about who was or wasn’t “wrong” during the war continue to this day.

                                                        I don’t think GitHub is necessarily “neutral to injustice”, just like the mayors during the war weren’t. I know people love to portray GitHub as this big evil company, but my impression is that GitHub is actually not all that bad; I mean, how many other CEOs would have joined youtube-dl’s IRC channel to apologize for the shitty situation they’re in? Or would have spent time securing a special contract to provide service to Iranian people? Or went out of their way to add features to rename the default branch?

                                                        But there is a limit to what is reasonable; no person or company can be unneutral to all forms of injustice; it would be debilitating. You have to pick your battles; ICE is a battle people picked, and IMO it’s completely the wrong one: what good would cutting a contract with ICE do? I don’t see it, and I do see a lot of risk in alienating the government of the country you’re based in, especially considering that the Trump administration was not exactly know for its cool, level-headed, and calm responses to (perceived) sleights. Besides, in the grand scheme of injustices present in the world ICE seems small fries.

                                                        And maybe all tech companies putting pressure on ICE would have made an impact in changing ICE’s practices, I don’t really think it would but let’s assume it would. But what does that mean? A bunch of undemocratic companies exerting pressure to change the policy of a democratically elected government. Yikes? Most of the time I see corporate influence on government it’s not for the better and I would rather we reduce this across the board, which would also reduce the potential “good influences”, but the bad influences vastly outnumber the good ones that this is a good trade.

                                                        1. 6

                                                          Yes, those are all fair and thoughtful points. I agree very much that with any system, no matter how oppressive, if one has a position of power within the system it’s important to weigh how much good one can do by staying in, against how much they can do by leaving. I rather wish I were living in times that didn’t require making such decisions in practice so frequently, but none of us get to choose when we’re born.

                                                          On the strategic point you raise, I disagree: I do think the GitHub/ICE issue is a valuable one to push on, precisely because it prompts conversations like this. Tech workers might be tempted to dismiss our own role in these atrocities; I think it’s important to have that reminder. However, I very much acknowledge that it’s hard to know whether there’s some other way that might be better, and there’s plenty of room for disagreement, even among people who agree on the goals.

                                                          When I was young, I was highly prone to taking absolute positions that weren’t warranted. I hope if I ever fall back into those old habits, you and others will call me out. I do think it’s really important for people who disagree to hear each other out, whenever that’s feasible, and I also think it’s important for us all to acknowledge the limits of our own arguments. So, overall, thank you for your thoughts.

                                                          1. 2

                                                            I recently read a really approachable article article from Stanford Encyclopedia of Philosophy (via HN), which I found really interesting and balanced in highlighting the tensions between (in this case study) “free speech” and other values. To me it also helps to understand that those apparent “conflicts of interest” are still rather possible to balance (if not trivially) given good will; and IMO that the “extreme positions” are something of a possibly unavoidable simplifications - given that even analyzing the positions of renowned philosophers, skilled at precise expression, it’s not always completely clear where they sat.

                                                            https://plato.stanford.edu/entries/freedom-speech/

                                                            edit: though I am totally worried when people refuse to even discuss those nuances and to explore their position in this space of values.

                                                            1. 7

                                                              Anyone with a sincere interest in educating themselves about the concept of free speech and other contentious issues will quickly learn about the nuances of the concepts. Some people will however not give a fig about these nuances and continue to argue absolutist positions on the internet, either to advance unrelated political positions or simply to wind people up.

                                                              Engaging with these people (on these issues) is generally a waste of time. It’s like wrestling with a pig - you’ll get dirty and the pig enjoys it.

                                                              1. 3

                                                                I’m not sure I agree that anyone who makes a sincere effort will learn about the nuances. The nuance is there, but whether people have the chance to learn it is largely a function of whether the social spaces they’re in give them the chance to. I’m really worried about how absolutist, reactionary positions are the bulk of discussion on social media today. I think we all have an obligation to try to steer discussions away from reductive absolutism, in every aspect of our lives.

                                                                With that said, it’s clear you’re coming from a good place and I sympathize. I only wish I felt that not engaging is clearly the right way; it would be easier.

                                                                1. 5

                                                                  I’ll have to admit that my comment was colored by my jaundiced view of the online conversation at this point in time. “Free speech” has become a shibboleth among groups who loudly demand immunity from criticism, and who expect their wares to be subsidized in the Marketplace of Ideas, but who would not hesitate to restrict the speech of their enemies should they attain power.

                                                                  I’m all for nuanced discussion, but some issues are just so hot button it’s functionally useless in a public forum.

                                                                  1. 4

                                                                    I completely understand, and that’s very fair.

                                                                    I agree with your assessment but, purely for myself and not as something I’d push on others, I refuse to accept the outcome of stepping back from discussion - because that would be a win for reactionary forms of engagement, and a loss for anyone with a sincere, thought-out position, wherever they might fall on the political spectrum.

                                                                    It’s fine to step back and say that for your own well being, you can’t dedicate your efforts to being part of the solution to that. You can only do what you can do, and no person or cause has a right to demand more than that. For myself, only, I haven’t given up and I’ll continue to look for solutions.

                                                        2. 6

                                                          There are a lot of people in the OSS community who don’t agree with your first point. You might find it contradictory, or “wrong” (And sure, I guess it wouldn’t be OSI certified if you codified it in a license). But it’s what a decent part of the community thinks.

                                                          And the easy answer to your comment about helping, let’s do the contrary. ICE has policies. Selling them tools to make it easier is clearly helping them to move forward on those policies. Just like AWS was helping Parler exist by offering its infrastructure. You can have value judgements or principles regarding those decisions, but you can’t say that it doesn’t matter at all.

                                                          And yeah, maybe there’s someone else who can offer the services. But maybe there are only so many Github-style services out there! And at one point it starts actually weighing on ICE’s ability to do stuff.

                                                          Of course people want to fix the politics. But lacking that power, people will still try to do something. And, yeah, people are allowed to be mad that a company is doing something, even they probably shouldn’t be surprised.

                                                          1. 4

                                                            And yeah, maybe there’s someone else who can offer the services. But maybe there are only so many Github-style services out there! And at one point it starts actually weighing on ICE’s ability to do stuff.

                                                            I’d expect ICE to be more than capable of self-hosting GitLab or some other free software project.

                                                            Of course people want to fix the politics. But lacking that power, people will still try to do something.

                                                            I don’t think it’s outside of people’s power to do that, but it is a lot harder, and requires more organisation and dedication. And “doing something” is not the same as “doing something useful”.

                                                            As for the rest, I already addressed most of that in my reply to Irene’s comment, so I won’t repeat that here.

                                                        3. 12

                                                          no disagreement with your main point, but… a crime against neoliberalism?

                                                          1. 4

                                                            I think they mean against the newest wave of liberal politics in the US. Not the actual term neoliberalism which—as you clearly know—refers to something completely different, if not totally opposite.

                                                          2. 10

                                                            there are active campaigns inside and outside most companies about those issues. It’s not like https://notechforice.com/ exists in a bubble. Amazon, Google, Microsoft, Palantir, Salesforce and many others have been attacked for this. Clearly the DoD created the Silicon Valley and the connections run deep since the beginning, but these campaigns are to raise awareness and build consensus against tech supporting imperialism, concentration camps and many other crimes committed by the American Government against its citizens or foreign countries. But you have to start somewhere: political change is not like compiling a program, it’s not on and off, it’s nuanced and complex. Attacking (and winning) stuff like Project Maven or ICE concentration camps is a way to show that you can achieve something, break the tip of the iceberg and use that to build bigger organizations and bigger support for bigger actions.

                                                            1. 1

                                                              Clearly the DoD created the Silicon Valley and the connections run deep since the beginning

                                                              Oh, I’d love to be red-pilled into that!

                                                          3. 22

                                                            This makes it sound worse than it actually was, ICE bought a Github Enterprise Server license through a reseller.

                                                            LA Times:

                                                            In a fact sheet circulating within GitHub, employees opposing the ICE contract wrote that the GitHub sales team actively pursued the contract renewal with ICE. The Times reviewed screenshots of an internal Slack channel after the contract was renewed on Sept. 4 that appear to show sales employees celebrating a $56,000 upgrade of the contract with ICE. The message, which congratulated four employees for the sale and was accompanied by emojis of a siren, bald eagle and American flag, read “stay out of their way. $56k upgrade at DHS ICE.” Five people responded with an American flag emoji.

                                                            It was not as at arm’s length as they’d like you to believe. Several prominent organisations rejected offers of parts of the $500k donation because they didn’t want to be associated with the ICE contract. Internally the company was shredded as it became clear that GitHub under MSFT would rather be torn apart inside than listen to employees and customers and commit to stop serving ICE in the future.

                                                            There were plenty of calls to cancel the contract immediately, which might’ve been a pipedream, but even the more realistic “could we just not renew it in future” was met with silence and corporatespeak. Long-serving employees asking “well, if this isn’t too far for us, what concretely would be over the line?” in Q&A’s were labelled hostile, and most certainly not answered.

                                                            1. 16

                                                              We could debate the relative weight of these and other grievances here, but I’d rather not. My point is simply that the ethical concerns are based on reason, and Daniel’s blithe dismissal of them is inappropriate.

                                                              1. 7

                                                                Could you elaborate on the reasons?

                                                                You state that the reasons exist, and you give an example of someone you think github should reject as a customer. But you don’t talk about what those reasons are, or really go into principles, rationales or philosophy at all.

                                                                I worry that without a thought-through framework, your attitude degenerates into mindless shitstorms.

                                                                1. 4

                                                                  He has not engaged with the ethical concerns you raise. That may well be because he is simply not aware of them. You are overinterpreting that as “blithe dismissal”.

                                                              2. 10

                                                                The firing of the employee has been reversed.

                                                                1. 10

                                                                  Just a honest question: does this poop management actually makes them look better to you? Despite this being a reaction to public outrage that would have hurt the company? Like, do you think they that out of guilt or something like that?

                                                                  1. 3

                                                                    Considering the fired employee was reinstated and the head of HR resigned, this looks like a much more substantive concession than the employment status Ctrl-Z that internet outrages usually produce.

                                                                    1. 3

                                                                      how? isn’t the “let’s sacrifice a scapegoat without fundamentally changing anything” a quite common strategy?

                                                                      1. 2

                                                                        None of us know the details of this case. It’s way too easy to form a conclusion from one party, especially if they’re not bound by law from discussing sensitive HR details openly.

                                                                        So while I can project a hope that this is a lasting change at GH, you are free to cynically dismiss it as window dressing. The facts, as we know them, support either view.

                                                                  2. 17

                                                                    Aye, and I commend them for that. But that doesn’t change the fact that “retaliated against an employee who spoke out against Nazism” is a permanent stain on their reputation which rightfully angers many people, who rightfully may wish to cease using the platform as a result. Daniel’s portrayal of their concerns as petty and base is not right.

                                                                    1. 2

                                                                      Not only that but the HR person who fired him was fired.

                                                                      1. 4

                                                                        Probably out of convenience and not actually the person who gave the order. At least, I think that’s the case more than we know.

                                                                        1. 5

                                                                          The person who resigned was the head of HR. It almost certainly wasn’t the person who made the call, or even their manager, it was likely their manager’s manager. That sends a pretty strong signal to the rest of HR that there will be consequences for this kind of thing in the future.

                                                                          1. 1

                                                                            Damn, the head of HR!? What a turnover. Maybe that means they’re taking this more seriously than I thought at first.

                                                                    2. 7

                                                                      Every time someone asked me to move away from GitHub it’s been because “it’s not Free Software” and various variants of “vendor lock-in” and “it’s centralized”. I am aware there are also other arguments, but those have not been stated in the two instances people asked me to move away from GitHub. What (probably) prompted this particular Twitter thread and that doesn’t mention ICE or anything like that (also: 1 2). Most comments opposed to GitHub on HN or Lobsters don’t focus on ICE either.

                                                                      That you personally care a great deal about this is all very fine, but it’s not the most commonly used argument against GitHub.

                                                                      There are valid ethical and philosophical complaints about GitHub

                                                                      According to your view of ethics, which many don’t share.

                                                                      1. 2

                                                                        I think that asking someone to change their infrastructure based solely on personal preferences is a step or two too far, be it based on ethics or ergonomics (“all the other code I use is on GitHub, yours should be too”).

                                                                        It’s at the very least a bunch of work to move, and the benefit is likely small. You’ve already made a choice when deciding to put your code where it is, so why would you want to change it?

                                                                        If asked, I’d recommend using something other than Github to work against the monoculture we’re already pretty deep in, but I don’t see myself actively trying to persuade others to abandon them.

                                                                      2. 4

                                                                        Isn’t sr.ht hosted and incorporated in the US? Or are only points (1) and (2) valid? :-D

                                                                        GitHub also fought the US Gov to get the Iranian developer access to their platform, which is also helping your platform as far as I know. https://github.blog/2021-01-05-advancing-developer-freedom-github-is-fully-available-in-iran/

                                                                        Any organization that is large enough will have some incidents which, when cherry-picked, can be used to paint the organization as evil. But really what happens is that they represent humanity. In terms of evil, you don’t have to look far to see much worse groups of people than GitHub.

                                                                        IMO a more compelling argument would be centered around how he is an open-source developer, depending on a closed platform. Daniel’s utilitarian view is understandable but also short-thinking. He is contributing towards building this monolith just by using it.

                                                                        1. 21

                                                                          Or are only points (1) and (2) valid? :-D

                                                                          None of the points Daniel raises are valid, because they’re strawmen, and bad-faith portrayals of actual positions.

                                                                          Actual argument: “GitHub, an American company, is choosing to cooperate with ICE, an American instutition which is controversial for its ethical problems”

                                                                          Bad faith re-stating: “GitHub is American thus evil”

                                                                          There is nuance here, and indeed you’ve found some of it, but a nuanced argument is not what Daniel is making.

                                                                        2. 6

                                                                          collaborated with US immigration and customs enforcement

                                                                          I think “is American and thus evil” definitely covers this.

                                                                          1. 2

                                                                            Why are two [1, 2] of your most popular projects primarily hosted on github?

                                                                            1. https://github.com/swaywm/sway

                                                                            2. https://github.com/swaywm/wlroots

                                                                            1. 19

                                                                              I have been gradually moving off of GitHub, but not all at once. A few months ago I finished migrating all of the projects under my user namespace (github.com/ddevault) to SourceHut. Last week I also announced to my GitHub Sponsors supporters that I intend to leave the program, which is almost certain to cause me to lose money when many of them choose not to move to my personal donation platform (which has higher payment processing fees than GitHub does, so even if they all moved I would still lose money). If you intend to imply that I am a hypocrite for still using GitHub, I don’t think that holds very much weight.

                                                                              Regarding those two projects in particular, some discussion was held about moving to gitlab.freedesktop.org last year, but it was postponed until the CI can be updated accordingly. In any case, I am no longer the maintainer of either project, and at best only an occasional contributor, so it’s not really my place nor my responsibility to move the projects elsewhere. I think that they should move, and perhaps a renewed call for doing so should be made, but it’s ultimately not my call anymore.

                                                                              1. 10

                                                                                If you intend to imply that I am a hypocrite for still using GitHub, I don’t think that holds very much weight.

                                                                                Nope, I was just genuinely curious since I don’t follow you that closely, and hadn’t heard any explanation or reasoning why those repos are still on github when I have heard you explain your position regarding github multiple times. So it seemed odd, so I asked.

                                                                                In any case, thanks for explaining! I hope those projects are moved off too (@emersion !)

                                                                                1. 6

                                                                                  Cool, makes sense. Thanks for clarifying.

                                                                                2. 2

                                                                                  I love that you represent another point of view here. I firmly believe that free software needs free tools. We don’t want history to repeat. And Yes, there will be some sacrifice for the switch.

                                                                                  Watching your actions closely for months, You represent how a free software leader should be.

                                                                            1. 37

                                                                              Hello, I am here to derail the Rust discussion before it gets started. The culprit behind sudo’s vast repertoire of vulnerabilities, and more broadly of bugs in general, is accountable almost entirely to one matter: its runaway complexity.

                                                                              We have another tool which does something very similar to sudo which we can compare with: doas. The portable version clocks in at about 500 lines of code, its man pages are a combined 157 lines long, and it has had two CVEs (only one of which Rust would have prevented), or approximately one every 30 months.

                                                                              sudo is about 120,000 lines of code (100x more), its had 140 CVEs, or about one every 2 months since the CVE database came into being 21 years ago. Its man pages are about 10,000 lines and include the following:

                                                                              $ man sudoers | grep -C1 despair
                                                                              The sudoers file grammar will be described below in Extended Backus-Naur
                                                                              Form (EBNF).  Don't despair if you are unfamiliar with EBNF; it is fairly
                                                                              simple, and the definitions below are annotated.
                                                                              

                                                                              If you want programs to be more secure, stable, and reliable, the key metric to address is complexity. Rewriting it in Rust is not the main concern.

                                                                              1. 45

                                                                                its had 140 CVEs

                                                                                Did you even look at that list? Most of those are not sudo vulnerabilities but issues in sudo configurations distros ship with. The actual list is more like 39, and a number of them are “disputed” and most are low-impact. I didn’t do a full detailed analysis of the issues, but the implication that it’s had “140 security problems” is simply false.

                                                                                sudo is about 120,000 lines of code

                                                                                More like 60k if you exclude the regress (tests) and lib directories, and 15k if you exclude the plugins (although the sudoers plugin is 40k lines, which most people use). Either way, it’s at least half of 120k.

                                                                                Its man pages are about 10,000 lines and include the following:

                                                                                12k, but this also includes various technical documentation (like the plugin API); the main documentation in sudoers(1) is 741 lines, and sudoers(5) is 3,255 lines. Well under half of 10,000.

                                                                                We have another tool which does something very similar to sudo which we can compare with: doas.

                                                                                Except that it only has 10% of the features, or less. This is good if you don’t use them, and bad if you do. But I already commented on this at HN so no need to repeat that here.

                                                                                1. 12

                                                                                  You’re right about these numbers being a back-of-the-napkin analysis. But even your more detailed analysis shows that the situation is much graver with sudo. I am going to include plugins, becuase if they ship, they’re a liability. And their docs, because they felt the need to write them. You can’t just shove the complexity you don’t use and/or like under the rug. Heartbleed brought the internet to its knees because of a vulnerability in a feature no one uses.

                                                                                  And yes, doas has 10% of the features by count - but it has 99% of the features by utility. If you need something in the 1%, what right do you have to shove it into my system? Go make your own tool! Your little feature which is incredibly useful to you is incredibly non-useful to everyone else, which means fewer eyes on it, and it’s a security liability to 99% of systems as such. Not every feature idea is meritous. Scope management is important.

                                                                                  1. 9

                                                                                    it has 99% of the features by utility

                                                                                    Citation needed.

                                                                                    what right do you have to shove it into my system?

                                                                                    Nobody is shoving anything into your system. The sudo maintainers have the right to decide to include features, and they’ve been exercising that right. You have the right to skip sudo and write your own - and you’ve been exercising that right too.

                                                                                    Go make your own tool!

                                                                                    You’re asking people to undergo the burden of forking or re-writing all of the common functionality of an existing tool just so they can add their one feature. This imposes a great cost on them. Meanwhile, including that code or feature into an existing tool imposes only a small (or much smaller) cost, if done correctly - the incremental cost of adding a new feature to an existing system.

                                                                                    The key phrase here is “if done correctly”. The consensus seems to be that sudo is suffering from poor engineering practices - few or no tests, including with the patch that (ostensibly) fixes this bug. If your software engineering practices are bad, then simpler programs will have fewer bugs only because there’s less code to have bugs in. This is not a virtue. Large, complex programs can be built to be (relatively) safe by employing tests, memory checkers, good design practices, good architecture (which also reduces accidental complexity) code reviews, and technologies that help mitigate errors (whether that be a memory-safe GC-less language like Rust or a memory-safe GC’ed language like Python). Most features can (and should) be partitioned off from the rest of the design, either through compile-time flags or runtime architecture, which prevents them from incurring security or performance penalties.

                                                                                    Software is meant to serve the needs of users. Users have varied use-cases. Distinct use-cases require more code to implement, and thereby incur complexity (although, depending on how good of an engineer one is, additional accidental complexity above the base essential complexity may be added). If you want to serve the majority of your users, you must incur some complexity. If you want to still serve them, then start by removing the accidental complexity. If you want to remove the essential complexity, then you are no longer serving your users.

                                                                                    The sudo project is probably designed to serve the needs of the vast majority of the Linux user-base, and it succeeds at that, for the most part. doas very intentionally does not serve the needs of the vast majority of the linux user-base. Don’t condemn a project for trying to serve more users than you are.

                                                                                    Not every feature idea is meritous.

                                                                                    Serving users is meritous - or do you disagree?

                                                                                    1. 6

                                                                                      Heartbleed brought the internet to its knees because of a vulnerability in a feature no one uses.

                                                                                      Yes, but the difference is that these are features people actually use, which wasn’t the case with Heartleed. Like I mentioned, I think doas is great – I’ve been using it for years and never really used (or liked) sudo because I felt it was far too complex for my needs, before doas I just used su. But I can’t deny that for a lot of other people (mainly organisations, which is the biggest use-case for sudo in the first place) these features are actually useful.

                                                                                      Go make your own tool! Your little feature which is incredibly useful to you is incredibly non-useful to everyone else

                                                                                      A lot of these things aren’t “little” features, and many interact with other features. What if I want doas + 3 flags from sudo + LDAP + auditing? There are many combinations possible, and writing a separate tool for every one of them isn’t really realistic, and all of this also required maintenance and reliable consistent long-term maintainers are kind of rare.

                                                                                      Scope management is important.

                                                                                      Yes, I’m usually pretty explicit about which use cases I want to solve and which I don’t want to solve. But “solving all the use cases” is also a valid scope. Is this a trade-off? Sure. But everything here is.

                                                                                      The real problem isn’t so much sudo; but rather that sudo is the de-facto default in almost all Linux distros (often installed by default, too). Ideally, the default should be the simplest tool which solves most of the common use cases (i.e. doas), and people with more complex use cases can install sudo if they need it. I don’t know why there aren’t more distros using doas by default (probably just inertia?)

                                                                                      1. 0

                                                                                        What if I want doas + 3 flags from sudo + LDAP + auditing?

                                                                                        Tough shit? I want a pony, and a tuba, and barbie doll…

                                                                                        But “solving all the use cases” is also a valid scope.

                                                                                        My entire thesis is that it’s not a valid scope. This fallacy leads to severe and present problems like the one we’re discussing today. You’re begging the question here.

                                                                                        1. 4

                                                                                          Tough shit? I want a pony, and a tuba, and barbie doll…

                                                                                          This is an extremely user-hostile attitude to have (and don’t try claiming that telling users with not-even-very-obscure use-cases to write their own tools isn’t user-hostile).

                                                                                          I’ve noticed that some programmers are engineers that try to build tools to solve problems for users, and some are artists that build programs that are beautiful or clever, or just because they can. You appear to be one of the latter, with your goal being crafting simple, beautiful systems. This is fine. However, this is not the mindset that allows you to build either successful systems (in a marketshare sense) or ones that are useful for many people other than yourself, for previously-discussed reasons. The sudo maintainers are trying to build software for people to use. Sure, there’s more than one way to do that (integration vs composition), but there are ways to do both poorly, and claiming the moral high ground for choosing simplicity (composition) is not only poor form but also kind of bad optics when you haven’t even begun to demonstrate that it’s a better design strategy.

                                                                                          My entire thesis is that it’s not a valid scope.

                                                                                          A thesis which you have not adequately defended. Your statements have amounted to “This bug is due to sudo’s complexity which is driven by the target scope/number of features that it has”, while both failing to provide any substantial evidence that this is the case (e.g. showing that sudo’s bugs are due to feature-driven essential complexity alone, and not use of a memory-unsafe language, poor software engineering practices (which could lead to either accidental complexity or directly to bugs themselves), or simple chance/statistics) and not actually providing any defense for the thesis as stated. Assume that @arp242 didn’t mean “all” the usecases, but instead “the vast majority” of them - say, enough that it works for 99.9% of users. Why is this “invalid”, exactly? It’s easy for me to imagine the argument being “this is a bad idea”, but I can’t imagine why you would think that it’s logically incoherent.

                                                                                          Finally, you have repeatedly conflated “complexity” and “features”. Your entire argument is, again, invalid if you can’t show that sudo’s complexity is purely (or even mostly) essential complexity, as opposed to accidental complexity coming from being careless etc.

                                                                                    2. 9

                                                                                      I dont’t think “users (distros) make a lot of configuration mistakes” is a good defence when arguing if complexity is the issue.

                                                                                      But I do agree about feature set. And I feel like arguing against complexity for safety is wrong (like ddevault was doing), because systems inevitably grow complex. We should still be able to build safe, complex systems. (Hence why I’m a proponent of language innovation and ditching C.)

                                                                                      1. 11

                                                                                        I dont’t think “users (distros) make a lot of configuration mistakes” is a good defence when arguing if complexity is the issue.

                                                                                        It’s silly stuff like (ALL : ALL) NOPASSWD: ALL. “Can run sudo without a password” seems like a common theme: some shell injection is found in the web UI and because the config is really naïve (which is definitely not the sudo default) it’s escalated to root.

                                                                                        Others aren’t directly related to sudo configuration as such; for example this one has a Perl script which is run with sudo that can be exploited to run arbitrary shell commands. This is also a common theme: some script is run with sudo, but the script has some vulnerability and is now escalated to root as it’s run with sudo.

                                                                                        I didn’t check all of the issues, but almost all that I checked are one of the above; I don’t really see any where the vulnerability is caused directly by the complexity of sudo or its configuration; it’s just that running anything as root is tricky: setuid returns 432 results, three times that of sudo, and I don’t think that anyone can argue that setuid is complex or that setuid implementations have been riddled with security bugs.

                                                                                        Other just mention sudo in passing by the way; this one is really about an unrelated remote exec vulnerability, and just mentions “If QCMAP_CLI can be run via sudo or setuid, this also allows elevating privileges to root”. And this one isn’t even about sudo at all, but about a “sudo mode” plugin for TYPO3, presumably to allow TYPO3 users some admin capabilities without giving away the admin password. And who knows why this one is even returned in a search for “sudo” as it’s not mentioned anywhere.

                                                                                        1. 3

                                                                                          it’s just that running anything as root is tricky: setuid returns 432 results, three times that of sudo

                                                                                          This is comparing apples to oranges. setuid affects many programs, so obviously it would have more results than a single program would. If you’re going to attack my numbers than at least run the same logic over your own.

                                                                                          1. 2

                                                                                            It is comparing apples to apples, because many of the CVEs are about other program’s improper sudo usage, similar to improper/insecure setuid usage.

                                                                                            1. 2

                                                                                              Well, whatever we’re comparing, it’s not making much sense.

                                                                                              1. If sudo is hard to use and that leads to security problems through its misusage, that’s sudo’s fault. Or do you think that the footguns in C are not C’s fault, either? I thought you liked Rust for that very reason. For this reason the original CVE count stands.
                                                                                              2. But fine, let’s move on on the presumption that the original CVE count is not appropriate to use here, and instead reference your list of 39 Ubuntu vulnerabilities. 39 > 2, Q.E.D. At this point we are comparing programs to programs.
                                                                                              3. You now want to compare this with 432 setuid results. You are comparing programs with APIs. Apples to oranges.

                                                                                              But, if you’re trying to bring this back and compare it with my 140 CVE number, it’s still pretty damning for sudo. setuid is an essential and basic feature of Unix, which cannot be made any smaller than it already is without sacrificing its essential nature. It’s required for thousands of programs to carry out their basic premise, including both sudo and doas! sudo, on the other hand, can be made much simpler and still address its most common use-cases, as demonstrated by doas’s evident utility. It also has a much smaller exposure: one non-standard tool written in the 80’s and shunted along the timeline of Unix history every since, compared to a standardized Unix feature introduced by DMR himself in the early 70’s. And setuid somehow has only 4x the number of footgun incidents? sudo could do a hell of a lot better, and it can do so by trimming the fat - a lot of it.

                                                                                              1. 3

                                                                                                If sudo is hard to use and that leads to security problems through its misusage, that’s sudo’s fault.

                                                                                                It’s not because it’s hard to use, it’s just that its usage can escalate other more (relatively) benign security problems, just like setuid can. This is my point, as a reply to stephank’s comment. This is inherent to running anything as root, with setuid, sudo, or doas, and why we have capabilities on Linux now. I bet that if doas would be the default instead of sudo we’d have a bunch of CVEs about improper doas usage now, because people do stupid things like allowing anyone to run anything without password and then write a shitty web UI in front of that. That particular problem is not doas’s (or sudo’s) fault, just as cutting myself with the kitchen knife isn’t the knife’s fault.

                                                                                                reference your list of 39 Ubuntu vulnerabilities. 39 > 2, Q.E.D.

                                                                                                Yes, sudo has had more issues in total; I never said it doesn’t. It’s just a lot lower than what you said, and quite a number are very low-impact, so I just disputed the implication that sudo is a security nightmare waiting to happen: it’s track record isn’t all that bad. As always, more features come with more (security) bugs, but use cases do need solving somehow. As I mentioned, it’s a trade-off.

                                                                                                sudo, on the other hand, can be made much simpler and still address its most common use-cases, as demonstrated by doas’s evident utility

                                                                                                We already agreed on this yesterday on HN, which I repeated here as well; all I’m adding is “but sudo is still useful, as it solves many more use cases” and “sudo isn’t that bad”.

                                                                                                Interesting thing to note: sudo was removed from OpenBSD by millert@openbsd.org; who is also the sudo maintainer. I think he’ll agree that “sudo is too complex for it to the default”, which we already agree on, but not that sudo is “too complex to exist”, which is where we don’t agree.

                                                                                                Could sudo be simpler or better architectured to contain its complexity? Maybe. I haven’t looked at the source or use cases in-depth, and I’m not really qualified to make this judgement.

                                                                                        2. 5

                                                                                          I think arguing against complexity is one of the core principles of UNIX philosophy, and it’s gotten us quite far on the operating system front.

                                                                                          If simplicity was used in sudo, this particular vulnerability would not have been possible to trigger it: why have sudoedit in the first place, which just implies the -e flag? This statement is a guarantee.

                                                                                          If it would’ve ditched C, there is no guarantee that this issue wouldn’t have happened.

                                                                                        3. 2

                                                                                          Did you even look at that list? Most of those are not sudo vulnerabilities but issues in sudo configurations distros ship with.

                                                                                          If even the distros can’t understand the configuration well enough to get it right, what hope do I have?

                                                                                        4. 16

                                                                                          OK maybe here’s a more specific discussion point:

                                                                                          There can be logic bugs in basically any language, of course. However, the following classes of bugs tend to be steps in major exploits:

                                                                                          • Bounds checking issues on arrays
                                                                                          • Messing around with C strings at an extremely low level

                                                                                          It is hard to deny that, in a universe where nobody ever messed up those two points, there are a lot less nasty exploits in the world in systems software in particular.

                                                                                          Many other toolchains have decided to make the above two issues almost non-existent through various techniques. A bunch of old C code doesn’t handle this. Is there not something that can be done here to get the same productivity and safety advantages found in almost every other toolchain for tools that form the foundation of operating computers? Including a new C standard or something?

                                                                                          I can have a bunch of spaghetti code in Python, but turning that spaghetti into “oh wow argv contents ran over some other variables and messed up the internal state machine” is a uniquely C problem, but if everyone else can find solutions, I feel like C could as well (including introducing new mechanisms to the language. We are not bound by what is printed in some 40-year-old books, and #ifdef is a thing).

                                                                                          EDIT: forgot to mention this, I do think that sudo is a bit special given that its default job is to take argv contents and run them. I kinda agree that sudo is a bit special in terms of exploitability. But hey, the logic bugs by themselves weren’t enough to trigger the bug. When you have a multi-step exploit, anything on the path getting stopped is sufficient, right?

                                                                                          1. 14

                                                                                            +1. Lost in the noise of “but not all CVEs…” is the simple fact that this CVE comes from an embarrassing C string fuckup that would be impossible, or at least caught by static analysis, or at very least caught at runtime, in most other languages. If “RWIIR” is flame bait, then how about “RWIIP” or at least “RWIIC++”?

                                                                                            1. 1

                                                                                              I be confused… what does the P in RWIIP mean?

                                                                                              1. 3

                                                                                                Pascal?

                                                                                                1. 1

                                                                                                  Python? Perl? Prolog? PL/I?

                                                                                                2. 2

                                                                                                  Probably Python, given the content of the comment by @rtpg. Python is also memory-safe, while it’s unclear to me whether Pascal is (a quick search reveals that at least FreePascal is not memory-safe).

                                                                                                  Were it not for the relative (accidental, non-feature-providing) complexity of Python to C, I would support RWIIP. Perhaps Lua would be a better choice - it has a tiny memory and disk footprint while also being memory-safe.

                                                                                                  1. 2

                                                                                                    Probably Python, given the content of the comment by @rtpg. Python is also memory-safe, while it’s unclear to me whether Pascal is (a quick search reveals that at least FreePascal is not memory-safe).

                                                                                                    That’s possibly it.

                                                                                                    Perhaps Lua would be a better choice - it has a tiny memory and disk footprint while also being memory-safe.

                                                                                                    Not to mention that Lua – even when used without LuaJIT – is simply blazingly fast compared to other scripting languages (Python, Perl, &c)!

                                                                                                    For instance, see this benchmark I did sometime ago: https://0x0.st/--3s.txt. I had implemented Ackermann’s function in various languages (the “./ack” file is the one in C) to get a rough idea on their execution speed, and lo and behold Lua turned out to be second only to the C implementation.

                                                                                            2. 15

                                                                                              I agree that rewriting things in Rust is not always the answer, and I also agree that simpler software makes for more secure software. However, I think it is disingenuous to compare the overall CVE count for the two programs. Would you agree that sudo is much more widely installed than doas (and therefore is a larger target for security researchers)? Additionally, most of the 140 CVEs linked were filed before October 2015, which is when doas was released. Finally, some of the linked CVEs aren’t even related to code vulnerabilities in sudo, such as the six Quest DR Series Disk Backup CVEs (example).

                                                                                              1. 4

                                                                                                I would agree that sudo has a bigger target painted on its back, but it’s also important to acknowledge that it has a much bigger back - 100× bigger. However, I think the comparison is fair. doas is the default in OpenBSD and very common in NetBSD and FreeBSD systems as well, which are at the heart of a lot of high-value operations. I think it’s over the threshold where we can consider it a high-value target for exploitation. We can also consider the kinds of vulnerabilities which have occured internally within each project, without comparing their quantity to one another, to characterize the sorts of vulnerabilities which are common to each project, and ascertain something interesting while still accounting for differences in prominence. Finally, there’s also a bias in the other direction: doas is a much simpler tool, shipped by a team famed for its security prowess. Might this not dissuade it as a target for security researchers just as much?

                                                                                                Bonus: if for some reason we believed that doas was likely to be vulnerable, we could conduct a thorough audit on its 500-some lines of code in an hour or two. What would the same process look like for sudo?

                                                                                                1. -1

                                                                                                  but it’s also important to acknowledge that it has a much bigger back - 100× bigger.

                                                                                                  Sorry but I miss the mass of users pretesting on the streets for tools that have 100x code compare to other tools providing similar functionality.

                                                                                                  1. 10

                                                                                                    What?

                                                                                              2. 10

                                                                                                So you’re saying that 50% of the CVEs in doas would have been prevented by writing it in Rust? Seems like a good reason to write it in Rust.

                                                                                                1. 11

                                                                                                  Another missing point is that Rust is only one of many memory safe languages. Sudo doesn’t need to be particularly performant or free of garbage collection pauses. It could be written in your favorite GCed language like Go, Java, Scheme, Haskell, etc. Literally any memory safe language would be better than C for something security-critical like sudo, whether we are trying to build a featureful complex version like sudo or a simpler one like doas.

                                                                                                  1. 2

                                                                                                    Indeed. And you know, Unix in some ways have been doing this for years anyway with Perl, python and shell scripts.

                                                                                                    1. 2

                                                                                                      I’m not a security expert, so I’m be happy to be corrected, but if I remember correctly, using secrets safely in a garbage collected language is not trivial. Once you’ve finished working with some secret, you don’t necessarily know how long it will remain in memory before it’s garbage collected, or whether it will be securely deleted or just ‘deallocated’ and left in RAM for the next program to read. There are ways around this, such as falling back to manual memory control for sensitive data, but as I say, it’s not trivial.

                                                                                                      1. 2

                                                                                                        That is true, but you could also do the secrets handling in a small library written in C or Rust and FFI with that, while the rest of your bog-standard logic not beholden to the issues that habitually plague every non-trivial C codebase.

                                                                                                        1. 2

                                                                                                          Agreed.

                                                                                                          Besides these capabilities, ideally a language would also have ways of expressing important security properties of code. For example, ways to specify that a certain piece of data is secret and ensure that it can’t escape and is properly overwritten when going out of scope instead of simply being dropped, and ways to specify a requirement for certain code to use constant time to prevent timing side channels. Some languages are starting to include things like these.

                                                                                                          Meanwhile when you try to write code with these invariants in, say, C, the compiler might optimize these desired constraints away (overwriting secrets is a dead store that can be eliminated, the password checker can abort early when the Nth character of the hash is wrong, etc) because there is no way to actually express those invariants in the language. So I understand that some of these security-critical things are written in inline assembly to prevent these problems.

                                                                                                          1. 1

                                                                                                            overwriting secrets is a dead store that can be eliminated

                                                                                                            I believe that explicit_bzero(3) largely solves this particular issue in C.

                                                                                                            1. 1

                                                                                                              Ah, yes, thanks!

                                                                                                              It looks like it was added to glibc in 2017. I’m not sure if I haven’t looked at this since then, if the resources I was reading were just not up to date, or if I just forgot about this function.

                                                                                                  2. 8

                                                                                                    I do think high complexity is the source of many problems in sudo and that doas is a great alternative to avoid many of those issues.

                                                                                                    I also think sudo will continue being used by many people regardless. If somebody is willing to write an implementation in Rust which might be just as complex but ensures some level of safety, I don’t see why that wouldn’t be an appropriate solution to reducing the attack surface. I certainly don’t see why we should avoid discussing Rust just because an alternative to sudo exists.

                                                                                                    1. 2

                                                                                                      Talking about Rust as an alternative is missing the forest for the memes. Rust is a viral language (in the sense of internet virality), and a brain worm that makes us all want to talk about it. But in actual fact, C is not the main reason why anything is broken - complexity is. We could get much more robust and reliable software if we focused on complexity, but instead everyone wants to talk about fucking Rust. Rust has its own share of problems, chief among them its astronomical complexity. Rust is not a moral imperative, and not even the best way of solving these problems, but it does have a viral meme status which means that anyone who sees through its bullshit has to proactively fend off the mob.

                                                                                                      1. 32

                                                                                                        But in actual fact, C is not the main reason why anything is broken - complexity is.

                                                                                                        Offering opinions as facts. The irony of going on to talk about seeing through bullshit.

                                                                                                        1. 21

                                                                                                          I don’t understand why you hate Rust so much but it seems as irrational as people’s love for it. Rust’s main value proposition is that it allows you to write more complex software that has fewer bugs, and your point is that this is irrelevant because the software should just be less complex. Well I have news for you, software is not going to lose any of its complexity. That’s because we want software to do stuff, the less stuff it does the less useful it becomes, or you have to replace one tool with two tools. The ecosystem hasn’t actually become less complex when you do that, you’re just dividing the code base into two chunks that don’t really do what you want. I don’t know why you hate Rust so much to warrant posting anywhere the discussion might come up, but I would suggest if you truly cannot stand it that you use some of your non-complex software to filter out related keywords in your web browser.

                                                                                                          1. 4

                                                                                                            Agree with what you’ve wrote, but just to pick at a theme that’s bothering me on this thread…

                                                                                                            I don’t understand why you hate Rust so much but it seems as irrational as people’s love for it.

                                                                                                            This is obviously very subjective, and everything below is anecdotal, but I don’t agree with this equivalence.

                                                                                                            In my own experience, everyone I’ve met who “loves” or is at least excited about rust seems to feel so for pretty rational reasons: they find the tech interesting (borrow checking, safety, ML-inspired type system), or they enjoy the community (excellent documentation, lots of development, lots of online community). Or maybe it’s their first foray into open source, and they find that gratifying for a number of reasons. I’ve learned from some of these people, and appreciate the passion for what they’re doing. Not to say they don’t exist, but I haven’t really seen anyone “irrationally” enjoy rust - what would that mean? I’ve seen floating around a certain spiteful narrative of the rust developer as some sort of zealous online persona that engages in magical thinking around the things rust can do for them, but I haven’t really seen this type of less-than-critical advocacy any more for rust than I have seen for other technologies.

                                                                                                            On the other hand I’ve definitely seen solid critiques of rust in terms of certain algorithms being tricky to express within the constraints of the borrow checker, and I’ve also seen solid pushback against some of the guarantees that didn’t hold up in specific cases, and to me that all obviously falls well within the bounds of “rational”. But I do see a fair amount of emotionally charged language leveled against not just rust (i.e. “bullshit” above) but the rust community as well (“the mob”), and I don’t understand what that’s aiming to accomplish.

                                                                                                            1. 3

                                                                                                              I agree with you, and I apologize if it came across that I think rust lovers are irrational - I for one am a huge rust proselytizer. I intended for the irrationality I mentioned to be the perceived irrationality DD attributes to the rust community

                                                                                                              1. 2

                                                                                                                Definitely no apology needed, and to be clear I think the rust bashing was coming from elsewhere, I just felt like calling it to light on a less charged comment.

                                                                                                              2. 1

                                                                                                                I think the criticism isn’t so much that people are irrational in their fondness of Rust, but rather that there are some people who are overly zealous in their proselytizing, as well as a certain disdain for everyone who is not yet using Rust.

                                                                                                                Here’s an example comment from the HN thread on this:

                                                                                                                Another question is who wants to maintain four decades old GNU C soup? It was written at a different time, with different best practices.

                                                                                                                In some point someone will rewrite all GNU/UNIX user land in modern Rust or similar and save the day. Until this happens these kind of incidents will happen yearly.

                                                                                                                There are a lot of things to say about this comment, and it’s entirely false IMO, but it’s not exactly a nice comment, and why Rust? Why not Go? Or Python? Or Zig? Or something else.

                                                                                                                Here’s another one:

                                                                                                                Rust is modernized C. You are looking for something that already exists. If C programmers would be looking for tools to help catch bugs like this and a better culture of testing and accountability they would be using Rust.

                                                                                                                The disdain is palatable in this one, and “Rust is modernized C” really misses the mark IMO; Rust has a vastly different approach. You can consider this a good or bad thing, but it’s really not the only approach towards memory-safe programming languages.


                                                                                                                Of course this is not representative for the entire community; there are plenty of Rust people that I like and have considerably more nuanced views – which are also expressed in that HN thread – but these comments certainly are frequent enough to give a somewhat unpleasant taste.

                                                                                                              3. 2

                                                                                                                While I don’t approve of the deliberately inflammatory form of the comments, and don’t agree with the general statement that all complexity is eliminateable, I personally agree that, in this particular case, simplicity > Rust.

                                                                                                                As a thought experiment, world 1 uses sudo-rs as a default implementation of sudo, while world 2 uses 500 lines of C which is doas. I do think that world 2 would be generally more secure. Sure, it’ll have more segfaults, but fewer logical bugs.

                                                                                                                I also think that the vast majority of world 2 populace wouldn’t notice the absence of advanced sudo features. To be clear, the small fraction that needs those features would have to install sudo, and they’ll use the less tested implementation, so they will be less secure. But that would be more than offset by improved security of all the rest.

                                                                                                                Adding a feature to a program always has a cost for those who don’t use this feature. If the feature is obscure, it might be overall more beneficial to have a simple version which is used by the 90% of the people, and a complex for the rest 10%. The 10% would be significantly worse off in comparison to the unified program. The 90% would be slightly better off. But 90% >> 10%.

                                                                                                                1. 2

                                                                                                                  Rust’s main value proposition is that it allows you to write more complex software that has fewer bugs

                                                                                                                  I argue that it’s actually that it allows you to write fast software with fewer bugs. I’m not entirely convinced that Rust allows you to manage complexity better than, say, Common Lisp.

                                                                                                                  That’s because we want software to do stuff, the less stuff it does the less useful it becomes

                                                                                                                  Exactly. Software is written for people to use. (technically, only some software - other software (such as demoscenes) is written for the beauty of it, or the enjoyment of the programmer; but in this discussion we only care about the former)

                                                                                                                  The ecosystem hasn’t actually become less complex when you do that

                                                                                                                  Even worse - it becomes more complex. Now that you have two tools, you have two userbases, two websites, two source repositories, two APIs, two sets of file formats, two packages, and more. If the designs of the tools begin to differ substantially, you have significantly more ecosystem complexity.

                                                                                                                  1. 2

                                                                                                                    You’re right about Rust value proposition, I should have added performance to that sentence. Or, I should have just said managed language, because as another commenter pointed out Rust is almost irrelevant to this whole conversation when it comes to preventing these type of CVEs

                                                                                                                  2. 1

                                                                                                                    The other issue is that it is a huge violation of principle of least privilege. Those other features are fine, but do they really need to be running as root?

                                                                                                              4. 7

                                                                                                                Just to add to that: In addition to having already far too much complexity, it seems the sudo developers have a tendency to add even more features: https://computingforgeeks.com/better-secure-new-sudo-release/

                                                                                                                Plugins, integrated log server, TLS support… none of that are things I’d want in a tool that should be simple and is installed as suid root.

                                                                                                                (Though I don’t think complexity vs. memory safety are necessarily opposed solutions. You could easily imagine a sudo-alike too that is written in rust and does not come with unnecessary complexity.)

                                                                                                                1. 4

                                                                                                                  What’s wrong with EBNF and how is it related to security? I guess you think EBNF is something the user shouldn’t need to concern themselves with?

                                                                                                                  1. 6

                                                                                                                    There’s nothing wrong with EBNF, but there is something wrong with relying on it to explain an end-user-facing domain-specific configuration file format for a single application. It speaks to the greater underlying complexity, which is the point I’m making here. Also, if you ever have to warn your users not to despair when reading your docs, you should probably course correct instead.

                                                                                                                    1. 2

                                                                                                                      Rewrite: The point that you made in your original comment is that sudo has too many features (disguising it as a point about complexity). The manpage snippet that you’re referring to has nothing to do with features - it’s a mix between (1) the manpage being written poorly and (2) a bad choice of configuration file format resulting in accidental complexity increase (with no additional features added).

                                                                                                                    2. 1

                                                                                                                      EBNF as a concept aside; the sudoers manpage is terrible.

                                                                                                                    3. 3

                                                                                                                      Hello, I am here to derail the Rust discussion before it gets started.

                                                                                                                      I am not sure what you are trying to say, let me guess with runaway complexity.

                                                                                                                      • UNIX is inherently insecure and it cannot be made secure by any means
                                                                                                                      • sudo is inherently insecure and it cannot be made secure by any means

                                                                                                                      Something else maybe?

                                                                                                                      1. 4

                                                                                                                        Technically I agree with both, though my arguments for the former are most decidedly off-topic.

                                                                                                                        1. 5

                                                                                                                          Taking Drew’s statement at face value: There’s about to be another protracted, pointless argument about rewriting things in rust, and he’d prefer to talk about something more practically useful?

                                                                                                                          1. 7

                                                                                                                            I don’t understand why you would care about preventing a protracted, pointless argument on the internet. Seems to me like trying to nail jello to a tree.

                                                                                                                        2. 3

                                                                                                                          This is a great opportunity to promote doas. I use it everywhere these days, and though I don’t consider myself any sort of Unix philosophy purist, it’s a good example of “do one thing well”. I’ll call out Ted Unangst for making great software. Another example is signify. Compared to other signing solutions, there is much less complexity, much less attack surface, and a far shallower learning curve.

                                                                                                                          I’m also a fan of tinyssh. It has almost no knobs to twiddle, making it hard to misconfigure. This is what I want in security-critical software.

                                                                                                                          Relevant link: Features Are Faults.

                                                                                                                          All of the above is orthogonal to choice of implementation language. You might have gotten a better response in the thread by praising doas and leaving iron oxide out of the discussion. ‘Tis better to draw flies with honey than with vinegar. Instead, you stirred up the hornets’ nest by preemptively attacking Rust.

                                                                                                                          PS. I’m a fan of your work, especially Sourcehut. I’m not starting from a place of hostility.

                                                                                                                          1. 3

                                                                                                                            If you want programs to be more secure, stable, and reliable, the key metric to address is complexity. Rewriting it in Rust is not the main concern.

                                                                                                                            Why can’t we have the best of both worlds? Essentially a program copying the simplicity of doas, but written in Rust.

                                                                                                                            1. 2

                                                                                                                              Note that both sudo and doas originated in OpenBSD. :)

                                                                                                                              1. 9

                                                                                                                                Got a source for the former? I’m pretty sure sudo well pre-dates OpenBSD.

                                                                                                                                Sudo was first conceived and implemented by Bob Coggeshall and Cliff Spencer around 1980 at the Department of Computer Science at SUNY/Buffalo. It ran on a VAX-11/750 running 4.1BSD. An updated version, credited to Phil Betchel, Cliff Spencer, Gretchen Phillips, John LoVerso and Don Gworek, was posted to the net.sources Usenet newsgroup in December of 1985.

                                                                                                                                The current maintainer is also an OpenBSD contributor, but he started maintaining sudo in the early 90s, before OpenBSD forked from NetBSD. I don’t know when he started contributing to OpenBSD.

                                                                                                                                So I don’t think it’s fair to say that sudo originated in OpenBSD :)

                                                                                                                                1. 1

                                                                                                                                  Ah, looks like I was incorrect. I misinterpreted OpenBSD’s innovations page. Thanks for the clarification!

                                                                                                                            1. 1

                                                                                                                              As an interested non-Go-user, could someone tell me what this offers that pkg.go.dev doesn’t? Forking and continuing an existing codebase that’s being put down by its original developers? Offering a third-party version of a service that isn’t controlled by Google? Something else?

                                                                                                                              1. 5

                                                                                                                                All of those. It’s also simpler and lighter weight than pkg.go.dev, and gives people a non-Google route to contribute (i.e. no CLA). The codebase is also cleaner and easier to deploy yourself, for example on an intranet, than pkg.go.dev is, and even more so than the original godoc.org codebase after our efforts to clean it up.

                                                                                                                                1. 1

                                                                                                                                  and gives people a non-Google route to contribute (i.e. no CLA)

                                                                                                                                  Thank you so much for this.

                                                                                                                                2. 5

                                                                                                                                  Compare https://pkg.go.dev/github.com/BurntSushi/xgbutil vs https://godocs.io/github.com/BurntSushi/xgbutil

                                                                                                                                  One has docs. The other doesn’t.

                                                                                                                                  (This is not an invitation to discuss licensing choices.)

                                                                                                                                1. 1

                                                                                                                                  Will it be feasible to pull new content from pkg.go.dev? Or is this a sort of soft fork of the Go ecosystem?

                                                                                                                                  Either way I am delighted with this effort. I am always apprehensive about Go due to the specter of control by Google; building independent community infrastructure is an important bulwark against that. Fighting the good fight!

                                                                                                                                  Would also like to register my disgust at the .dev TLD.

                                                                                                                                  1. 3

                                                                                                                                    This doesn’t host packages, it pulls packages from other sources and displays documentation for them. This is also true of pkg.go.dev. Neither have forked the Go ecosystem.

                                                                                                                                    1. 1

                                                                                                                                      Who maintains the list of where to pull from?

                                                                                                                                      1. 3

                                                                                                                                        It pulls any package from any source as soon as you type the import path into the search box, or navigate to its URL. There’s no global package list for Go.

                                                                                                                                        1. 1

                                                                                                                                          So could it just as well be a client program that you run on your own machine?

                                                                                                                                          1. 1

                                                                                                                                            Yes. Every package on godocs.io works on pkg.go.dev and vice versa.

                                                                                                                                            1. 1

                                                                                                                                              Almost seems like people would be better off with a local client program rather than a web app, as with email clients vs. web mail.

                                                                                                                                              1. 1

                                                                                                                                                Perhaps! The main advantage is the ability to link to it from anywhere. Many projects link to it from their README files, or their website, lots of people answer questions on forums or IRC by linking to a specific function’s docs, and so on.

                                                                                                                                                1. 1

                                                                                                                                                  Then you’d want a standard URL scheme that is recognized by every client, maybe something like godoc://9fans.net/go/acme

                                                                                                                                                  By the way, looking at https://9fans.net/go/acme I am a bit confused. Where does godoc.org pull from if the source redirects to godoc.org?

                                                                                                                                                  1. 1

                                                                                                                                                    Oh, I don’t know, installing a new global URL handler just for documentation isn’t really to my taste. I think the web is fine for this situation.

                                                                                                                                                    Regarding 9fans.net/go/acme, try this: curl https://9fans.net/go/acme

                                                                                                                                                    1. 1

                                                                                                                                                      I understand using the web for the sake of practicality, but having to maintain an entire web app with HTTPS and Postres just so users don’t have to install a client application feels like an ugly artifact of the web which could be overcome.

                                                                                                                                                      Maybe a better approach would be to design a protocol that could apply to a broader class of documents but would work just as well for godocs.

                                                                                                                                                      1. 2

                                                                                                                                                        Nothing stops you from running go doc 9fans.net/go/acme locally and achieving the same effect. It’s a lot harder to link to from the web however.

                                                                                                                                                        1. 1

                                                                                                                                                          Web browsers are fickle: RSS feeds were viewable until recently. Gopher support was possible with an addon until Firefox Quantum. FTP links work now but won’t for much longer.

                                                                                                                                                          Keeping up with the Chrome hamster wheel does not feel worth while, though practical concerns may require it.

                                                                                                                                  1. 3

                                                                                                                                    The only thing that is kinda a problem is the fact that I believe that godoc doesn’t support modules/displaying help for a specific version, not that I would need this often as in Go you usually just use the latest one and bump it often.

                                                                                                                                    1. 2

                                                                                                                                      We’re interested in developing those features. Want to help?

                                                                                                                                    1. 7

                                                                                                                                      Hopefully it will soon support Go modules, as properly supporting separate versions is one of the biggest improvements pkg.go.dev has.

                                                                                                                                      1. 5

                                                                                                                                        Aye, I intend on looking into this feature when I next need something to do on a rainy day. I think someone else was interested as well. If you’d like to see it sooner rather than later, you might find it easy to do - the gddo codebase, once purged of Google crap, is reasonably accesisble to the typical Go programmer.

                                                                                                                                        1. 2

                                                                                                                                          I read through the changelog when I found the project last week; I had previously done some work to cleanup gddo for our internal deployment a few years ago. You and akavel have done a much better job than I (though I was trying to retain AppEngine support, to try to upstream interfaces, alas).

                                                                                                                                          I thought about adding support for modules, but the implementation details haven’t formed in my mind just yet.

                                                                                                                                          1. 1

                                                                                                                                            I didn’t touch the codebase, you probably meant Alexey Yerin, Adnan Maolood and others.

                                                                                                                                            1. 1

                                                                                                                                              Yes. I copy and pasted from the wrong spot. I’m sorry for the misattribution.

                                                                                                                                      1. 4

                                                                                                                                        Eh, this is something I’ve always wondered: if I release software under the AGPL/GPLV3, does that bind only the licensees and not me, given that I own the copyright? I could develop proprietary enterprise features and not be obliged to release their sources, right, but not be bound by the license. This is allowed, right? So in theory I could have the following business model:

                                                                                                                                        1. The software is free software under AGPL
                                                                                                                                        2. I register a trademark for the software, so others can’t redistribute the software under the same name without explicit permissions
                                                                                                                                        3. I offer closed source proprietary “enterprise features” that don’t really respect the AGPL, but I don’t have to
                                                                                                                                        4. I can also offer commercial proprietary licenses to users that don’t want to license under the AGPL

                                                                                                                                        Right, this is a bit lawful evil, and it wouldn’t be my ideal business model. I would prefer offering services and support (hosted SaaS offerings etc.) instead of this but the model above is definitely easier to make money with, if it’s legal.

                                                                                                                                        1. 10

                                                                                                                                          Eh, this is something I’ve always wondered: if I release software under the AGPL/GPLV3, does that bind only the licensees and not me, given that I own the copyright? I could develop proprietary enterprise features and not be obliged to release their sources, right, but not be bound by the license. This is allowed, right?

                                                                                                                                          Yes, but there’s a big catch: if you have accepted contributions from third parties who licensed their changes under the GPL and retain copyright, those changes virally “infect” your own code within the project, too. So you can distribute a project under a GPL-family license and change the license terms at your pleasure, if and only if you never accept outside contributions. And that’s a good thing IMO.

                                                                                                                                          1. 1

                                                                                                                                            Yes, right, that’s the caveat. Suppose I get contributors to sign some sort of CLA granting me the copyright? Again, lawful evil, but that would be the way around it?

                                                                                                                                            For this very reason, CLAs are evil incarnate, because such contributors would essentially be working for me for free.

                                                                                                                                            The only “CLAs” I’ve ever signed are the FSF ones. The FSF is probably the only entity in the world to which I’d gladly transfer my copyright.

                                                                                                                                        1. 7

                                                                                                                                          That sounds suspiciously like one of the main motivations for zig, from the Zig main page:

                                                                                                                                          Focus on debugging your application rather than debugging your programming language knowledge.
                                                                                                                                          
                                                                                                                                              No hidden control flow.
                                                                                                                                              No hidden memory allocations.
                                                                                                                                              No preprocessor, no macros.
                                                                                                                                          

                                                                                                                                          @ddevault, any thoughts there?

                                                                                                                                          1. 2

                                                                                                                                            Zig does well in this regard, but comptime - one of my biggest complaints about Zig - betrays this somewhat.

                                                                                                                                            1. 3

                                                                                                                                              Really honestly curious what aspect of comptime are you thinking of here? I haven’t dove deep enough into Zig yet to have my own solid opinions, so I’m really interested esp. in potential pitfalls to watch out for.

                                                                                                                                              1. 2

                                                                                                                                                It’s hard to reason about when a line of code will run - at runtime? Build time? What conseqeunces will that have on the program’s behavior? What if one of those consequences leads to, for example, a timing attack?

                                                                                                                                                1. 7

                                                                                                                                                  Why do you need to know whether a line of code will run at runtime or compile-time? Even with C and similar languages the optimizer may turn a runtime-known value into a compile-time known value. If you’re concerned about a timing attack, it sounds like you would be interested in this issue, which, again, plagues all C-like languages.

                                                                                                                                                  1. 2

                                                                                                                                                    I wonder whether Zig could support E-style auditors, which are objects that can prove arbitrary properties about other objects. During an audition, an auditor receives an AST for an object, and the binding guards for incoming values (but not the values themselves). This sounds abstract and fancy, but for Zig, the binding guards would ultimately just be type annotations; Zig-compatible auditors would simply receive a typed AST for a routine.

                                                                                                                                                    1. 2

                                                                                                                                                      I don’t want to get into the weeds with you about Zig today, but I think it’s fair to state that the comptime semantics of Zig far exceed those of e.g. C, otherwise they would not be a compelling value-add over C when pitching Zig; and mention that adding a language-level feature specifically to address constant time operations is an unpleasant solution to my sensibilities.

                                                                                                                                                      1. 3

                                                                                                                                                        adding a language-level feature specifically to address constant time operations is an unpleasant solution to my sensibilities

                                                                                                                                                        For any optimizing / code-transforming compiler, you need some feature to opt out in some way in order to guarantee constant time operation. Otherwise your code as written can be changed in ways that break your constant time guarantees.

                                                                                                                                                        In-line assembly solves the problem, albeit in an architecture dependent way. Arguably this could be a linker feature rather than language level. Regardless, you need something.

                                                                                                                                                2. 1

                                                                                                                                                  Would you say it’s not worth the tradeoff?

                                                                                                                                                  1. 1

                                                                                                                                                    Yes.

                                                                                                                                                    1. 1

                                                                                                                                                      Isn’t that up to you to decide within the context of your specific constraints?

                                                                                                                                                1. 17

                                                                                                                                                  Here’s the summary of the licencing change:

                                                                                                                                                  We are moving our Apache 2.0-licensed source code in Elasticsearch and Kibana to be dual licensed under Server Side Public License (SSPL) and the Elastic License, giving users the choice of which license to apply. This license change ensures our community and customers have free and open access to use, modify, redistribute, and collaborate on the code. It also [restricts] cloud service providers from offering Elasticsearch and Kibana as a service without contributing back

                                                                                                                                                  I’m not clear on how this pairs with Drew’s summary of:

                                                                                                                                                  Elastic is no longer open source [and] has spit in the face of every single one of 1,573 contributors

                                                                                                                                                  The code that those folk wrote and contributed to still exists, right? Elastic is saying that further code that they themselves distribute, which includes code from those 1,573 contributors, will now be under some other licence. And I’ll grant that that’s not great, but Elastic isn’t sshing into all of your servers and deleting the code running on them. They aren’t retroactively undistributing or relicencing old code. They’re just releasing new code that they write under some other, less good, licence. Potential contributors going forward know that those are the terms going forward but the rug wasn’t pulled out from other them and they’re free to contribute to a fork instead, under whatever terms they like.

                                                                                                                                                  I’m undoutedly missing all of the context so I feel like I’m missing out on what’s happening that’s so terrible.

                                                                                                                                                  Yesterday I was offering beer at my bar for free and some folk kindly helped me develop the recipe and brew it. Today I’m going to start charging for it, and hey if you want to keep helping you’re welcome to but the old recipe’s still on the door outside if you want it. I don’t follow what’s worth being up in arms about.

                                                                                                                                                  1. 7

                                                                                                                                                    I was just trying to answer a similar comment in the HN thread when I got rate limited, so convenient that I can almost paste it into this thread instead :)

                                                                                                                                                    The community is built around the upstream project, and has a lot invested in their continued existence as an open source entity. This move will force the community to start from scratch to build a new entity which continues to meet those guarantees - something Elastic plans to capitalize on by capturing more users on its paid offering while the open source community struggles to set up a new entity at the same level of sophistication from scratch.

                                                                                                                                                    What Elastic done was within their rights, but it wasn’t right - so to speak.

                                                                                                                                                    1. 5

                                                                                                                                                      when I got rate limited

                                                                                                                                                      Fun fact, HN does not actually have rate limiting for regular users. It’s only applied to a few users who are flagged by the mods as annoying, but where a ban would cause more trouble than it’s worth. Beyond that there’s the shadowban or insta-flagging of comments.

                                                                                                                                                      If you get rate limited, it might be worth sending an email to the mods and asking what to change to get rid of it - that said, being rate limited on HN is something to be proud of ;)

                                                                                                                                                      1. 13
                                                                                                                                                        1. 9

                                                                                                                                                          https://jcs.org/2012/06/13/hellbanned_from_hacker_news

                                                                                                                                                          The last sentence of that post is the kicker.

                                                                                                                                                          1. 4
                                                                                                                                                          2. 6

                                                                                                                                                            They have a clickthrough rate limit, and because everything is form-based every click is a new page load. You can hit it pretty easily by hitting the comments link on like 10 posts. I’m not sure sure on how aggressive it is though, or if that’s what Drew is talking about.

                                                                                                                                                            1. 1

                                                                                                                                                              I can middle-click open the comments of the entire frontpage as fast as I can click and I don’t get rate limited. That said, the rate-limit for “troublesome” users applies not just to actions, but also to comments.

                                                                                                                                                              I do have enough karma to be able to downvote, though, so maybe that helps.

                                                                                                                                                              Your experience sounds like you’re on one of the lower ban levels.

                                                                                                                                                              1. 2

                                                                                                                                                                I wonder if it’s load-based. I was able to click through to the comments for nearly every article on the front page without issue. It may also depend on if you commented recently or something…

                                                                                                                                                      1. 9

                                                                                                                                                        If this were written in C, without knowing anything other than the fact that this code compiles correctly, I can tell you that x and y are numeric types, and the result is their sum. I can even make an educated guess about the CPU instructions which will be generated to perform this task.

                                                                                                                                                        x could be a pointer and y could be an int. A pointer isn’t a numeric type. On most architectures, it kind of is numeric; it stores an address – which is just a numeric index into memory. FWIW I never fully grokked pointers until I learned assembly.

                                                                                                                                                        C hides a lot of details too. That’s what programming languages at a higher level than assembly are all about: abstracting over details.

                                                                                                                                                        Regardless of what language you write in, it’s good to be aware of the performance characteristics of your code. And it’s totally possible to cultivate that sort of awareness, even in the face of things like operator overloading. Start by learning some assembly.

                                                                                                                                                        1. 7

                                                                                                                                                          Not to mention that even if they’re numeric types, then “the result is their sum” can still not be true either - IIRC:

                                                                                                                                                          • if the types are unsigned, this line could be an overflow and thus the result would be sum modulo;
                                                                                                                                                          • even worse, if the types are signed, this line could be an undefined behavior, and thus a spooky eldritch Cthulhu action crawling slowly throughout your app and devouring any logic.

                                                                                                                                                          Also, the malloc line does not check the returned value, so:

                                                                                                                                                          • in case you run close to an out-of-memory condition, the strcpy and strcat could overwrite memory, also doing spooky eldritch Cthulhu corruption action;
                                                                                                                                                          • in case x or y come from your users, they could be maliciously crafted such that strlen could overflow int, leading again to spooky eldritch Cthulhu corruption action, security breach etc. etc. (I think; not 100% sure if some other effect doesn’t accidentally neutralize this, I don’t care enough about C anymore to try and track precisely what would happen there in such case).
                                                                                                                                                          1. 1

                                                                                                                                                            in case you run close to an out-of-memory condition, the strcpy and strcat could overwrite memory, also doing spooky eldritch Cthulhu corruption action;

                                                                                                                                                            if there is no memory, it returns NULL - this is probably just gonna segfault and not corrupt anything.

                                                                                                                                                          2. 5

                                                                                                                                                            I think this comment, and @akavel’s follow-up, are both missing the point. Yes, it’s more complex than my brief summary affords. However, the action is well-defined and constrained by the specification, and you can learn what it entails, then apply that knowledge to your understanding of all C programs. On the contrary, for languages which have operator overloading, you can never fully understand the behavior of the + operator without potentially consulting project-specific code.

                                                                                                                                                            1. 3

                                                                                                                                                              I totally get the point of “spooky action at a distance”, and totally agree with it; I learnt of this idea in programming quite some time ago and do look at it similarly since. I also honestly appreciate the look at #define in C as a feature that can be said to exhibit a similar mechanism; I never thought of it like this, and I find it a really cool and interesting take.

                                                                                                                                                              Yet, I still do find the claims about C’s simplicity that this article makes highly misleading. Especially in this context of subtle hidden difficulties. If it at least provided some disclaimer footnote (or whatever else form given gemini), I could swallow this. But as shown in the comments, some of the sentences are currently just plain wrong.

                                                                                                                                                              Let me phrase this differently: I believe this could be a great article. I could recommend it to others. However, currently I see it as, looking at a whole, an okay-ish article. Some good points, some false claims. I don’t plan to advertise it to others, given the dangerous and unfortunately seductive ideas it also conveys (esp. to non-expert or just would-be C programmers) “piggybacked” with the main argument (e.g. that “C is easy and simple”). I lament that the article is not better. And I’m painfully aware I need to actually warn some people against naively succumbing to the overt simplifications presented in this article. Because I’ve seen so many unaware and sincerely gullible people, and myself have been unaware for ohhhh so long

                                                                                                                                                              1. 1

                                                                                                                                                                I don’t think the omission is consequential to the point of the article. In fact, it criticizes C for its role in this “spooky action at a distance”. The point is not to explain how C works.

                                                                                                                                                                1. 2

                                                                                                                                                                  Given that this reply basically rehashes parts of what both of us already wrote above, and I apparently read some parts of the article differently than you, I’m assuming neither of us nor anyone else will probably gain anything from any further discussion by us on this matter.

                                                                                                                                                              2. 1

                                                                                                                                                                What C code seemingly free of function calls does depends on platform as well, for instance see:

                                                                                                                                                                https://godbolt.org/z/jsWqzM

                                                                                                                                                                Here binary operations on 64-bit operands compiled as 32-bit code may result in function calls. The other example is initializing a structure of a certain size, which may also result in a function call.

                                                                                                                                                              3. 1

                                                                                                                                                                Right. Try this for fun,

                                                                                                                                                                cat <<'.'|cc -x c - && ./a.out && rm a.out
                                                                                                                                                                #include<stdio.h>
                                                                                                                                                                #define P(T) do{T*x=0;printf("%lx\n",(unsigned long)(x+y));}while(0)
                                                                                                                                                                int main(int c,char**v){int y=1;P(char);P(short);P(int);P(long);P(void);return 0;}
                                                                                                                                                                .
                                                                                                                                                                
                                                                                                                                                              1. 9

                                                                                                                                                                IRC revival? As far as I can tell it never died.

                                                                                                                                                                1. 2

                                                                                                                                                                  It did in popular circles. The thinking here was to try to rekindle some interest … and also reimagine what social media might have been like if it involved an account on a Unix box rather than a SPOF owned by a publicly listed, VC-funded, company …

                                                                                                                                                                  1. 2

                                                                                                                                                                    Is “a Unix box” not a SPOF?

                                                                                                                                                                    1. 1

                                                                                                                                                                      Sure, but many Unix boxen wouldn’t be. I meant SPOF in the sense of a single monolithic entity that acts as a choke point for censorship, trolling, etc.

                                                                                                                                                                  2. 2

                                                                                                                                                                    To anybody who thinks IRC is “dead”, see the activity I’ve witnessed over the past 24 hours, excluding bots and grouping consecutive messages from the same nick:

                                                                                                                                                                    Analyzing logs from 2020-12-30 12:32:03.572224 till 2020-12-31 12:32:03.572224
                                                                                                                                                                    total messages: 29579
                                                                                                                                                                    RANK CHANNEL                   MSGS NICKS
                                                                                                                                                                    1.   freenode.##linux          3575 178
                                                                                                                                                                    2.   tilde_chat.#meta          1980 72
                                                                                                                                                                    3.   2600net.#2600             1905 37
                                                                                                                                                                    4.   snoonet.#gnulag           1815 37
                                                                                                                                                                    5.   freenode.#anime           1812 44
                                                                                                                                                                    6.   freenode.#reddit-sysadmin 1506 47
                                                                                                                                                                    7.   ircnet.#worldchat         1287 64
                                                                                                                                                                    8.   slashnet.#xkcd            1167 46
                                                                                                                                                                    9.   efnet.#lrh                1122 76
                                                                                                                                                                    10.  freenode.#reddit          1012 30
                                                                                                                                                                    
                                                                                                                                                                    If you use WeeChat then you can generate this yourself:
                                                                                                                                                                    	https://sr.ht/~seirdy/clogstats
                                                                                                                                                                    	https://github.com/Seirdy/clogstats
                                                                                                                                                                    If you'd like your channel excluded from this list, contact me.
                                                                                                                                                                    My contact info is at https://seirdy.one/about.html and gemini://seirdy.one/about/
                                                                                                                                                                    
                                                                                                                                                                  1. 5

                                                                                                                                                                    This article nicely illustrates how easy the git email flow is for developers/contributors, but how about for maintainers?

                                                                                                                                                                    For instance, how do I easily apply a patch sent to me, if I use the gmail/fastmail web interface? If web-mail is frowned upon, does it work if I use thunderbird/outlook/something-else? Or does it basically require me to use mutt?

                                                                                                                                                                    How about managing different patches from different sources at the same time? Github conveniently suggests you create a local copy of the PR’ed branch, meaning you can easily change between different PRs. How does this work with git email flow? Do I just create those branches myself, or is there an easier way to do it?

                                                                                                                                                                    What about very large patches? I recently applied a code formatter to a project I contribute to, resulting in a 4MB diff. Not all email clients/servers handle arbitrarily large files well.

                                                                                                                                                                    I’ve seen enough descriptions about how git-send-email works from a contributors perspective, but I would really appreciate it, if someone would write down how they then receive and review patches in this flow.

                                                                                                                                                                    1. 6

                                                                                                                                                                      Although his workflow is obviously not representative of most maintainers, Greg K-H’s writeup was nonetheless fascinating.

                                                                                                                                                                      1. 1

                                                                                                                                                                        That does indeed look fascinating, and like it addresses some of my questions. Will take a closer look later.

                                                                                                                                                                      2. 6

                                                                                                                                                                        As long as your e-mail client/interface doesn’t somehow mangle the original message, the simplest solution would probably be to simply copy the message as a whole and then run something like xsel -ob | git am in the repository. I’d reckon this is much easier than setting up some more UNIX-like e-mail client.

                                                                                                                                                                        1. 3

                                                                                                                                                                          For instance, how do I easily apply a patch sent to me, if I use the gmail/fastmail web interface? If web-mail is frowned upon, does it work if I use thunderbird/outlook/something-else? Or does it basically require me to use mutt?

                                                                                                                                                                          You can certainly use mutt or Gnus if you want. Most projects using git-by-email use mailing lists, some are fancy with download buttons to get the patches. Most of the time you can pass the output of curl fetching the mailing list page directly to patch. Quoting the manual

                                                                                                                                                                          patch tries to skip any leading garbage, apply the diff, and then skip any trailing garbage. Thus you could feed an article or message containing a diff listing to patch, and it should work.

                                                                                                                                                                          I’ve done this as recently as this week, when I found an interesting patch to u-boot.

                                                                                                                                                                          If a mailing list or other web view isn’t available, then you can either set one up (patchwork is the defacto standard) or find a client that doesn’t break plaintext emails. Last I checked, receiving patches via Gmail’s web interface was difficult (but sometimes possible if you hunt down the “raw message” link), I haven’t checked Fastmail. If you don’t want to use a TUI or GUI email client, you can configure getmail to only fetch emails from specific folders or labels, and that will create mbox files you can use with git am.

                                                                                                                                                                          How about managing different patches from different sources at the same time? Github conveniently suggests you create a local copy of the PR’ed branch, meaning you can easily change between different PRs. How does this work with git email flow? Do I just create those branches myself, or is there an easier way to do it?

                                                                                                                                                                          Branches are lightweight, create them for however many things you want to work on at the same time. I usually don’t use branches at all, instead using stgit, and just import patches and push and pop them as needed.

                                                                                                                                                                          What about very large patches? I recently applied a code formatter to a project I contribute to, resulting in a 4MB diff. Not all email clients/servers handle arbitrarily large files well.

                                                                                                                                                                          You can breakup that work into smaller chunks, to avoid too much disruption to other patches that might be in flight. Nothing stops you from linking to an external patch, though I would probably prefer instructions on how to run the tool myself, and just forgo the patch in that case.

                                                                                                                                                                          1. 3

                                                                                                                                                                            What about very large patches?

                                                                                                                                                                            git request-pull can be used for this purpose, it generates emails with a URL from which the reviewer can pull the changes. It’s generally used for subsystem maintainers in large projects to merge their (independent) histories upstream, but it can also be used to handle large changes which would be unwieldly in patch form.

                                                                                                                                                                            1. 2
                                                                                                                                                                              1. 1

                                                                                                                                                                                I wondered about the same thing. After a very long unfruitful search on the internet, @fkooman pointed me to git am - Apply a series of patches from a mailbox, which pretty much takes care of everything. It would have been helpful if git-send-email.io had a pointer to that, or maybe I missed it..

                                                                                                                                                                              1. 2

                                                                                                                                                                                Great article, the perspective is good even if you don’t end up building your compiler in exactly these steps. You had me at C and YACC. It’s weird to see a chicken developer hate on C. I thought that was chicken’s superpower. From the highest level of abstraction to the lowest (figuratively at least) it has you covered. (I really like programming in chicken along with C.) To all you vader haters out there, we’ll blow you’re planet up.

                                                                                                                                                                                1. 4

                                                                                                                                                                                  haha, hi :)

                                                                                                                                                                                  C is (or: was? before all the UB exploitation) a nice language to work with, but it is not sacred. It has a lot of issues (and we’ve certainly run into our share of issues particularly due to using it as a compilation target). It is great for low-level operations and for interfacing with existing operating systems. But for prototyping a new compiler? That’s silly. Manual memory management and its impoverished string processing functionality alone should put anyone off of using it for prototyping. At that stage, you don’t need performance or low-level control over anything, really.

                                                                                                                                                                                  1. 3

                                                                                                                                                                                    C’s string handling sucks (“impoversed” is harsh), but you can learn to use it. However, string handling really doesn’t matter for a compiler (really, I’m serious). The only time the compiler ever handles strings is in the lexer, and it’s not especially more difficult in C to convert input text into a token stream. From that point forward, you’re just working with types.

                                                                                                                                                                                    The language which inspired this post has a lexer which is about 1 kloc, and literally none of the remaining stages use strings at all. The last stage, emit, does deal with strings insofar as it translates an AST back into text, but that’s accomplished entirely with fprintf calls and it’s quite straightforward.

                                                                                                                                                                                1. 13

                                                                                                                                                                                  As @andyc points out, I think it’s bad advice to tell beginner programming language creators to start with C/C++ and Lex and Yacc. The words “extraneous cognitive load” rush to mind immediately, especially for the first goal of a “prototype implementation”. When I first started working on compilers in high school, I only knew C and some C++, and I wasted so much time writing the most minute algorithms and transformations, messing with pointers and memory allocation and reference cycles and segmentation faults. If you want to design a brand new language, these should be the least of your worries. Dealing with these issues just distracts you from designing your actual language.

                                                                                                                                                                                  It was not until I took a programming languages course in university that I realized just how much easier it is to write programming languages in something like Haskell or OCaml: I can write an imperative language with functions, closures, different data types and much more in about an hour; it would still take me days to make the same work in C++, and although I’m no professional C++ developer, I doubt that it’s purely a matter of skill. The ability to rapidly prototype and explore ideas for your language is a huge positive, since it drastically lowers the cost of making an incorrect design decision. Even Rust, a very much imperative language, had OCaml as its first compiler implementation language.

                                                                                                                                                                                  I wonder if the article’s author has ever tried functional languages, especially as metalanguages, and if they did, I wonder what they thought of them.

                                                                                                                                                                                  1. 9

                                                                                                                                                                                    I agree that C/C++ and Lex/Yacc is a bad idea. It reeks of cargo culting – there is generally no reason to write any new code for a general purpose machine in C/C++. Sure, you can LARP the 70ies, but it detracts massively from the article.

                                                                                                                                                                                    Also, writing a self-hosting compiler is neither a requirement nor particularly useful; it certainly doesn’t deserve it’s own bullet point.

                                                                                                                                                                                    I sincerely hope people avoid doing the things that are recommended in the article. Perhaps “(1990)” could be added to the title, such that people see at a glance that the advice is 30 years out of date?

                                                                                                                                                                                    1. 7

                                                                                                                                                                                      Author here. First of all, C and C++ are completely different programming languages. Please do not conflate them. I didn’t mention C++ at all in my article, and I will reply to you from the C perspective only.

                                                                                                                                                                                      I think it’s bad advice to tell beginner programming language creators to start with C/C++ and Lex and Yacc. The words “extraneous cognitive load” rush to mind immediately, especially for the first goal of a “prototype implementation”. When I first started working on compilers in high school

                                                                                                                                                                                      Quotes from my article:

                                                                                                                                                                                      1. “Designing and implementing a new programming language from scratch is one of the most challenging tasks a programmer can undertake.”
                                                                                                                                                                                      2. “This post is targeted at motivated programmers who want to make a serious attempt at designing a useful programming language.”
                                                                                                                                                                                      3. “You [the reader] already know a few programming languages[..]”

                                                                                                                                                                                      This is not geared towards “beginners”. This is geared towards someone who is making a serious attempt at a language which they believe will be practical and useful beyond their own personal needs as a learning project. I don’t expect such a programmer to get bogged down in learning C, because I expect such a programmer to have already learned C, perhaps to an expert level.

                                                                                                                                                                                      The ability to rapidly prototype and explore ideas for your language is a huge positive, since it drastically lowers the cost of making an incorrect design decision.

                                                                                                                                                                                      I spoke to much of the same advantages in the article when talking about the sacrificial implementation. You can write this in whatever you want, and OCaml is a fine choice. I only mention C for the second compiler, which would evolve to become the bootstrap compiler for your future, self-hosted toolchain. Having a bootstrap compiler written in C has a lot of practical benefits, which I could elaborate on more in a future article, but at this point you are no longer in the prototyping phase where the advantage of rapid design iteration is entirely necessary. Hell, I also stated that you should be writing a specification at this point.

                                                                                                                                                                                      I wonder if the article’s author has ever tried functional languages, especially as metalanguages, and if they did, I wonder what they thought of them.

                                                                                                                                                                                      Yes, I have. I hate them passionately.

                                                                                                                                                                                      1. 6

                                                                                                                                                                                        This is not geared towards “beginners”. This is geared towards someone who is making a serious attempt at a language which they believe will be practical and useful beyond their own personal needs as a learning project. I don’t expect such a programmer to get bogged down in learning C, because I expect such a programmer to have already learned C

                                                                                                                                                                                        Knowing languages doesn’t make you any better at designing programming languages. You are, for all intents and purposes, still a “beginner” when it comes to compiler implementation until you’ve written a compiler. So, you are still talking to people who are “beginners”, just not beginner programmers. Furthermore, it’s not about “not knowing C”. You won’t get bogged down because you don’t know the language; you will get bogged down because the language is made to require far more precision than is needed for implementing a language.

                                                                                                                                                                                        You can write this in whatever you want, and OCaml is a fine choice. I only mention C for the second compiler . . .

                                                                                                                                                                                        I assumed that you would want the sacrificial implementation written in C because you mentioned Yacc, which ismade for the C and C++ languages.

                                                                                                                                                                                        Yes, I have. I hate them passionately.

                                                                                                                                                                                        May I ask why? And also, may I ask why you think C is good for writing languages?

                                                                                                                                                                                        1. 5

                                                                                                                                                                                          Knowing languages doesn’t make you any better at designing programming languages.

                                                                                                                                                                                          But not knowing languages does make you worse at this.

                                                                                                                                                                                          You are, for all intents and purposes, still a “beginner” when it comes to compiler implementation until you’ve written a compiler.

                                                                                                                                                                                          This is why you should be prepared to throw your first one away. It has nothing to do with your C skills.

                                                                                                                                                                                          You won’t get bogged down because you don’t know the language; you will get bogged down because the language is made to require far more precision than is needed for implementing a language.

                                                                                                                                                                                          I don’t agree that this is a symptom of using C.

                                                                                                                                                                                          I assumed that you would want the sacrificial implementation written in C because you mentioned Yacc, which ismade for the C and C++ languages.

                                                                                                                                                                                          I happened to write my sacrificial compiler in C, using yacc, but any parser generator would do. My article only mentions it as one possible option for this use-case. I understand how you could interpret it differently, though. To be clear: I don’t think it matters what language your first compiler is written in. You’re going to throw it away.

                                                                                                                                                                                          May I ask why [you don’t like functional programming languages]?

                                                                                                                                                                                          This is a deeper question than I have the patience to answer in an unrelated discussion.

                                                                                                                                                                                          May I ask why you think C is good for writing languages?

                                                                                                                                                                                          Again, it doesn’t really matter what language you use for the sacrificial compiler. But I do think that C has some advantages for a second compiler. It is universally supported, which makes your language easy to bootstrap. It’s also quite fast, so as long as your algorithms are fast then your compiler will be fast. Writing parsers, representing an AST, representing your type systems, this is all fairly straightforward to do in C. It does have its limitations and it would get annoying if you tried to solve every problem in the language design space with C, but that don’t really matter for a simple compiler. Simple is all this compiler will ever need to be, because its primary purpose is to build your third compiler. Your third compiler should be written in your own language, and at that point it’s on you to make your langauge suitable for any level of advanced compiler development.

                                                                                                                                                                                          1. 3

                                                                                                                                                                                            May I ask why [you don’t like functional programming languages]?

                                                                                                                                                                                            This is a deeper question than I have the patience to answer in an unrelated discussion.

                                                                                                                                                                                            As a regular (and happy) OCaml user, I’d be interested in your opinion as well. If you could write something about it on your blog some day, that would be great.

                                                                                                                                                                                        2. 4

                                                                                                                                                                                          I hate [“functional languages, especially as metalanguagues”] passionately.

                                                                                                                                                                                          Can I ask for more information on this perspective? I’ve found functional programming protects me from so many bugs when mutating state, and I’m a little surprised that you are not on the hype train slash bandwagon.

                                                                                                                                                                                      1. 23

                                                                                                                                                                                        I hope no one actually takes this article seriously, because:

                                                                                                                                                                                        • I can’t think of ANY language that was implemented this way (writing 3 compilers, the second in C, the 3rd being bootstrapped). Bootstrapping is something I’ve researched [1], and I’m pretty confident that the number of projects that have done this is exactly zero.
                                                                                                                                                                                        • It doesn’t appear that the author has done anything close to this himself. It’s basically hand waving. (Though I definitely respect the other things he’s done; I’m using parts of sourcehut recently and it’s great)

                                                                                                                                                                                        A partial list of what’s missing (in my experience, which I don’t claim to be representative):

                                                                                                                                                                                        • Underestimating how much more realistic it is to write 2 compilers than 3 (I actually made somewhat of the same mistake early in Oil, i.e. thinking I could improve a Python compiler more than was feasible in a reasonable amount of time)
                                                                                                                                                                                        • Underestimating how bad a language C is to write compilers. C++ is not great but it’s better than C in this domain.
                                                                                                                                                                                        • Not mentioning the requirement to make your language useful in its nascent state, because you can’t do this without help, and people probably won’t help you without something to run and play with

                                                                                                                                                                                        If you want to bootstrap, which is not a given (and that’s another flaw with the article), then Rust’s strategy is pretty decent and proven to have worked: OCaml then Rust.

                                                                                                                                                                                        Go has a similar strategy, except they didn’t start from scratch (that is important!), and they used a few bespoke tools to convert the compiler in C to the compiler in Go.

                                                                                                                                                                                        [1] https://github.com/oilshell/oil/wiki/BootstrappingCaseStudies

                                                                                                                                                                                        1. 9

                                                                                                                                                                                          I can’t think of ANY language that was implemented this way (writing 3 compilers, the second in C, the 3rd being bootstrapped). Bootstrapping is something I’ve researched [1], and I’m pretty confident that the number of projects that have done this is exactly zero.

                                                                                                                                                                                          The first compiler is just a prototyping and language design tool, not a practical compiler. I can guarantee you that many of the projects on your page started with small experiments to test out the ideas they were working on before they went and wrote the compiler that they eventually shipped. Odds are that hardly anyone outside of your direct group of collaborators would use or even see your first compiler.

                                                                                                                                                                                          It doesn’t appear that the author has done anything close to this himself.

                                                                                                                                                                                          I have done exactly what this article describes, as a matter of fact. I wrote it based on my own experiences.

                                                                                                                                                                                          Underestimating how much more realistic it is to write 2 compilers than 3 (I actually made somewhat of the same mistake early in Oil, i.e. thinking I could improve a Python compiler more than was feasible in a reasonable amount of time)

                                                                                                                                                                                          Underestimating how bad a language C is to write compilers. C++ is not great but it’s better than C in this domain.

                                                                                                                                                                                          If these are problems for you, your language is probably too complicated. Remember: the first implementation can be (and inevitably will be) a great big mess. It doesn’t have to work well. It can have crippling bugs and limitations. It doesn’t necessarily have to implement the entire language. Its sole purpose is to reinforce your language design with proof that the ideas can be put into practice.

                                                                                                                                                                                          C is a great language for writing compilers.

                                                                                                                                                                                          Not mentioning the requirement to make your language useful in its nascent state, because you can’t do this without help, and people probably won’t help you without something to run and play with

                                                                                                                                                                                          Not true on any of these counts. It helps to have people interested, and it helps gather interest if you have something they can pick up and mess around with, but you can (1) build it without help, it just takes longer, (2) get people interested without having something readily available to play with. You should be able to elevator pitch your programming language to people you want to work with, and get them on board. If you can’t, your pitch (or your idea (or your associates)) needs work. And if you prioritize getting it useful ASAP then you’re likely to rush it and lay a poor foundation upon which the rest of your language can be built.

                                                                                                                                                                                          This is a guide for serious projects which want to design a robust, self-consistent programming langauges. If you’re just making a toy, then this level of rigor is not called for. A lot of what you’re arguing for is a reduction in rigor because, well, this is difficult. But difficult though it may be, the demanded rigor will create a significantly better result. Writing a sacrificial compiler as you develop the language design proves your ideas as you work. Throwing it away and starting over lets you create a fresh codebase using only proven ideas, and shrug off the tech debt that your first compiler has in spades. Doing it in C instead of C++ (or whatever else, you were nonspecific) ensures a broadly compatible bootstrap route - you called out Rust as doing this well, but I suspect you’ve never actually bootstrapped Rust if you think it works well.

                                                                                                                                                                                          1. 10

                                                                                                                                                                                            C is a great language for writing compilers.

                                                                                                                                                                                            So great that even gcc moved to C++ :-). Snark aside, the ML family is probably the best for writing compilers, they’ve been designed for decades for that very purpose. SML is barely younger than C, has a standard, multiple implementations, and is very simple; I’d say SML or OCaml are great for writing a bootstrapping compiler or even the main compiler.

                                                                                                                                                                                            Also, if you want to bootstrap, look at what OCaml itself does: rather than writing the whole compiler in C (😱), the compiler is written in itself, but can produce either native code, or bytecode for a quite simple virtual machine that is implemented in C. If you want to bootstrap on a new architecture you just have to port the virtual machine. If the only goal is to bootstrap (i.e. run literally one program before you get the main compiler), any form of simple interpreter should suffice.

                                                                                                                                                                                            1. 4

                                                                                                                                                                                              So great that even gcc moved to C++ :-)

                                                                                                                                                                                              One of the great tragedies of our time. Let’s not pretend that GCC is a shining example of good design and compiler philosophy, though.

                                                                                                                                                                                              Also, if you want to bootstrap, look at what OCaml itself does

                                                                                                                                                                                              The OCaml approach is pretty interesting, and definitely better than some approaches (cough Rust cough). I would prefer to be able to bootstrap compilers without cross-compiling them, however, and C is nice for that because it’s already universally supported. If there’s any issues during the bootstrapping progress, you don’t have that extra layer of indirection to debug through that the VM design introduces.

                                                                                                                                                                                              1. 8

                                                                                                                                                                                                One of the great tragedies of our time.

                                                                                                                                                                                                I had thought the same thing when Mesa3D started allowing C++, too, but later in life I realized that C and C++ have different angles of repose. There are real size limits on what can be built with C while maintaining an acceptable defect rate. C++ isn’t perfect, just not as bad.

                                                                                                                                                                                                C is nice for [some purpose] because it’s already universally supported.

                                                                                                                                                                                                This is modern-day vaxocentrism; just like how GNU/Linux is not the only production runtime, and the world’s not a VAX, C is not universally supported. For example, since C is not capability-safe or memory-safe, any systems which provably don’t have capability or memory errors will have to emulate parts of the C virtual machine if they want to host C programs; there will usually be a more direct and native language which is supported first, with C support being an extra feature which might not manifest.

                                                                                                                                                                                                1. 2

                                                                                                                                                                                                  C currently compiles to more systems than any other language. For bootstrapping, the compiler doesn’t need to run well, it has to run at all. C gives the most chances to do that on an arbitrary system.

                                                                                                                                                                                                  1. 3

                                                                                                                                                                                                    The bootstrapping compiler doesn’t even need to run on an arbitrary system; it merely needs to run on the bootstrapping compiler author’s system. I personally have used Python as my bootstrapping environment, and I’ve seen other folks use D, Haskell, OCaml, Prolog, and Scheme.

                                                                                                                                                                                                    1. 0

                                                                                                                                                                                                      That’s how you get trust problems. I trust the C compiler on my system. I can check the source of the C programs and compile them with my trusted compiler to get trusted binaries. If you have a C bootstrapping compiler, I can compile it, and with it compile the final compiler without breaking the trust chain and me only needing to compile two things. If you need to use Python or some other language to bootstrap, now I need yet another thing to compile and, potentially, port to my system. Or, I could do as you suggest and break the trust chain I had by using the compiled binary from the developer. Now I with anything that compiler produces I innately trust the developer that provided me that binary.

                                                                                                                                                                                            2. 8

                                                                                                                                                                                              Have you published the source to your compilers? It would have been helpful to mention them in the article, and refer to those experiences.

                                                                                                                                                                                              I disagree with most of the above, but open source has a good way of resolving these arguments (“show me the code”). That said, some short comments:

                                                                                                                                                                                              • I agree with the need to prototype the language in any way possible at first
                                                                                                                                                                                              • Throwing out the code and starting again IMO is a sign that the metalanguage was too inflexible (e.g. it can be hard to quickly refactor C code, in part because memory management is a concern at every function boundary; the language encourages long functions for the same reason)
                                                                                                                                                                                              • There are multiple dimensions of rigor, and they can be at odds: efficiency is one, but correctness is another. C is better for the former; ML dialects are better for the latter. (If you don’t believe this, let me test your compiler.)
                                                                                                                                                                                              • I think you implicitly understand the need to get feedback, but what you wrote is at odds with that. Why did you release sourcehut alpha? Why not just wait until it’s finished? That principle doesn’t change just because you’re writing a language.
                                                                                                                                                                                              • Bootstrapping means two different things: building a compiler from source without binaries, and writing the language in itself. Ironically the latter bootstrapping makes the former much harder!
                                                                                                                                                                                              • There are other options for bootstrapping, which are left out of the article: not writing the language in itself (Swift); making the language popular enough that somebody else will write an independent compiler (somebody started an independent C++ implementation of the Rust compiler).
                                                                                                                                                                                                • Aside: I think that translating to C++ can be a good bootstrapping method for certain kinds of languages; I think Oil’s metalanguage might do that
                                                                                                                                                                                              • Go is one of the smallest languages that’s useful and deployed. Even so, their compiler was big enough that they didn’t throw it out when bootstrapping (IIRC it was something like 60-80K lines of C, i.e. not trivial to rewrite in Go).
                                                                                                                                                                                                • They also didn’t start from scratch. They did zero complete rewrites, not 2, and they were a a team paid to work on the compiler. They also bootstrapped after about ~9 years IIRC. The time frame is missing from the blog post; you’re describing 15-20 years of work IMO, if you have the requirement “people use this language for real work”.
                                                                                                                                                                                              • Conjecture: languages have harsher “illusion of being close to done” than other software projects. One example is that I remember watching early videos about Jai by Jon Blow, where he claimed that “making is a language is less work than an AAA game”. That may or may not be true, and by all accounts he’s a productive programmer, but I’d be curious about how he feels about that now …
                                                                                                                                                                                                • Part of the reason is that a language relies on a huge ecosystem, whereas a game doesn’t. Also, I think there was some evidence that library code is 3x more expensive to develop than application code, and I believe that. Generality, documentation, testing, edge cases, backward compatibility, etc. matter in library code.

                                                                                                                                                                                              Anyway I wish you luck on your project and I hope to see blog posts with concrete experiences!

                                                                                                                                                                                            3. 4

                                                                                                                                                                                              I don’t understand. You’re pretty confident that the number of bootstrapped programming language projects is exactly zero… then link to a list you wrote of a bunch of bootstrapped programming language projects? Am I misunderstanding something?

                                                                                                                                                                                              1. 5

                                                                                                                                                                                                They give three criteria:

                                                                                                                                                                                                • Writing three compilers
                                                                                                                                                                                                • The second compiler is written in C
                                                                                                                                                                                                • The third compiler is self-hosting and bootstrapped from the second compiler

                                                                                                                                                                                                This is indeed a very rare collection of circumstances, because it implies that C is the preferred high-level prototyping language, which hasn’t been true for decades and in fact might have never been true. Combined with the original article’s advice, we might be misled to imagine that the second compiler is going to be thrown away immediately.

                                                                                                                                                                                                As we’ve bootstrapped the Monte programming language, we’ve basically had three compilers. The first compiler was written in Python, generated more Python, and was barely able to run basic Monte expressions; we used it to rewrite the parser and AST handling in Monte. It existed for around two years in toto, not including time spent on pre-Monte research. The second compiler, montec, is written in pure Monte and sends Monte source code to binary bytecode. This compiler has existed for about half a decade and probably will continue existing for another few years. Our third and final compiler will send pre-linked bytecode to C or assembly and use GCC or LLVM to do native compilation.

                                                                                                                                                                                                So, while it might seem like we followed the outlined pattern, our second compiler is written in pure Monte and uses standard Monte modules for linkage. Only the runtime interpreter has any C, and that particular codebase continues to shrink over time; it’s almost entirely Python except for libsodium and libuv.

                                                                                                                                                                                                If the second compiler were to be written in C, then the project would be moribund. Indeed, my partner in writing Monte wrote a C runtime for E as part of their pre-Monte research, and it turns out that even implementing a small slice of E’s relatively demanding runtime is a tall task for C. It is generally accepted knowledge in the E community that compiling E to native code requires a garbage collector, and at some point it is too difficult to implement capability-safety and memory-safety simultaneously in one single runtime.

                                                                                                                                                                                                1. 2

                                                                                                                                                                                                  The first compiler is the prototype compiler, written in whichever language. The article is pretty clear that the first implementation is to be thrown away. Further, the second compiler is written in C, but the second compiler isn’t a prototype; the article is pretty clear that it’s a form of reference implementation, which should be kept up to date as the formal specification evolves. I don’t see anything which demands C to be used as a prototyping language, or anything which leads one to believe that the second compiler is to be thrown away.

                                                                                                                                                                                                  Though the rest of what you’re saying makes sense, and my original misunderstanding was that I interpreted “Bootstrapping is something I’ve researched [1], and I’m pretty confident that the number of projects that have done this is exactly zero” to mean that “the number of projects which have bootstrapped is exactly zero”. I realize now that they meant that “the number of projects which have gone through all three steps is exactly zero”.

                                                                                                                                                                                            1. 5

                                                                                                                                                                                              Related: Simon Ser wrote an article about his experience actually saying “no”, and the (lack) of negative consequences for him which resulted from this decision. Saying “no” to unethical tasks.

                                                                                                                                                                                              Disclaimer: I have a business relationship with Simon.

                                                                                                                                                                                              1. 7

                                                                                                                                                                                                And yet, i915 has DRM upstreamed.

                                                                                                                                                                                                So what are to make of this? Saying “no” may not have direct impact on your own career, and it may allow you to not do something you find personally distasteful, but is it enough?

                                                                                                                                                                                                1. 4

                                                                                                                                                                                                  That’s obviously not what I’m saying. This article demonstrates that you can say “no” and it will probably be just fine for your career.

                                                                                                                                                                                                  The next trick is getting a majority of engineers to realize this, and start exercising their influence on the ethical direction of the companies they work for.

                                                                                                                                                                                                  1. 12

                                                                                                                                                                                                    The next trick is getting a majority of engineers to realize this, and start exercising their influence on the ethical direction of the companies they work for.

                                                                                                                                                                                                    Engineers[1] realized back in the 1800s that they could exercise their influence on the direction of the railroads that hired them. This quickly lead to the birth of the Pinkerton agency and its often violent deployment against workers all over the country for over a century afterwards.

                                                                                                                                                                                                    Saying “no” in isolation will probably not impact your career, especially if you are a valuable and skilled software engineer. But saying no and trying to educate/empower/motivate others? The evidence of history suggests your career will be in jeopardy.

                                                                                                                                                                                                    This is why I find the author’s idea of a professional code of ethics to be interesting, though I doubt it will be something that can ever be formalized in our industry.

                                                                                                                                                                                                    [1] For the pedants reading this: I’m sure it was not just engineers looking to unionize the railroads – but the parallel was too fun to ignore :)

                                                                                                                                                                                                    1. 1

                                                                                                                                                                                                      something that can ever be formalized in our industry.

                                                                                                                                                                                                      It can with the stroke of a pen.

                                                                                                                                                                                                      But it shouldn’t happen. We as a profession would be closing the doors to the next generation of programmers who would see adhering to a professional organization as a needless barrier.

                                                                                                                                                                                                    2. 5

                                                                                                                                                                                                      If you say ‘no’ and someone else is willing to say ‘yes’, the outcome will be that the other person does the job. If you’re in an industry with an overall skills shortage (kernel programming definitely counts!) then you will end up working on something not directly the same as the unethical thing. The company has a vested interest in not losing talented employees and in shipping their product. In this case, the two are not in conflict, they keep you happy and someone else who doesn’t consider it unethical does the work.

                                                                                                                                                                                                      If; however, everyone says ‘no’, the balance shifts. Now the company has a big incentive to ensure that someone is willing to do the unethical thing. This may mean higher salaries for people working on unethical projects or workplace bullying / intimidation. The latter can be solved with a strong union (you bully your workers? Good luck ever making them show up to work), the former can be addressed by personal liability.

                                                                                                                                                                                                      DRM is an interesting example because it’s not universally agreed that things like to HDCP are unethical. They prop up broken business models but all of the folks employed in those industries are likely to be pretty happy with them. Personally, I’d love to see DRM aggressively regulated and, at a minimum, require that any consumer product distributed with DRM must file a DRM-free copy with a copyright library, to be publicly distributed as soon as the DRM’d system stops working. If the owner of the copyrighted material does not comply with this, then they should immediately and permanently forfeit copyright protection on the work, such that anyone who bypasses the DRM is legally able to freely distribute the work. I’d also like to see large statutory fines when DRM prevents activities guaranteed by Fair Use / Fair Dealings or equivalent parts of copyright law. I see DRM for preventing copyright violation in much the same light as putting bear traps around your property to prevent trespassers: you may be allowed to do it but you are opening yourself up to a load of liability when a child steps in one.

                                                                                                                                                                                                      That really comes back to the ‘personal liability’ issue. There is no personal or corporate liability for ‘unethical’ behaviour, there is only liability for illegal behaviour. If there is a social consensus that something is unethical, then it eventually becomes illegal. Many of the problems with big tech today are caused by that lag. Pervasive surveillance, DRM, and so on may be broadly perceived within the tech industry as unethical, but they’re legal. Most people outside of the industry don’t think about them at all (ask your Facebook-using friends some time how many of them have actually read the Facebook T&Cs and how many are happy with the company knowing their address, voting preference, top 5 political issues, and being willing to sell that information to political parties). Until they’re a significant issue for a large percentage of the population, they’re a low priority for lawmakers on democratic countries. You don’t get re-elected by championing issues that no one cares about. I used to have an MEP who was an active campaigner against software patents. I would be willing to bet that no more than 5% of the electorate in her constituency even knew what a software patent was and so I doubt it did very much for her re-election chances.

                                                                                                                                                                                                      1. 1

                                                                                                                                                                                                        If; however, everyone says ‘no’, the balance shifts. Now the company has a big incentive to ensure that someone is willing to do the unethical thing.

                                                                                                                                                                                                        Two points: First, humans are social creatures, and companies are run by humans. if a huge number of workers start complaining about ethics, it will give some of the leadership pause.

                                                                                                                                                                                                        Second, companies have option to find a more ethical way of doing something. If doing ethical things turns out to be lower friction, lower cost, and leads to better publicity and easier hiring than doing unethical things, then it becomes the pragmatic business decision.

                                                                                                                                                                                                    3. 3

                                                                                                                                                                                                      It’s absolutely not enough; for it to have an impact you need to organize with other workers and present a united front of refusal. Otherwise it’s a token effort that just helps you sleep better at night.